- developers - lists.mariadb.org

Re: [Maria-developers] 2849076672d: MDEV-16470: switch off user variables (and fixes of its support)
by Sergei Golubchik 12 Jun '20

12 Jun '20

Hi, Oleksandr! On Jun 12, Oleksandr Byelkin wrote: > revision-id: 2849076672d (mariadb-10.5.2-322-g2849076672d) > parent(s): bb47050e1fd > author: Oleksandr Byelkin <sanja(a)mariadb.com> > committer: Oleksandr Byelkin <sanja(a)mariadb.com> > timestamp: 2020-06-12 08:32:56 +0200 > message: > > MDEV-16470: switch off user variables (and fixes of its support) > > diff --git a/client/mysqltest.cc b/client/mysqltest.cc > index 55c4a66a89d..24e0cd25108 100644 > --- a/client/mysqltest.cc > +++ b/client/mysqltest.cc > &data, &data_length)) > { > dynstr_append(ds, "-- "); > - switch (type) > + if (type <= SESSION_TRACK_END) > { > - case SESSION_TRACK_SYSTEM_VARIABLES: > - dynstr_append(ds, "Tracker : SESSION_TRACK_SYSTEM_VARIABLES\n"); > - break; > - case SESSION_TRACK_SCHEMA: > - dynstr_append(ds, "Tracker : SESSION_TRACK_SCHEMA\n"); > - break; > - case SESSION_TRACK_STATE_CHANGE: > - dynstr_append(ds, "Tracker : SESSION_TRACK_STATE_CHANGE\n"); > - break; > - case SESSION_TRACK_GTIDS: > - dynstr_append(ds, "Tracker : SESSION_TRACK_GTIDS\n"); > - break; > - case SESSION_TRACK_TRANSACTION_CHARACTERISTICS: > - dynstr_append(ds, "Tracker : SESSION_TRACK_TRANSACTION_CHARACTERISTICS\n"); > - break; > - case SESSION_TRACK_TRANSACTION_TYPE: > - dynstr_append(ds, "Tracker : SESSION_TRACK_TRANSACTION_TYPE\n"); > - break; > - default: > - DBUG_ASSERT(0); > - dynstr_append(ds, "\n"); > + dynstr_append(ds, trking_info_desc[type]); > + } > + else > + { > + DBUG_ASSERT(0); > + dynstr_append(ds, "Trecker???\n"); "Tracker" > } > - > > dynstr_append(ds, "-- "); > dynstr_append_mem(ds, data, data_length); > diff --git a/libmariadb b/libmariadb > index ce74fd0c400..db7adf63fbf 160000 > --- a/libmariadb > +++ b/libmariadb > @@ -1 +1 @@ > -Subproject commit ce74fd0c4009ed9f4bcbdb4a01e96c823e961dc3 > +Subproject commit db7adf63fbf242a07c355f192a4d8c60323b5d89 No, I don't see any need for that now. C/C does not know anything about SESSION_TRACK_USER_VARIABLES, let's just keep it that way. > diff --git a/sql/item_func.cc b/sql/item_func.cc > index da185f9878b..448bfd380a2 100644 > --- a/sql/item_func.cc > +++ b/sql/item_func.cc > @@ -4702,10 +4702,12 @@ update_hash(user_var_entry *entry, bool set_null, void *ptr, size_t length, > entry->unsigned_flag= unsigned_arg; > } > entry->type=type; > +#ifdef USER_VAR_TACKING TRACKING > #ifndef EMBEDDED_LIBRARY > THD *thd= current_thd; > thd->session_tracker.user_variables.mark_as_changed(thd, entry); > #endif > +#endif // USER_VAR_TACKING > return 0; > } > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] 4f528bb514c: MDEV-22011: DISTINCT with JSON_ARRAYAGG gives wrong results
by Sergey Petrunia 11 Jun '20

11 Jun '20

Hi Varun, I only have input about comments/variable names. Please find it below. On Tue, Mar 24, 2020 at 10:14:38AM +0530, Varun wrote: > revision-id: 4f528bb514c607d848b30119cea135e37e0d9c69 (mariadb-10.5.0-425-g4f528bb514c) > parent(s): 1cd8e8213b6019794ca7f01e8276cfd1ba6408cf > author: Varun Gupta > committer: Varun Gupta > timestamp: 2020-03-24 10:13:28 +0530 > message: > > MDEV-22011: DISTINCT with JSON_ARRAYAGG gives wrong results > > For JSON_ARRAYAGG with DISTINCT also store the null bytes for the record > inside the Unique object > > --- > mysql-test/main/func_json.result | 4 +- > sql/item_jsonfunc.cc | 36 ++++++++++++-- > sql/item_jsonfunc.h | 9 ++-- > sql/item_sum.cc | 101 ++++++++++++++++++++++++++++++++++----- > sql/item_sum.h | 13 +++-- > 5 files changed, 133 insertions(+), 30 deletions(-) > > diff --git a/sql/item_jsonfunc.cc b/sql/item_jsonfunc.cc > index a3b5d3b7fac..ed496df8169 100644 > --- a/sql/item_jsonfunc.cc > +++ b/sql/item_jsonfunc.cc > @@ -1452,6 +1452,31 @@ static int append_json_value(String *str, Item *item, String *tmp_val) > } > > > +bool append_json_value(String *str, String *res, Item *item, bool null_value) There is another append_json_value right above this one. That function has no comments but let's not follow that bad pattern. - Please add a function comment describing this function: @brief should describe this function @detail should describe the difference from the one above. It is very counter-intuitive that the parameter named "res" does not hold the result of this function. Please rename it (to value?) (can it be changed to be const String* ?) > +{ > + if (null_value) > + return str->append("null", 4); > + > + if (item->type_handler()->is_bool_type()) > + { > + return ((*res)[0] == '1') ? > + str->append("true", 4) : > + str->append("false", 5); > + } > + > + if (item->is_json_type()) > + return str->append(res->ptr(), res->length()); > + > + if (item->result_type() == STRING_RESULT) > + { > + return str->append("\"", 1) || > + st_append_escaped(str, res) || > + str->append("\"", 1); > + } > + return st_append_escaped(str, res); This made me wonder why something is escaped if it is not enclosed in quotes, but now I see this is just copied from above. > +} > + > + > static int append_json_keyname(String *str, Item *item, String *tmp_val) > { > String *sv= item->val_str(tmp_val); > diff --git a/sql/item_sum.cc b/sql/item_sum.cc > index 5c732095573..a04dbdc19e2 100644 > --- a/sql/item_sum.cc > +++ b/sql/item_sum.cc > @@ -3552,6 +3552,54 @@ int group_concat_key_cmp_with_distinct(void* arg, const void* key1, > } > > > +int group_concat_key_cmp_with_distinct_with_nulls(void* arg, > + const void* key1_arg, > + const void* key2_arg) > +{ > + Item_func_group_concat *item_func= (Item_func_group_concat*)arg; > + > + uchar *key1= (uchar*)key1_arg + item_func->table->s->null_bytes; > + uchar *key2= (uchar*)key2_arg + item_func->table->s->null_bytes; > + > + for (uint i= 0; i < item_func->arg_count_field; i++) This loop is actually not needed, right? JSON_ARRAYAGG only accepts one argument. Please add a note. > + { > + Item *item= item_func->args[i]; > + /* > + If item is a const item then either get_tmp_table_field returns 0 > + or it is an item over a const table. > + */ > + if (item->const_item()) > + continue; > + /* > + We have to use get_tmp_table_field() instead of > + real_item()->get_tmp_table_field() because we want the field in > + the temporary table, not the original field > + */ > + Field *field= item->get_tmp_table_field(); > + > + if (!field) > + continue; > + > + if (field->is_null_in_record((uchar*)key1_arg) && > + field->is_null_in_record((uchar*)key2_arg)) > + continue; > + > + if (field->is_null_in_record((uchar*)key1_arg)) > + return -1; > + > + if (field->is_null_in_record((uchar*)key2_arg)) > + return 1; > + > + uint offset= (field->offset(field->table->record[0]) - > + field->table->s->null_bytes); > + int res= field->cmp(key1 + offset, key2 + offset); > + if (res) > + return res; > + } > + return 0; > +} > + > + > /** > function of sort for syntax: GROUP_CONCAT(expr,... ORDER BY col,... ) > */ > @@ -3650,9 +3699,11 @@ int dump_leaf_key(void* key_arg, element_count count __attribute__((unused)), > result->append(*item->separator); > > > + bool is_null= false; > for (; arg < arg_end; arg++) > { > String *res; > + is_null= false; (1) > /* > We have to use get_tmp_table_field() instead of > real_item()->get_tmp_table_field() because we want the field in > @@ -3661,7 +3712,11 @@ int dump_leaf_key(void* key_arg, element_count count __attribute__((unused)), > because it contains both order and arg list fields. > */ > if ((*arg)->const_item()) > + { > res= (*arg)->val_str(&tmp); > + if ((*arg)->null_value) > + is_null= TRUE; (2) Please either use true/false (preferred) or TRUE / FALSE. > + } > else > { > Field *field= (*arg)->get_tmp_table_field(); ... > diff --git a/sql/item_sum.h b/sql/item_sum.h > index b10d9f5974d..908dbe3cce9 100644 > --- a/sql/item_sum.h > +++ b/sql/item_sum.h > @@ -1897,14 +1899,19 @@ class Item_func_group_concat : public Item_sum > */ > Item_func_group_concat *original; > > + bool exclude_nulls; > + > /* > Used by Item_func_group_concat and Item_func_json_arrayagg. The latter > needs null values but the former doesn't. > */ Is this comment relevant here? I think it's better to remove this and instead add a comment for the exclude_nulls above. > - bool add(bool exclude_nulls); > + bool add(); > > friend int group_concat_key_cmp_with_distinct(void* arg, const void* key1, > const void* key2); > + friend int group_concat_key_cmp_with_distinct_with_nulls(void* arg, > + const void* key1, > + const void* key2); > friend int group_concat_key_cmp_with_order(void* arg, const void* key1, > const void* key2); > friend int dump_leaf_key(void* key_arg, BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Improving `source` in mysqltest
by Nikita Malyavin 11 Jun '20

11 Jun '20

Hello mariadb developers! This year we have a student on mysqltest parser improvements, mentored by me and @serg. The major improvement will be proper expressions support, which you'll soon receive an email about. Another a bit more minor-ish improvement we want our student Igor to make is parameters support for source command, which I want to ask you about, which way is better to implement. It's quite spread in the tests to parametrize source with variables, for example > --let SEARCH_FILE=$MYSQLTEST_VARDIR/log/mysqld.1.err > --let SEARCH_PATTERN= This connection closed normally without > authentication > --source include/search_pattern_in_file.inc Another example is period/create_triggers.inc. So, we want make it possible to write > --source do_something.inc A=1 B=foobar And the question is, how should it behave. One way is to make it a strict equivalent of > let A=1; > let B=foobar; > source do_something.inc; This allows to reuse previously set values, for example: > --source do_something.inc A=1 B=foobar > ... > --source do_something.inc A=2 Which literally means > let A=1; > let B=foobar; > --source do_something.inc; ... let A=2; > --source do_something.inc; Another way is to make it work like local arguments, so the arguments are put in the stack, and as a result > let A=qwe; > --source do_something.inc A=1 B=foobar > --echo $A; outputs qwe. What do you think of this, what's more suitable for us? This is an important question, since all of us are using mysqltest, and we probably wouldn't change the behavior ever later. -- Yours truly, Nikita Malyavin

1 0

Re: [Maria-developers] MDEV-21831: Assertion `length == pack_length()' failed in Field_inet6::sort_string upon INSERT into RocksDB table
by Sergey Petrunia 11 Jun '20

11 Jun '20

On Thu, Jun 11, 2020 at 03:41:24PM +0530, Varun Gupta wrote: > > /* > > pack_length() returns size (in bytes) used to store field data in > > memory > > (i.e. it returns the maximum size of the field in a row of the table, > > which is located in RAM). > > */ > > virtual uint32 pack_length() const { return (uint32) field_length; } > > > > so, it is size in bytes, while strnxfrmlen() accepts length *in > > characters*. > > That is, strnxfrmlen(field->pack_length()) is certainly not correct. > > > > > This is not correct, strnxfrmlen accepts length in bytes and there we > change the length to characters and then calculate the weights. You're right. I've missed this part. The patch is ok to push, then. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Progress report week 5
by Mohammed Hammaad Mateen 10 Jun '20

10 Jun '20

Greetings, Hope you are safe and doing great, This post describes the things I've done during 1-7 June under the mentor-ship of Sergei Golubchik and Oleksandr Byelkin for GSoC-20. The tasks taken up for this week Greetings mentors, I've realized that to extend the parser for multi delete, I should be changing the sql_yacc.yy file, for incorporating opt_returning rule in multi-delete. opt_returning rule works completely fine with single table delete, but to tweak it to adapt for multi-delete, required me to learn some bison grammar syntax and semantics and that took me a couple of days. Moreover everytime I made changes to the grammar, the server used to crash rendering all changes invalid. I'll try to push the code that was triggering crashes, shortly. Thank you Regards, Mohammed Hammaad Mateen

1 0

Re: [Maria-developers] 5ff9525ff86: MDEV-22722 Assertion "inited==NONE" failed in handler::ha_index_init on the slave during UPDATE
by Sergei Golubchik 10 Jun '20

10 Jun '20

Hi, Sachin! On Jun 10, Sachin wrote: > revision-id: 5ff9525ff86 (mariadb-10.5.2-290-g5ff9525ff86) > parent(s): ff881056b8d > author: Sachin <sachin.setiya(a)mariadb.com> > committer: Sachin <sachin.setiya(a)mariadb.com> > timestamp: 2020-06-07 16:45:01 +0530 > message: > > diff --git a/sql/log_event_server.cc b/sql/log_event_server.cc > index 5910ece12f0..ffda352c866 100644 > --- a/sql/log_event_server.cc > +++ b/sql/log_event_server.cc > @@ -5570,6 +5570,12 @@ int Rows_log_event::do_apply_event(rpl_group_info *rgi) > { > master_had_triggers= table->master_had_triggers; > bool transactional_table= table->file->has_transactions_and_rollback(); > + if (get_general_type_code() == WRITE_ROWS_EVENT) > + table->file->prepare_for_insert(0); > + else if (get_general_type_code() == UPDATE_ROWS_EVENT || > + get_general_type_code() == DELETE_ROWS_EVENT) > + table->file->prepare_for_insert(1); > + You can simplify it as table->file->prepare_for_insert(get_general_type_code() != WRITE_ROWS_EVENT); otherwise ok to push Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] MDEV-21831: Assertion `length == pack_length()' failed in Field_inet6::sort_string upon INSERT into RocksDB table
by Sergey Petrunia 09 Jun '20

09 Jun '20

Hi Varun, > MDEV-21831: Assertion `length == pack_length()' failed in Field_inet6::sort_string upon INSERT into RocksDB table > > For INET6 columns the values are stored as BINARY columns and returned to the client in TEXT format. > For rocksdb the indexes store mem-comparable images for columns, so use the pack_length() to store > the mem-comparable form for INET6 columns. This would also remain consistent with CHAR columns. > > diff --git a/storage/rocksdb/rdb_datadic.cc b/storage/rocksdb/rdb_datadic.cc > index 21a9b57b555e..6e73535d1de7 100644 > --- a/storage/rocksdb/rdb_datadic.cc > +++ b/storage/rocksdb/rdb_datadic.cc > @@ -3317,7 +3317,9 @@ bool Rdb_field_packing::setup(const Rdb_key_def *const key_descr, > field->field_length = field->char_length() * cs->mbmaxlen. > */ > const CHARSET_INFO *cs = field->charset(); > - m_max_image_len = cs->strnxfrmlen(field->field_length); > + m_max_image_len = cs->strnxfrmlen(type == MYSQL_TYPE_STRING ? > + field->pack_length() : > + field->field_length); Field::pack_length is defined as: /* pack_length() returns size (in bytes) used to store field data in memory (i.e. it returns the maximum size of the field in a row of the table, which is located in RAM). */ virtual uint32 pack_length() const { return (uint32) field_length; } so, it is size in bytes, while strnxfrmlen() accepts length *in characters*. That is, strnxfrmlen(field->pack_length()) is certainly not correct. Rdb_field_packing::m_max_image_len is defined as: /* Length of mem-comparable image of the field, in bytes */ int m_max_image_len; while pack_length() seems to describe table->record[0] format (or is it KeyTupleFormat) ? Field::sort_length() doesn't have a definition but it seems to be a closer match as it describes the length of the mem-comparable form? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] 06942363ebc: MDEV-14347 CREATE PROCEDURE returns no error when using an unknown variable
by Sergei Golubchik 09 Jun '20

09 Jun '20

Hi, Alexander! On Jun 07, Alexander Barkov wrote: > revision-id: 06942363ebc (mariadb-10.5.2-311-g06942363ebc) > parent(s): 0d6d63e1505 > author: Alexander Barkov <bar(a)mariadb.com> > committer: Alexander Barkov <bar(a)mariadb.com> > timestamp: 2020-06-03 12:11:07 +0400 > message: > > MDEV-14347 CREATE PROCEDURE returns no error when using an unknown variable > > diff --git a/sql/item.cc b/sql/item.cc > index 8ea6366e6c4..5800b9bc95e 100644 > --- a/sql/item.cc > +++ b/sql/item.cc > @@ -764,6 +764,29 @@ bool Item_field::collect_item_field_processor(void *arg) > } > > > +void Item_ident::undeclared_spvar_error() const Currently it's used only in Item_field::unknown_splocal_processor(). Do you plan to use this somewhere else? Why is it a separate method and in Item_ident? > +{ > + /* > + We assume this is an unknown SP variable, possibly a ROW variable. > + Print the leftmost name in the error: > + SET var=a; -> a > + SET var=a.b; -> a > + SET var=a.b.c; -> a > + */ > + my_error(ER_SP_UNDECLARED_VAR, MYF(0), db_name.str ? db_name.str : > + table_name.str ? table_name.str : > + field_name.str); > +} > + > +bool Item_field::unknown_splocal_processor(void *arg) > +{ > + DBUG_ENTER("Item_field::unknown_splocal_processor"); > + DBUG_ASSERT(type() == FIELD_ITEM); > + undeclared_spvar_error(); > + DBUG_RETURN(true); > +} > + > + > bool Item_field::add_field_to_set_processor(void *arg) > { > DBUG_ENTER("Item_field::add_field_to_set_processor"); > @@ -8012,6 +8035,27 @@ void Item_ref::cleanup() > } > > > +bool Item_ref::unknown_splocal_processor(void *arg) > +{ > + DBUG_ENTER("Item_ref::unknown_splocal_processor"); > + DBUG_ASSERT(type() == REF_ITEM); > + DBUG_ASSERT(ref_type() == REF); > + undeclared_spvar_error(); > + DBUG_RETURN(true); > +} > + > + > +bool Item_ref::walk(Item_processor processor, bool walk_subquery, void *arg) > +{ > + if (processor == &Item::unknown_splocal_processor) > + return unknown_splocal_processor(arg); Why is that? This is very unusual, normally walk() should not change its behavior based on the processor. > + if (ref && *ref) > + return (*ref)->walk(processor, walk_subquery, arg) || > + (this->*processor)(arg); > + return false; > +} > + > + > /** > Transform an Item_ref object with a transformer callback function. > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 2

Re: [Maria-developers] JSON aggregate functions working on JSON documents
by Sergey Petrunia 08 Jun '20

08 Jun '20

On Mon, Jun 08, 2020 at 04:05:48PM +0400, Alexey Botchkov wrote: > So the problem is that we (MariaDB) doesn't consider the JSON field content > as a JSON, but as a plain text. > That affects not only aggregate functions, but any other expecting JSON > arguments. > To me it's the bug, should be fixed. Will you report it, Sergei? Filed: https://jira.mariadb.org/browse/MDEV-22837 BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Progress Report GSoC 2020 (1-6 June)
by Anton Mikhaylenko 08 Jun '20

08 Jun '20

Good day, everybody. In this post, I want to describe what I did this week, June 1-6. I have added the initial implementation of the hill_climbing class. It includes an update function - the main function that changes the required number of threads. To implement this function, I added the Complex class - the base class for working with complex numbers. https://github.com/sapiest/server/commits/10.5-update_threadpool_concurrency The main work is not completed yet, as the algorithm requires adjustments and adaptation in MariaDB environment. Regards, Anton MIkhailenko

1 0

[Maria-developers] JSON aggregate functions working on JSON documents
by Sergey Petrunia 08 Jun '20

08 Jun '20

Hi Alexey, I'm looking at how JSON_ARRAYAGG and JSON_OBJECTAGG work at processing JSON documents. I'm trying these examples, first on MySQL-8: create table t30 (col1 json); insert into t30 values('{"color":"red", "size":1}' ); insert into t30 values('{"color":"blue", "size":2}' ); select JSON_ARRAYAGG(col1) from t30; +-------------------------------------------------------------+ | JSON_ARRAYAGG(col1) | +-------------------------------------------------------------+ | [{"size": 1, "color": "red"}, {"size": 2, "color": "blue"}] | +-------------------------------------------------------------+ Good. create table t31 (name varchar(100), contents json); insert into t31 values('obj1', '{"color":"blue", "size":2}' ); insert into t31 values('obj2', '{"color":"red", "size":1}' ); select JSON_OBJECTAGG(name, contents) from t31; +-----------------------------------------------------------------------------+ | JSON_OBJECTAGG(name, contents) | +-----------------------------------------------------------------------------+ | {"obj1": {"size": 2, "color": "blue"}, "obj2": {"size": 1, "color": "red"}} | +-----------------------------------------------------------------------------+ Good. Now, trying these on MariaDB, I get: MariaDB [test]> select JSON_ARRAYAGG(col1) from t30; +------------------------------------------------------------------------+ | JSON_ARRAYAGG(col1) | +------------------------------------------------------------------------+ | ["{\"color\":\"red\", \"size\":1}","{\"color\":\"blue\", \"size\":2}"] | +------------------------------------------------------------------------+ MariaDB [test]> select JSON_OBJECTAGG(name, contents) from t31; +---------------------------------------------------------------------------------------+ | JSON_OBJECTAGG(name, contents) | +---------------------------------------------------------------------------------------+ | {"obj1":"{\"color\":\"blue\", \"size\":2}", "obj2":"{\"color\":\"red\", \"size\":1}"} | +---------------------------------------------------------------------------------------+ The JSON documents are coverted into strings. I think, the users will prefer MySQL's behavior. But which behaivor is "correct"? (i.e. required by the standard, etc) Is MariaDBs's behaivor a bug? Is this expected? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] MDEV-11563: GROUP_CONCAT(DISTINCT ...) may produce a non-distinct list
by Sergey Petrunia 07 Jun '20

07 Jun '20

Hi Varun, > revision-id: c408c41230c4a9de0114d13aaf861d3ced65f4c7 (mariadb-10.5.0-424-gc408c41230c) > parent(s): 95da2113a050ad739fdaf60ee871329468a01554 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2020-03-31 11:57:37 +0530 > message: > > MDEV-11563: GROUP_CONCAT(DISTINCT ...) may produce a non-distinct list > The commit message should describe the error and the fix made. (especially important here as this patch fixes only a part of the issue with GROUP_CONCAT) > --- > mysql-test/main/func_gconcat.result | 53 ++++++++++++++++++++++--------------- > mysql-test/main/func_gconcat.test | 7 +++++ > sql/item_sum.cc | 40 +++++++++++++++++----------- > sql/item_sum.h | 3 ++- > 4 files changed, 65 insertions(+), 38 deletions(-) > > diff --git a/sql/item_sum.h b/sql/item_sum.h > index e221d04397d..b10d9f5974d 100644 > --- a/sql/item_sum.h > +++ b/sql/item_sum.h > @@ -1879,7 +1879,8 @@ class Item_func_group_concat : public Item_sum > bool warning_for_row; > bool always_null; > bool force_copy_fields; > - bool no_appended; > + /** True if result has been written to output buffer. */ > + bool m_result_finalized; Why does this variable use the m_ convention when others do not? I think this is confusing (I stop for a moment to wonder how is it different from all others every time I see the name). Please remove this. What's "output buffer"? this->result ? And what does "result" mean here - the entire return value of GROUP_CONCAT function or some part of it? Please improve the comment. > /** Limits the rows in the result */ > Item *row_limit; > /** Skips a particular number of rows in from the result*/ Ok to push after this is addressed. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Is it okay that JSON_OBJECTAGG creates objects with duplicate elements?
by Sergey Petrunia 07 Jun '20

07 Jun '20

Hi Alexey, Consider this example: create table t20 (a varchar(10), b int); insert into t20 values ('red', 1), ('blue', 2), ('red',1001), ('blue', 1002); select json_objectagg(a,b) from t20; +----------------------------------------------+ | json_objectagg(a,b) | +----------------------------------------------+ | {"red":1, "blue":2, "red":1001, "blue":1002} | +----------------------------------------------+ the created object has multiple members with the same name. Is this expected or should be reported as a bug? In MySQL-8, the above example produces this: +-----------------------------+ | json_objectagg(a,b) | +-----------------------------+ | {"red": 1001, "blue": 1002} | +-----------------------------+ BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] 0f91c62d332: MDEV-22722 Assertion "inited==NONE" failed in handler::ha_index_init on the slave during UPDATE
by Sergei Golubchik 07 Jun '20

07 Jun '20

Hi, Sachin! Two questions: 1. This looks like there was no replication tests for periods of long uniques at all? Please add them (create an MDEV, like "replication tests of long unique") and add various relevant replication tests in 10.4. 2. It feels a bit risky to backport my changes to 10.4. At least you definitely should take the latest code from 10.5. Not cherry-pick one commit, but copy handler::prepare_for_insert() and other relevant code from 10.5. May be it'd be safer not to backport? What would your fix look like if you wouldn't have prepare_for_insert()? Also, a couple of comments to the code, see below. On Jun 03, Sachin wrote: > revision-id: 0f91c62d332 (mariadb-10.4.11-225-g0f91c62d332) > parent(s): c01b3d91690 > author: Sachin <sachin.setiya(a)mariadb.com> > committer: Sachin <sachin.setiya(a)mariadb.com> > timestamp: 2020-06-02 08:24:24 +0530 > message: > > MDEV-22722 Assertion "inited==NONE" failed in handler::ha_index_init on the slave during UPDATE > > Add missing call for handler->prepare_for_insert() in rows event. > > diff --git a/mysql-test/main/long_unique_bugs_replication.test b/mysql-test/main/long_unique_bugs_replication.test > new file mode 100644 > index 00000000000..5348be3e7d3 > --- /dev/null > +++ b/mysql-test/main/long_unique_bugs_replication.test > @@ -0,0 +1,22 @@ > +# > +# Long unique bugs related to master slave replication > +# > + > +# > +# MDEV-22722 Assertion "inited==NONE" failed in handler::ha_index_init on the slave during UPDATE > +# > + > +--source include/master-slave.inc > +--source include/have_binlog_format_row.inc master-slave.inc should always be included last, after all have_xxx.inc files that may cause the test to be skipped. > + > +create table t1 (i1 int, a1 text, unique key i1 (a1)) engine=myisam; > +insert into t1 values (1,1); > +update t1 set a1 = 'd' limit 1; > + > +sync_slave_with_master; > +connection slave; > + > +connection master; > +drop table t1; > + > +--source include/rpl_end.inc > diff --git a/sql/log_event.cc b/sql/log_event.cc > index 3707f73a716..f4c0c546a38 100644 > --- a/sql/log_event.cc > +++ b/sql/log_event.cc > @@ -11568,6 +11568,7 @@ int Rows_log_event::do_apply_event(rpl_group_info *rgi) > { > master_had_triggers= table->master_had_triggers; > bool transactional_table= table->file->has_transactions(); > + table->file->prepare_for_insert(); > /* > table == NULL means that this table should not be replicated > (this was set up by Table_map_log_event::do_apply_event() > @@ -14344,6 +14345,7 @@ int Rows_log_event::find_row(rpl_group_info *rgi) > issue_long_find_row_warning(get_general_type_code(), m_table->alias.c_ptr(), > is_index_scan, rgi); > table->default_column_bitmaps(); > + table->file->prepare_for_insert(); Why do you need it here, I'd think that do_apply_event() should be enough? > DBUG_RETURN(error); > } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] MDEV-16230: Server crashes when Analyze format=json is run with a window function with empty PARTITION BY and ORDER BY clauses
by Sergey Petrunia 03 Jun '20

03 Jun '20

Hi Varun, * Please limit the commit comment to ~80 chars wide. * The .result file has r_total_time_ms. Use analyze-format.inc to avoid this. Ok to push after this is addressed. > commit 9e755e21799695b9a837a240a86ba5ef863ae7b3 > Author: Varun Gupta <varun.gupta(a)mariadb.com> > Date: Tue Jun 2 16:31:53 2020 +0530 > > MDEV-16230: Server crashes when Analyze format=json is run with a window function with empty PARTITION BY and ORDER BY clauses > > Currently when both PARTITION BY and ORDER BY clauses are empty then we create a Item with the first field > in the select list and sort with that field. It should be created as an Item_temptable_field instead of > Item_field because the print() function continues to work even if the table has been dropped. > ... BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] e14dcfe0bc7: MDEV-22715: SIGSEGV in radixsort_for_str_ptr and in native_compare/my_qsort2 (optimized builds)
by Sergey Petrunia 03 Jun '20

03 Jun '20

Hi Varun, Please add a comment above the declaration of Field::sort_string() that some implementations actually may write up to 8 bytes regardless of what size was requested. Ok to push after that. > revision-id: e14dcfe0bc743aa6b4ed2b3c4f7e9314aa01d6eb (mariadb-10.1.43-158-ge14dcfe0bc7) > parent(s): db1f73b26edaace41345f19bca507a7848aa39a8 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2020-05-29 21:13:38 +0530 > message: > > MDEV-22715: SIGSEGV in radixsort_for_str_ptr and in native_compare/my_qsort2 (optimized builds) > > For DECIMAL[(M[,D])] datatype max_sort_length was not being honoured which was leading to buffer > overflow while making the sort key. The fix to this problem would be to create sort keys for decimals > with atmost max_sort_key bytes > > Important: > The minimum value of max_sort_length has been raised to 8 (previously was 4), > so fixed size datatypes like DOUBLE and BIGINIT are not truncated for > lower values of max_sort_length. > ... BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] 7df57cc8206: MDEV-22719 Long unique keys are not created when individual key_part->length < max_key_length but SUM(key_parts->length) > max_key_length
by Sergei Golubchik 03 Jun '20

03 Jun '20

Hi, Sachin! ok to push On Jun 03, Sachin wrote: > revision-id: 7df57cc8206 (mariadb-10.4.11-226-g7df57cc8206) > parent(s): 0f91c62d332 > author: Sachin <sachin.setiya(a)mariadb.com> > committer: Sachin <sachin.setiya(a)mariadb.com> > timestamp: 2020-06-03 13:47:39 +0530 > message: > > MDEV-22719 Long unique keys are not created when individual key_part->length < max_key_length but SUM(key_parts->length) > max_key_length > > Make UNIQUE HASH key in case when key_info->key_length > max_key_length > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 1e9ea675a18: MDEV-21282 Assertion 'mariadb_table' failed in gcol.innodb_virtual_debug_purge
by Sergei Golubchik 03 Jun '20

03 Jun '20

Hi, Thirunarayanan! Sure, it's purely InnoDB functionality. If you say that purge thread already hold MDL for the table, then open_purge_table() doesn't need to. Ok to push as you see fit. On Jun 03, Thirunarayanan Balathandayuthapani wrote: > revision-id: 1e9ea675a18 (mariadb-10.5.2-304-g1e9ea675a18) > parent(s): f22093ad392 > author: Thirunarayanan Balathandayuthapani <thiru(a)mariadb.com> > committer: Thirunarayanan Balathandayuthapani <thiru(a)mariadb.com> > timestamp: 2020-06-01 18:02:17 +0530 > message: > > MDEV-21282 Assertion 'mariadb_table' failed in gcol.innodb_virtual_debug_purge > > - commit ea37b144094a0c2ebfc6774047fd473c1b2a8658 (MDEV-16678) caused > a regression. when purge thread tries to open the table for virtual > column computation, there is no need to acquire MDL for the table. > Because purge thread already hold MDL for the table Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] c01b3d91690: MDEV-22551 Server crashes in skip_trailing_space upon ANALYZE on table with long unique, or Assertion `marked_for_read()' fails
by Sergei Golubchik 03 Jun '20

03 Jun '20

Hi, Sachin! On Jun 03, Sachin wrote: > revision-id: c01b3d91690 (mariadb-10.4.11-224-gc01b3d91690) > parent(s): 0552ed86782 > author: Sachin <sachin.setiya(a)mariadb.com> > committer: Sachin <sachin.setiya(a)mariadb.com> > timestamp: 2020-06-02 08:24:13 +0530 > message: > > MDEV-22551 Server crashes in skip_trailing_space upon ANALYZE on table with long unique, or Assertion `marked_for_read()' fails > > If Blob(or parent) field read_set is not set then Do not calc hash in > that case. No, I think this is wrong. The problem, again, is that read_set bits are set incorrectly, and the fix is, again, to set them correctly, not to perform more read_set checks at run time. Think about what's happening here. The server tries to calculate a generated column value (because gcol is present in read_set) when not all argument columns were read (because they are not in read_set). This is an impossible situation, a gcol can _only_ be in read_set if all its dependencies are also there, you've seen in TABLE::mark_virtual_columns_for_write() that for every marked gcol all dependencies are also recursively marked. So in this case ANALYZE breaks this rule and should be fixed not to do it. There are two possible fixes - either ANALYZE should mark dependencies for all gcols (marking blobs in this bug) or ANALYZE should skip and not mark gcols. Better ask somebody who's more famiiar with EITS about what fix is preferred. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] e2017fc3a5f: MDEV-22441 implement a generic way to change a value of a variable in scope
by Sergei Golubchik 03 Jun '20

03 Jun '20

Hi, Eugene! On Jun 03, Eugene Kosov wrote: > revision-id: e2017fc3a5f (mariadb-10.2.31-219-ge2017fc3a5f) This should go into 10.5, not into old-old GA 10.2. Besides, 10.2 cannot use C++11 features, 10.5 can. > parent(s): 1b3adaab25c > author: Eugene Kosov <claprix(a)yandex.ru> > committer: Eugene Kosov <eugene.kosov(a)mariadb.com> > timestamp: 2020-05-27 12:45:41 +0300 > message: > > MDEV-22441 implement a generic way to change a value of a variable in scope > > Scope_value<T> class changes a variable value to some temporary value while > it exits and restores an old value in it's destructor. > > make_scope_value() is a function targeted for C++11 which allows to omit > a template parameter which is required by Scope_value<T> I think I'd name it backup_and_change_scoped() or something like that. But it's just a suggestion, feel free to keep make_scope_value() name if you like it better. > Sql_mode_save removed and the only usage is replaced with Scope_value<T> > > Sql_mode_save removal should not be merged in newer branches! > > --- > include/scope.h | 46 ++++++++++++++++++++++++++++++++++++++++++++++ > sql/sql_acl.cc | 7 ++++--- > sql/sql_class.h | 16 ---------------- > 3 files changed, 50 insertions(+), 19 deletions(-) > > diff --git a/include/scope.h b/include/scope.h > new file mode 100644 > index 00000000000..81061ceed37 > --- /dev/null > +++ b/include/scope.h > @@ -0,0 +1,46 @@ > +/* > + Copyright (c) 2020, MariaDB Corporation. > + > + This program is free software; you can redistribute it and/or modify > + it under the terms of the GNU General Public License as published by > + the Free Software Foundation; version 2 of the License. > + > + This program is distributed in the hope that it will be useful, > + but WITHOUT ANY WARRANTY; without even the implied warranty of > + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + GNU General Public License for more details. > + > + You should have received a copy of the GNU General Public License > + along with this program; if not, write to the Free Software > + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1335 USA */ > + > +#pragma once use include guards, please, not a pragma > + > +namespace maria and, as you remember, we didn't quite decide on a global server-wide namespace, so it's not the MDEV that should introduce it. > +{ > + > +template <typename T> class Scope_value > +{ > +public: > + Scope_value(T &variable, const T &scope_value) > + : variable_(variable), saved_value_(variable) > + { > + variable= scope_value; > + } > + > + ~Scope_value() { variable_= saved_value_; } > + > +private: > + T &variable_; > + T saved_value_; > +}; > + > +// Starting from C++11 this allows to omit template argument like this: > +// auto _ = make_scope_value(var, tmp_value); > +template <typename T> > +Scope_value<T> make_scope_value(T &variable, const T &scope_value) > +{ > + return Scope_value<T>(variable, scope_value); > +} as you've seen from Sql_mode_save usage, sometimes the value is assigned later, not when it's saved. Please, add another helper that doesn't require to provide a value. Like template <typename T> Scope_value<T> backup_scoped(T &variable) { return Scope_value<T>(variable, variable); } > + > +} // namespace maria Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 0552ed86782: MDEV-21804 Assertion `marked_for_read()' failed upon INSERT into table with long unique blob under binlog_row_image=NOBLOB
by Sergei Golubchik 03 Jun '20

03 Jun '20

Hi, Sachin! ok to push with one change, see below. On Jun 03, Sachin wrote: > revision-id: 0552ed86782 (mariadb-10.4.11-223-g0552ed86782) > parent(s): 61b2cd38d48 > author: Sachin <sachin.setiya(a)mariadb.com> > committer: Sachin <sachin.setiya(a)mariadb.com> > timestamp: 2020-06-02 08:24:02 +0530 > message: > > MDEV-21804 Assertion `marked_for_read()' failed upon INSERT into table with long unique blob under binlog_row_image=NOBLOB > > Problem:- Calling mark_columns_per_binlog_row_image() earlier may change the > result of mark_virtual_columns_for_write() , Since it can set the bitmap on > for virtual column, and henceforth mark_virtual_column_deps(field) will > never be called in mark_virtual_column_with_deps. > > This bug is not specific for long unique, It also fails for this case > create table t2(id int primary key, a blob, b varchar(20) as (LEFT(a,2))); > > diff --git a/sql/table.cc b/sql/table.cc > index bc8ddbc0f9d..8edbbe8d599 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -7040,7 +7040,6 @@ void TABLE::mark_columns_needed_for_update() > DBUG_ENTER("TABLE::mark_columns_needed_for_update"); > bool need_signal= false; > > - mark_columns_per_binlog_row_image(); > > if (triggers) > triggers->mark_fields_used(TRG_EVENT_UPDATE); > @@ -7048,6 +7047,7 @@ void TABLE::mark_columns_needed_for_update() > mark_default_fields_for_write(FALSE); > if (vfield) > need_signal|= mark_virtual_columns_for_write(FALSE); > + mark_columns_per_binlog_row_image(); I think that's too early. In that if() below you can see > if (file->ha_table_flags() & HA_REQUIRES_KEY_COLUMNS_FOR_DELETE) > { > KEY *end= key_info + s->keys; ... if (bitmap_fast_test_and_set(read_set, idx)) continue; if (field[idx]->vcol_info) mark_virtual_col(field[idx]); and mark_columns_per_binlog_row_image() might break that. Better to put mark_columns_per_binlog_row_image() at the very end, just before signalling. > @@ -7132,7 +7132,6 @@ void TABLE::mark_columns_needed_for_update() > void TABLE::mark_columns_needed_for_insert() > { > DBUG_ENTER("mark_columns_needed_for_insert"); > - mark_columns_per_binlog_row_image(); > > if (triggers) > { > @@ -7152,6 +7151,7 @@ void TABLE::mark_columns_needed_for_insert() > /* Mark virtual columns for insert */ > if (vfield) > mark_virtual_columns_for_write(TRUE); > + mark_columns_per_binlog_row_image(); this looks ok. > if (check_constraints) > mark_check_constraint_columns_for_read(); > DBUG_VOID_RETURN; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] Progress report - Week 4
by Mohammed Hammaad Mateen 02 Jun '20

02 Jun '20

Greetings, Hope you are safe and doing great, This post describes the things I've done in my fourth week [25-31 May] of Community Bonding Period under the mentor-ship of Sergei Golubchik and Oleksandr Byelkin for GSoC-20. The tasks taken up for this week was to study and analyze *INSERT SELECT RETURNING* and developing a test suite for *MULTIPLE TABLE DELETE RETURNING.* The following attachment contains the test cases for evaluating Multiple Delete Returning, as they are developed before actual implementation of code, please do notify me of any improvement needed, I'd be glad to resolve. Regards, Mohammed Hammaad Mateen

2 1

Re: [Maria-developers] fab8349af86: MDEV-21804 Assertion `marked_for_read()' failed upon INSERT into table with long unique blob under binlog_row_image=NOBLOB
by Sergei Golubchik 01 Jun '20

01 Jun '20

Hi, Sachin! On May 31, Sachin wrote: > revision-id: fab8349af86 (mariadb-10.4.11-223-gfab8349af86) > parent(s): 61b2cd38d48 > author: Sachin <sachin.setiya(a)mariadb.com> > committer: Sachin <sachin.setiya(a)mariadb.com> > timestamp: 2020-05-29 11:56:45 +0530 > message: > > MDEV-21804 Assertion `marked_for_read()' failed upon INSERT into table with long unique blob under binlog_row_image=NOBLOB > > Problem:- > In the case of BINLOG_ROW_IMAGE_NOBLOB table->readset is PKE + non blob fields > But while updating virtual fields we need to read the blob field, which will > generate assert failure. > Solution:- > If binlog_row_image == NOBLOB, then set read_bit for the blob field. > This bug is not specific for long unique, It also fails for this case > create table t2(id int primary key, a blob, b varchar(20) as (LEFT(a,2))); > > diff --git a/sql/field.cc b/sql/field.cc > index 0a8fdc3d3f5..bcad056e76a 100644 > --- a/sql/field.cc > +++ b/sql/field.cc > @@ -8467,6 +8467,10 @@ longlong Field_blob::val_int(void) > String *Field_blob::val_str(String *val_buffer __attribute__((unused)), > String *val_ptr) > { > + ulong binlog_row_image= table->in_use->variables.binlog_row_image; > + if (table->read_set && !bitmap_is_set(table->read_set, field_index) && > + binlog_row_image == BINLOG_ROW_IMAGE_NOBLOB) > + bitmap_set_bit(table->read_set, field_index); > DBUG_ASSERT(marked_for_read()); > char *blob; > memcpy(&blob, ptr+packlength, sizeof(char*)); A couple of red flags here. First, you don't fix BINLOG_ROW_IMAGE_MINIMAL. And generally you should not modify read/write sets from val_str, if should be done earlier before the execution phase. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] Review for: MDEV-17399 Add support for JSON_TABLE, part #5
by Sergey Petrunia 26 May '20

26 May '20

Hi Alexey, On Fri, May 15, 2020 at 04:26:02PM +0400, Alexey Botchkov wrote: > See the next iteration here > https://github.com/MariaDB/server/commit/692cb566096d61b240ec26e84fc7d3c7d1… > So the ha_json_table;:position() was implemented. > The size of the reference depends on the nested path depth - each level > adds some 9 bytes to the initial 5. > That can be decreased but i decided to keep it simple initially and i doubt > we're going to have really deep > nesting in realistic scenarios. This is ok. I'm not sure if the server has a limit on the rowid size. Perhaps there is, and in that case we just need to limit the depth we allow. > And i'd like to ask you for some testing > queries here. Or just the model how > to produce queries that will be using the ::position() extensively. I've provided a testcase for the position() call in my previous email. https://gist.github.com/spetrunia/a905d51731c58f5439bd9f70c64cdc43 As for rnd_pos(), one case that I'm aware of is when the query does a filesort, and the row being sorted either includes a blob, or has the total length exceeding certain limit. I've tried constructing an example with blobs, and it fails with an assertion before the execution reaches rnd_pos() calls: select * from json_table('[{"color": "blue", "price": 50}, {"color": "red", "price": 100}, {"color": "rojo", "price": 10.0}, {"color": "blanco", "price": 11.0}]', '$[*]' columns( color varchar(100) path '$.color', price text path '$.price', seq for ordinality ) ) as T order by color desc; fails an assertion: mysqld: /home/psergey/dev-git/10.5-json/sql/field.cc:8309: virtual int Field_blob::store(const char*, size_t, CHARSET_INFO*): Assertion `marked_for_write_or_computed()' failed. Once that is fixed, this should use position() and rnd_pos() calls. This will likely expose more issues with rnd_pos(), see my comments to the Json_table_nested_path::set_position below. == A problem with VIEWs == mysql> select * -> from -> json_table( -> '[ {"color": "red", "price": 1}, {"color": "black", "price": 2}]', -> '$[*]' columns( color varchar(100) path '$.color')) as `T_A` where T_A.color<>'azul' ; +-------+ | color | +-------+ | red | | black | +-------+ 2 rows in set (6.34 sec) Good so far. Now, let's try creating a VIEW from this: create view v1 as select * from json_table( '[ {"color": "red", "price": 1}, {"color": "black", "price": 2}]', '$[*]' columns( color varchar(100) path '$.color')) as `T_A` where T_A.color<>'azul' ; select * from v1; ERROR 4041 (HY000): Unexpected end of JSON path in argument 2 to function 'JSON_TABLE' Examining the .frm file, I see something that looks like garbage data: query=select `T_A`.`color` AS `color` from JSON_TABLE(\'[ {"color": "red", "price": 1}, {"color": "black", "price": 2}]\', \'\'\0<D8>7^A<9C><FF>^?\0\0\0\0\0\0\0\0\0\0\0\0^A<A5><A5><A5><A5><A5>^H<D0>kWUU\0\0[ {"color": "red", "price": 1}, {"color": "black", "price": 2}]\0$[*]\'\' COLUMNS (`color` PATH \'^A<9C><FF>^?\0\0\0\0\0\0^A\0\0\0\0\0\0\0<A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5>\0\0\0\0<A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5> <A5><A5><A5><A5><A5><A5> D^A<9C><FF>^?\0\0H:^A<9C><FF>^?\0\0<A5><A5><A5><A5><A5><A5><A5><A5>100\0<8F><8F><8F><8F>$.color\')) T_A where `T_A`.`color` <> \'azul\' EXPLAIN EXTENDED ...; SHOW WARNINGS; - also print something odd. == EXPLAIN [FORMAT=JSON] == I think EXPLAIN output should provide an indication that a table function is used. MySQL does it like so: <TODO> == Unneeded recursive rules in grammar == I've already complained about this: the grammar has recursive ON EMPTY, ON ERROR rules, which cause the following to be accepted (note the two ON EMPTY clauses) : select * from json_table('[{"color": "blue", "price": 50},{"color": "red"}]', '$[*]' columns( price varchar(255) path '$.price' default 'xyz' on empty default 'abc' on empty ) ) as T; > commit 692cb566096d61b240ec26e84fc7d3c7d13f024c > Author: Alexey Botchkov <holyfoot(a)askmonty.org> > Date: Fri May 15 15:25:28 2020 +0400 > > MDEV-17399 Add support for JSON_TABLE. > > Syntax for JSON_TABLE added. > The ha_json_table handler added. Executing the JSON_TABLE we > create the temporary table of the ha_json_table, add dependencies > of other tables and sometimes conditions to WHERE. > and sometimes conditions to WHERE. I think this is not true anymore? ... > diff --git a/include/my_base.h b/include/my_base.h > index 7efa5eb9673..89ef3e8e7c1 100644 > --- a/include/my_base.h > +++ b/include/my_base.h > @@ -523,6 +523,13 @@ enum ha_base_keytype { > #define HA_ERR_TABLESPACE_MISSING 194 /* Missing Tablespace */ > #define HA_ERR_SEQUENCE_INVALID_DATA 195 > #define HA_ERR_SEQUENCE_RUN_OUT 196 > + > +/* > + Share the error code to not increment the HA_ERR_LAST for now, > + as it disturbs some storage engine's tests. > + Probably should be fixed later. > +*/ > +#define HA_ERR_INVALID_JSON HA_ERR_TABLE_IN_FK_CHECK We definitely can't have this in the final patch. We need to either: A. Use an engine-specific error code. Check out MyRocks and ha_rocksdb::get_error_message() for an example of how this is done B. Indeed introduce a generic "Invalid JSON" error code. I'm hesitant about B, let's discuss this with other developers. ... > diff --git a/sql/table_function.cc b/sql/table_function.cc > new file mode 100644 > index 00000000000..71f2378ce7d > --- /dev/null > +++ b/sql/table_function.cc ... > + > +class ha_json_table: public handler > +{ Please add comments about these > +protected: > + Table_function_json_table *m_jt; > + String m_tmps; > + String *m_js; > + uchar *m_cur_pos; > +public: > + ha_json_table(TABLE_SHARE *share_arg, Table_function_json_table *jt): > + handler(&table_function_hton.m_hton, share_arg), m_jt(jt) > + { > + /* > + set the mark_trx_read_write_done to avoid the > + handler::mark_trx_read_write_internal() call. > + It relies on &ha_thd()->ha_data[ht->slot].ha_info[0] to be set. > + But we don't set the ha_data for the ha_json_table, and > + that call makes no sence for ha_json_table. > + */ > + mark_trx_read_write_done= 1; > + ref_length= (jt->m_depth+1) * (4 + 1) + > + jt->m_depth * (sizeof(Json_table_nested_path *)); > + } > + ~ha_json_table() {} > + handler *clone(const char *name, MEM_ROOT *mem_root) { return NULL; } > + const char *index_type(uint inx) { return "NONE"; } > + /* Rows also use a fixed-size format */ > + enum row_type get_row_type() const { return ROW_TYPE_FIXED; } > + ulonglong table_flags() const > + { > + return (HA_FAST_KEY_READ | /*HA_NO_BLOBS |*/ HA_NULL_IN_KEY | > + HA_CAN_SQL_HANDLER | > + HA_REC_NOT_IN_SEQ | HA_NO_TRANSACTIONS | > + HA_HAS_RECORDS | HA_CAN_HASH_KEYS); > + } > + ulong index_flags(uint inx, uint part, bool all_parts) const > + { > + return HA_ONLY_WHOLE_INDEX | HA_KEY_SCAN_NOT_ROR; > + } > + uint max_supported_keys() const { return 1; } > + uint max_supported_key_part_length() const { return MAX_KEY_LENGTH; } > + double scan_time() { return 1000000.0; } > + double read_time(uint index, uint ranges, ha_rows rows) { return 0.0; } The above two functions are never called. Please * add a comment saying that. * add a DBUG_ASSERT() into them to back the point made by the comment :-) > + int open(const char *name, int mode, uint test_if_locked); > + int close(void) { return 0; } > + int rnd_init(bool scan); > + int rnd_next(uchar *buf); > + int rnd_pos(uchar * buf, uchar *pos); > + void position(const uchar *record); > + int can_continue_handler_scan() { return 1; } > + int info(uint); > + int extra(enum ha_extra_function operation); > + THR_LOCK_DATA **store_lock(THD *thd, THR_LOCK_DATA **to, > + enum thr_lock_type lock_type) > + { return NULL; } > + int create(const char *name, TABLE *form, HA_CREATE_INFO *create_info) > + { return 1; } > +private: > + void update_key_stats(); > +}; > + ... > +void Json_table_nested_path::set_position(const char *j_start, const uchar *pos) > +{ * This needs a comment. * I don't see where the value of m_ordinality_counter is restored? * The same about m_null - its value is not restored either? > + if (m_nested) > + { > + memcpy(&m_cur_nested, pos, sizeof(m_cur_nested)); > + pos+= sizeof(m_cur_nested); > + } > + > + m_engine.s.c_str= (uchar *) j_start + sint4korr(pos); > + m_engine.state= (int) pos[4]; > + if (m_cur_nested) > + m_cur_nested->set_position(j_start, pos); > +} > + ... > +int ha_json_table::info(uint) > +{ > + /* We don't want 0 or 1 in stats.records. */ > + stats.records= 4; > + return 0; Does this value matter? As far as I understand it doesn't, as the optimizer will use the estimates obtained from Table_function_json_table::get_estimates. Please add a comment about this. > +} > + ... Please document this function. What's last_column? Why does print() method get it and return it? > +int Json_table_nested_path::print(THD *thd, TABLE_LIST *sql_table, String *str, > + List_iterator_fast<Json_table_column> &it, > + Json_table_column **last_column) > +{ > + Json_table_nested_path *c_path= this; ... > diff --git a/sql/table_function.h b/sql/table_function.h > new file mode 100644 > index 00000000000..49d650cb7b2 > --- /dev/null > +++ b/sql/table_function.h > @@ -0,0 +1,168 @@ ... > +#include <json_lib.h> > + > +/* > + The Json_table_nested_path represents the 'current nesting' level > + for a set of JSON_TABLE columns. > + Each column (Json_table_column instance) is linked with corresponding > + 'nested path' object and gets it's piece of JSON to parse during the computation > + phase. > + The root 'nested_path' is always present as a part of Table_function_json_table, > + then other 'nested_paths' can be created and linked into a tree structure when new > + 'NESTED PATH' is met. The nested 'nested_paths' are linked with 'm_nested', the same-level > + 'nested_paths' are linked with 'm_next_nested'. > + So for instance > + JSON_TABLE( '...', '$[*]' > + COLUMNS( a INT PATH '$.a' , > + NESTED PATH '$.b[*]' COLUMNS (b INT PATH '$', > + NESTED PATH '$.c[*]' COLUMNS(x INT PATH '$')), > + NESTED PATH '$.n[*]' COLUMNS (z INT PAHT '$')) > + results in 4 'nested_path' created: > + root nested_b nested_c nested_n > + m_path '$[*]' '$.b[*]' '$.c[*]' '$.n[*] > + m_nested &nested_b &nested_c NULL NULL > + n_next_nested NULL &nested_n NULL NULL > + > +and 4 columns created: > + a b x z > + m_nest &root &nested_b &nested_c &nested_n > +*/ > + > + > +class Json_table_column; > + > +class Json_table_nested_path : public Sql_alloc > +{ > +public: > + bool m_null; > + json_path_t m_path; > + json_engine_t m_engine; > + json_path_t m_cur_path; > + > + /* Counts the rows produced. Value is set to the FOR ORDINALITY coluns */ > + longlong m_ordinality_counter; > + > + Json_table_nested_path *m_parent; > + Json_table_nested_path *m_nested, *m_next_nested; > + Json_table_nested_path **m_nested_hook; > + Json_table_nested_path *m_cur_nested; Please add documentation about the above members. I see the diagram above, but I think text descriptions are also needed for each member. What's m_nested_hook? > + Json_table_nested_path(Json_table_nested_path *parent_nest): > + m_parent(parent_nest), m_nested(0), m_next_nested(0), > + m_nested_hook(&m_nested) {} > + int set_path(THD *thd, const LEX_CSTRING &path); > + void scan_start(CHARSET_INFO *i_cs, const uchar *str, const uchar *end); > + int scan_next(); > + int print(THD *thd, TABLE_LIST *sql_table, String *str, > + List_iterator_fast<Json_table_column> &it, > + Json_table_column **last_column); > + void get_current_position(const char *j_start, uchar *pos) const; > + void set_position(const char *j_start, const uchar *pos); > +}; ... > +class Table_function_json_table : public Sql_alloc Please document the class and the data members. > +{ > +public: > + Item *m_json; > + Json_table_nested_path m_nested_path; > + List<Json_table_column> m_columns; > + table_map m_dep_tables; > + uint m_depth, m_cur_depth; > + > + Table_function_json_table(Item *json): m_json(json), m_nested_path(0), > + m_depth(0), m_cur_depth(0) {} > + > + /* > + Used in sql_yacc.yy. > + Represents the current NESTED PATH level being parsed. > + */ > + Json_table_nested_path *m_sql_nest; > + void add_nested(Json_table_nested_path *np); > + void leave_nested(); BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] 070df171c1e: MDEV-16937 Strict SQL with system versioned tables causes issues
by Sergei Golubchik 26 May '20

26 May '20

Hi, Aleksey! ok to push On May 26, Aleksey Midenkov wrote: > revision-id: 070df171c1e (mariadb-10.3.21-91-g070df171c1e) > parent(s): ad41da5c934 > author: Aleksey Midenkov <midenok(a)gmail.com> > committer: Aleksey Midenkov <midenok(a)gmail.com> > timestamp: 2020-05-25 14:47:26 +0300 > message: > > MDEV-16937 Strict SQL with system versioned tables causes issues > > Respect system fields in NO_ZERO_DATE mode. > > This is the subject for refactoring in MDEV-19597 > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] cd9cab54aac: MDEV-20015 Assertion `!in_use->is_error()' failed in TABLE::update_virtual_field
by Sergei Golubchik 26 May '20

26 May '20

Hi, Aleksey! Ok to push, thanks! On May 26, Aleksey Midenkov wrote: > revision-id: cd9cab54aac (mariadb-10.2.31-123-gcd9cab54aac) > parent(s): d275ecbd208 > author: Aleksey Midenkov <midenok(a)gmail.com> > committer: Aleksey Midenkov <midenok(a)gmail.com> > timestamp: 2020-05-25 20:56:31 +0300 > message: > > MDEV-20015 Assertion `!in_use->is_error()' failed in TABLE::update_virtual_field > > update_virtual_field() is called as part of index rebuild in > ha_myisam::repair() (MDEV-5800) which is done on bulk INSERT finish. > > Assertion in update_virtual_field() was put as part of MDEV-16222 > because update_virtual_field() returns in_use->is_error(). The idea: > wrongly mixed semantics of error status before update_virtual_field() > and the status returned by update_virtual_field(). The former can > falsely influence the latter. > > diff --git a/sql/table.cc b/sql/table.cc > index d6d86d96016..2429bb12abe 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -7707,15 +7707,17 @@ int TABLE::update_virtual_fields(handler *h, enum_vcol_update_mode update_mode) > > int TABLE::update_virtual_field(Field *vf) > { > - DBUG_ASSERT(!in_use->is_error()); > - Query_arena backup_arena; > DBUG_ENTER("TABLE::update_virtual_field"); > + Query_arena backup_arena; > + Counting_error_handler count_errors; > + in_use->push_internal_handler(&count_errors); > in_use->set_n_backup_active_arena(expr_arena, &backup_arena); > bitmap_clear_all(&tmp_set); > vf->vcol_info->expr->walk(&Item::update_vcol_processor, 0, &tmp_set); > vf->vcol_info->expr->save_in_field(vf, 0); > in_use->restore_active_arena(expr_arena, &backup_arena); > - DBUG_RETURN(in_use->is_error()); > + in_use->pop_internal_handler(); > + DBUG_RETURN(count_errors.errors); > } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 7d593466a22: MDEV-20015 Assertion `!in_use->is_error()' failed in TABLE::update_virtual_field
by Sergei Golubchik 25 May '20

25 May '20

Hi, Aleksey! I don't see what you've changed. We've discussed that fix and that one isn't supposed to swap Diagnostics_area's like that. And in your new patch you do exactly the same. Possible correct approaches: * don't return in_use->is_error(), return the return value of vf->vcol_info->expr->walk() || vf->vcol_info->expr->save_in_field() This means that Item_field::update_vcol_processor() should also do the same, I suspect * Use thd->push_internal_handler() and Counting_error_handler. Or, better, Turn_errors_to_warnings_handler with counting. This is the simplest one. there's a third option: * always return 0, because, looking how it's used, I don't really see how update_virtual_field() can ever get an error. But it's not a particularly future-proof approach. And I just might be wrong about errors. On Apr 23, Aleksey Midenkov wrote: > revision-id: 7d593466a22 (mariadb-10.2.28-4-g7d593466a22) > parent(s): 7bc26de591c > author: Aleksey Midenkov <midenok(a)gmail.com> > committer: Aleksey Midenkov <midenok(a)gmail.com> > timestamp: 2019-11-07 10:45:21 +0300 > message: > > MDEV-20015 Assertion `!in_use->is_error()' failed in TABLE::update_virtual_field > > Preserve and restore statement DA. > > update_virtual_field() is called as part of index rebuild in > ha_myisam::repair() (MDEV-5800) which is done on bulk INSERT finish. > > Assertion in update_virtual_field() was put as part of MDEV-16222 > because update_virtual_field() returns in_use->is_error(). The idea: > wrongly mixed semantics of error status before update_virtual_field() > and the status returned by update_virtual_field(). The former can > falsely influence the latter. > > Preserve global error status and run update_virtual_field() with clear > DA since no matter how SQL command is finished it must update the > index after bulk INSERT. > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

[Maria-developers] Progress Report - Week 3
by Mohammed Hammaad Mateen 25 May '20

25 May '20

Greetings, Hope you are safe and doing great, This post describes the things I've done in my third week [ 18-24 May ] of Community Bonding Period under the mentor-ship of Sergei Golubchik and Oleksandr Byelkin for GSoC-20. The tasks taken up for this week was to study and analyze: *INSERT.... RETURNING:* • *TableList* consists of 1 table instance to use, where each instance is a structure consisting of db argument, table name argument, alias argument and lock type argument. *Referenced this from the table.h file.* • In *sql_parse.cc* <http://sql_parse.cc> under *case SQLCOM_INSERT*: we do a DBUG_ASSERT where first_table == all_table && first_table!=0 // we check if only 1 table is used and the table used is already created. • if lex-> has_returning() we increment the system status var by 1 and perform analyze.. insert.. returning. • we compute result by mysql_insert function by passing the following arguments. - Thread Handler thd - all table instances - list of all the fields used - the values we want to insert - the list of update feilds - and update value list - duplicate flag - ignore - result of select • if Inserting fails due to some reason we equate result to send_explain(thd) • we also update the MYSQL_INSERT_DONE with result of insert and the the row count. *INSERT.. SELECT.. RETURNING:* • We fix the lock on first table. • lock other tables until command is written to the binary log. • the procedure is same as discussed earlier with respect to insert.... returning. • To switch to the second table we traverse from first_table to (->) next_local and we compute select result with the help of select insert function. • we can now unlock the tables and we also need to check if something changed after unlocking, of that happens we should invalidate the table from the query cache using query_cache_invalidate3 function. • Manual cleaning of select result obtained from select insert must be done. Regards, Mohammed Hammaad Mateen

1 0

Re: [Maria-developers] MDEV-22461: JOIN::make_aggr_tables_info(): Assertion `select_options & (1ULL << 17)' failed.
by Sergey Petrunia 24 May '20

24 May '20

Hi Varun, - Please add a testcase to the testsuite - Please address a few cosmetic comments below - Ok to push after addressed. > commit 5e448b77a3812e65623c0a1214049322ace3aacf > Author: Varun Gupta <varun.gupta(a)mariadb.com> > Date: Tue May 5 20:44:43 2020 +0530 > > MDEV-22461: JOIN::make_aggr_tables_info(): Assertion `select_options & (1ULL << 17)' failed. > > A temporary table is needed for window function computation but if only a NAMED WINDOW SPEC > is used and there is no window function, then there is no need to create a temporary > table as there is no stage to compute WINDOW FUNCTION > > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index c601946cfa0..0a1d0c2dbcc 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -2014,11 +2014,16 @@ JOIN::optimize_inner() > } > > need_tmp= test_if_need_tmp_table(); > - //TODO this could probably go in test_if_need_tmp_table. > - if (this->select_lex->window_specs.elements > 0) { > - need_tmp= TRUE; > + > + /* > + If window functions are present then we can't have simple_order set to > + TRUE as the window function needs a temp table for compuatation. typo: compuatation. > + ORDER BY is computed after the window function computation is done, so > + the sort would be done on the temp table. > + */ > + if (this->select_lex->have_window_funcs()) Is there a need to have "this->select_lex"? I think not, please remove, as it only confuses the reader. > simple_order= FALSE; > - } > + > > /* > If the hint FORCE INDEX FOR ORDER BY/GROUP BY is used for the table > diff --git a/sql/sql_select.h b/sql/sql_select.h > index 0e011c9267a..7a892c1af89 100644 > --- a/sql/sql_select.h > +++ b/sql/sql_select.h > @@ -1645,7 +1645,8 @@ class JOIN :public Sql_alloc > ((select_distinct || !simple_order || !simple_group) || > (group_list && order) || > MY_TEST(select_options & OPTION_BUFFER_RESULT))) || > - (rollup.state != ROLLUP::STATE_NONE && select_distinct)); > + (rollup.state != ROLLUP::STATE_NONE && select_distinct) || > + select_lex->have_window_funcs()); Please amend the function comment above this code to cover the new condition as well. > } > bool choose_subquery_plan(table_map join_tables); > void get_partial_cost_and_fanout(int end_tab_idx, > BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] b22a28c2295: fixup! 3fe5cd5e1785e3e8de7add9977a1c2ddd403538b
by Michael Widenius 22 May '20

22 May '20

Hi! On Fri, May 22, 2020 at 3:27 PM Andrei Elkin <andrei.elkin(a)mariadb.com> wrote: <cut> CORNER CASES: read-only, pure myisam, binlog-*, @@skip_log_bin, etc > > Aria just makes yet another previously unknown use case of an engine > that produces THD::ha_info but does not support 2pc, which the assert > implied. > > To explain more, the original block > > #ifndef DBUG_OFF > for (ha_info= thd->transaction.all.ha_list; rw_count > 1 && ha_info; > ha_info= ha_info->next()) > DBUG_ASSERT(ha_info->ht() != binlog_hton); > #endif > > claims there most be no binlog hton in a transaction consisting of more > than 1 hton:s, *when* (at this point) this transaction has not been > binlogged yet. > So combination of binlog + Innodb hton would be raise the assert, to > question "why the heck binlogging has not been done yet?!". > > Aria is different from Innodb in this context in that binlogging > was done at the end of the statement, so to miss > `cache_mngr->need_unlog' flagging (which is at xa prepare time logging). Thanks for the explanation, we should have had that in the code. > I think we should fix the assert rather than to remove. This way: > > cat > assert.diff <<. > diff --git a/sql/log.cc b/sql/log.cc > index 792c6bb1a99..aaf1fae1cd6 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -10124,6 +10124,16 @@ int TC_LOG_BINLOG::unlog_xa_prepare(THD *thd, bool all) > Ha_trx_info *ha_info; > uint rw_count= ha_count_rw_all(thd, &ha_info); > bool rc= false; > +#ifndef DBUG_OFF > + bool no_binlog= true, exist_no_2pc= false; > + for (ha_info= thd->transaction->all.ha_list; rw_count > 1 && ha_info; > + ha_info= ha_info->next()) > + { > + no_binlog= no_binlog && ha_info->ht() != binlog_hton; > + exist_no_2pc= exist_no_2pc || !ha_info->ht()->prepare; > + } > + DBUG_ASSERT(no_binlog || exist_no_2pc); > +#endif > > if (rw_count > 0) > { > . > > I tested it briefly with running XA:s on combination of engines > including. On which tree did you test? bb-10.5-aria? In the end, after discussions on slack, we ended with: #ifndef DBUG_OFF if (rw_count > 1) { /* There must be no binlog_hton used in a transaction consisting of more than 1 engine, *when* (at this point) this transaction has not been binlogged. The one exception is if there is an engine without a prepare method, as in this case the engine doesn't support XA and we have to ignore this check. */ bool binlog= false, exist_hton_without_prepare= false; for (ha_info= thd->transaction->all.ha_list; ha_info; ha_info= ha_info->next()) { if (ha_info->ht() == binlog_hton) binlog= true; if (!ha_info->ht()->prepare) exist_hton_without_prepare= true; } DBUG_ASSERT(!binlog || exist_hton_without_prepare); } #endif > > The reason it was not hit > > is that before Aria was not treated as transactional we could only > > come here in case of errors > > and that code was was apparently not tested, at least with binary logging on. > > > > > With Aria we could come here in case of rollback and we got assert for > > cases that was perfectly ok. > > Well, what 'rollback' do you mean? The function is invoked only > for ha_prepare(). As far as I remember, I come into this code in some edge case when something failed that normally never fails. I think the issue was that Aria doesn't have a prepare handler, which caused issues in ha_prepare(), but not sure. Anyway, we now have a solution for this. Regards, Monty

1 0

Re: [Maria-developers] acbe14b122c: Aria will now register it's transactions
by Sergei Golubchik 22 May '20

22 May '20

Hi, Michael! See below On May 20, Michael Widenius wrote: > revision-id: acbe14b122c (mariadb-10.5.2-255-gacbe14b122c) > parent(s): 48a758696bf > author: Michael Widenius <monty(a)mariadb.com> > committer: Michael Widenius <monty(a)mariadb.com> > timestamp: 2020-05-19 17:52:17 +0300 > message: > > Aria will now register it's transactions > > MDEV-22531 Remove maria::implicit_commit() > diff --git a/sql/sql_class.h b/sql/sql_class.h > index d4a95fa3fd8..a7a071f7cdb 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -5149,6 +5151,40 @@ class THD: public THD_count, /* this must be first */ > > }; > > + > +/* > + Start a new independent transaction for the THD. > + The old one is stored in this object and restored when calling > + restore_old_transaction() or when the object is freed > +*/ > + > +class start_new_trans > +{ > + /* container for handler's private per-connection data */ > + Ha_data old_ha_data[MAX_HA]; > + struct THD::st_transactions *old_transaction, new_transaction; > + Open_tables_backup open_tables_state_backup; > + MDL_savepoint mdl_savepoint; > + PSI_transaction_locker *m_transaction_psi; > + THD *org_thd; > + uint in_sub_stmt; > + uint server_status; > + > +public: > + start_new_trans(THD *thd); > + ~start_new_trans() > + { > + destroy(); > + } > + void destroy() > + { > + if (org_thd) // Safety > + restore_old_transaction(); > + new_transaction.free(); > + } > + void restore_old_transaction(); interesting. You made restore_old_transaction() public and you use it in many places. Why not to use destroy() instead? When one would want to use restore_old_transaction() instead of destroy()? > +}; > + > /** A short cut for thd->get_stmt_da()->set_ok_status(). */ > > inline void > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index dda8e00f6bf..51d7380f622 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -5742,6 +5744,90 @@ void THD::mark_transaction_to_rollback(bool all) > } > > > +/** > + Commit the whole transaction (both statment and all) > + > + This is used mainly to commit an independent transaction, > + like reading system tables. > + > + @return 0 0k > + @return <>0 error code. my_error() has been called() > +*/ > + > +int THD::commit_whole_transaction_and_close_tables() > +{ > + int error, error2; > + DBUG_ENTER("THD::commit_whole_transaction_and_close_tables"); > + > + /* > + This can only happened if we failed to open any table in the > + new transaction > + */ > + if (!open_tables) > + DBUG_RETURN(0); Generally, I think, it should still end the transaction here. Or add an assert that there is no active transaction at the moment. > + > + /* > + Ensure table was locked (opened with open_and_lock_tables()). If not > + the THD can't be part of any transactions and doesn't have to call > + this function. > + */ > + DBUG_ASSERT(lock); > + > + error= ha_commit_trans(this, FALSE); > + /* This will call external_lock to unlock all tables */ > + if ((error2= mysql_unlock_tables(this, lock))) > + { > + my_error(ER_ERROR_DURING_COMMIT, MYF(0), error2); > + error= error2; > + } > + lock= 0; > + if ((error2= ha_commit_trans(this, TRUE))) > + error= error2; > + close_thread_tables(this); I wonder why you're doing it in that specific order. commit(stmt)-unlock-commit(all)-close > + DBUG_RETURN(error); > +} > + > +/** > + Start a new independent transaction > +*/ > + > +start_new_trans::start_new_trans(THD *thd) > +{ > + org_thd= thd; > + mdl_savepoint= thd->mdl_context.mdl_savepoint(); > + memcpy(old_ha_data, thd->ha_data, sizeof(old_ha_data)); > + thd->reset_n_backup_open_tables_state(&open_tables_state_backup); > + bzero(thd->ha_data, sizeof(thd->ha_data)); > + old_transaction= thd->transaction; > + thd->transaction= &new_transaction; > + new_transaction.on= 1; > + in_sub_stmt= thd->in_sub_stmt; > + thd->in_sub_stmt= 0; > + server_status= thd->server_status; > + m_transaction_psi= thd->m_transaction_psi; > + thd->m_transaction_psi= 0; > + thd->server_status&= ~(SERVER_STATUS_IN_TRANS | > + SERVER_STATUS_IN_TRANS_READONLY); > + thd->server_status|= SERVER_STATUS_AUTOCOMMIT; > +} Few thoughts: 1. If you need to save and restore _all that_ then, perhaps, all that should be inside st_transactions ? 2. strictly speaking, ha_data is _per connection_. If you just bzero it, the engine will think it's a new connection, and you cannot just overwrite it on restore without hton->close_connection. A strictly "proper" solution would be to introduce ha_data per transaction in addition to what we have now. But it looks like an overkill. So I'd just add close_system_tables() now into restore_old_transaction() A strictly "proper" solution would be to introduce ha_data per transaction in addition to what we have now. But it looks like an overkill. So I'd just add ha_close_connection() now into restore_old_transaction() and that's all. I see that you've added free_transaction() call, but isn't it redundant? ha_close_connection() does that now. Otherwise very good, thanks, I cannot wait to start using it more for other features. > + > + > +void start_new_trans::restore_old_transaction() > +{ > + org_thd->transaction= old_transaction; > + org_thd->restore_backup_open_tables_state(&open_tables_state_backup); > + ha_free_transactions(org_thd); > + memcpy(org_thd->ha_data, old_ha_data, sizeof(old_ha_data)); > + org_thd->mdl_context.rollback_to_savepoint(mdl_savepoint); > + org_thd->in_sub_stmt= in_sub_stmt; > + org_thd->server_status= server_status; > + if (org_thd->m_transaction_psi) > + MYSQL_COMMIT_TRANSACTION(org_thd->m_transaction_psi); > + org_thd->m_transaction_psi= m_transaction_psi; > + org_thd= 0; > +} > + > + > /** > Decide on logging format to use for the statement and issue errors > or warnings as needed. The decision depends on the following > diff --git a/sql/event_db_repository.cc b/sql/event_db_repository.cc > index af43d92dea7..82b3968de85 100644 > --- a/sql/event_db_repository.cc > +++ b/sql/event_db_repository.cc > @@ -742,7 +748,8 @@ Event_db_repository::create_event(THD *thd, Event_parse_data *parse_data, > ret= 0; > > end: > - close_thread_tables(thd); > + if (table) > + thd->commit_whole_transaction_and_close_tables(); You're checking twice. Here in the caller, and the first thing inside commit_whole_transaction_and_close_tables(). Here and in many places. If you want to check in the caller, there's no need to check inside? You can add an assert instead, can't you? > thd->mdl_context.rollback_to_savepoint(mdl_savepoint); > > thd->variables.sql_mode= saved_mode; > @@ -1117,22 +1124,20 @@ update_timing_fields_for_event(THD *thd, > TABLE *table= NULL; > Field **fields; > int ret= 1; > - enum_binlog_format save_binlog_format; > MYSQL_TIME time; > DBUG_ENTER("Event_db_repository::update_timing_fields_for_event"); > > - /* > - Turn off row binlogging of event timing updates. These are not used > - for RBR of events replicated to the slave. > - */ > - save_binlog_format= thd->set_current_stmt_binlog_format_stmt(); > - > DBUG_ASSERT(thd->security_ctx->master_access & PRIV_IGNORE_READ_ONLY); > > if (open_event_table(thd, TL_WRITE, &table)) > goto end; Why do you reuse a current transaction, instead of creating a new one as everywhere above? > > fields= table->field; > + /* > + Turn off row binlogging of event timing updates. These are not used > + for RBR of events replicated to the slave. > + */ > + table->file->row_logging= 0; > > if (find_named_event(event_db_name, event_name, table)) > goto end; > diff --git a/sql/handler.h b/sql/handler.h > index e4903172c33..ebe23b97062 100644 > --- a/sql/handler.h > +++ b/sql/handler.h > @@ -1765,6 +1766,12 @@ handlerton *ha_default_tmp_handlerton(THD *thd); > */ > #define HTON_TABLE_MAY_NOT_EXIST_ON_SLAVE (1 << 15) > > +/* > + True if handler cannot roolback transactions. If not true, the transaction > + will be put in the transactional binlog cache. > +*/ > +#define HTON_NO_ROLLBACK (1 << 16) How is it different from HA_PERSISTENT_TABLE? > + > class Ha_trx_info; > > struct THD_TRANS > diff --git a/sql/ha_partition.cc b/sql/ha_partition.cc > index 582c9bb110b..0c53bdc5bbe 100644 > --- a/sql/ha_partition.cc > +++ b/sql/ha_partition.cc > @@ -11016,7 +11016,7 @@ int ha_partition::check_misplaced_rows(uint read_part_id, bool do_repair) > If the engine supports transactions, the failure will be > rollbacked. if you're changing it anyway, can you also change rollbacked -> rolled back ? thanks. > */ > - if (!m_file[correct_part_id]->has_transactions()) > + if (!m_file[correct_part_id]->has_transactions_and_rollback()) > { > /* Log this error, so the DBA can notice it and fix it! */ > sql_print_error("Table '%-192s' failed to move/insert a row" > diff --git a/sql/handler.cc b/sql/handler.cc > index 39841cc28d7..e3f5773717d 100644 > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -4514,7 +4515,6 @@ void handler::mark_trx_read_write_internal() > */ > if (ha_info->is_started()) > { > - DBUG_ASSERT(has_transaction_manager()); there should be *some* assert here, I think > /* > table_share can be NULL in ha_delete_table(). See implementation > of standalone function ha_delete_table() in sql_base.cc. > diff --git a/sql/share/errmsg-utf8.txt b/sql/share/errmsg-utf8.txt > index 250da9948a0..1f3bf212d3f 100644 > --- a/sql/share/errmsg-utf8.txt > +++ b/sql/share/errmsg-utf8.txt > @@ -7961,3 +7961,5 @@ ER_KEY_CONTAINS_PERIOD_FIELDS > eng "Key %`s cannot explicitly include column %`s" > ER_KEY_CANT_HAVE_WITHOUT_OVERLAPS > eng "Key %`s cannot have WITHOUT OVERLAPS" > +ER_DATA_WAS_COMMITED_UNDER_ROLLBACK > + eng "Engine %s does not support rollback. Changes where commited during rollback call" A confusing error message. I'd just say "table is not transactional". This is the user point of view difference between a "transactional engine" like InnoDB and "crash safe engine" like Aria. It's very confusing to invent a new category of "transactional engines that cannot roll back" even if internally Aria is treated like one. > diff --git a/sql/sp.cc b/sql/sp.cc > index 51bbeeef368..1629290eb73 100644 > --- a/sql/sp.cc > +++ b/sql/sp.cc > @@ -470,27 +471,29 @@ static Proc_table_intact proc_table_intact; > currently open tables will be saved, and from which will be > restored when we will end work with mysql.proc. > > + NOTES > + On must have a start_new_trans object active when calling this function I think this: DBUG_ASSERT(thd->transcation != &thd->default_transaction); could be a bit safer than a NOTE :) > + > @retval > 0 Error > @retval > \# Pointer to TABLE object of mysql.proc > */ > > -TABLE *open_proc_table_for_read(THD *thd, Open_tables_backup *backup) > +TABLE *open_proc_table_for_read(THD *thd) > { > TABLE_LIST table; > - > DBUG_ENTER("open_proc_table_for_read"); > > table.init_one_table(&MYSQL_SCHEMA_NAME, &MYSQL_PROC_NAME, NULL, TL_READ); > > - if (open_system_tables_for_read(thd, &table, backup)) > + if (open_system_tables_for_read(thd, &table)) > DBUG_RETURN(NULL); > > if (!proc_table_intact.check(table.table, &proc_table_def)) > DBUG_RETURN(table.table); > > - close_system_tables(thd, backup); > + thd->commit_whole_transaction_and_close_tables(); > > DBUG_RETURN(NULL); > } > @@ -517,7 +520,8 @@ static TABLE *open_proc_table_for_update(THD *thd) > MDL_savepoint mdl_savepoint= thd->mdl_context.mdl_savepoint(); > DBUG_ENTER("open_proc_table_for_update"); > same? also need start_new_trans? > > - table_list.init_one_table(&MYSQL_SCHEMA_NAME, &MYSQL_PROC_NAME, NULL, TL_WRITE); > + table_list.init_one_table(&MYSQL_SCHEMA_NAME, &MYSQL_PROC_NAME, NULL, > + TL_WRITE); > > if (!(table= open_system_table_for_update(thd, &table_list))) > DBUG_RETURN(NULL); > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > index 6c938670fdc..325a0f1d41d 100644 > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -4259,7 +4259,7 @@ bool open_tables(THD *thd, const DDL_options_st &options, > list, we still need to call open_and_process_routine() to take > MDL locks on the routines. > */ > - if (thd->locked_tables_mode <= LTM_LOCK_TABLES) > + if (thd->locked_tables_mode <= LTM_LOCK_TABLES && *sroutine_to_open) why? > { > /* > Process elements of the prelocking set which are present there > @@ -8887,17 +8887,16 @@ bool is_equal(const LEX_CSTRING *a, const LEX_CSTRING *b) > open_system_tables_for_read() > thd Thread context. > table_list List of tables to open. > - backup Pointer to Open_tables_state instance where > - information about currently open tables will be > - saved, and from which will be restored when we will > - end work with system tables. > > NOTES > + Caller should have used start_new_trans object to start a new > + transcation when reading system tables. assert, may be? > + > Thanks to restrictions which we put on opening and locking of > system tables for writing, we can open and lock them for reading > - even when we already have some other tables open and locked. One > - must call close_system_tables() to close systems tables opened > - with this call. > + even when we already have some other tables open and locked. > + One should call thd->commit_whole_transaction_and_close_tables() > + to close systems tables opened with this call. > > NOTES > In some situations we use this function to open system tables for > diff --git a/sql/sql_help.cc b/sql/sql_help.cc > index c9307b578fc..3ccab553bfe 100644 > --- a/sql/sql_help.cc > +++ b/sql/sql_help.cc > @@ -709,8 +709,9 @@ static bool mysqld_help_internal(THD *thd, const char *mask) > Reset and backup the current open tables state to > make it possible. > */ > - Open_tables_backup open_tables_state_backup; > - if (open_system_tables_for_read(thd, tables, &open_tables_state_backup)) > + start_new_trans new_trans(thd); I'm starting to think that this (and in some other places) is rather redundant. Transactions can be expensive, the overhead being quite big. It seems like a overkill to wrap read-only system table accesses in a separate transaction, they could as well be performed as a part of the parent transaction. Write accesses has to be wrapped in a dedicated transaction, of course. > + > + if (open_system_tables_for_read(thd, tables)) > goto error2; > > /* > diff --git a/sql/sql_show.cc b/sql/sql_show.cc > index 2528134f4ee..db5b4d1c5fd 100644 > --- a/sql/sql_show.cc > +++ b/sql/sql_show.cc > @@ -6105,7 +6105,7 @@ static my_bool iter_schema_engines(THD *thd, plugin_ref plugin, > table->field[1]->store(option_name, strlen(option_name), scs); > table->field[2]->store(plugin_decl(plugin)->descr, > strlen(plugin_decl(plugin)->descr), scs); > - tmp= &yesno[MY_TEST(hton->commit)]; > + tmp= &yesno[MY_TEST(hton->commit && !(hton->flags & HTON_NO_ROLLBACK))]; Why not to say that hton->rollback==NULL means no rollback? Then you won't need a new flag for that. And you'll remove redundancy, what would it mean if HTON_NO_ROLLBACK is present, but hton->rollback!=NULL? Or the other way around, HTON_NO_ROLLBACK is not present, but hton->rollback==NULL? > table->field[3]->store(tmp->str, tmp->length, scs); > table->field[3]->set_notnull(); > tmp= &yesno[MY_TEST(hton->prepare)]; > diff --git a/sql/sql_statistics.cc b/sql/sql_statistics.cc > index 55e8e52c052..7a817aea97e 100644 > --- a/sql/sql_statistics.cc > +++ b/sql/sql_statistics.cc > @@ -230,17 +230,17 @@ index_stat_def= {INDEX_STAT_N_FIELDS, index_stat_fields, 4, index_stat_pk_col}; > Open all statistical tables and lock them > */ > > -static int open_stat_tables(THD *thd, TABLE_LIST *tables, > - Open_tables_backup *backup, bool for_write) > +static int open_stat_tables(THD *thd, TABLE_LIST *tables, bool for_write) > { > int rc; > - > Dummy_error_handler deh; // suppress errors > + DBUG_ASSERT(thd->transaction != &thd->default_transaction); I see, you already use such an assert :) > + > thd->push_internal_handler(&deh); > init_table_list_for_stat_tables(tables, for_write); > init_mdl_requests(tables); > thd->in_sub_stmt|= SUB_STMT_STAT_TABLES; > - rc= open_system_tables_for_read(thd, tables, backup); > + rc= open_system_tables_for_read(thd, tables); > thd->in_sub_stmt&= ~SUB_STMT_STAT_TABLES; > thd->pop_internal_handler(); > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] b22a28c2295: fixup! 3fe5cd5e1785e3e8de7add9977a1c2ddd403538b
by Sergei Golubchik 22 May '20

22 May '20

Hi, Michael! On May 20, Michael Widenius wrote: > revision-id: b22a28c2295 (mariadb-10.5.2-256-gb22a28c2295) > parent(s): acbe14b122c > author: Michael Widenius <monty(a)mariadb.com> > committer: Michael Widenius <monty(a)mariadb.com> > timestamp: 2020-05-19 17:52:18 +0300 > message: > > fixup! 3fe5cd5e1785e3e8de7add9977a1c2ddd403538b > > MDEV-22607 Assertion `ha_info->ht() != binlog_hton' failed in > MYSQL_BIN_LOG::unlog_xa_prepare > > diff --git a/sql/log.cc b/sql/log.cc > index 7e9e231358a..792c6bb1a99 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -10128,11 +10128,6 @@ int TC_LOG_BINLOG::unlog_xa_prepare(THD *thd, bool all) > if (rw_count > 0) > { > /* an empty XA-prepare event group is logged */ > -#ifndef DBUG_OFF > - for (ha_info= thd->transaction->all.ha_list; rw_count > 1 && ha_info; > - ha_info= ha_info->next()) > - DBUG_ASSERT(ha_info->ht() != binlog_hton); > -#endif > rc= write_empty_xa_prepare(thd, cache_mngr); // normally gains need_unlog > trans_register_ha(thd, true, binlog_hton, 0); // do it for future commmit > } What does this assertion mean and what does it mean that it no longer holds? Regards, Sergei

2 1

Re: [Maria-developers] 20d54b09830: Update galera to work with independent sub transactions
by Sergei Golubchik 22 May '20

22 May '20

Hi, Michael! I see, you basically disable wsrep for these out-of-band transactions. I'm sure it works fine now, with Aria. But will it work with InnoDB? On May 20, Michael Widenius wrote: > revision-id: 20d54b09830 (mariadb-10.5.2-257-g20d54b09830) > parent(s): b22a28c2295 > author: Michael Widenius <monty(a)mariadb.com> > committer: Michael Widenius <monty(a)mariadb.com> > timestamp: 2020-05-19 17:55:00 +0300 > message: > > Update galera to work with independent sub transactions > > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index 51d7380f622..4c87cbeee59 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -5806,6 +5806,8 @@ start_new_trans::start_new_trans(THD *thd) > server_status= thd->server_status; > m_transaction_psi= thd->m_transaction_psi; > thd->m_transaction_psi= 0; > + wsrep_on= thd->variables.wsrep_on; > + thd->variables.wsrep_on= 0; > thd->server_status&= ~(SERVER_STATUS_IN_TRANS | > SERVER_STATUS_IN_TRANS_READONLY); > thd->server_status|= SERVER_STATUS_AUTOCOMMIT; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] c5238d7b3b7: MDEV-22545: my_vsnprintf behaves not as in C standard
by Sergei Golubchik 20 May '20

20 May '20

Hi, Oleksandr! On May 20, Oleksandr Byelkin wrote: > revision-id: c5238d7b3b7 (mariadb-10.5.3-200-gc5238d7b3b7) > parent(s): 8f9c53586bf > author: Oleksandr Byelkin <sanja(a)mariadb.com> > committer: Oleksandr Byelkin <sanja(a)mariadb.com> > timestamp: 2020-05-20 08:35:12 +0200 > message: > > MDEV-22545: my_vsnprintf behaves not as in C standard > > Added parameter %T for string which should be visibly truncated. > > diff --git a/include/mysql/service_my_snprintf.h b/include/mysql/service_my_snprintf.h > index bd1f069c527..94a9e07dcbb 100644 > --- a/include/mysql/service_my_snprintf.h > +++ b/include/mysql/service_my_snprintf.h > @@ -69,6 +70,9 @@ > Format 'M': takes one integer, prints this integer, space, double quote > error message, double quote. In other words > printf("%M", n) === printf("%d \"%s\"", n, strerror(n)) > + > + Format 'T': takes string and print it like s but if the strints shoud be should > + truncated puts "..." at the end. > */ > > #ifdef __cplusplus > diff --git a/mysql-test/main/frm-debug.result b/mysql-test/main/frm-debug.result > index caf344e241d..332d7e00a8f 100644 > --- a/mysql-test/main/frm-debug.result > +++ b/mysql-test/main/frm-debug.result > @@ -13,8 +13,8 @@ CREATE TABLE t1 (c01 INT, c02 CHAR(20), c03 TEXT, c04 DOUBLE); > Warnings: > Note 1105 build_frm_image: Field data type info length: 14 > Note 1105 DBUG: [0] name='c01' type_info='' > -Note 1105 DBUG: [1] name='c02' type_info='xc...' > -Note 1105 DBUG: [2] name='c03' type_info='xb...' > +Note 1105 DBUG: [1] name='c02' type_info='xchar' > +Note 1105 DBUG: [2] name='c03' type_info='xblob' good > Note 1105 DBUG: [3] name='c04' type_info='' > SET SESSION debug_dbug="-d,frm_data_type_info_emulate"; > SET SESSION debug_dbug="-d,frm_data_type_info"; > diff --git a/mysql-test/main/xml.result b/mysql-test/main/xml.result > index efaca961b4a..84dbd68438f 100644 > --- a/mysql-test/main/xml.result > +++ b/mysql-test/main/xml.result > @@ -1007,7 +1007,7 @@ Warning 1292 Truncated incorrect INTEGER value: 'string ' > Warning 1292 Truncated incorrect INTEGER value: 'string ' > DROP PROCEDURE spxml; > select UpdateXML('<a>a</a>',repeat('a b ',1000),''); > -ERROR HY000: XPATH syntax error: 'b a b a b a b a b a b a b a b...' > +ERROR HY000: XPATH syntax error: 'b a b a b a b a b a b a b a b a ' this doesn't look right, the string is truncated, isn't it? > select ExtractValue('<a>a</a>', '/a[@x=@y0123456789_0123456789_0123456789_0123456789]'); > ERROR HY000: XPATH error: comparison of two nodesets is not supported: '=@y0123456789_0123456789_0123...' > select ExtractValue('<a>a</a>', '/a[@x=$y0123456789_0123456789_0123456789_0123456789]'); > @@ -1110,9 +1110,9 @@ NULL > # Bug#57279 updatexml dies with: Assertion failed: str_arg[length] == 0 > # > SELECT UPDATEXML(NULL, (LPAD(0.1111E-15, '2011', 1)), 1); > -ERROR 22007: Illegal double '111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111...' value found during parsing > +ERROR 22007: Illegal double '111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111' value found during parsing same here, lpad appends 2011 characters, so this "double" is clearly truncated. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] b97e45651d1: MDEV-16937 Strict SQL with system versioned tables causes issues
by Sergei Golubchik 19 May '20

19 May '20

Hi, Aleksey! First - is commit b97e45651d1 the one I was supposed to review? On May 17, Aleksey Midenkov wrote: > revision-id: b97e45651d1 (mariadb-10.4.7-33-gb97e45651d1) > parent(s): 7587975bf06 > author: Aleksey Midenkov <midenok(a)gmail.com> > committer: Aleksey Midenkov <midenok(a)gmail.com> > timestamp: 2019-08-19 11:58:07 +0300 > message: > > MDEV-16937 Strict SQL with system versioned tables causes issues > > Vers SQL: respect system fields in NO_ZERO_DATE mode. > > This is the subject for refactoring in tempesta-tech/mariadb#379 > > diff --git a/mysql-test/suite/versioning/engines.combinations b/mysql-test/suite/versioning/engines.combinations > index 26b5bab23f1..57e2af6cd06 100644 > --- a/mysql-test/suite/versioning/engines.combinations > +++ b/mysql-test/suite/versioning/engines.combinations > @@ -7,5 +7,10 @@ default-storage-engine=innodb > [myisam] > default-storage-engine=myisam > > +[traditional] > +default-storage-engine=myisam > +sql-mode=traditional I'm not sure about it. It's, of course, safe to run all tests with every possible settings, but it really blows up testing times. May be it'd make sense to have a more targeted test here? > + > [heap] > default-storage-engine=memory > + > diff --git a/mysql-test/suite/versioning/r/select.result b/mysql-test/suite/versioning/r/select.result > index 3569268ce1d..68df246af6b 100644 > --- a/mysql-test/suite/versioning/r/select.result > +++ b/mysql-test/suite/versioning/r/select.result > @@ -45,7 +45,7 @@ ASOF_x y > 7 107 > 8 108 > 9 109 > -select x as FROMTO_x, y from t1 for system_time from timestamp '0-0-0 0:0:0' to timestamp @t1; > +select x as FROMTO_x, y from t1 for system_time from timestamp '1970-01-01 00:00:00' to timestamp @t1; Does this test use a fixed time zone? If not, is 1970-01-01 00:00:00 a valid timestamp in all time zones? If unsure, better use 1971, that will definitely be fine everywhere. > FROMTO_x y > 0 100 > 1 101 > diff --git a/sql/field.cc b/sql/field.cc > index 969c32a5180..f5580239a9f 100644 > --- a/sql/field.cc > +++ b/sql/field.cc > @@ -11139,6 +11139,13 @@ bool Field::save_in_field_default_value(bool view_error_processing) > { > THD *thd= table->in_use; > > + /* > + TODO: refactor setting the system fields via default_value mechanism. > + This condition will go away as well as other conditions with VERS_SYSTEM_FIELD. > + */ MDEV? There's a pull request, but it is marked as obsolete > + if (flags & VERS_SYSTEM_FIELD) > + return false; > + > if (unlikely(flags & NO_DEFAULT_VALUE_FLAG && > real_type() != MYSQL_TYPE_ENUM)) > { > diff --git a/sql/unireg.cc b/sql/unireg.cc > index d019b5f8a75..484ad0334e4 100644 > --- a/sql/unireg.cc > +++ b/sql/unireg.cc > @@ -1060,7 +1060,8 @@ static bool make_empty_rec(THD *thd, uchar *buff, uint table_options, > !f_bit_as_char(field->pack_flag)) > null_count+= field->length & 7; > > - error= make_empty_rec_store_default(thd, regfield, field->default_value); > + if (!(field->flags & VERS_SYSTEM_FIELD)) > + error= make_empty_rec_store_default(thd, regfield, field->default_value); will the field be initialized at all? It should have something, even bzero will do, but we shouldn't dump uninited memory to frm. > delete regfield; // Avoid memory leaks > if (error) > goto err; > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 3

[Maria-developers] Progress report - Week 2
by Mohammed Hammaad Mateen 18 May '20

18 May '20

Greetings, Hope you are safe and doing great, This post describes the things I've done in my second week [ 11-17 May ] of Community Bonding Period under the mentor-ship of Sergei Golubchik and Oleksandr Byelkin for GSoC-20. Tasks done this week were mostly the continuation of the leftover tasks from the previous week and some related study. Things I've done include: 1:* I was caught up in analysing how was DELETE .... RETURNING implemented, which led me to read from:* • *sp_head.cc* : this made me understand that sp actually meant stored procedures but there is another called PS which I'm yet to figure out. • *sql_parse.cc* : from here I understood that delete return as actually { Analyse->Delete->Return } and all the logic for returning is executed if(lex->has_returning()). 2: Though I've raised the pull request, *MDEV-14558* is not yet completely tested as it does not include tests for *mysql \s*, moreover, I'm facing problems to escape the backslash inorder to write a test-case for *mysql \s* . Regards, Mohammed Hammaad Mateen

1 0

Re: [Maria-developers] 82b6a5cb85f: MDEV-22556: Incorrect result for window function when using encrypt-tmp-files=ON
by Sergei Golubchik 17 May '20

17 May '20

Hi, Varun! Ok to push. One comment below On May 17, Varun Gupta wrote: > revision-id: 82b6a5cb85f (mariadb-10.4.11-192-g82b6a5cb85f) > parent(s): 3bfe305c5cd > author: Varun Gupta <varun.gupta(a)mariadb.com> > committer: Varun Gupta <varun.gupta(a)mariadb.com> > timestamp: 2020-05-15 02:37:16 +0530 > message: > > MDEV-22556: Incorrect result for window function when using encrypt-tmp-files=ON > > diff --git a/mysql-test/suite/encryption/t/tempfiles_encrypted.opt b/mysql-test/suite/encryption/t/tempfiles_encrypted.opt > new file mode 100644 > index 00000000000..f515f7b8689 > --- /dev/null > +++ b/mysql-test/suite/encryption/t/tempfiles_encrypted.opt > @@ -0,0 +1,5 @@ > +--encrypt-tmp_files=ON > +--loose-file-key-management-encryption-algorithm=aes_cbc > +--file-key-management=FORCE > +--loose-file-key-management-filename=$MTR_SUITE_DIR/t/filekeys-data.enc > +--loose-file-key-management-filekey=FILE:$MTR_SUITE_DIR/t/filekeys-data.key this is very redundant, you include have_file_key_management_plugin.inc and it already sets all options for the file_key_management plugin to work. so you only need --encrypt-tmp_files=ON here. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] Progress Report - Week 1-2
by Anton Mikhaylenko 17 May '20

17 May '20

Good day, everyone. In this post, I want to describe the things I've done in the 1-2 weeks of GSoC-20. For begin, I messaged with my mentor Vladislav Vaintroub and I got some tips on where to start. The first step was to run the server and tests for MariaDB on Linux and Windows. At this stage, a little bug was found, which was documented by my mentor https://jira.mariadb.org/browse/MDEV-22479. The next step was to look at the theoretical part of my work. Specifically, what algorithm to use for dynamic concurrency. This is the algorithm "hill-climbing", which is used in .NET ThreadPool https://www.researchgate.net/publication/228977836_Optimizing_concurrency_l… . In the last step, I started to get into the work of the current ThreadPool in MariaDB. At the moment this step is not completed yet. What needs to be done next: Fully understand the theoretic part and how the current thread pool works. Regards, Anton MIkhailenko

1 0

Re: [Maria-developers] Review for MDEV-19751.
by Aleksey Midenkov 14 May '20

14 May '20

Hi, Alexey! Actually the bug is applicable only to primary key and only when default field list is set (i.e. empty KEY() clause). So updated the patch accordingly. Please peek again. On Wed, May 13, 2020 at 2:40 PM Alexey Botchkov <holyfoot(a)mariadb.com> wrote: > > Hello, Aleksey! > > > commit 5ef168e00f7b8c821463e46535a4beccb47ec8ea > + if (key_info->flags & HA_INVISIBLE_KEY) > + continue; > > I'd add || !(key_info->flags & HA_NOSAME). > Can the non-unique index participate in the KEY partitioning? > > + for (uint kp= 0; kp < key_info->user_defined_key_parts; ++kp) > + { > + const KEY_PART_INFO &key_part= key_info->key_part[kp]; > + for (Field **part_field= tab_part_info->part_field_array; > + *part_field; ++part_field) > + { > + if (*part_field == key_part.field) > + { > + *partition_changed= TRUE; > + goto search_finished; > + } > + } // for (part_field) > + } // for (key_part) > > So you check if the key has any of the partitioning fields in it. > In this case we are going to disallow INPLACE with no reason if the > unrelated key is removed just containing the primary key column. > I'd say we should check for the exact match, when the key has > all the part_field_array in that exact order. Disagree? > > And for the test - please add the case for a table with no PRIMARY KEY, bug with the UNIQUE instead. > > Best regards! > HF > -- All the best, Aleksey Midenkov @midenok

1 0

[Maria-developers] Progress Report - Week 1
by Mohammed Hammaad Mateen 12 May '20

12 May '20

Greetings, Hope you are safe and doing great, This post describes the things I've done in my first week [ 4-10 May ] of Community Bonding Period under the mentor-ship of Sergei Golubchik and Oleksandr Byelkin for GSoC-20 and I must acknowledge that it was really an interactive experience and at the same time informative as well. Things I've done include: 1: *Raising my first ever pull request for MariaDB under MDEV-14558 *: Learnt about *strnxmov*, a function of C++ which I had never used earlier. I also came across the procedure of testing my implemented code. Wrote some test cases for the same and generated their corresponding results. 2:* Understand the working of the functions under multi_delete*: Ran it in the debugger and tried to familiarize myself with the internals of how Multiple Table Delete actually works. Things to-do: 1: Though I've raised the pull request, MDEV-14558 is not completed tested as it does not include tests for *mysql \s*, moreover, test cases have to be refactored. 2: Learn more about the working of *multi_delete* so as to develop a parser for Multiple Table Delete Returning. 3: Write some test cases for Multiple Table Delete Returning before hand. Regards, Mohammed Hammaad Mateen

2 1

[Maria-developers] Review for: MDEV-17399 Add support for JSON_TABLE, part #3
by Sergey Petrunia 10 May '20

10 May '20

Hi Alexey, Please find the next batch of input below. (I haven't finished the review but I'm sending it now so you can start addressing it) > commit 68ca18518337443090bdeddf6d612129b6843c21 > Author: Alexey Botchkov <holyfoot(a)askmonty.org> > Date: Fri Apr 17 14:42:40 2020 +0400 > > MDEV-17399 Add support for JSON_TABLE. > > Syntax for JSON_TABLE added. > The ha_json_table handler added. Executing the JSON_TABLE we > create the temporary table of the ha_json_table, add dependencies > of other tables and sometimes conditions to WHERE. > ... > +public: > + ha_json_table(TABLE_SHARE *share_arg, Table_function_json_table *jt): > + handler(&table_function_hton.m_hton, share_arg), m_jt(jt) > + { > + mark_trx_read_write_done= 1; This definitely needs a comment. > + ~ha_json_table() {} > + handler *clone(const char *name, MEM_ROOT *mem_root) { return NULL; } > + const char *index_type(uint inx) { return "HASH"; } Why is ha_json_table still trying to pretend it supports indexes? Is this a left-over from the old way of interfacing with the optimizer that can now be removed? > +int Table_function_json_table::setup(THD *thd, TABLE_LIST *sql_table, > + COND **cond) > +{ > + if (m_json->fix_fields(thd, &m_json)) > + return TRUE; > + > + m_dep_tables= m_json->used_tables(); Why does this function have a 'cond' argument? Please remove it if it's not necessary. == Duplicate column names are allowed == select * from JSON_TABLE('{"color": "black", "price": 1000}', "$" COLUMNS (rowSeq FOR ORDINALITY, color VARCHAR(100) PATH '$.color', color VARCHAR(100) PATH '$.price') ) as T; +--------+-------+-------+ | rowSeq | color | color | +--------+-------+-------+ | 0 | black | 1000 | +--------+-------+-------+ Is it really okay that duplicate column names are allowed? == RIGHT JOIN is allowed == create table t1 (item_name varchar(32), item_props varchar(1024)); insert into t1 values ('Laptop', '{"color": "black", "price": 1000}'); insert into t1 values ('Jeans', '{"color": "blue", "price": 50}'); This query should not be allowed, because JSON_TABLE has a dependency on an inner side of an outer join: select * from t1 right join json_table(t1.item_props, '$' columns(color varchar(100) path '$.color', f2 for ordinality) ) as T on 1; === NATURAL JOIN crashes == create table t10 (color varchar(100), f2 int); select * from t10 natural join json_table(t1.item_props, '$' columns(color varchar(100) path '$.color', f2 for ordinality) ) as JT ; Thread 31 "mysqld" received signal SIGSEGV, Segmentation fault. 0x0000555555d8ac12 in Select_limit_counters::get_select_limit (this=0xa5a5a5a5a5a5ad2d) at /home/psergey/dev-git/10.5-json/sql/sql_limit.h:67 (gdb) wher #0 0x0000555555d8ac12 in Select_limit_counters::get_select_limit (this=0xa5a5a5a5a5a5ad2d) at /home/psergey/dev-git/10.5-json/sql/sql_limit.h:67 #1 0x0000555555e2a4a0 in JOIN::optimize_inner (this=0x7fffac018520) at /home/psergey/dev-git/10.5-json/sql/sql_select.cc:1805 #2 0x0000555555e29b12 in JOIN::optimize (this=0x7fffac018520) at /home/psergey/dev-git/10.5-json/sql/sql_select.cc:1612 BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 1

[Maria-developers] OSX fix need on MariaDB-10.5
by Daniel Black 09 May '20

09 May '20

Mac lovers (who can use C/C++ and a debugger), As your MariaDB developers are hard at work getting 10.5 ready for you, unfortunately a bug has slipped into the 10.5 branch making it unusable on OSX (at least that's what it looks like on the CI infrastructure). The bug report is here: https://jira.mariadb.org/browse/MDEV-22173 As the time of developers is a little short as is the availability of OSX hardware, I'm hoping some of you are able to spend a few hours tackling this task to understand it better, and if possible, create patch. Steps: 1. build environment https://mariadb.com/kb/en/Build_Environment_Setup_for_Mac/ 2. clone the source git clone --branch 10.5 https://github.com/MariaDB/server.git mariadb-server cd mariadb-server git submodule update --recursive --init 3. build instructions https://mariadb.com/kb/en/generic-build-instructions/ (you don't need to install) 4. test (from build directory) mysql-test/mysql-test-run main.select might be useful here: --debugger=NAME or --gdb client-debugger=NAME or client-gdb To collaborative share your findings and to ask for further help MariaDB's community chat stream has been created for you. https://mariadb.zulipchat.com/#narrow/stream/118759-general/topic/Get.20OSX… To paraphrase J.F.Kennedy ask not what your database server community can do for you — ask what you can do for your database server community. Keep the community spirit going, Daniel Black -- Sent as a community member who noticed OSX failures and that everyone is busy

2 3

Re: [Maria-developers] 4312f382b6f: MDEV-22312: Bad error message for SET DEFAULT ROLE when user account is not granted the role
by Sergei Golubchik 08 May '20

08 May '20

Hi, Anel! On May 08, Anel Husakovic wrote: > revision-id: 4312f382b6f (mariadb-10.1.43-138-g4312f382b6f) > parent(s): 8c4b5261210 > author: Anel Husakovic <anel(a)mariadb.org> > committer: Anel Husakovic <anel(a)mariadb.org> > timestamp: 2020-05-07 16:49:33 +0200 > message: > > MDEV-22312: Bad error message for SET DEFAULT ROLE when user account is not granted the role > > diff --git a/mysql-test/suite/roles/set_role-recursive.test b/mysql-test/suite/roles/set_role-recursive.test > index 23d623e1966..dcf3dd5b6fc 100644 > --- a/mysql-test/suite/roles/set_role-recursive.test > +++ b/mysql-test/suite/roles/set_role-recursive.test > @@ -44,7 +44,7 @@ select * from mysql.roles_mapping; > > --sorted_result > show grants; > ---error ER_INVALID_ROLE > +--error ER_SET_DEFAULT_ROLE_FOR_USER > set role test_role2; > select current_user(), current_role(); > --sorted_result > diff --git a/sql/share/errmsg-utf8.txt b/sql/share/errmsg-utf8.txt > index c5e83062f0f..83ecd4735e8 100644 > --- a/sql/share/errmsg-utf8.txt > +++ b/sql/share/errmsg-utf8.txt > @@ -7144,3 +7144,5 @@ ER_WARN_AGGFUNC_DEPENDENCE > ukr "Агрегатна функція '%-.192s)' з SELECTу #%d належить до SELECTу #%d" > WARN_INNODB_PARTITION_OPTION_IGNORED > eng "<%-.64s> option ignored for InnoDB partition" > +ER_SET_DEFAULT_ROLE_FOR_USER > + eng "User '%s' has not been granted a role '%s'." Sorry, this is an absolute no-no. We cannot add new error messages to the old GA versions. Only to the very last RC/GA or to any alpha/beta. When you add an error message, all error messages added after it in newer versions will shift, change their numbers. And error numbers in RC/GA versions cannot change. Instead of a new error message and my_error(ER_SET_DEFAULT_ROLE_FOR_USER, MYF(0), c_usr.ptr(), rolename); you have to write my_printf_error(ER_INVALID_ROLE, "User %`s has not been granted a role %`s", MYF(0), c_usr.c_ptr(), rolename); note other changes: * %`s instead of '%s' for correct identifier quoting * no dot at the end * c_ptr() because printf expects a zero-terminated C string, and ptr() doesn't guarantee that. > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index b6a6f806e50..37141dd2390 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -2041,6 +2041,9 @@ static int check_user_can_set_role(const char *user, const char *host, > ACL_USER_BASE *acl_user_base; > ACL_USER *UNINIT_VAR(acl_user); > bool is_granted= FALSE; > + bool user_can_see_role= FALSE; > + char buf[1024]; > + String c_usr(buf, sizeof(buf), &my_charset_bin); use StringBuffer<1024> c_str; > int result= 0; > > /* clear role privileges */ > @@ -2083,14 +2086,39 @@ static int check_user_can_set_role(const char *user, const char *host, > is_granted= TRUE; > break; > } > + > + for (uint i=0; i < acl_user->role_grants.elements; i++) > + { > + ACL_ROLE *r= *(dynamic_element(&acl_user->role_grants, i, ACL_ROLE**)); > + if (!strcmp(safe_str(r->user.str), safe_str(rolename))) > + { > + size_t len= acl_user->user.length + acl_user->hostname_length + 2; > + c_usr.alloc(len); > + const char c= '@'; > + strmov(strmov(strmov(const_cast<char*>(c_usr.ptr()), > + safe_str(user)), &c), safe_str(host)); > + user_can_see_role= TRUE; > + } > + } No, you repeat this for() block for evert role grant of every role's grantee. And you only use it when there was an error, an uncommon case. Do not do the work that you aren't going to use. And the check doesn't seem correct. You want to use the same rule as for applicable_roles, you need to traverse the graph. Perhaps you can simply use check_role_is_granted() though. And also you need to check whether the current user has read access on mysql.user table - in that case it can see all roles, granted or not. > } > > /* According to SQL standard, the same error message must be presented */ this comment now belongs inside the if(), above ER_INVALID_ROLE > if (!is_granted) > { > - my_error(ER_INVALID_ROLE, MYF(0), rolename); > - result= 1; > - goto end; > + /* Handling of the bad error message - MDEV-22312 */ don't put MDEV numbers into the code, please. > + /* If the current user can see the role generate different type of error. */ > + if(user_can_see_role) > + { > + my_error(ER_SET_DEFAULT_ROLE_FOR_USER, MYF(0), c_usr.ptr(), rolename); > + result= 1; > + goto end; > + } > + else > + { > + my_error(ER_INVALID_ROLE, MYF(0), rolename); > + result= 1; > + goto end; > + } > } > > if (access) Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] Welcome to Google Summer of Code 2020 with MariaDB
by Vicențiu Ciorbaru 04 May '20

04 May '20

Hi! GSoC 2020 accepted proposals have just been announced and you are part of the successful applicants. This year we had a lot of very good proposals and so we have a one of the highest number of students (7) to mentor for GSoC. Congratulations on the effort to learn about the program, get involved and submit successful proposals! The fun part begins now! If you have not done so already, please reach out to your mentor and seek further advice about your project. Each one of your projects can be implemented in multiple ways, some more complex than others. I strongly advise you to create a concrete plan based off of your proposal with your mentor and aim to stick to it. The MariaDB Server codebase is a complex one with lots of different parts, however your mentor can guide you to ensure you have a successful GSoC. Remember that you are here to learn about Open Source, what it means to be part of an Open Source community, to interact with the community, all while becoming better programmers. Please take advantage of this opportunity and keep your technical discussions public, on Zulip https://mariadb.zulipchat.com, and on our developer mailing list https://launchpad.net/~maria-developers. By keeping your discussions public, you get the benefit of not only having your mentor see your questions but all of the MariaDB community. You never know when someone might lend a hand in case your mentor is not yet online. :) The mantra of our community should be that we are always welcoming to new people. Do not be afraid to ask questions. There are no bad or stupid or silly questions. The only key requirement is that you first make an effort to find the answer yourself and only afterwards seek outside help. If you can show a bit of involvement, the community will show you the same. I recommend that you agree with your mentor on some form of weekly progress reporting, via either a blog-post, an email, or something similar that can be publicly shared. I can not stress this enough: we are working on an open source project, our work must be done in the open and it is important to *show the world the good we are doing*. Hope all of you will have a great GSoC! Vicențiu MariaDB Foundation GSoC Admin

1 0

Re: [Maria-developers] d08860b28f3: MDEV-22374: VIEW with security definer require FILE privilege from definer not invoker in case of INTO OUTFILE
by Sergei Golubchik 04 May '20

04 May '20

Hi, Oleksandr! On May 04, Oleksandr Byelkin wrote: > revision-id: d08860b28f3 (mariadb-5.5.67-15-gd08860b28f3) > parent(s): ac2604f923f > author: Oleksandr Byelkin <sanja(a)mariadb.com> > committer: Oleksandr Byelkin <sanja(a)mariadb.com> > timestamp: 2020-04-28 09:16:33 +0200 > message: > > MDEV-22374: VIEW with security definer require FILE privilege from definer not invoker in case of INTO OUTFILE > > Check INTO OUTFILE clause always from invoker. > > --- > mysql-test/r/view_grant.result | 21 +++++++++++++++++++++ > mysql-test/t/view_grant.test | 33 +++++++++++++++++++++++++++++++++ > sql/sql_parse.cc | 18 +++++++++--------- > 3 files changed, 63 insertions(+), 9 deletions(-) > > diff --git a/mysql-test/r/view_grant.result b/mysql-test/r/view_grant.result > index b2d3a0b8ca4..37dc6adc978 100644 > --- a/mysql-test/r/view_grant.result > +++ b/mysql-test/r/view_grant.result > @@ -1587,3 +1587,24 @@ USE test; > DROP DATABASE mysqltest1; > DROP USER 'mysqluser1'@'%'; > DROP USER 'mysqluser2'@'%'; > +# > +# MDEV-22374: VIEW with security definer require FILE privilege from definer > +# not invoker in case of INTO OUTFILE > +# > +create user test@localhost; > +grant select on test.* to test@localhost; > +create table t1 (a int); > +create definer=test@localhost sql security definer view v1 as select * from t1; > +# check that ot works without view typo "it" > +select * INTO OUTFILE '../..//tmp/test_out_txt' from t1; > +# check that ot works without file same why "//" ? > +select * from v1; > +a > +# rights for file should be taken from current user not view > +select * INTO OUTFILE '../../tmp/test_out_txt' from (select count(*) from v1) as dv1; > +select * INTO OUTFILE '../..//tmp/test_out_txt' from (select * from v1) as dv1; > +select * INTO OUTFILE '../..//tmp/test_out_txt' from v1; same > +drop view v1; please add reverse tests too: +create sql security definer view v1 as select * from t1; +connect(con1, localhost, test); +error 1045; +select * INTO OUTFILE '../..//tmp/test_out_txt' from t1; +select * from v1; +error 1045; +select * INTO OUTFILE '../../tmp/test_out_txt' from (select count(*) from v1) as dv1; +error 1045; +select * INTO OUTFILE '../..//tmp/test_out_txt' from (select * from v1) as dv1; +error 1045; +select * INTO OUTFILE '../..//tmp/test_out_txt' from v1; +connection default; > +drop table t1; > +drop user test@localhost; > +# End of 5.5 tests > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > index ae5a6b4cd35..515990c879f 100644 > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -2206,15 +2206,15 @@ mysql_execute_command(THD *thd) > lex->exchange != NULL implies SELECT .. INTO OUTFILE and this > requires FILE_ACL access. > */ > - ulong privileges_requested= lex->exchange ? SELECT_ACL | FILE_ACL : > - SELECT_ACL; > - > - if (all_tables) > - res= check_table_access(thd, > - privileges_requested, > - all_tables, FALSE, UINT_MAX, FALSE); > - else > - res= check_access(thd, privileges_requested, any_db, NULL, NULL, 0, 0); > + if (lex->exchange) > + res= check_access(thd, FILE_ACL, any_db, NULL, NULL, 0, 0); > + if (!res) > + { > + if (all_tables) > + res= check_table_access(thd, SELECT_ACL, all_tables, FALSE, UINT_MAX, FALSE); > + else > + res= check_access(thd, SELECT_ACL, any_db, NULL, NULL, 0, 0); > + } No, I think this special check for a special SELECT ... INTO OUTFILE case is wrong. Why does the server even checks global privileges via views? It should not do it, I think, not for FILE not for any other global or db level privilege. I'd try something like diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc --- a/sql/sql_acl.cc +++ b/sql/sql_acl.cc @@ -4699,7 +4699,7 @@ bool check_grant(THD *thd, ulong want_access, TABLE_LIST > Save a copy of the privileges without the SHOW_VIEW_ACL attribute. It will be checked during making view. */ - tl->grant.orig_want_privilege= (want_access & ~SHOW_VIEW_ACL); + tl->grant.orig_want_privilege= want_access & ~SHOW_VIEW_ACL & TABLE_ACLS; } mysql_rwlock_rdlock(&LOCK_grant); diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc --- a/sql/sql_parse.cc +++ b/sql/sql_parse.cc @@ -5204,7 +5204,8 @@ check_table_access(THD *thd, ulong requirements,TABLE_LIS> Register access for view underlying table. Remove SHOW_VIEW_ACL, because it will be checked during making view */ - table_ref->grant.orig_want_privilege= (want_access & ~SHOW_VIEW_ACL); + table_ref->grant.orig_want_privilege= + want_access & ~SHOW_VIEW_ACL & TABLE_ACLS; if (table_ref->schema_table_reformed) { Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] 3ead2cea95c: MDEV-13266: Race condition in ANALYZE TABLE / statistics collection
by Sergey Petrunia 04 May '20

04 May '20

Hi Varun, On Sun, Apr 12, 2020 at 09:08:37PM +0530, Varun wrote: > revision-id: 3ead2cea95c32b7ceaf6e6ec81f7afbd9137cfe9 (mariadb-10.2.31-103-g3ead2cea95c) > parent(s): d565895bbd8e2a163be48b9bac51fccbf3949c80 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2020-04-12 21:05:36 +0530 > message: > > MDEV-13266: Race condition in ANALYZE TABLE / statistics collection > > Fixing a race condition while collecting the engine independent statistics. > The issue here was when the statistics was collected on specific indexes > then because of some race condition the statistics for indexes was not > collected. The TABLE::keys_in_use_for_query was set to 0 in such cases. > > This happens when the table is opened from TABLE_SHARE instead of the > table share stored in table_cache. It would be nice to write down the race condition that we've caught (in case we need this info later) Is my understanding correct: the error scenario is as follows: Thread1> start running "ANALYZE TABLE t PERISTENT FOR COLUMNS (..) INDEXES ($list)" Thread1> Walk through $list and save it in TABLE::keys_in_use_for_query Thread1> Close/re-open tables Thread2> Make some use of table t. This involves taking table t from Thread2> the table cache, and putting it back (with TABLE::keys_in_use_for_query reset to 0) Thread1> continue collecting EITS stats. Since TABLE::keys_in_use_for_query we will not collect statistics for indexes in $list. Please confirm (and if not, describe the race condition). The patch also introduces this behaviour: analyze table ... persistent for ... indexes(no_such_index); will now cause engine statistics to be still collected. Before the patch it exited with an error. But then, this is consistent with what happens for analyze table ... persistent for columns(no_such_column) ... So I guess this is ok. > --- > sql/sql_admin.cc | 46 +++++++++++++++++++++------------------------- > 1 file changed, 21 insertions(+), 25 deletions(-) > > diff --git a/sql/sql_admin.cc b/sql/sql_admin.cc > index ab95fdc340c..0613495f202 100644 > --- a/sql/sql_admin.cc > +++ b/sql/sql_admin.cc > @@ -769,31 +769,6 @@ static bool mysql_admin_table(THD* thd, TABLE_LIST* tables, > (table->table->s->table_category == TABLE_CATEGORY_USER && > (get_use_stat_tables_mode(thd) > NEVER || > lex->with_persistent_for_clause)); > - > - > - if (!lex->index_list) > - { > - tab->keys_in_use_for_query.init(tab->s->keys); > - } > - else > - { > - int pos; > - LEX_STRING *index_name; > - List_iterator_fast<LEX_STRING> it(*lex->index_list); > - > - tab->keys_in_use_for_query.clear_all(); > - while ((index_name= it++)) > - { > - if (tab->s->keynames.type_names == 0 || > - (pos= find_type(&tab->s->keynames, index_name->str, > - index_name->length, 1)) <= 0) > - { > - compl_result_code= result_code= HA_ADMIN_INVALID; > - break; > - } > - tab->keys_in_use_for_query.set_bit(--pos); > - } > - } > } > > if (result_code == HA_ADMIN_OK) > @@ -878,6 +853,27 @@ static bool mysql_admin_table(THD* thd, TABLE_LIST* tables, > } > tab->file->column_bitmaps_signal(); > } > + if (!lex->index_list) > + tab->keys_in_use_for_query.init(tab->s->keys); > + else > + { > + int pos; > + LEX_STRING *index_name; > + List_iterator_fast<LEX_STRING> it(*lex->index_list); > + > + tab->keys_in_use_for_query.clear_all(); > + while ((index_name= it++)) > + { > + if (tab->s->keynames.type_names == 0 || > + (pos= find_type(&tab->s->keynames, index_name->str, > + index_name->length, 1)) <= 0) > + { > + compl_result_code= result_code= HA_ADMIN_INVALID; > + break; > + } > + tab->keys_in_use_for_query.set_bit(--pos); > + } > + } > if (!(compl_result_code= > alloc_statistics_for_table(thd, table->table)) && > !(compl_result_code= > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

2 2

[Maria-developers] Igor please review: MDEV-22377: Subquery in an [UPDATE] query uses full scan instead of range
by Sergey Petrunia 29 Apr '20

29 Apr '20

Hello Igor, Could you please review the below: commit 793b100cc01228f01ead841d6c45f9e2b07786f9 Author: Sergei Petrunia <psergey(a)askmonty.org> Date: Thu Apr 30 01:25:11 2020 +0300 MDEV-22377: Subquery in an [UPDATE] query uses full scan instead of range When doing IN->EXISTS rewrite, Item_in_subselect::inject_in_to_exists_cond injects equalities into subquery's WHERE condition. The problem is that build_equal_items() has already been called for the subquery's WHERE, and tampering with the WHERE condition can prevent equality propagation from working. If the subquery's WHERE is an Item_cond_and with multiple equalities: - Item_equal objects form a suffix sub-list of the list in WHERE's Item_cond_and::list. The suffix is stored in JOIN::cond_equal->current_level. - Item_cond_and::m_cond_equal must also be preserved. This patch makes inject_in_to_exists_cond() not to break these properties diff --git a/mysql-test/main/subselect2.result b/mysql-test/main/subselect2.result index a3d7fda7abc..c54d635230f 100644 --- a/mysql-test/main/subselect2.result +++ b/mysql-test/main/subselect2.result @@ -262,7 +262,7 @@ id select_type table type possible_keys key key_len ref rows filtered Extra 2 DEPENDENT SUBQUERY t2c ALL NULL NULL NULL NULL 2 100.00 Using where; Using join buffer (incremental, BNL join) Warnings: Note 1276 Field or reference 'test.t1.a' of SELECT #2 was resolved in SELECT #1 -Note 1003 /* select#1 */ select `test`.`t1`.`a` AS `a`,`test`.`t1`.`c1` AS `c1` from `test`.`t1` where !<expr_cache><`test`.`t1`.`c1`,`test`.`t1`.`a`>(<in_optimizer>(`test`.`t1`.`c1`,<exists>(/* select#2 */ select `test`.`t2a`.`c2` from `test`.`t2` `t2a` join `test`.`t2` `t2b` join `test`.`t2` `t2c` where (`test`.`t2b`.`m` <> `test`.`t1`.`a` or `test`.`t2b`.`m` = `test`.`t2a`.`m`) and trigcond(<cache>(`test`.`t1`.`c1`) = `test`.`t2a`.`c2` or `test`.`t2a`.`c2` is null) and `test`.`t2c`.`c2` = `test`.`t2b`.`c2` and `test`.`t2b`.`n` = `test`.`t2a`.`m` having trigcond(`test`.`t2a`.`c2` is null)))) +Note 1003 /* select#1 */ select `test`.`t1`.`a` AS `a`,`test`.`t1`.`c1` AS `c1` from `test`.`t1` where !<expr_cache><`test`.`t1`.`c1`,`test`.`t1`.`a`>(<in_optimizer>(`test`.`t1`.`c1`,<exists>(/* select#2 */ select `test`.`t2a`.`c2` from `test`.`t2` `t2a` join `test`.`t2` `t2b` join `test`.`t2` `t2c` where `test`.`t2c`.`c2` = `test`.`t2b`.`c2` and `test`.`t2b`.`n` = `test`.`t2a`.`m` and (`test`.`t2b`.`m` <> `test`.`t1`.`a` or `test`.`t2b`.`m` = `test`.`t2a`.`m`) and trigcond(<cache>(`test`.`t1`.`c1`) = `test`.`t2a`.`c2` or `test`.`t2a`.`c2` is null) having trigcond(`test`.`t2a`.`c2` is null)))) DROP TABLE t1,t2; # # MDEV-614, also MDEV-536, also LP:1050806: diff --git a/mysql-test/main/subselect3.result b/mysql-test/main/subselect3.result index 5c4544a1b05..0034f61ac23 100644 --- a/mysql-test/main/subselect3.result +++ b/mysql-test/main/subselect3.result @@ -169,7 +169,7 @@ id select_type table type possible_keys key key_len ref rows filtered Extra 2 DEPENDENT SUBQUERY t2 ref a a 5 test.t1.b 1 100.00 Using where Warnings: Note 1276 Field or reference 'test.t3.oref' of SELECT #2 was resolved in SELECT #1 -Note 1003 /* select#1 */ select `test`.`t3`.`a` AS `a`,`test`.`t3`.`oref` AS `oref`,<expr_cache><`test`.`t3`.`a`,`test`.`t3`.`oref`>(<in_optimizer>(`test`.`t3`.`a`,<exists>(/* select#2 */ select `test`.`t1`.`a` from `test`.`t1` join `test`.`t2` where `test`.`t2`.`b` = `test`.`t3`.`oref` and trigcond(<cache>(`test`.`t3`.`a`) = `test`.`t1`.`a` or `test`.`t1`.`a` is null) and `test`.`t2`.`a` = `test`.`t1`.`b` having trigcond(`test`.`t1`.`a` is null)))) AS `Z` from `test`.`t3` +Note 1003 /* select#1 */ select `test`.`t3`.`a` AS `a`,`test`.`t3`.`oref` AS `oref`,<expr_cache><`test`.`t3`.`a`,`test`.`t3`.`oref`>(<in_optimizer>(`test`.`t3`.`a`,<exists>(/* select#2 */ select `test`.`t1`.`a` from `test`.`t1` join `test`.`t2` where `test`.`t2`.`a` = `test`.`t1`.`b` and `test`.`t2`.`b` = `test`.`t3`.`oref` and trigcond(<cache>(`test`.`t3`.`a`) = `test`.`t1`.`a` or `test`.`t1`.`a` is null) having trigcond(`test`.`t1`.`a` is null)))) AS `Z` from `test`.`t3` drop table t1, t2, t3; create table t1 (a int NOT NULL, b int NOT NULL, key(a)); insert into t1 values @@ -197,7 +197,7 @@ id select_type table type possible_keys key key_len ref rows filtered Extra 2 DEPENDENT SUBQUERY t2 ref a a 4 test.t1.b 1 100.00 Using where Warnings: Note 1276 Field or reference 'test.t3.oref' of SELECT #2 was resolved in SELECT #1 -Note 1003 /* select#1 */ select `test`.`t3`.`a` AS `a`,`test`.`t3`.`oref` AS `oref`,<expr_cache><`test`.`t3`.`a`,`test`.`t3`.`oref`>(<in_optimizer>(`test`.`t3`.`a`,<exists>(/* select#2 */ select `test`.`t1`.`a` from `test`.`t1` join `test`.`t2` where `test`.`t2`.`b` = `test`.`t3`.`oref` and trigcond(<cache>(`test`.`t3`.`a`) = `test`.`t1`.`a`) and `test`.`t2`.`a` = `test`.`t1`.`b`))) AS `Z` from `test`.`t3` +Note 1003 /* select#1 */ select `test`.`t3`.`a` AS `a`,`test`.`t3`.`oref` AS `oref`,<expr_cache><`test`.`t3`.`a`,`test`.`t3`.`oref`>(<in_optimizer>(`test`.`t3`.`a`,<exists>(/* select#2 */ select `test`.`t1`.`a` from `test`.`t1` join `test`.`t2` where `test`.`t2`.`a` = `test`.`t1`.`b` and `test`.`t2`.`b` = `test`.`t3`.`oref` and trigcond(<cache>(`test`.`t3`.`a`) = `test`.`t1`.`a`)))) AS `Z` from `test`.`t3` drop table t1,t2,t3; create table t1 (oref int, grp int); insert into t1 (oref, grp) values diff --git a/mysql-test/main/subselect3_jcl6.result b/mysql-test/main/subselect3_jcl6.result index 4260676cc37..b7b18bf80e0 100644 --- a/mysql-test/main/subselect3_jcl6.result +++ b/mysql-test/main/subselect3_jcl6.result @@ -172,7 +172,7 @@ id select_type table type possible_keys key key_len ref rows filtered Extra 2 DEPENDENT SUBQUERY t2 ref a a 5 test.t1.b 1 100.00 Using where; Using join buffer (flat, BKA join); Key-ordered Rowid-ordered scan Warnings: Note 1276 Field or reference 'test.t3.oref' of SELECT #2 was resolved in SELECT #1 -Note 1003 /* select#1 */ select `test`.`t3`.`a` AS `a`,`test`.`t3`.`oref` AS `oref`,<expr_cache><`test`.`t3`.`a`,`test`.`t3`.`oref`>(<in_optimizer>(`test`.`t3`.`a`,<exists>(/* select#2 */ select `test`.`t1`.`a` from `test`.`t1` join `test`.`t2` where `test`.`t2`.`b` = `test`.`t3`.`oref` and trigcond(<cache>(`test`.`t3`.`a`) = `test`.`t1`.`a` or `test`.`t1`.`a` is null) and `test`.`t2`.`a` = `test`.`t1`.`b` having trigcond(`test`.`t1`.`a` is null)))) AS `Z` from `test`.`t3` +Note 1003 /* select#1 */ select `test`.`t3`.`a` AS `a`,`test`.`t3`.`oref` AS `oref`,<expr_cache><`test`.`t3`.`a`,`test`.`t3`.`oref`>(<in_optimizer>(`test`.`t3`.`a`,<exists>(/* select#2 */ select `test`.`t1`.`a` from `test`.`t1` join `test`.`t2` where `test`.`t2`.`a` = `test`.`t1`.`b` and `test`.`t2`.`b` = `test`.`t3`.`oref` and trigcond(<cache>(`test`.`t3`.`a`) = `test`.`t1`.`a` or `test`.`t1`.`a` is null) having trigcond(`test`.`t1`.`a` is null)))) AS `Z` from `test`.`t3` drop table t1, t2, t3; create table t1 (a int NOT NULL, b int NOT NULL, key(a)); insert into t1 values @@ -200,7 +200,7 @@ id select_type table type possible_keys key key_len ref rows filtered Extra 2 DEPENDENT SUBQUERY t2 ref a a 4 test.t1.b 1 100.00 Using where; Using join buffer (flat, BKA join); Key-ordered Rowid-ordered scan Warnings: Note 1276 Field or reference 'test.t3.oref' of SELECT #2 was resolved in SELECT #1 -Note 1003 /* select#1 */ select `test`.`t3`.`a` AS `a`,`test`.`t3`.`oref` AS `oref`,<expr_cache><`test`.`t3`.`a`,`test`.`t3`.`oref`>(<in_optimizer>(`test`.`t3`.`a`,<exists>(/* select#2 */ select `test`.`t1`.`a` from `test`.`t1` join `test`.`t2` where `test`.`t2`.`b` = `test`.`t3`.`oref` and trigcond(<cache>(`test`.`t3`.`a`) = `test`.`t1`.`a`) and `test`.`t2`.`a` = `test`.`t1`.`b`))) AS `Z` from `test`.`t3` +Note 1003 /* select#1 */ select `test`.`t3`.`a` AS `a`,`test`.`t3`.`oref` AS `oref`,<expr_cache><`test`.`t3`.`a`,`test`.`t3`.`oref`>(<in_optimizer>(`test`.`t3`.`a`,<exists>(/* select#2 */ select `test`.`t1`.`a` from `test`.`t1` join `test`.`t2` where `test`.`t2`.`a` = `test`.`t1`.`b` and `test`.`t2`.`b` = `test`.`t3`.`oref` and trigcond(<cache>(`test`.`t3`.`a`) = `test`.`t1`.`a`)))) AS `Z` from `test`.`t3` drop table t1,t2,t3; create table t1 (oref int, grp int); insert into t1 (oref, grp) values diff --git a/mysql-test/main/subselect4.result b/mysql-test/main/subselect4.result index 3dc019cc6c1..e355cfc60f0 100644 --- a/mysql-test/main/subselect4.result +++ b/mysql-test/main/subselect4.result @@ -2349,7 +2349,7 @@ id select_type table type possible_keys key key_len ref rows filtered Extra 1 PRIMARY t2 system NULL NULL NULL NULL 1 100.00 2 DEPENDENT SUBQUERY t1 ALL NULL NULL NULL NULL 2 100.00 Using where Warnings: -Note 1003 /* select#1 */ select 3 AS `f` from dual where !<expr_cache><3>(<in_optimizer>(3,<exists>(/* select#2 */ select `test`.`t1`.`b` from `test`.`t1` where (`test`.`t1`.`c` = 'USA' or `test`.`t1`.`c` <> 'USA') and trigcond(<cache>(3) = `test`.`t1`.`b` or `test`.`t1`.`b` is null) and `test`.`t1`.`b` = `test`.`t1`.`a` having trigcond(`test`.`t1`.`b` is null)))) +Note 1003 /* select#1 */ select 3 AS `f` from dual where !<expr_cache><3>(<in_optimizer>(3,<exists>(/* select#2 */ select `test`.`t1`.`b` from `test`.`t1` where `test`.`t1`.`b` = `test`.`t1`.`a` and (`test`.`t1`.`c` = 'USA' or `test`.`t1`.`c` <> 'USA') and trigcond(<cache>(3) = `test`.`t1`.`b` or `test`.`t1`.`b` is null) having trigcond(`test`.`t1`.`b` is null)))) SELECT * FROM t2 WHERE f NOT IN (SELECT b FROM t1 WHERE 0 OR (c IN ('USA') OR c NOT IN ('USA')) AND a = b); @@ -2582,3 +2582,29 @@ id select_type table type possible_keys key key_len ref rows Extra 1 PRIMARY <subquery2> eq_ref distinct_key distinct_key 4 func 1 2 MATERIALIZED t2 ALL NULL NULL NULL NULL 100 drop table t0, t1, t2; +# +# MDEV-22377: Subquery in an UPDATE query uses full scan instead of range +# +create table t0(a int); +insert into t0 values (0),(1),(2),(3),(4),(5),(6),(7),(8),(9); +create table t1(nkey1 int, col1 int); +insert into t1 select A.a+10*B.a+100*C.a, A.a+10*B.a+100*C.a from t0 A, t0 B, t0 C; +create table t2(key1 int, col1 int, key(key1)); +insert into t2 select A.a+10*B.a, A.a+10*B.a from t0 A, t0 B; +alter table t0 add b int; +# This must use the join order of t0,t2,t1 +# And table t2 must use range access, not full scan: +explain +select * +from t0 +where +a not in (SELECT t1.col1 +FROM t1 JOIN t2 ON t2.key1 = t1.nkey1 +WHERE +t1.nkey1 IN (1,2,3,4,5) and t2.col1>t0.b +); +id select_type table type possible_keys key key_len ref rows Extra +1 PRIMARY t0 ALL NULL NULL NULL NULL 10 Using where +2 DEPENDENT SUBQUERY t2 range key1 key1 5 NULL 5 Using index condition; Using where +2 DEPENDENT SUBQUERY t1 ALL NULL NULL NULL NULL 1000 Using where; Using join buffer (flat, BNL join) +drop table t0, t1, t2; diff --git a/mysql-test/main/subselect4.test b/mysql-test/main/subselect4.test index bc8db41c505..e3d8b5be832 100644 --- a/mysql-test/main/subselect4.test +++ b/mysql-test/main/subselect4.test @@ -2115,6 +2115,32 @@ where exists (select * from t2 where t2.a=t1.a order by t2.b limit 2,3); explain select * from t1 where t1.a in (select t2.a from t2 order by t2.b); +drop table t0, t1, t2; + +--echo # +--echo # MDEV-22377: Subquery in an UPDATE query uses full scan instead of range +--echo # +create table t0(a int); +insert into t0 values (0),(1),(2),(3),(4),(5),(6),(7),(8),(9); +create table t1(nkey1 int, col1 int); +insert into t1 select A.a+10*B.a+100*C.a, A.a+10*B.a+100*C.a from t0 A, t0 B, t0 C; + +create table t2(key1 int, col1 int, key(key1)); +insert into t2 select A.a+10*B.a, A.a+10*B.a from t0 A, t0 B; + +alter table t0 add b int; +--echo # This must use the join order of t0,t2,t1 +--echo # And table t2 must use range access, not full scan: +explain +select * +from t0 +where + a not in (SELECT t1.col1 + FROM t1 JOIN t2 ON t2.key1 = t1.nkey1 + WHERE + t1.nkey1 IN (1,2,3,4,5) and t2.col1>t0.b + ); drop table t0, t1, t2; + diff --git a/mysql-test/main/subselect_mat_cost_bugs.result b/mysql-test/main/subselect_mat_cost_bugs.result index a18c5e608f1..755f1e2ad13 100644 --- a/mysql-test/main/subselect_mat_cost_bugs.result +++ b/mysql-test/main/subselect_mat_cost_bugs.result @@ -100,7 +100,7 @@ id select_type table type possible_keys key key_len ref rows filtered Extra 2 DEPENDENT SUBQUERY t2 index c3 c3 9 NULL 2 100.00 Using where; Using index; Using join buffer (flat, BNL join) Warnings: Note 1276 Field or reference 'test.t1.pk' of SELECT #2 was resolved in SELECT #1 -Note 1003 /* select#1 */ select `test`.`t1`.`pk` AS `pk` from `test`.`t1` where <expr_cache><`test`.`t1`.`c1`,`test`.`t1`.`pk`>(<in_optimizer>(`test`.`t1`.`c1`,<exists>(/* select#2 */ select `test`.`t1a`.`c1` from `test`.`t1b` join `test`.`t2` left join `test`.`t1a` on(`test`.`t1a`.`c2` = `test`.`t1b`.`pk` and 2) where `test`.`t1`.`pk` <> 0 and <cache>(`test`.`t1`.`c1`) = `test`.`t1a`.`c1` and `test`.`t2`.`c3` = `test`.`t1b`.`c4`))) +Note 1003 /* select#1 */ select `test`.`t1`.`pk` AS `pk` from `test`.`t1` where <expr_cache><`test`.`t1`.`c1`,`test`.`t1`.`pk`>(<in_optimizer>(`test`.`t1`.`c1`,<exists>(/* select#2 */ select `test`.`t1a`.`c1` from `test`.`t1b` join `test`.`t2` left join `test`.`t1a` on(`test`.`t1a`.`c2` = `test`.`t1b`.`pk` and 2) where `test`.`t2`.`c3` = `test`.`t1b`.`c4` and `test`.`t1`.`pk` <> 0 and <cache>(`test`.`t1`.`c1`) = `test`.`t1a`.`c1`))) SELECT pk FROM t1 WHERE c1 IN @@ -364,7 +364,7 @@ AND a = SOME (SELECT b FROM t5)); id select_type table type possible_keys key key_len ref rows Extra 1 PRIMARY t3 ALL NULL NULL NULL NULL 2 Using where 2 DEPENDENT SUBQUERY t5 index c c 10 NULL 2 Using where; Using index; Start temporary -2 DEPENDENT SUBQUERY t4 eq_ref PRIMARY PRIMARY 4 test.t5.b 1 Using index condition; Using where; End temporary +2 DEPENDENT SUBQUERY t4 eq_ref PRIMARY PRIMARY 4 test.t5.b 1 Using where; End temporary SELECT * FROM t3 WHERE t3.b > ALL ( diff --git a/sql/item_subselect.cc b/sql/item_subselect.cc index 736dfbd33b4..726c58e49a2 100644 --- a/sql/item_subselect.cc +++ b/sql/item_subselect.cc @@ -2704,9 +2704,28 @@ bool Item_in_subselect::inject_in_to_exists_cond(JOIN *join_arg) and_args= ((Item_cond*) join_arg->conds)->argument_list(); if (join_arg->cond_equal) and_args->disjoin((List<Item> *) &join_arg->cond_equal->current_level); + + /* + If subquery's WHERE is an Item_cond_and, add the injected conditions + into it manually. If we just called and_items(...) that could + potentially replace the top-level Item_cond_and and current top-level + Item_cond_and::m_cond_equal will be lost. + */ + if (where_item->type() == Item::COND_ITEM && + ((Item_cond*)where_item)->functype() == Item_func::COND_AND_FUNC) + { + List_iterator<Item> it(*((Item_cond*)where_item)->argument_list()); + while (Item *item=it++) + and_args->push_back(item); + } + else + and_args->push_back(where_item); + + where_item= join_arg->conds; } + else + where_item= and_items(thd, join_arg->conds, where_item); - where_item= and_items(thd, join_arg->conds, where_item); if (where_item->fix_fields_if_needed(thd, 0)) DBUG_RETURN(true); // TIMOUR TODO: call optimize_cond() for the new where clause @@ -2719,12 +2738,15 @@ bool Item_in_subselect::inject_in_to_exists_cond(JOIN *join_arg) { /* The argument list of the top-level AND may change after fix fields. */ and_args= ((Item_cond*) join_arg->conds)->argument_list(); - List_iterator<Item_equal> li(join_arg->cond_equal->current_level); - Item_equal *elem; - while ((elem= li++)) - { - and_args->push_back(elem, thd->mem_root); - } + + /* + Note that Item_equal objects must be present in this form: + Item_cond::conds list must have the end of the list to be the + same linked list as join_arg->cond_equal->current_level. + The code in substitute_for_best_equal_field depends on it (see the + disjoin() call). + */ + and_args->append((List<Item> *)&join_arg->cond_equal->current_level); } } BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] 96aee3229a2: MENT-703: Disable rpl.rpl_skip_replication, rpl.rpl_set_statement_default_master, rpl.rpl_extra_col_master_myisam, rpl.rpl_slave_load_tmpdir_not_exist tests in ES 10.2 and above
by sujatha 29 Apr '20

29 Apr '20

revision-id: 96aee3229a288dae52c457ca808966092da6d821 (mariadb-10.4.5-669-g96aee3229a2) parent(s): a284ea1909d66e4284a964a4ca7d45b75c450ed7 author: Sujatha committer: Sujatha timestamp: 2020-04-29 13:28:23 +0530 message: MENT-703: Disable rpl.rpl_skip_replication, rpl.rpl_set_statement_default_master, rpl.rpl_extra_col_master_myisam, rpl.rpl_slave_load_tmpdir_not_exist tests in ES 10.2 and above Problem: ======= Disable following tests in ES 10.2 and above, till MDEV-13258 and MDEV-14203 are fixed. rpl.rpl_skip_replication, rpl.rpl_set_statement_default_master, rpl.rpl_extra_col_master_myisam, rpl.rpl_slave_load_tmpdir_not_exist --- mysql-test/suite/rpl/disabled.def | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/mysql-test/suite/rpl/disabled.def b/mysql-test/suite/rpl/disabled.def index 77b01bd8d1d..5a04f4ccd15 100644 --- a/mysql-test/suite/rpl/disabled.def +++ b/mysql-test/suite/rpl/disabled.def @@ -19,3 +19,7 @@ rpl_parallel2 : fails after MDEV-16172 rpl_semi_sync_after_sync : fails after MDEV-16172 rpl_slave_grp_exec: MDEV-10514 rpl_auto_increment_update_failure : disabled for now +rpl_skip_replication : MDEV-13258 +rpl_set_statement_default_master : MDEV-13258 +rpl_extra_col_master_myisam : MDEV-14203 +rpl_slave_load_tmpdir_not_exist : MDEV-14203

1 0

[Maria-developers] 7f8dd7b2044: MENT-703: Disable rpl.rpl_skip_replication, rpl.rpl_set_statement_default_master, rpl.rpl_extra_col_master_myisam, rpl.rpl_slave_load_tmpdir_not_exist tests in ES 10.2 and above
by sujatha 29 Apr '20

29 Apr '20

revision-id: 7f8dd7b20443e0247ca478cbfd98c43749acdae0 (mariadb-10.2.24-722-g7f8dd7b2044) parent(s): 52cb9e87e1d23aeb0958966807050a6a843c181c author: Sujatha committer: Sujatha timestamp: 2020-04-29 11:33:41 +0530 message: MENT-703: Disable rpl.rpl_skip_replication, rpl.rpl_set_statement_default_master, rpl.rpl_extra_col_master_myisam, rpl.rpl_slave_load_tmpdir_not_exist tests in ES 10.2 and above Problem: ======= Disable following tests in ES 10.2 and above, till MDEV-13258 and MDEV-14203 are fixed. rpl.rpl_skip_replication, rpl.rpl_set_statement_default_master, rpl.rpl_extra_col_master_myisam, rpl.rpl_slave_load_tmpdir_not_exist --- mysql-test/suite/rpl/disabled.def | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/mysql-test/suite/rpl/disabled.def b/mysql-test/suite/rpl/disabled.def index bdefb1660bd..951f8a6aa0f 100644 --- a/mysql-test/suite/rpl/disabled.def +++ b/mysql-test/suite/rpl/disabled.def @@ -16,3 +16,7 @@ rpl_partition_archive : MDEV-5077 2013-09-27 svoj Cannot exchange partition rpl_row_binlog_max_cache_size : MDEV-11092 rpl_row_index_choice : MDEV-11666 rpl_slave_grp_exec: MDEV-10514 +rpl_skip_replication : MDEV-13258 +rpl_set_statement_default_master : MDEV-13258 +rpl_extra_col_master_myisam : MDEV-14203 +rpl_slave_load_tmpdir_not_exist : MDEV-14203

1 0

Re: [Maria-developers] [Commits] 85b116d1ab9: MDEV-22317: SIGSEGV in my_free/delete_dynamic in optimized builds (ARIA)
by sujatha 28 Apr '20

28 Apr '20

Hello Andrei, Good morning. Thank you for the review comments. Please find my replies inline. On 27/04/20 11:26 pm, Andrei Elkin wrote: > Sujatha, howdy. > > Thanks for a prompt checking the case! > The patch looks good, only > >> - if (!wild_do_table.elements) >> + if (status && wild_do_table_inited) > I would suggest to reorder the && arg:s to be then > in temporal wrt their evaluation order. Sure Andrei. I have refined changes as shown below. diff --git a/sql/rpl_filter.cc b/sql/rpl_filter.cc index 635a0f4e2d6..8035763bf35 100644 --- a/sql/rpl_filter.cc +++ b/sql/rpl_filter.cc @@ -416,10 +416,13 @@ Rpl_filter::set_wild_do_table(const char* table_spec) status= parse_filter_rule(table_spec, &Rpl_filter::add_wild_do_table); - if (!wild_do_table.elements) + if (wild_do_table_inited && status) { - delete_dynamic(&wild_do_table); - wild_do_table_inited= 0; + if (!wild_do_table.elements) + { + delete_dynamic(&wild_do_table); + wild_do_table_inited= 0; + } } > >> revision-id: 85b116d1ab9ea85dcef63d259b8f6366466e2750 (mariadb-10.5.2-185-g85b116d1ab9) >> parent(s): fbe2712705d464bf8488df249c36115e2c1f63f7 >> author: Sujatha >> committer: Sujatha >> timestamp: 2020-04-27 17:43:51 +0530 >> message: >> >> MDEV-22317: SIGSEGV in my_free/delete_dynamic in optimized builds (ARIA) >> >> Problem: >> ======= >> SET @@GLOBAL.replicate_wild_ignore_table=''; >> SET @@GLOBAL.replicate_wild_do_table=''; >> >> Reports following valgrind error. >> >> Conditional jump or move depends on uninitialised value(s) >> Rpl_filter::set_wild_ignore_table(char const*) (rpl_filter.cc:439) >> >> Conditional jump or move depends on uninitialised value(s) >> at 0xF60390: delete_dynamic (array.c:304) >> by 0x74F3F2: Rpl_filter::set_wild_do_table(char const*) (rpl_filter.cc:421) >> >> Analysis: >> ======== >> List of values provided for options "wild_do_table" and "wild_ignore_table" are >> stored in DYNAMIC_ARRAYS. When an empty list is provided these dynamic arrays >> are not initialized. > Correct. And that's 'cos > > Rpl_filter::Rpl_filter() : > parallel_mode(SLAVE_PARALLEL_CONSERVATIVE), > table_rules_on(0), > do_table_inited(0), ignore_table_inited(0), > wild_do_table_inited(0), wild_ignore_table_inited(0) > { > do_db.empty(); > ignore_db.empty(); > rewrite_db.empty(); > } > > does not initialize the two members of your patch's interest with > empty()-ing. > I wonder if > > + wild_do_table.empty() > + wild_ignore_table.empty() > ? > > > that could work out as well which looks as more consistent solution to > me. [It feels like binlog_filter object might object this idea though]. 'do_db' and 'ignore_db' are of type list iterators. Hence they have 'empty' defined for them. 'empty' is not a member of DYNAMIC_ARRAY. I_List<i_string> do_db; I_List<i_string> ignore_db; /home/sujatha/bug_repo/MDEV-22317-10.5/sql/rpl_filter.cc:36:17: error: ‘DYNAMIC_ARRAY {aka struct st_dynamic_array}’ has no member named ‘empty’ wild_do_table.empty(); ^~~~~ /home/sujatha/bug_repo/MDEV-22317-10.5/sql/rpl_filter.cc:37:21: error: ‘DYNAMIC_ARRAY {aka struct st_dynamic_array}’ has no member named ‘empty’ wild_ignore_table.empty(); ^~~~~ "wild_do_table" and "wild_ignore_table" arrays get initialized as part of "add_wild_do_table" and "add_wild_ignore_table". The initialization is done through following calls. status= parse_filter_rule(table_spec, &Rpl_filter::add_wild_do_table); status= parse_filter_rule(table_spec, &Rpl_filter::add_wild_ignore_table); When default empty string is provided for replicate_wild_ignore_table='' and replicate_wild_do_table='' the dynamic array is not initialized. 'wild_do_table.elements' will be '0'. No cleanup is required in this case. Hence we should do the cleanup only when 'wild_do_table_inited' is true. Please let me know your thoughts. Thank you S.Sujatha > > Cheers, > > Andrei > > >> Existing code treats empty element list as an error and >> tries to clean the uninitialized list. This results in above valgrind issue. >> >> Fix: >> === >> The clean up should be initiated only when there is an error while parsing the >> 'wild_do_table' or 'wild_ignore_table' list and the dynamic_array is in >> initialized state. Otherwise for empty list it should simply return success. >> >> --- >> mysql-test/suite/rpl/r/rpl_filter_wild_tables_dynamic.result | 2 ++ >> mysql-test/suite/rpl/t/rpl_filter_wild_tables_dynamic.test | 2 ++ >> sql/rpl_filter.cc | 4 ++-- >> 3 files changed, 6 insertions(+), 2 deletions(-) >> >> diff --git a/mysql-test/suite/rpl/r/rpl_filter_wild_tables_dynamic.result b/mysql-test/suite/rpl/r/rpl_filter_wild_tables_dynamic.result >> index 47cd362a549..fe0b283fc7c 100644 >> --- a/mysql-test/suite/rpl/r/rpl_filter_wild_tables_dynamic.result >> +++ b/mysql-test/suite/rpl/r/rpl_filter_wild_tables_dynamic.result >> @@ -7,6 +7,8 @@ SET @@GLOBAL.replicate_wild_ignore_table="test.b%"; >> ERROR HY000: This operation cannot be performed as you have a running slave ''; run STOP SLAVE '' first >> connection slave; >> include/stop_slave.inc >> +SET @@GLOBAL.replicate_wild_do_table=""; >> +SET @@GLOBAL.replicate_wild_ignore_table=""; >> SET @@GLOBAL.replicate_wild_do_table="test.a%"; >> SET @@GLOBAL.replicate_wild_ignore_table="test.b%"; >> include/start_slave.inc >> diff --git a/mysql-test/suite/rpl/t/rpl_filter_wild_tables_dynamic.test b/mysql-test/suite/rpl/t/rpl_filter_wild_tables_dynamic.test >> index 6db61927eec..657a95cec15 100644 >> --- a/mysql-test/suite/rpl/t/rpl_filter_wild_tables_dynamic.test >> +++ b/mysql-test/suite/rpl/t/rpl_filter_wild_tables_dynamic.test >> @@ -13,6 +13,8 @@ SET @@GLOBAL.replicate_wild_ignore_table="test.b%"; >> >> connection slave; >> source include/stop_slave.inc; >> +SET @@GLOBAL.replicate_wild_do_table=""; >> +SET @@GLOBAL.replicate_wild_ignore_table=""; >> SET @@GLOBAL.replicate_wild_do_table="test.a%"; >> SET @@GLOBAL.replicate_wild_ignore_table="test.b%"; >> source include/start_slave.inc; >> diff --git a/sql/rpl_filter.cc b/sql/rpl_filter.cc >> index 635a0f4e2d6..49b498d3568 100644 >> --- a/sql/rpl_filter.cc >> +++ b/sql/rpl_filter.cc >> @@ -416,7 +416,7 @@ Rpl_filter::set_wild_do_table(const char* table_spec) >> >> status= parse_filter_rule(table_spec, &Rpl_filter::add_wild_do_table); >> >> - if (!wild_do_table.elements) >> + if (status && wild_do_table_inited) >> { >> delete_dynamic(&wild_do_table); >> wild_do_table_inited= 0; >> @@ -436,7 +436,7 @@ Rpl_filter::set_wild_ignore_table(const char* table_spec) >> >> status= parse_filter_rule(table_spec, &Rpl_filter::add_wild_ignore_table); >> >> - if (!wild_ignore_table.elements) >> + if (status && wild_ignore_table_inited) >> { >> delete_dynamic(&wild_ignore_table); >> wild_ignore_table_inited= 0; >> _______________________________________________ >> commits mailing list >> commits(a)mariadb.org >> https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

[Maria-developers] 85b116d1ab9: MDEV-22317: SIGSEGV in my_free/delete_dynamic in optimized builds (ARIA)
by sujatha 27 Apr '20

27 Apr '20

revision-id: 85b116d1ab9ea85dcef63d259b8f6366466e2750 (mariadb-10.5.2-185-g85b116d1ab9) parent(s): fbe2712705d464bf8488df249c36115e2c1f63f7 author: Sujatha committer: Sujatha timestamp: 2020-04-27 17:43:51 +0530 message: MDEV-22317: SIGSEGV in my_free/delete_dynamic in optimized builds (ARIA) Problem: ======= SET @@GLOBAL.replicate_wild_ignore_table=''; SET @@GLOBAL.replicate_wild_do_table=''; Reports following valgrind error. Conditional jump or move depends on uninitialised value(s) Rpl_filter::set_wild_ignore_table(char const*) (rpl_filter.cc:439) Conditional jump or move depends on uninitialised value(s) at 0xF60390: delete_dynamic (array.c:304) by 0x74F3F2: Rpl_filter::set_wild_do_table(char const*) (rpl_filter.cc:421) Analysis: ======== List of values provided for options "wild_do_table" and "wild_ignore_table" are stored in DYNAMIC_ARRAYS. When an empty list is provided these dynamic arrays are not initialized. Existing code treats empty element list as an error and tries to clean the uninitialized list. This results in above valgrind issue. Fix: === The clean up should be initiated only when there is an error while parsing the 'wild_do_table' or 'wild_ignore_table' list and the dynamic_array is in initialized state. Otherwise for empty list it should simply return success. --- mysql-test/suite/rpl/r/rpl_filter_wild_tables_dynamic.result | 2 ++ mysql-test/suite/rpl/t/rpl_filter_wild_tables_dynamic.test | 2 ++ sql/rpl_filter.cc | 4 ++-- 3 files changed, 6 insertions(+), 2 deletions(-) diff --git a/mysql-test/suite/rpl/r/rpl_filter_wild_tables_dynamic.result b/mysql-test/suite/rpl/r/rpl_filter_wild_tables_dynamic.result index 47cd362a549..fe0b283fc7c 100644 --- a/mysql-test/suite/rpl/r/rpl_filter_wild_tables_dynamic.result +++ b/mysql-test/suite/rpl/r/rpl_filter_wild_tables_dynamic.result @@ -7,6 +7,8 @@ SET @@GLOBAL.replicate_wild_ignore_table="test.b%"; ERROR HY000: This operation cannot be performed as you have a running slave ''; run STOP SLAVE '' first connection slave; include/stop_slave.inc +SET @@GLOBAL.replicate_wild_do_table=""; +SET @@GLOBAL.replicate_wild_ignore_table=""; SET @@GLOBAL.replicate_wild_do_table="test.a%"; SET @@GLOBAL.replicate_wild_ignore_table="test.b%"; include/start_slave.inc diff --git a/mysql-test/suite/rpl/t/rpl_filter_wild_tables_dynamic.test b/mysql-test/suite/rpl/t/rpl_filter_wild_tables_dynamic.test index 6db61927eec..657a95cec15 100644 --- a/mysql-test/suite/rpl/t/rpl_filter_wild_tables_dynamic.test +++ b/mysql-test/suite/rpl/t/rpl_filter_wild_tables_dynamic.test @@ -13,6 +13,8 @@ SET @@GLOBAL.replicate_wild_ignore_table="test.b%"; connection slave; source include/stop_slave.inc; +SET @@GLOBAL.replicate_wild_do_table=""; +SET @@GLOBAL.replicate_wild_ignore_table=""; SET @@GLOBAL.replicate_wild_do_table="test.a%"; SET @@GLOBAL.replicate_wild_ignore_table="test.b%"; source include/start_slave.inc; diff --git a/sql/rpl_filter.cc b/sql/rpl_filter.cc index 635a0f4e2d6..49b498d3568 100644 --- a/sql/rpl_filter.cc +++ b/sql/rpl_filter.cc @@ -416,7 +416,7 @@ Rpl_filter::set_wild_do_table(const char* table_spec) status= parse_filter_rule(table_spec, &Rpl_filter::add_wild_do_table); - if (!wild_do_table.elements) + if (status && wild_do_table_inited) { delete_dynamic(&wild_do_table); wild_do_table_inited= 0; @@ -436,7 +436,7 @@ Rpl_filter::set_wild_ignore_table(const char* table_spec) status= parse_filter_rule(table_spec, &Rpl_filter::add_wild_ignore_table); - if (!wild_ignore_table.elements) + if (status && wild_ignore_table_inited) { delete_dynamic(&wild_ignore_table); wild_ignore_table_inited= 0;

1 0

Re: [Maria-developers] 8a990ad1774: MDEV-18319 BIGINT UNSIGNED Performance issue
by Sergei Golubchik 27 Apr '20

27 Apr '20

Hi, Alexander! On Apr 26, Alexander Barkov wrote: > revision-id: 8a990ad1774 > author: Alexander Barkov <bar(a)mariadb.com> > committer: Alexander Barkov <bar(a)mariadb.com> > timestamp: 2019-03-25 19:19:48 +0400 > message: MDEV-18319 BIGINT UNSIGNED Performance issue > > diff --git a/mysql-test/main/errors.result b/mysql-test/main/errors.result > index ba05a2b37d45..96ad96395aab 100644 > --- a/mysql-test/main/errors.result > +++ b/mysql-test/main/errors.result > @@ -32,16 +32,21 @@ set sql_mode=default; > CREATE TABLE t1 (a INT); > SELECT a FROM t1 WHERE a IN(1, (SELECT IF(1=0,1,2/0))); > a > +Warnings: > +Warning 1365 Division by 0 Interesting. There was no warning before. > INSERT INTO t1 VALUES(1); > SELECT a FROM t1 WHERE a IN(1, (SELECT IF(1=0,1,2/0))); > a > 1 > diff --git a/mysql-test/main/func_in.result b/mysql-test/main/func_in.result > index 65313148bf80..59822ae1049c 100644 > --- a/mysql-test/main/func_in.result > +++ b/mysql-test/main/func_in.result > @@ -489,6 +489,7 @@ SELECT id FROM t1 WHERE id IN(4564, (SELECT IF(1=0,1,1/0)) ); > id > Warnings: > Warning 1365 Division by 0 > +Warning 1365 Division by 0 This is not very nice. May be we should use Item_cache in these cases? > DROP TABLE t1; > End of 5.0 tests > create table t1(f1 char(1)); > @@ -909,3 +910,71 @@ Warnings: > Warning 1292 Truncated incorrect time value: '' > Warning 1292 Truncated incorrect time value: '' > Warning 1292 Truncated incorrect time value: '' > +# > +# MDEV-18319 BIGINT UNSIGNED Performance issue > +# > +CREATE TABLE t1 ( > +id bigint(20) unsigned NOT NULL AUTO_INCREMENT PRIMARY KEY > +); > +FOR i IN 0..255 > +DO > +INSERT INTO t1 VALUES (); > +END FOR > +$$ > +SELECT MIN(id), MAX(id), COUNT(*) FROM t1; > +MIN(id) MAX(id) COUNT(*) > +1 256 256 interesting. why it's min=1 and max=255, if your loop is in 0..255? > +EXPLAIN SELECT id FROM t1 WHERE id IN (1,2); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 range PRIMARY PRIMARY 8 NULL 2 Using where; Using index > +EXPLAIN SELECT id FROM t1 WHERE id IN (9223372036854775806, 9223372036854775807); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 range PRIMARY PRIMARY 8 NULL 2 Using where; Using index > +EXPLAIN SELECT id FROM t1 WHERE id IN (9223372036854775807, 9223372036854775808); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 range PRIMARY PRIMARY 8 NULL 2 Using where; Using index > +EXPLAIN SELECT id FROM t1 WHERE id IN (1.0,2.0); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 range PRIMARY PRIMARY 8 NULL 2 Using where; Using index > +EXPLAIN SELECT id FROM t1 WHERE id IN (9223372036854775806.0, 9223372036854775807.0); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 range PRIMARY PRIMARY 8 NULL 2 Using where; Using index > +# Cannot compare this as INT (yet) > +EXPLAIN SELECT id FROM t1 WHERE id IN (9223372036854775807.0, 9223372036854775808.0); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 index PRIMARY PRIMARY 8 NULL 256 Using where; Using index > +DROP TABLE t1; > +CREATE TABLE t1 ( > +id bigint(20) NOT NULL AUTO_INCREMENT PRIMARY KEY > +); > +FOR i IN 0..255 > +DO > +INSERT INTO t1 VALUES (); > +END FOR > +$$ > +SELECT MIN(id), MAX(id), COUNT(*) FROM t1; > +MIN(id) MAX(id) COUNT(*) > +1 256 256 > +EXPLAIN SELECT id FROM t1 WHERE id IN (-1,-2); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 range PRIMARY PRIMARY 8 NULL 2 Using where; Using index > +EXPLAIN SELECT id FROM t1 WHERE id IN (-9223372036854775806, -9223372036854775807); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 range PRIMARY PRIMARY 8 NULL 2 Using where; Using index > +EXPLAIN SELECT id FROM t1 WHERE id IN (-9223372036854775807, -9223372036854775808); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 range PRIMARY PRIMARY 8 NULL 2 Using where; Using index > +EXPLAIN SELECT id FROM t1 WHERE id IN (-1.0,-2.0); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 range PRIMARY PRIMARY 8 NULL 2 Using where; Using index > +EXPLAIN SELECT id FROM t1 WHERE id IN (-9223372036854775806.0, -9223372036854775807.0); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 range PRIMARY PRIMARY 8 NULL 2 Using where; Using index > +EXPLAIN SELECT id FROM t1 WHERE id IN (-9223372036854775807.0, -9223372036854775808.0); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 range PRIMARY PRIMARY 8 NULL 2 Using where; Using index > +# Cannot compare this as INT (yet) > +EXPLAIN SELECT id FROM t1 WHERE id IN (-9223372036854775808.0, -9223372036854775809.0); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 index PRIMARY PRIMARY 8 NULL 256 Using where; Using index > +DROP TABLE t1; > diff --git a/mysql-test/main/func_in.test b/mysql-test/main/func_in.test > index b99fad159c22..04b0b328c27b 100644 > --- a/mysql-test/main/func_in.test > +++ b/mysql-test/main/func_in.test > @@ -690,3 +690,51 @@ SELECT > TIME'00:00:00'='' AS c1_true, > TIME'00:00:00' IN ('', TIME'10:20:30') AS c2_true, > TIME'00:00:00' NOT IN ('', TIME'10:20:30') AS c3_false; > + > +--echo # > +--echo # MDEV-18319 BIGINT UNSIGNED Performance issue > +--echo # > + > +CREATE TABLE t1 ( > + id bigint(20) unsigned NOT NULL AUTO_INCREMENT PRIMARY KEY > +); > +DELIMITER $$; > +FOR i IN 0..255 > +DO > + INSERT INTO t1 VALUES (); > +END FOR > +$$ > +DELIMITER ;$$ A couple of one-liners instead: insert t1 select seq from seq_1_to_256; or, in pure SQL insert t1 with recursive g(i) as (select 1 union select i+1 from g where i < 257) select * from g; > +SELECT MIN(id), MAX(id), COUNT(*) FROM t1; > +EXPLAIN SELECT id FROM t1 WHERE id IN (1,2); > +EXPLAIN SELECT id FROM t1 WHERE id IN (9223372036854775806, 9223372036854775807); > +EXPLAIN SELECT id FROM t1 WHERE id IN (9223372036854775807, 9223372036854775808); > + > +EXPLAIN SELECT id FROM t1 WHERE id IN (1.0,2.0); > +EXPLAIN SELECT id FROM t1 WHERE id IN (9223372036854775806.0, 9223372036854775807.0); > +--echo # Cannot compare this as INT (yet) > +EXPLAIN SELECT id FROM t1 WHERE id IN (9223372036854775807.0, 9223372036854775808.0); > +DROP TABLE t1; > + > + > +CREATE TABLE t1 ( > + id bigint(20) NOT NULL AUTO_INCREMENT PRIMARY KEY > +); > +DELIMITER $$; > +FOR i IN 0..255 > +DO > + INSERT INTO t1 VALUES (); > +END FOR > +$$ > +DELIMITER ;$$ > +SELECT MIN(id), MAX(id), COUNT(*) FROM t1; > +EXPLAIN SELECT id FROM t1 WHERE id IN (-1,-2); > +EXPLAIN SELECT id FROM t1 WHERE id IN (-9223372036854775806, -9223372036854775807); > +EXPLAIN SELECT id FROM t1 WHERE id IN (-9223372036854775807, -9223372036854775808); > + > +EXPLAIN SELECT id FROM t1 WHERE id IN (-1.0,-2.0); > +EXPLAIN SELECT id FROM t1 WHERE id IN (-9223372036854775806.0, -9223372036854775807.0); > +EXPLAIN SELECT id FROM t1 WHERE id IN (-9223372036854775807.0, -9223372036854775808.0); > +--echo # Cannot compare this as INT (yet) > +EXPLAIN SELECT id FROM t1 WHERE id IN (-9223372036854775808.0, -9223372036854775809.0); > +DROP TABLE t1; > diff --git a/sql/item.h b/sql/item.h > index 883cc791f384..9f215c74864b 100644 > --- a/sql/item.h > +++ b/sql/item.h > @@ -70,6 +70,11 @@ class Value: public st_value > bool is_temporal() const { return m_type == DYN_COL_DATETIME; } > bool is_string() const { return m_type == DYN_COL_STRING; } > bool is_decimal() const { return m_type == DYN_COL_DECIMAL; } > + Longlong_hybrid to_longlong_hybrid_native() const > + { > + DBUG_ASSERT(is_longlong()); > + return Longlong_hybrid(value.m_longlong, m_type == DYN_COL_UINT); > + } > }; > > > diff --git a/sql/item_cmpfunc.cc b/sql/item_cmpfunc.cc > index b2fa753f2bd2..c7176df43d6a 100644 > --- a/sql/item_cmpfunc.cc > +++ b/sql/item_cmpfunc.cc > @@ -3747,11 +3748,45 @@ bool Predicant_to_list_comparator::add_value(const char *funcname, > Item *tmpargs[2]; > tmpargs[0]= args->arguments()[m_predicant_index]; > tmpargs[1]= args->arguments()[value_index]; > - if (tmp.aggregate_for_comparison(funcname, tmpargs, 2, true)) > + if (tmp.aggregate_for_comparison(funcname, tmpargs, 2, false)) How would that work? Are you going to compare signed to unsigned as longlongs? > { > DBUG_ASSERT(current_thd->is_error()); > return true; > } > + /* > + Try to compare using type handler of the predicant when possible, > + as this can use indexes for conditions like: > + WHERE field IN (const1, .. constN) > + */ > + if (prefer_predicant_type_handler && > + args->arguments()[value_index]->const_item() && > + !args->arguments()[value_index]->is_expensive() && > + tmp.type_handler()->cmp_type() == DECIMAL_RESULT && > + args->arguments()[m_predicant_index]->cmp_type() == INT_RESULT) > + { > + /* > + For now we catch only one special case: > + an INT predicant can be compared to a DECIMAL constant > + using Longlong_hybrid comparison, when the DECIMAL constant: > + a. has no significant fractional digits, and > + b. is within the signed longlong range > + For DECIMAL constants in the range LONGLONG_MAX..ULONGLONG_MAX > + we cannot call to_longlong_hybrid() safely, because DECIMAL type > + Items usually set their "unsigned_flag" to "false", so val_int() > + will truncate such constants to LONGLONG_MAX (instead of ULONGLONG_MAX). > + TODO: fix to_longlong_hybrid() for DECIMAL LONGLONG_MAX..ULONGLONG_MAX > + TODO: move this code to Type_handler (will be done by MDEV-18898) > + TODO: skip constants outside of LONGLONG_MIN..ULONGLONG_MAX, as > + such conditon can never be true, e.g.: > + WHERE int_expr IN (.. -9223372036854775809 ..) > + WHERE int_expr IN (.. 18446744073709551616 ..) > + */ > + my_decimal buf, *dec= args->arguments()[value_index]->val_decimal(&buf); > + longlong res; > + if (dec && decimal_actual_fraction(dec) == 0 && > + my_decimal2int(0, dec, false /*SIGNED*/, &res) == 0) > + tmp.set_handler(&type_handler_longlong); why not to replace the item with Item_int or Item_cache_int here? Then you'll be able to handle decimals up to ULONGLONG_MAX, that's your first TODO. For the last TODO, you can simply remove the always-false element from the arguments[] array. > + } > m_comparators[m_comparator_count].m_handler= tmp.type_handler(); > m_comparators[m_comparator_count].m_arg_index= value_index; > m_comparator_count++; > diff --git a/sql/item_cmpfunc.h b/sql/item_cmpfunc.h > index 06f15503258a..4e84c3008578 100644 > --- a/sql/item_cmpfunc.h > +++ b/sql/item_cmpfunc.h > @@ -1588,29 +1588,28 @@ class cmp_item_sort_string :public cmp_item_string > > class cmp_item_int : public cmp_item_scalar > { > - longlong value; > + Longlong_hybrid m_value; > public: > - cmp_item_int() {} /* Remove gcc warning */ > + cmp_item_int(): m_value(0, false) {} > void store_value(Item *item) > { > - value= item->val_int(); > + m_value= item->to_longlong_hybrid(); I wonder if to_longlong_hybrid() returns a Longlong_hybrid like that, where the compiler will allocate it? Could you check it, please? > m_null_value= item->null_value; > } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 00f9df29224: MDEV-20787: Script dgcov.pl does not work
by Sergei Golubchik 23 Apr '20

23 Apr '20

Hi, Anel! On Apr 23, Anel Husakovic wrote: > revision-id: 00f9df29224 (mariadb-10.2.29-23-g00f9df29224) > parent(s): 6718d3bc324 > author: Anel Husakovic <anel(a)mariadb.org> > committer: Anel Husakovic <anel(a)mariadb.org> > timestamp: 2019-11-19 15:00:48 +0100 > message: > > MDEV-20787: Script dgcov.pl does not work > > Let's change CMakeList with `--coverage` flag as an alias for > `-fprofile-arcs -ftest-coverage -lgcov` in addition. > When the server is compiled with `-DENABLE_GCOV=ON`, from object files are generated > `.gcno` and `.gcda` files. > `./mtr --gcov is_check_constraint` is invoking the script calls > `./dgcov.pl --purge`, `./mtr is_check_constraint`, > `./dgcov.pl --generate>/var/last_changes.dgcov`. > The `purge` flag is clearing `.gcda` files (and others extensions), > while running the test new `.gcda` files are obtained. > With `generate` flag, `gcov -i` (`intermediate format`) is called > on obtained `<object-files-name>.gcda` files (`dbug.c.gcda` e.g.). > The patch is tested on `gcov 6.3` and `gcov 7.4` versions > and can be seen that resulting `.gcov` file for `6.3` creates > `<full path>.gcov` (`dbug.c.gcda.gcov` e.g.) file, > where `gcov 7.4` is still creating `object-file-names.gcov`(`dbug.c.gcov`) files > as `gcov` in general is doing. > > diff --git a/CMakeLists.txt b/CMakeLists.txt > index 0dcc2a75587..af025a0312f 100644 > --- a/CMakeLists.txt > +++ b/CMakeLists.txt > @@ -249,7 +249,7 @@ SET(CMAKE_C_FLAGS_DEBUG "${CMAKE_C_FLAGS_DEBUG} -DENABLED_DEBUG_SYNC") > > OPTION(ENABLE_GCOV "Enable gcov (debug, Linux builds only)" OFF) > IF (ENABLE_GCOV) > - MY_CHECK_AND_SET_COMPILER_FLAG("-fprofile-arcs -ftest-coverage -lgcov" DEBUG) > + MY_CHECK_AND_SET_COMPILER_FLAG("--coverage" DEBUG) Why? Since what version does gcc support it? > ENDIF() > > MY_CHECK_AND_SET_COMPILER_FLAG(-ggdb3 DEBUG) > diff --git a/mysql-test/dgcov.pl b/mysql-test/dgcov.pl > index fbc5540e697..47ffaca04ef 100755 > --- a/mysql-test/dgcov.pl > +++ b/mysql-test/dgcov.pl > @@ -68,8 +68,11 @@ if ($opt_purge) > system($cmd)==0 or die "system($cmd): $? $!"; > exit 0; > } > - > +my $gcov_vers= `gcov -v`; > +$gcov_vers=~ s/\D//g; > +$gcov_vers= substr($gcov_vers, 0, 1); my gcov is 9.3.0. gcc version 10 is already out. You get just the first digit, 10 will become 1. Better to something like $gcov_vers ~= s/^.*(\d+\.\d+\.\d+).*$/\1/s; > find(\&gcov_one_file, $root); > +undef $gcov_vers; why? > find(\&write_coverage, $root) if $opt_generate; > exit 0 if $opt_only_gcov; > > @@ -162,7 +165,16 @@ sub gcov_one_file { > } > > # now, read the generated file > - open FH, '<', "$_.gcov" or die "open(<$_.gcov): $!"; > + if($gcov_vers<7) > + { > + open FH, '<', "$_.gcov" or die "open(<$_.gcov): $!"; > + } > + else > + { > + my $f=substr $_, 0, -5; > + open FH, '<', "$f.gcov" or die "open(<$f.gcov): $!"; > + undef $f; > + } I'd rather write it more compact, in two lines like my $f = $gcov_vers < 7 ? $_ : substr $_, 0, -5; open FH, '<', "$f.gcov" or die "open(<$f.gcov): $!"; > my $fname; > while (<FH>) { > chomp; > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] fb963a7fd3b: MDEV-21117: --tc-heuristic-recover=rollback is not replication safe
by sujatha 21 Apr '20

21 Apr '20

revision-id: fb963a7fd3b3e0733fb7ad9ece6256aebd8c176c (mariadb-10.1.43-112-gfb963a7fd3b) parent(s): ad4b70562bb94dd063eebde5189c6e730d3120a2 author: Sujatha committer: Sujatha timestamp: 2020-04-21 18:19:34 +0530 message: MDEV-21117: --tc-heuristic-recover=rollback is not replication safe Problem: ======= When run after master server crash --tc-heuristic-recover=rollback produces inconsistent server state with binlog still containing transactions that were rolled back by the option. Such way recovered server may not be used for replication. Fix: === During "--tc-heuristic-recover=ROLLBACK", query the storage engine to get binlog file name and position corresponding to the last committed transaction. This marks the consistent binlog state. If last_commit_file is not set then checkpoint binary log file is considered as starting point. Look for first transactional event beyond the consistent binlog state. This will be the starting point for heuristic rollback. Consider this event specific starting point as binlog truncation position. Now traverse the rest of binlogs beyond this point. During this traversal check for the presence of DDL or non transactional operations, as they cannot be safely rolled back. If such events are found the truncation will not happen it will return an error. If only transactional events are found beyond the binlog truncation position it is safe to truncate binlog. The log gets truncated to the identified position and the GTID state is adjusted accordingly. If there are more binary logs beyond the being truncated file they are all removed. If recovery is initiated based on last committed transaction specific binary log name and position, and there is failure during recovery, then recovery may be retried by remove the source of the failure. i.e --tc-heuristic-recover=rollback may be retried. Retry will not be successful if recovery is based on checkpoint file. As at the end of tc-heuristic-recover a new binary log file is created which will change the checkpoint file. Appropriate error message is written to error log for each case. --- .../r/binlog_heuristic_rollback_active_log.result | 18 + .../binlog/r/binlog_truncate_multi_log.result | 33 ++ .../r/binlog_truncate_multi_log_unsafe.result | 30 ++ .../suite/binlog/r/binlog_truncate_none.result | 42 ++ .../binlog/r/binlog_truncate_retry_success.result | 28 + .../t/binlog_heuristic_rollback_active_log.test | 75 +++ .../suite/binlog/t/binlog_truncate_multi_log.test | 87 +++ .../binlog/t/binlog_truncate_multi_log_unsafe.test | 106 ++++ .../suite/binlog/t/binlog_truncate_none.test | 130 +++++ .../binlog/t/binlog_truncate_retry_success.test | 84 +++ .../suite/rpl/r/rpl_heuristic_fail_over.result | 53 ++ .../suite/rpl/t/rpl_heuristic_fail_over.test | 160 ++++++ sql/log.cc | 589 ++++++++++++++++++++- sql/log.h | 7 +- sql/mysqld.h | 3 +- storage/innobase/handler/ha_innodb.cc | 7 + storage/innobase/log/log0recv.cc | 5 +- storage/xtradb/handler/ha_innodb.cc | 7 + storage/xtradb/log/log0recv.cc | 5 +- 19 files changed, 1457 insertions(+), 12 deletions(-) diff --git a/mysql-test/suite/binlog/r/binlog_heuristic_rollback_active_log.result b/mysql-test/suite/binlog/r/binlog_heuristic_rollback_active_log.result new file mode 100644 index 00000000000..eff95d05aac --- /dev/null +++ b/mysql-test/suite/binlog/r/binlog_heuristic_rollback_active_log.result @@ -0,0 +1,18 @@ +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); +RESET MASTER; +CREATE TABLE t ( f INT ) ENGINE=INNODB; +INSERT INTO t VALUES (10); +SET DEBUG_SYNC= "commit_before_update_end_pos SIGNAL con1_ready WAIT_FOR con1_go"; +INSERT INTO t VALUES (20); +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; +# Kill the server +"One row should be present in table 't'" +SELECT COUNT(*) FROM t; +COUNT(*) +1 +"gtid_binlog_state should be 0-1-2 +SELECT @@GLOBAL.gtid_binlog_state; +@@GLOBAL.gtid_binlog_state +0-1-2 +DROP TABLE t; diff --git a/mysql-test/suite/binlog/r/binlog_truncate_multi_log.result b/mysql-test/suite/binlog/r/binlog_truncate_multi_log.result new file mode 100644 index 00000000000..1a3d49b5463 --- /dev/null +++ b/mysql-test/suite/binlog/r/binlog_truncate_multi_log.result @@ -0,0 +1,33 @@ +SET @old_max_binlog_size= @@GLOBAL.max_binlog_size; +SET GLOBAL max_binlog_size= 4096; +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); +RESET MASTER; +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +FLUSH LOGS; +"List of binary logs before rotation" +show binary logs; +Log_name File_size +master-bin.000001 # +master-bin.000002 # +SET DEBUG_SYNC= "commit_after_release_LOCK_log SIGNAL con1_ready WAIT_FOR con1_go"; +INSERT INTO t1 VALUES (2, REPEAT("x", 4100)); +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; +"List of binary logs after rotation" +show binary logs; +Log_name File_size +master-bin.000001 # +master-bin.000002 # +master-bin.000003 # +# Kill the server +"Zero rows shoule be present in table" +SELECT COUNT(*) FROM t1; +COUNT(*) +0 +SELECT @@GLOBAL.gtid_current_pos; +@@GLOBAL.gtid_current_pos +0-1-1 +DROP TABLE t1; +SELECT @@GLOBAL.gtid_binlog_state; +@@GLOBAL.gtid_binlog_state +0-1-2 diff --git a/mysql-test/suite/binlog/r/binlog_truncate_multi_log_unsafe.result b/mysql-test/suite/binlog/r/binlog_truncate_multi_log_unsafe.result new file mode 100644 index 00000000000..5b7d45d81de --- /dev/null +++ b/mysql-test/suite/binlog/r/binlog_truncate_multi_log_unsafe.result @@ -0,0 +1,30 @@ +SET @old_max_binlog_size= @@global.max_binlog_size; +SET GLOBAL max_binlog_size= 4096; +call mtr.add_suppression("Table '.*tm' is marked as crashed and should be repaired"); +call mtr.add_suppression("Got an error from unknown thread"); +call mtr.add_suppression("Checking table: '.*tm'"); +call mtr.add_suppression("Recovering table: '.*tm'"); +call mtr.add_suppression("tc-heuristic-recover cannot trim the binary log to"); +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); +RESET MASTER; +CREATE TABLE ti (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +CREATE TABLE tm (f INT) ENGINE=MYISAM; +SET DEBUG_SYNC= "commit_before_get_LOCK_commit_ordered SIGNAL con1_ready WAIT_FOR con1_go"; +INSERT INTO ti VALUES (2, REPEAT("x", 4100)); +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; +INSERT INTO tm VALUES (30);; +# Kill the server +FOUND /tc-heuristic-recover cannot trim the binary log to File/ in mysqld.1.err +"Zero rows shoule be present in 'ti' table." +SELECT COUNT(*) FROM ti; +COUNT(*) +0 +"One row must be present in 'tm' table." +SELECT COUNT(*) FROM tm; +COUNT(*) +1 +SELECT @@GLOBAL.gtid_current_pos; +@@GLOBAL.gtid_current_pos +0-1-4 +DROP TABLE ti, tm; diff --git a/mysql-test/suite/binlog/r/binlog_truncate_none.result b/mysql-test/suite/binlog/r/binlog_truncate_none.result new file mode 100644 index 00000000000..bae87985208 --- /dev/null +++ b/mysql-test/suite/binlog/r/binlog_truncate_none.result @@ -0,0 +1,42 @@ +SET @old_max_binlog_size= @@global.max_binlog_size; +SET GLOBAL max_binlog_size= 4096; +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); +RESET MASTER; +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +INSERT INTO t1 VALUES (1, REPEAT("x", 4100)); +CREATE TABLE t2 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +show binary logs; +Log_name File_size +master-bin.000001 # +master-bin.000002 # +# Kill the server +"Zero records should be there." +SELECT COUNT(*) FROM t1; +COUNT(*) +1 +show binary logs; +Log_name File_size +master-bin.000001 # +master-bin.000002 # +master-bin.000003 # +master-bin.000004 # +DROP TABLE t1,t2; +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); +RESET MASTER; +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +CREATE TABLE t2 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +show binary logs; +Log_name File_size +master-bin.000001 # +# Kill the server +"Zero records should be there." +SELECT COUNT(*) FROM t1; +COUNT(*) +0 +SHOW TABLES; +Tables_in_test +t1 +t2 +DROP TABLE t1,t2; diff --git a/mysql-test/suite/binlog/r/binlog_truncate_retry_success.result b/mysql-test/suite/binlog/r/binlog_truncate_retry_success.result new file mode 100644 index 00000000000..cd0c4c23d05 --- /dev/null +++ b/mysql-test/suite/binlog/r/binlog_truncate_retry_success.result @@ -0,0 +1,28 @@ +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); +RESET MASTER; +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +INSERT INTO t1 VALUES (1,'dummy'); +SET DEBUG_SYNC= "commit_after_release_LOCK_log SIGNAL con1_ready WAIT_FOR con1_go"; +INSERT INTO t1 VALUES (2,'dummy'); +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; +"List of binary logs" +show binary logs; +Log_name File_size +master-bin.000001 # +# Kill the server +TEST 1: Recovery with error +TEST 2: Recovery without error +"One row shoule be present in table" +SELECT COUNT(*) FROM t1; +COUNT(*) +1 +"Two gtids should be present 0-1-2" +SELECT @@GLOBAL.gtid_current_pos; +@@GLOBAL.gtid_current_pos +0-1-2 +"Two gtids should be present 0-1-3" +DROP TABLE t1; +SELECT @@GLOBAL.gtid_binlog_state; +@@GLOBAL.gtid_binlog_state +0-1-3 diff --git a/mysql-test/suite/binlog/t/binlog_heuristic_rollback_active_log.test b/mysql-test/suite/binlog/t/binlog_heuristic_rollback_active_log.test new file mode 100644 index 00000000000..87af83a7a92 --- /dev/null +++ b/mysql-test/suite/binlog/t/binlog_heuristic_rollback_active_log.test @@ -0,0 +1,75 @@ +# ==== Purpose ==== +# +# Test verifies the truncation of single binary log file. +# +# ==== Implementation ==== +# +# Steps: +# 0 - Create table t1 and insert a row. +# 1 - Insert an another row such that it gets written to binlog but commit +# in engine fails as server crashed at this point. +# 2 - Restart server with --tc-heuristic-recover=ROLLBACK +# 3 - Upon server start 'master-bin.000001' will be truncated to contain +# only the first insert +# +# ==== References ==== +# +# MDEV-21117: --tc-heuristic-recover=rollback is not replication safe + + +--source include/have_innodb.inc +--source include/have_log_bin.inc +--source include/have_debug.inc +--source include/have_binlog_format_statement.inc + +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); + +connect(master,localhost,root,,); +connect(master1,localhost,root,,); + +--connection master +RESET MASTER; +CREATE TABLE t ( f INT ) ENGINE=INNODB; +INSERT INTO t VALUES (10); + +--connection master1 +# Hold insert after write to binlog and before "run_commit_ordered" in engine +SET DEBUG_SYNC= "commit_before_update_end_pos SIGNAL con1_ready WAIT_FOR con1_go"; +send INSERT INTO t VALUES (20); + +--connection master +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; + +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +wait +EOF + +--source include/kill_mysqld.inc +--source include/wait_until_disconnected.inc +# +# Server restart +# +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: --tc-heuristic-recover=ROLLBACK +EOF +--source include/wait_until_disconnected.inc + +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: +EOF + +connection default; +--enable_reconnect +--source include/wait_until_connected_again.inc + +--connection master +--enable_reconnect +--source include/wait_until_connected_again.inc + +--echo "One row should be present in table 't'" +SELECT COUNT(*) FROM t; + +--echo "gtid_binlog_state should be 0-1-2 +SELECT @@GLOBAL.gtid_binlog_state; +DROP TABLE t; diff --git a/mysql-test/suite/binlog/t/binlog_truncate_multi_log.test b/mysql-test/suite/binlog/t/binlog_truncate_multi_log.test new file mode 100644 index 00000000000..0a6e011627e --- /dev/null +++ b/mysql-test/suite/binlog/t/binlog_truncate_multi_log.test @@ -0,0 +1,87 @@ +# ==== Purpose ==== +# +# Test verifies truncation of multiple binary logs. +# +# ==== Implementation ==== +# +# Steps: +# 0 - Create a table in innodb engine and execute FLUSH LOGS command to +# generate a new binary log. +# 1 - Set max_binlog_size= 4096. Insert a row such that the max_binlog_size +# is reached and binary log gets rotated. +# 2 - Using debug simulation make the server crash at a point where the DML +# transaction is written to binary log but not committed in engine. +# 3 - At the time of crash three binary logs will be there +# master-bin.0000001, master-bin.000002 and master-bin.000003. +# 4 - Restart server with --tc-heuristic-recover=ROLLBACK +# 5 - Since the prepared DML in master-bin.000002 the binary log will be +# truncated and master-bin.000003 will be removed. +# +# ==== References ==== +# +# MDEV-21117: --tc-heuristic-recover=rollback is not replication safe + + +--source include/have_innodb.inc +--source include/have_log_bin.inc +--source include/have_debug.inc +--source include/have_debug_sync.inc +--source include/have_binlog_format_row.inc + +SET @old_max_binlog_size= @@GLOBAL.max_binlog_size; +SET GLOBAL max_binlog_size= 4096; +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); + +RESET MASTER; + +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +FLUSH LOGS; +connect(master1,localhost,root,,); +connect(master2,localhost,root,,); + +--connection master1 +# Hold insert after write to binlog and before "run_commit_ordered" in engine. +# Use "commit_after_release_LOCK_log" sync point. This point is reached after +# the binary log end position is updated which actually triggers binlog to be +# rotated. +--echo "List of binary logs before rotation" +--source include/show_binary_logs.inc +SET DEBUG_SYNC= "commit_after_release_LOCK_log SIGNAL con1_ready WAIT_FOR con1_go"; +send INSERT INTO t1 VALUES (2, REPEAT("x", 4100)); + +--connection master2 +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; +--echo "List of binary logs after rotation" +--source include/show_binary_logs.inc +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +wait +EOF + +--source include/kill_mysqld.inc +--source include/wait_until_disconnected.inc + +# +# Server restart +# +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: --tc-heuristic-recover=ROLLBACK --debug-dbug=d,simulate_innodb_forget_commit_pos +EOF +--source include/wait_until_disconnected.inc + +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: +EOF + +connection default; +--enable_reconnect +--source include/wait_until_connected_again.inc + +--echo "Zero rows shoule be present in table" +SELECT COUNT(*) FROM t1; + +SELECT @@GLOBAL.gtid_current_pos; + +DROP TABLE t1; +SELECT @@GLOBAL.gtid_binlog_state; + diff --git a/mysql-test/suite/binlog/t/binlog_truncate_multi_log_unsafe.test b/mysql-test/suite/binlog/t/binlog_truncate_multi_log_unsafe.test new file mode 100644 index 00000000000..cf4f3d34a24 --- /dev/null +++ b/mysql-test/suite/binlog/t/binlog_truncate_multi_log_unsafe.test @@ -0,0 +1,106 @@ +# ==== Purpose ==== +# +# Test verifies truncation of multiple binary logs will report an error on +# unsafe scenario. +# +# ==== Implementation ==== +# +# Steps: +# 0 - Set max_binlog_size= 4096. Create a table 'ti' using transactional +# storage engine. Do an insert such that the max_binlog_size is reached +# and binary log gets rotated. Hold this thread at a position where +# transaction is written to binary log but not committed in engine. +# 1 - Create a table named 'tm' using non transactional storage engine. +# 2 - Insert a row in 'tm' table. The DML will reach the engine and it is +# also written to binary log. +# 3 - Kill and restart the server with --tc-heuristic-recover=ROLLBACK +# 4 - Check for binary log truncation unsafe message in error log. +# 5 - No rows should be present in 'ti' table. One row should be present in +# 'tm' table. +# +# ==== References ==== +# +# MDEV-21117: --tc-heuristic-recover=rollback is not replication safe + + +--source include/have_innodb.inc +--source include/have_log_bin.inc +--source include/have_debug.inc +--source include/have_debug_sync.inc +--source include/have_binlog_format_row.inc + +SET @old_max_binlog_size= @@global.max_binlog_size; +SET GLOBAL max_binlog_size= 4096; + +call mtr.add_suppression("Table '.*tm' is marked as crashed and should be repaired"); +call mtr.add_suppression("Got an error from unknown thread"); +call mtr.add_suppression("Checking table: '.*tm'"); +call mtr.add_suppression("Recovering table: '.*tm'"); +call mtr.add_suppression("tc-heuristic-recover cannot trim the binary log to"); +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); + +RESET MASTER; + +CREATE TABLE ti (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +CREATE TABLE tm (f INT) ENGINE=MYISAM; + +connect(master1,localhost,root,,); +connect(master2,localhost,root,,); + +--connection master1 +# Hold insert after write to binlog and before "run_commit_ordered" in engine +SET DEBUG_SYNC= "commit_before_get_LOCK_commit_ordered SIGNAL con1_ready WAIT_FOR con1_go"; +--send INSERT INTO ti VALUES (2, REPEAT("x", 4100)) + +--connection master2 +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; +--send INSERT INTO tm VALUES (30); + +--connection default +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +wait +EOF + +--source include/kill_mysqld.inc +--source include/wait_until_disconnected.inc + +# +# Server restart +# +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: --tc-heuristic-recover=ROLLBACK --debug-dbug=d,simulate_innodb_forget_commit_pos +EOF +--source include/wait_until_disconnected.inc + +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: +EOF + +connection default; +--enable_reconnect +--source include/wait_until_connected_again.inc + +# Check error log for correct messages. +let $log_error_= `SELECT @@GLOBAL.log_error`; +if(!$log_error_) +{ + # MySQL Server on windows is started with --console and thus + # does not know the location of its .err log, use default location + let $log_error_ = $MYSQLTEST_VARDIR/log/mysqld.1.err; +} +--let SEARCH_FILE=$log_error_ +--let SEARCH_RANGE=-50000 +--let SEARCH_PATTERN=tc-heuristic-recover cannot trim the binary log to File +--source include/search_pattern_in_file.inc + +--echo "Zero rows shoule be present in 'ti' table." +SELECT COUNT(*) FROM ti; +--echo "One row must be present in 'tm' table." +--replace_regex /Table '.*tm/Table 'tm/ +--disable_warnings +SELECT COUNT(*) FROM tm; +--enable_warnings +SELECT @@GLOBAL.gtid_current_pos; + +DROP TABLE ti, tm; diff --git a/mysql-test/suite/binlog/t/binlog_truncate_none.test b/mysql-test/suite/binlog/t/binlog_truncate_none.test new file mode 100644 index 00000000000..cc3fd4f0b37 --- /dev/null +++ b/mysql-test/suite/binlog/t/binlog_truncate_none.test @@ -0,0 +1,130 @@ +# ==== Purpose ==== +# +# Test case verifies no binlog truncation happens when non transactional +# events are found in binlog after the last committed transaction. +# +# ==== Implementation ==== +# +# Steps: +# 0 - Set max_binlog_size= 4096. Create a table and do an insert such that +# the max_binlog_size is reached and binary log gets rotated. +# 1 - Create a table in newly created binary log and crash the server +# 2 - Restart server with --tc-heuristic-recover=ROLLBACK +# 3 - Recovery code will get the last committed DML specific postion and +# will try to check if binlog can be truncated upto this position. +# Since a DDL is present beyond this no truncation will happen. +# ==== References ==== +# +# MDEV-21117: --tc-heuristic-recover=rollback is not replication safe + + +--source include/have_innodb.inc +--source include/have_log_bin.inc +--source include/have_debug.inc +--source include/have_binlog_format_row.inc + +SET @old_max_binlog_size= @@global.max_binlog_size; +SET GLOBAL max_binlog_size= 4096; + +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); + +RESET MASTER; + +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +INSERT INTO t1 VALUES (1, REPEAT("x", 4100)); +CREATE TABLE t2 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +--source include/show_binary_logs.inc + +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +wait +EOF + +--source include/kill_mysqld.inc +--source include/wait_until_disconnected.inc + +# +# Server restart +# +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: --tc-heuristic-recover=ROLLBACK +EOF +--source include/wait_until_disconnected.inc + +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: +EOF + +connection default; +--enable_reconnect +--source include/wait_until_connected_again.inc + +--echo "Zero records should be there." +SELECT COUNT(*) FROM t1; +--source include/show_binary_logs.inc +DROP TABLE t1,t2; + +# ==== Purpose ==== +# +# Test case verifies no binlog truncation happens when only DDLs are present in +# the binary log. Since none of the DMLs are performed in storage engine, +# Engine will not have last committed transaction file name or position. +# Truncation code should return success. +# +# ==== Implementation ==== +# +# Steps: +# 0 - Create two table t1, t2 +# 1 - Kill and restart server with --tc-heuristic-recover=ROLLBACK +# 2 - Only DDL statements are present in the binary log. Since +# no DML was performed engine will not have last commited transaction +# specific binary log name and position. Since no transactional events +# are found, truncation code should simply return. +# +# ==== References ==== +# +# MDEV-21117: --tc-heuristic-recover=rollback is not replication safe + + +--source include/have_innodb.inc +--source include/have_log_bin.inc +--source include/have_debug.inc +--source include/have_binlog_format_row.inc + +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); + +RESET MASTER; + +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +CREATE TABLE t2 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +--source include/show_binary_logs.inc + +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +wait +EOF + +--source include/kill_mysqld.inc +--source include/wait_until_disconnected.inc + +# +# Server restart +# +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: --tc-heuristic-recover=ROLLBACK --debug-dbug=d,simulate_innodb_forget_commit_pos +EOF +--source include/wait_until_disconnected.inc + +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: +EOF + +connection default; +--enable_reconnect +--source include/wait_until_connected_again.inc + +--echo "Zero records should be there." +SELECT COUNT(*) FROM t1; +SHOW TABLES; +DROP TABLE t1,t2; + diff --git a/mysql-test/suite/binlog/t/binlog_truncate_retry_success.test b/mysql-test/suite/binlog/t/binlog_truncate_retry_success.test new file mode 100644 index 00000000000..6e92667c9b4 --- /dev/null +++ b/mysql-test/suite/binlog/t/binlog_truncate_retry_success.test @@ -0,0 +1,84 @@ +# ==== Purpose ==== +# +# Test verifies that tc-heuristic-recover=ROLLBACK can be retried. +# +# ==== Implementation ==== +# +# Steps: +# 0 - Create a table in Innodb storage engine. Insert a row into the table. +# During the commit of this DML engine will persist last committed +# transaction specific binary log file name and position. +# 1 - Do an another DML into the table, and simulate a crash in the middle +# of DML commit, so that DML is present in binary log but not committed in +# the storage engine. +# 2 - Restart the server using --tc-heuristic-recover=ROLLBACK. Simulate an +# error during the binary log rollback operation. Verify appropriate +# error is reported in the error log. +# 3 - Retry the --tc-heuristic-recver=ROLLBACK operation. Verify that +# binary log gets truncated as expected. +# 4 - Verify that global gtid state is according to the rolled back +# transaction. +# ==== References ==== +# +# MDEV-21117: --tc-heuristic-recover=rollback is not replication safe + + +--source include/have_innodb.inc +--source include/not_embedded.inc +--source include/have_log_bin.inc +--source include/have_debug.inc +--source include/have_debug_sync.inc +--source include/have_binlog_format_row.inc + +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); + +RESET MASTER; + +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +INSERT INTO t1 VALUES (1,'dummy'); +connect(master1,localhost,root,,); +connect(master2,localhost,root,,); + +--connection master1 +SET DEBUG_SYNC= "commit_after_release_LOCK_log SIGNAL con1_ready WAIT_FOR con1_go"; +send INSERT INTO t1 VALUES (2,'dummy'); + +--connection master2 +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; +--echo "List of binary logs" +--source include/show_binary_logs.inc +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +wait +EOF + +--source include/kill_mysqld.inc +--source include/wait_until_disconnected.inc + +# +# Server restart +# +--echo TEST 1: Recovery with error +--error 1 +--exec $MYSQLD_LAST_CMD --tc-heuristic-recover=ROLLBACK --debug-dbug=d,fault_injection_opening_binlog > $MYSQLTEST_VARDIR/log/mysqld.1.err 2>&1 +--source include/wait_until_disconnected.inc + +--echo TEST 2: Recovery without error +--error 1 +--exec $MYSQLD_LAST_CMD --tc-heuristic-recover=ROLLBACK > $MYSQLTEST_VARDIR/log/mysqld.1.err 2>&1 +--source include/wait_until_disconnected.inc + +--source include/start_mysqld.inc + +--echo "One row shoule be present in table" +SELECT COUNT(*) FROM t1; + +--echo "Two gtids should be present 0-1-2" +SELECT @@GLOBAL.gtid_current_pos; + +--echo "Two gtids should be present 0-1-3" +DROP TABLE t1; +SELECT @@GLOBAL.gtid_binlog_state; + +--disconnect master1 +--disconnect master2 diff --git a/mysql-test/suite/rpl/r/rpl_heuristic_fail_over.result b/mysql-test/suite/rpl/r/rpl_heuristic_fail_over.result new file mode 100644 index 00000000000..02fc47def41 --- /dev/null +++ b/mysql-test/suite/rpl/r/rpl_heuristic_fail_over.result @@ -0,0 +1,53 @@ +include/rpl_init.inc [topology=1->2] +include/stop_slave.inc +set global rpl_semi_sync_slave_enabled = 1; +CHANGE MASTER TO master_use_gtid= current_pos; +include/start_slave.inc +ALTER TABLE mysql.gtid_slave_pos ENGINE=InnoDB; +set global rpl_semi_sync_master_enabled = 1; +set global rpl_semi_sync_master_wait_point=AFTER_SYNC; +SET @old_max_binlog_size= @@global.max_binlog_size; +SET GLOBAL max_binlog_size= 4096; +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +INSERT INTO t1 VALUES (1, 'dummy1'); +SET DEBUG_SYNC= "commit_before_update_end_pos SIGNAL con1_ready WAIT_FOR con1_go"; +INSERT INTO t1 VALUES (2, REPEAT("x", 4100)); +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; +# Kill the server +include/stop_slave.inc +SELECT @@GLOBAL.gtid_current_pos; +@@GLOBAL.gtid_current_pos +0-1-5 +FOUND /tc-heuristic-recover: Truncated binlog File: \.\/master\-bin\.000001 of Size:[0-9]*, to Position */ in mysqld.1.err +set global rpl_semi_sync_master_enabled = 1; +set global rpl_semi_sync_master_wait_point=AFTER_SYNC; +CHANGE MASTER TO master_host='127.0.0.1', master_port=SERVER_MYPORT_2, master_user='root', master_use_gtid=CURRENT_POS; +set global rpl_semi_sync_slave_enabled = 1; +include/start_slave.inc +INSERT INTO t1 VALUES (3, 'dummy3'); +SELECT COUNT(*) FROM t1; +COUNT(*) +2 +SHOW VARIABLES LIKE 'gtid_current_pos'; +Variable_name Value +gtid_current_pos 0-2-6 +SHOW VARIABLES LIKE 'gtid_current_pos'; +Variable_name Value +gtid_current_pos 0-2-6 +DROP TABLE t1; +set global rpl_semi_sync_master_enabled = 0; +set global rpl_semi_sync_slave_enabled = 0; +set global rpl_semi_sync_master_wait_point=default; +set global rpl_semi_sync_master_enabled = 0; +set global rpl_semi_sync_slave_enabled = 0; +set global rpl_semi_sync_master_wait_point=default; +include/stop_slave.inc +RESET MASTER; +RESET SLAVE; +RESET MASTER; +RESET SLAVE; +CHANGE MASTER TO master_host='127.0.0.1', master_port=SERVER_MYPORT_1, master_user='root', master_use_gtid=no; +include/start_slave.inc +include/rpl_end.inc diff --git a/mysql-test/suite/rpl/t/rpl_heuristic_fail_over.test b/mysql-test/suite/rpl/t/rpl_heuristic_fail_over.test new file mode 100644 index 00000000000..39b1f7545d0 --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_heuristic_fail_over.test @@ -0,0 +1,160 @@ +# ==== Purpose ==== +# +# Test verifies replication failover scenario. +# +# ==== Implementation ==== +# +# Steps: +# 0 - Have two servers with id's 1 and 2. Enable semi-sync based +# replication. Have semi sync master wait point as 'after_sync'. +# 1 - Create a table and insert a row. While inserting second row simulate +# a server crash at once the transaction is written to binlog, flushed +# and synced but the binlog position is not updated. +# 2 - Restart the server using tc-heuristic-recover=ROLLBACK +# 3 - Post restart switch the crashed master to slave. Execute CHANGE MASTER +# TO command to connect to server id 2. +# 4 - Slave should be able to connect to master. +# 5 - Execute some DML on new master with server id 2. Ensure that it gets +# replicated to server id 1. +# 6 - Verify the "gtid_current_pos" for correctness. +# 7 - Clean up +# +# ==== References ==== +# +# MDEV-21117: --tc-heuristic-recover=rollback is not replication safe + + +--source include/have_semisync.inc +--source include/have_innodb.inc +--source include/have_log_bin.inc +--source include/have_debug.inc +--source include/have_debug_sync.inc +--source include/have_binlog_format_row.inc +--let $rpl_topology=1->2 +--source include/rpl_init.inc + +--connection server_2 +--source include/stop_slave.inc +set global rpl_semi_sync_slave_enabled = 1; +CHANGE MASTER TO master_use_gtid= current_pos; +--source include/start_slave.inc + + +--connection server_1 +ALTER TABLE mysql.gtid_slave_pos ENGINE=InnoDB; +set global rpl_semi_sync_master_enabled = 1; +set global rpl_semi_sync_master_wait_point=AFTER_SYNC; +SET @old_max_binlog_size= @@global.max_binlog_size; +SET GLOBAL max_binlog_size= 4096; + +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); + +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +INSERT INTO t1 VALUES (1, 'dummy1'); +--save_master_pos + +--connection server_2 +--sync_with_master + +--connection server_1 +connect(master1,localhost,root,,); +connect(master2,localhost,root,,); + +--connection master1 +# Hold insert after write to binlog and before "run_commit_ordered" in engine +SET DEBUG_SYNC= "commit_before_update_end_pos SIGNAL con1_ready WAIT_FOR con1_go"; +send INSERT INTO t1 VALUES (2, REPEAT("x", 4100)); + +--connection master2 +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +wait +EOF + +--source include/kill_mysqld.inc +--source include/wait_until_disconnected.inc + +--connection server_2 +--error 2003 +--source include/stop_slave.inc +SELECT @@GLOBAL.gtid_current_pos; + +--connection server_1 +# +# Server restart +# +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: --tc-heuristic-recover=ROLLBACK +EOF +--source include/wait_until_disconnected.inc + +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: +EOF + +connection default; +--enable_reconnect +--source include/wait_until_connected_again.inc + +--connection server_1 +--enable_reconnect +--source include/wait_until_connected_again.inc + +# Check error log for correct messages. +let $log_error_= `SELECT @@GLOBAL.log_error`; +if(!$log_error_) +{ + # MySQL Server on windows is started with --console and thus + # does not know the location of its .err log, use default location + let $log_error_ = $MYSQLTEST_VARDIR/log/mysqld.1.err; +} +--let SEARCH_FILE=$log_error_ +--let SEARCH_RANGE=-50000 +--let SEARCH_PATTERN=tc-heuristic-recover: Truncated binlog File: \.\/master\-bin\.000001 of Size:[0-9]*, to Position * +--source include/search_pattern_in_file.inc + +--connection server_2 +set global rpl_semi_sync_master_enabled = 1; +set global rpl_semi_sync_master_wait_point=AFTER_SYNC; + +--connection server_1 +--replace_result $SERVER_MYPORT_2 SERVER_MYPORT_2 +eval CHANGE MASTER TO master_host='127.0.0.1', master_port=$SERVER_MYPORT_2, master_user='root', master_use_gtid=CURRENT_POS; +set global rpl_semi_sync_slave_enabled = 1; +--source include/start_slave.inc + +--connection server_2 +INSERT INTO t1 VALUES (3, 'dummy3'); +--save_master_pos + +--connection server_1 +--sync_with_master +SELECT COUNT(*) FROM t1; +SHOW VARIABLES LIKE 'gtid_current_pos'; + +--connection server_2 +SHOW VARIABLES LIKE 'gtid_current_pos'; +DROP TABLE t1; +set global rpl_semi_sync_master_enabled = 0; +set global rpl_semi_sync_slave_enabled = 0; +set global rpl_semi_sync_master_wait_point=default; +--save_master_pos + +--connection server_1 +--sync_with_master +set global rpl_semi_sync_master_enabled = 0; +set global rpl_semi_sync_slave_enabled = 0; +set global rpl_semi_sync_master_wait_point=default; +--source include/stop_slave.inc +RESET MASTER; +RESET SLAVE; + +--connection server_2 +RESET MASTER; +RESET SLAVE; +--replace_result $SERVER_MYPORT_1 SERVER_MYPORT_1 +eval CHANGE MASTER TO master_host='127.0.0.1', master_port=$SERVER_MYPORT_1, master_user='root', master_use_gtid=no; +--source include/start_slave.inc + +--source include/rpl_end.inc diff --git a/sql/log.cc b/sql/log.cc index 0efef6d1e29..66445944b40 100644 --- a/sql/log.cc +++ b/sql/log.cc @@ -3164,6 +3164,7 @@ MYSQL_BIN_LOG::MYSQL_BIN_LOG(uint *sync_period) checksum_alg_reset(BINLOG_CHECKSUM_ALG_UNDEF), relay_log_checksum_alg(BINLOG_CHECKSUM_ALG_UNDEF), description_event_for_exec(0), description_event_for_queue(0), + last_commit_pos_offset(0), current_binlog_id(0) { /* @@ -3173,6 +3174,7 @@ MYSQL_BIN_LOG::MYSQL_BIN_LOG(uint *sync_period) before main(). */ index_file_name[0] = 0; + last_commit_pos_file[0]= 0; bzero((char*) &index_file, sizeof(index_file)); bzero((char*) &purge_index_file, sizeof(purge_index_file)); } @@ -4576,8 +4578,8 @@ int MYSQL_BIN_LOG::open_purge_index_file(bool destroy) 0, 0, MYF(MY_WME | MY_NABP | MY_WAIT_IF_FULL))) { error= 1; - sql_print_error("MYSQL_BIN_LOG::open_purge_index_file failed to open register " - " file."); + sql_print_error("MYSQL_BIN_LOG::open_purge_index_file failed to open " + "register file."); } } DBUG_RETURN(error); @@ -4742,7 +4744,7 @@ int MYSQL_BIN_LOG::purge_index_entry(THD *thd, ulonglong *reclaimed_space, } goto err; } - + error= 0; DBUG_PRINT("info",("purging %s",log_info.log_file_name)); @@ -7877,6 +7879,7 @@ MYSQL_BIN_LOG::trx_group_commit_leader(group_commit_entry *leader) first= false; } + DEBUG_SYNC(leader->thd, "commit_before_update_end_pos"); /* update binlog_end_pos so it can be read by dump thread * * note: must be _after_ the RUN_HOOK(after_flush) or else @@ -8964,7 +8967,7 @@ int TC_LOG_MMAP::open(const char *opt_name) { if (my_errno != ENOENT) goto err; - if (using_heuristic_recover()) + if (using_heuristic_recover(opt_name)) return 1; if ((fd= mysql_file_create(key_file_tclog, logname, CREATE_MODE, O_RDWR | O_CLOEXEC, MYF(MY_WME))) < 0) @@ -9497,14 +9500,40 @@ TC_LOG_MMAP tc_log_mmap; 1 heuristic recovery was performed */ -int TC_LOG::using_heuristic_recover() +int TC_LOG::using_heuristic_recover(const char* opt_name) { + LOG_INFO log_info; + int error; + if (!tc_heuristic_recover) return 0; sql_print_information("Heuristic crash recovery mode"); + if (ha_recover(0)) + { sql_print_error("Heuristic crash recovery failed"); + } + + if (!strcmp(opt_name, opt_bin_logname) && + tc_heuristic_recover == TC_HEURISTIC_RECOVER_ROLLBACK) + { + if ((error= mysql_bin_log.find_log_pos(&log_info, NullS, 1))) + { + if (error != LOG_INFO_EOF) + sql_print_error("tc-heuristic-recover failed in find_log_pos(). " + "Error: %d", error); + } + else + { + if ((error= heuristic_binlog_rollback())) + { + sql_print_error("Heuristic crash recovery of binary log failed."); + return 1; + } + } + } + sql_print_information("Please restart mysqld without --tc-heuristic-recover"); return 1; } @@ -9512,6 +9541,554 @@ int TC_LOG::using_heuristic_recover() /****** transaction coordinator log for 2pc - binlog() based solution ******/ #define TC_LOG_BINLOG MYSQL_BIN_LOG +/** + Truncates the current binlog to specified position. Removes the rest of binlogs + which are present after this binlog file. + + @param truncate_file Holds the binlog name to be truncated + @param truncate_pos Position within binlog from where it needs to + truncated. + + @retval true ok + @retval false error + +*/ +bool MYSQL_BIN_LOG::truncate_and_remove_binlogs(const char *truncate_file, + my_off_t truncate_pos) +{ + int error= 0; +#ifdef HAVE_REPLICATION + LOG_INFO log_info; + THD *thd= current_thd; + my_off_t index_file_offset= 0; + File file= -1; + IO_CACHE cache; + MY_STAT s; + my_off_t binlog_size; + + if ((error= find_log_pos(&log_info, truncate_file, 1))) + { + sql_print_error("tc-heuristic-recover: Failed to locate binary log file:%s." + "Error:%d", truncate_file, error); + goto end; + } + + while (!(error= find_next_log(&log_info, 1))) + { + if (!index_file_offset) + { + index_file_offset= log_info.index_file_start_offset; + if ((error= open_purge_index_file(TRUE))) + { + sql_print_error("tc-heuristic-recover: Failed to open purge index " + "file:%s. Error:%d", purge_index_file_name, error); + goto end; + } + } + if ((error= register_purge_index_entry(log_info.log_file_name))) + { + sql_print_error("tc-heuristic-recover: Failed to copy %s to purge index" + " file. Error:%d", log_info.log_file_name, error); + goto end; + } + } + + if (error != LOG_INFO_EOF) + { + sql_print_error("tc-heuristic-recover: Failed to find the next binlog to " + "add to purge index register. Error:%d", error); + goto end; + } + + if (is_inited_purge_index_file()) + { + if (!index_file_offset) + index_file_offset= log_info.index_file_start_offset; + + if ((error= sync_purge_index_file())) + { + sql_print_error("tc-heuristic-recover: Failed to flush purge index " + "file. Error:%d", error); + goto end; + } + + // Trim index file + if ((error= + mysql_file_chsize(index_file.file, index_file_offset, '\n', + MYF(MY_WME))) || + (error= + mysql_file_sync(index_file.file, MYF(MY_WME|MY_SYNC_FILESIZE)))) + { + sql_print_error("tc-heuristic-recover: Failed to trim binlog index " + "file:%s to offset:%llu. Error:%d", index_file_name, + index_file_offset); + mysql_file_close(index_file.file, MYF(MY_WME)); + goto end; + } + + /* Reset data in old index cache */ + if ((error= reinit_io_cache(&index_file, READ_CACHE, (my_off_t) 0, 0, 1))) + { + sql_print_error("tc-heuristic-recover: Failed to reinit binlog index " + "file. Error:%d", error); + mysql_file_close(index_file.file, MYF(MY_WME)); + goto end; + } + + /* Read each entry from purge_index_file and delete the file. */ + if ((error= purge_index_entry(thd, NULL, TRUE))) + { + sql_print_error("tc-heuristic-recover: Failed to process registered " + "files that would be purged."); + goto end; + } + } + + DBUG_ASSERT(truncate_pos); + + if ((file= mysql_file_open(key_file_binlog, truncate_file, + O_RDWR | O_BINARY, MYF(MY_WME))) < 0) + { + error= 1; + sql_print_error("tc-heuristic-recover: Failed to open binlog file:%s for " + "truncation.", truncate_file); + goto end; + } + my_stat(truncate_file, &s, MYF(0)); + binlog_size= s.st_size; + + /* Change binlog file size to truncate_pos */ + if ((error= + mysql_file_chsize(file, truncate_pos, 0, MYF(MY_WME))) || + (error= mysql_file_sync(file, MYF(MY_WME|MY_SYNC_FILESIZE)))) + { + sql_print_error("tc-heuristic-recover: Failed to trim the " + "binlog file:%s to size:%llu. Error:%d", + truncate_file, truncate_pos, error); + goto end; + } + else + { + sql_print_information("tc-heuristic-recover: Truncated binlog " + "File: %s of Size:%llu, to Position:%llu.", + truncate_file, binlog_size, truncate_pos); + } + if (!(error= init_io_cache(&cache, file, IO_SIZE, WRITE_CACHE, + (my_off_t) truncate_pos, 0, MYF(MY_WME|MY_NABP)))) + { + /* + Write Stop_log_event to ensure clean end point for the binary log being + truncated. + */ + Stop_log_event se; + se.checksum_alg= (enum_binlog_checksum_alg) binlog_checksum_options; + if ((error= write_event(&se, &cache))) + { + sql_print_error("tc-heuristic-recover: Failed to write stop event to " + "binary log. Errno:%d", + (cache.error == -1) ? my_errno : error); + goto end; + } + if ((error= flush_io_cache(&cache)) || + (error= mysql_file_sync(file, MYF(MY_WME|MY_SYNC_FILESIZE)))) + { + sql_print_error("tc-heuristic-recover: Faild to write stop event to " + "binary log. Errno:%d", + (cache.error == -1) ? my_errno : error); + } + } + else + sql_print_error("tc-heuristic-recover: Failed to initialize binary log " + "cache for writing stop event. Errno:%d", + (cache.error == -1) ? my_errno : error); + +end: + if (file >= 0) + { + end_io_cache(&cache); + mysql_file_close(file, MYF(MY_WME)); + } + error= error || close_purge_index_file(); +#endif + return error > 0; +} + +/** + Returns the checkpoint binlog file name found in the lastest binlog file. + + @param checkpoint_file Holds the binlog checkpoint file name. + + @retval 0 ok + @retval 1 error + +*/ +int TC_LOG_BINLOG::get_binlog_checkpoint_file(char* checkpoint_file) +{ + Log_event *ev= NULL; + bool binlog_checkpoint_found= false; + LOG_INFO log_info; + const char *errmsg; + IO_CACHE log; + File file; + Format_description_log_event fdle(BINLOG_VERSION); + char log_name[FN_REFLEN]; + int error=1; + + if (!fdle.is_valid()) + return 1; + + if ((error= find_log_pos(&log_info, NullS, 1))) + { + sql_print_error("tc-heuristic-recover: find_log_pos() failed to read first " + "binary log entry from index file.(error: %d)", error); + return error; + } + + // Move to the last binary log. + do + { + strmake_buf(log_name, log_info.log_file_name); + } while (!(error= find_next_log(&log_info, 1))); + + if (error != LOG_INFO_EOF) + { + sql_print_error("tc-heuristic-recover: find_next_log() failed " + "(error: %d)", error); + return error; + } + if ((file= open_binlog(&log, log_name, &errmsg)) < 0) + { + sql_print_error("tc-heuristic-recover failed to open the binlog:%s for " + "reading checkpoint file name. Error: %s", + log_info.log_file_name, errmsg); + return error; + } + while (!binlog_checkpoint_found && + (ev= + Log_event::read_log_event(&log, 0, &fdle, opt_master_verify_checksum)) + && ev->is_valid()) + { + enum Log_event_type typ= ev->get_type_code(); + if (typ == BINLOG_CHECKPOINT_EVENT) + { + size_t dir_len; + Binlog_checkpoint_log_event *cev= (Binlog_checkpoint_log_event *)ev; + if (cev->binlog_file_len >= FN_REFLEN) + { + sql_print_error("tc-heuristic-recover: Incorrect binlog checkpoint " + "event with too long file name found."); + delete ev; + ev= NULL; + end_io_cache(&log); + mysql_file_close(file, MYF(MY_WME)); + return 1; + } + else + { + dir_len= dirname_length(log_name); + strmake(strnmov(checkpoint_file, log_name, dir_len), + cev->binlog_file_name, FN_REFLEN - 1 - dir_len); + binlog_checkpoint_found= true; + } + } + delete ev; + ev= NULL; + } // End of while + end_io_cache(&log); + mysql_file_close(file, MYF(MY_WME)); + file= -1; + /* + Old binary log without checkpoint found, binlog truncation is not + possible. Hence return error. + */ + if (!binlog_checkpoint_found) + return 1; + + return 0; +} + + +/** + Truncates the binary log, according to the transactions that got rolled + back from engine, during heuristic-recover=ROLLBACK. Global GTID state is + adjusted as per the truncated binlog. + + Called from @c TC_LOG::using_heuristic_recover(const char* opt_name) + + @param opt_name The base name of binary log. + + @return indicates success or failure of binlog rollback + @retval 0 success + @retval 1 failure + +*/ +int TC_LOG_BINLOG::heuristic_binlog_rollback() +{ + int error=0; +#ifdef HAVE_REPLICATION + Log_event *ev= NULL; + char engine_commit_file[FN_REFLEN]; + char binlog_truncate_file_name[FN_REFLEN]; + char checkpoint_file[FN_REFLEN]; + my_off_t binlog_truncate_pos= 0; + my_off_t engine_commit_pos= 0; + LOG_INFO log_info; + const char *errmsg; + IO_CACHE log; + File file=-1; + Format_description_log_event fdle(BINLOG_VERSION); + bool found_engine_commit_pos= false; + bool found_truncate_pos= false; + bool is_safe= true; + my_off_t tmp_truncate_pos=0; + rpl_gtid last_gtid; + bool last_gtid_standalone= false; + bool last_gtid_valid= false; + bool is_checkpoint_based_recovery= false; + + + DBUG_EXECUTE_IF("simulate_innodb_forget_commit_pos", + { + last_commit_pos_file[0]= 0; + };); + + // Initialize engine_commit_file/pos + if (last_commit_pos_file[0] != 0) + { + strmake_buf(engine_commit_file, last_commit_pos_file); + engine_commit_pos= last_commit_pos_offset; + sql_print_information("tc-heuristic-recover: Initialising heuristic " + "rollback of binary log using last committed " + "transaction specific binary log name:%s and " + "position:%llu", last_commit_pos_file, + last_commit_pos_offset); + } + else + { + if ((error= get_binlog_checkpoint_file(checkpoint_file))) + { + is_checkpoint_based_recovery= true; + sql_print_error("tc-heuristic-recover: Failed to read latest checkpoint " + "binary log name."); + goto end; + } + strmake_buf(engine_commit_file, checkpoint_file); + sql_print_information("tc-heuristic-recover: Initialising heuristic " + "rollback of binary log using last checkpoint " + "file:%s.", engine_commit_file); + /* + If there is no engine specific commit file we are doing checkpoint file + based recovery. Hence we mark "found_engine_commit_pos" true, and start + looking for the first transactional event group with is the candidate for + rollback. + */ + found_engine_commit_pos= true; + } + + if ((error= find_log_pos(&log_info, engine_commit_file, 1))) + { + sql_print_error("tc-heuristic-recover: Failed to locate binary log file:%s " + "in index file. Error:%d", engine_commit_file, error); + goto end; + } + if ((file= open_binlog(&log, log_info.log_file_name, &errmsg)) < 0 || + DBUG_EVALUATE_IF("fault_injection_opening_binlog", (errmsg="Unknown"), + FALSE)) + { + error= 1; + sql_print_error("tc-heuristic-recover: Failed to open the binlog:%s for " + "recovery. Error:%s", log_info.log_file_name, errmsg); + goto end; + } + + + error= read_state_from_file(); + if (error && error != 2) + { + sql_print_error("tc-heuristic-recover: Failed to load global gtid binlog " + "state from file"); + goto end; + } + if (!fdle.is_valid()) + { + error= 1; + sql_print_error("tc-heuristic-recover: Failed due to invalid format " + "description log event."); + goto end; + } + + for(;;) + { + while (is_safe && + (ev= Log_event::read_log_event(&log, 0, &fdle, + opt_master_verify_checksum)) && ev->is_valid()) + { + enum Log_event_type typ= ev->get_type_code(); + switch (typ) + { + case XID_EVENT: + if (ev->log_pos == engine_commit_pos) + { + found_engine_commit_pos= true; + } + break; + case GTID_LIST_EVENT: + { + Gtid_list_log_event *glev= (Gtid_list_log_event *)ev; + /* Initialise the binlog state from the Gtid_list event. */ + if (!found_truncate_pos && glev->count > 0 && + rpl_global_gtid_binlog_state.load(glev->list, glev->count)) + { + error= 1; + sql_print_error("tc-heuristic-recover: Failed to read GTID List " + "event."); + goto end; + } + } + break; + case GTID_EVENT: + { + Gtid_log_event *gev= (Gtid_log_event *)ev; + + /* Update the binlog state with any GTID logged after Gtid_list. */ + last_gtid.domain_id= gev->domain_id; + last_gtid.server_id= gev->server_id; + last_gtid.seq_no= gev->seq_no; + last_gtid_standalone= + ((gev->flags2 & Gtid_log_event::FL_STANDALONE) ? true : false); + last_gtid_valid= true; + if (gev->flags2 & Gtid_log_event::FL_TRANSACTIONAL && + !found_truncate_pos) + { + if ((engine_commit_pos == 0 || found_engine_commit_pos)) + { + found_truncate_pos= true; + strmake_buf(binlog_truncate_file_name, log_info.log_file_name); + binlog_truncate_pos= tmp_truncate_pos; + } + } + else + { + if (found_truncate_pos) + is_safe= false; + } + } + break; + default: + /* Nothing. */ + break; + }// End switch + if (!found_truncate_pos && last_gtid_valid && + ((last_gtid_standalone && !ev->is_part_of_group(typ)) || + (!last_gtid_standalone && + (typ == XID_EVENT || + (typ == QUERY_EVENT && + (((Query_log_event *)ev)->is_commit() || + ((Query_log_event *)ev)->is_rollback())))))) + { + if ((error= rpl_global_gtid_binlog_state.update_nolock(&last_gtid, + false))) + { + sql_print_error("tc-heuristic-recover: Failed to update GTID within " + "global gtid state."); + goto end; + } + last_gtid_valid= false; + } + // Used to identify the last group specific end position. + tmp_truncate_pos= ev->log_pos; + delete ev; + ev= NULL; + }// End While + if (file >= 0) + { + end_io_cache(&log); + mysql_file_close(file, MYF(MY_WME)); + file= -1; + } + if (is_safe) + { + if ((error= find_next_log(&log_info, 1))) + { + if (error != LOG_INFO_EOF) + { + sql_print_error("tc-heuristic-recover: Failed to read next binary " + "log during recovery."); + goto end; + } + else + { + error= 0; // LOG_INFO_EOF= -1 is not an error. + break; + } + } + if ((file= open_binlog(&log, log_info.log_file_name, &errmsg)) < 0) + { + error= 1; + sql_print_error("tc-heuristic-recover: Failed to open the binlog:%s for " + "recovery. Error:%s", log_info.log_file_name, errmsg); + goto end; + } + } + else + break; + } //end of for(;;) + sql_print_information("tc-heuristic-recover: Binary log to be truncated " + "File:%s Pos:%llu.", binlog_truncate_file_name, + binlog_truncate_pos); + if (!found_truncate_pos) + goto end; // Nothing to truncate + else + DBUG_ASSERT(binlog_truncate_pos > 0); + + if (is_safe) + { + if ((error= truncate_and_remove_binlogs(binlog_truncate_file_name, + binlog_truncate_pos))) + { + sql_print_error("tc-heuristic-recover: Failed to trim the binary log to " + "File:%s Pos:%llu.", binlog_truncate_file_name, + binlog_truncate_pos); + goto end; + } + } + else + { + sql_print_error("tc-heuristic-recover cannot trim the binary log to " + "File:%s Pos:%llu as unsafe statements (non-trans/DDL) " + "statements are found beyond the truncation position.", + binlog_truncate_file_name, binlog_truncate_pos); + } + if ((error= write_state_to_file())) + { + sql_print_error("tc-heuristic-recover: Failed to write global gtid state " + "to file"); + goto end; + } + +end: + if (file >= 0) + { + end_io_cache(&log); + mysql_file_close(file, MYF(MY_WME)); + } + if (error) + { + if (is_checkpoint_based_recovery) + { + sql_print_error("tc-heuristic-recover failed during binary log rollback." + "Further retry attemps won't succeed."); + } + else + sql_print_error("tc-heuristic-recover failed during binary log rollback. " + "After removing the source of the failure " + "--tc-heuristic-recover=rollback may be retried."); + } + +#endif + return error; +} + int TC_LOG_BINLOG::open(const char *opt_name) { int error= 1; @@ -9526,7 +10103,7 @@ int TC_LOG_BINLOG::open(const char *opt_name) return 1; } - if (using_heuristic_recover()) + if (using_heuristic_recover(opt_name)) { mysql_mutex_lock(&LOCK_log); /* generate a new binlog to mask a corrupted one */ diff --git a/sql/log.h b/sql/log.h index 277e5c6f69c..84c0a457b52 100644 --- a/sql/log.h +++ b/sql/log.h @@ -41,7 +41,8 @@ bool stmt_has_updated_non_trans_table(const THD* thd); class TC_LOG { public: - int using_heuristic_recover(); + int using_heuristic_recover(const char* opt_name); + virtual int heuristic_binlog_rollback() { return 0; }; TC_LOG() {} virtual ~TC_LOG() {} @@ -694,6 +695,7 @@ class MYSQL_BIN_LOG: public TC_LOG, private MYSQL_LOG void commit_checkpoint_notify(void *cookie); int recover(LOG_INFO *linfo, const char *last_log_name, IO_CACHE *first_log, Format_description_log_event *fdle, bool do_xa); + int heuristic_binlog_rollback(); int do_binlog_recovery(const char *opt_name, bool do_xa_recovery); #if !defined(MYSQL_CLIENT) @@ -794,6 +796,9 @@ class MYSQL_BIN_LOG: public TC_LOG, private MYSQL_LOG int purge_first_log(Relay_log_info* rli, bool included); int set_purge_index_file_name(const char *base_file_name); int open_purge_index_file(bool destroy); + bool truncate_and_remove_binlogs(const char *truncate_file, + my_off_t truncate_pos); + int get_binlog_checkpoint_file(char* checkpoint_file); bool is_inited_purge_index_file(); int close_purge_index_file(); int clean_purge_index_file(); diff --git a/sql/mysqld.h b/sql/mysqld.h index e939524dbff..515d5f39c3d 100644 --- a/sql/mysqld.h +++ b/sql/mysqld.h @@ -94,6 +94,7 @@ extern "C" MYSQL_PLUGIN_IMPORT CHARSET_INFO *system_charset_info; extern MYSQL_PLUGIN_IMPORT CHARSET_INFO *files_charset_info ; extern MYSQL_PLUGIN_IMPORT CHARSET_INFO *national_charset_info; extern MYSQL_PLUGIN_IMPORT CHARSET_INFO *table_alias_charset; +extern MYSQL_PLUGIN_IMPORT bool opt_bin_log; /** Character set of the buildin error messages loaded from errmsg.sys. @@ -104,7 +105,7 @@ extern CHARSET_INFO *character_set_filesystem; extern MY_BITMAP temp_pool; extern bool opt_large_files, server_id_supplied; -extern bool opt_update_log, opt_bin_log, opt_error_log; +extern bool opt_update_log, opt_error_log; extern my_bool opt_log, opt_bootstrap; extern my_bool opt_backup_history_log; extern my_bool opt_backup_progress_log; diff --git a/storage/innobase/handler/ha_innodb.cc b/storage/innobase/handler/ha_innodb.cc index 4de2cdbeaec..552348ee2c5 100644 --- a/storage/innobase/handler/ha_innodb.cc +++ b/storage/innobase/handler/ha_innodb.cc @@ -16493,6 +16493,13 @@ innobase_xa_recover( { DBUG_ASSERT(hton == innodb_hton_ptr); + if (opt_bin_log) + { + mysql_bin_log.last_commit_pos_offset= trx_sys_mysql_bin_log_pos; + strmake_buf(mysql_bin_log.last_commit_pos_file, + trx_sys_mysql_bin_log_name); + } + if (len == 0 || xid_list == NULL) { return(0); diff --git a/storage/innobase/log/log0recv.cc b/storage/innobase/log/log0recv.cc index 3b3c7c23224..c8a742d95e9 100644 --- a/storage/innobase/log/log0recv.cc +++ b/storage/innobase/log/log0recv.cc @@ -2,7 +2,7 @@ Copyright (c) 1997, 2017, Oracle and/or its affiliates. All Rights Reserved. Copyright (c) 2012, Facebook Inc. -Copyright (c) 2013, 2019, MariaDB Corporation. +Copyright (c) 2013, 2020, MariaDB Corporation. This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software @@ -3301,7 +3301,8 @@ void recv_recovery_from_checkpoint_finish(void) /*======================================*/ { - if (recv_needed_recovery) { + extern MYSQL_PLUGIN_IMPORT bool opt_bin_log; + if (opt_bin_log) { trx_sys_print_mysql_master_log_pos(); trx_sys_print_mysql_binlog_offset(); } diff --git a/storage/xtradb/handler/ha_innodb.cc b/storage/xtradb/handler/ha_innodb.cc index 2aafb1a44ee..a966f726a03 100644 --- a/storage/xtradb/handler/ha_innodb.cc +++ b/storage/xtradb/handler/ha_innodb.cc @@ -17165,6 +17165,13 @@ innobase_xa_recover( { DBUG_ASSERT(hton == innodb_hton_ptr); + if (opt_bin_log) + { + mysql_bin_log.last_commit_pos_offset= trx_sys_mysql_bin_log_pos; + strmake_buf(mysql_bin_log.last_commit_pos_file, + trx_sys_mysql_bin_log_name); + } + if (len == 0 || xid_list == NULL) { return(0); diff --git a/storage/xtradb/log/log0recv.cc b/storage/xtradb/log/log0recv.cc index dd55d31218a..a4844507328 100644 --- a/storage/xtradb/log/log0recv.cc +++ b/storage/xtradb/log/log0recv.cc @@ -2,7 +2,7 @@ Copyright (c) 1997, 2017, Oracle and/or its affiliates. All Rights Reserved. Copyright (c) 2012, Facebook Inc. -Copyright (c) 2013, 2019, MariaDB Corporation. +Copyright (c) 2013, 2020, MariaDB Corporation. This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software @@ -3397,7 +3397,8 @@ void recv_recovery_from_checkpoint_finish(void) /*======================================*/ { - if (recv_needed_recovery) { + extern MYSQL_PLUGIN_IMPORT bool opt_bin_log; + if (opt_bin_log) { trx_sys_print_mysql_master_log_pos(); trx_sys_print_mysql_binlog_offset(); }

1 0

Re: [Maria-developers] 784cc5970dd: MDEV-19650: Privilege bug on MariaDB 10.4
by Sergei Golubchik 21 Apr '20

21 Apr '20

Hi, Oleksandr! On Apr 21, Oleksandr Byelkin wrote: > > > OK, INSERT probably is not needed, but we have tons (more than 40 I think) > of DELETE in our tests (and so UPDATE should work). Yes, the view is delete-able and some columns are update-able. So, let's keep these privileges. INSERT and LOAD never worked. > Also we have not LOAD but this: > > rpl.rpl_current_user 'stmt' w7 [ fail ] > Test ended at 2020-04-21 09:34:44 > > CURRENT_TEST: rpl.rpl_current_user > mysqltest: In included file "./include/diff_tables.inc": > included from > /home/sanja/maria/git/10.4/mysql-test/suite/rpl/t/rpl_current_user.test at > line 65: > At line 170: query 'SELECT * INTO OUTFILE '$_dt_outfile' FROM > $_dt_database.$_dt_table ORDER BY `$_dt_column_list`' failed: 1356: View > 'test.v_user' references invalid table(s) or column(s) or function(s) or > definer/invoker of view lack rights to use them This must be a bug. If one does SELECT * INTO OUTFILE FROM some.view than the current user should have FILE, not the view definer. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 784cc5970dd: MDEV-19650: Privilege bug on MariaDB 10.4
by Sergei Golubchik 20 Apr '20

20 Apr '20

Hi, Oleksandr! On Apr 20, Oleksandr Byelkin wrote: > > > > > + > > > > > +CREATE TEMPORARY TABLE tmp_user_sys LIKE global_priv; > > > > > +INSERT INTO tmp_user_sys (Host,User,Priv) VALUES ('localhost','mariadb.sys','{"access":512,"plugin":"mysql_native_password","authentication_string":"","account_locked":true,"password_last_changed":0}'); > > > > > +INSERT INTO global_priv SELECT * FROM tmp_user_sys WHERE NOT @had_sys_user; > > > > > +DROP TABLE tmp_user_sys; > > > > > > > > 1. This could've been simply INSERT IGNORE, I suspect > > > > > > Nope, the idea is do not insert more than needed. > > > > Why would INSERT IGNORE insert more than needed? > > I thought you propose insert ignore instead of where clause, in any case I > do not see why it should help. I mean, instead of four lines above, I think, one can simply do INSERT IGNORE global_priv (Host,User,Priv) VALUES ('localhost','mariadb.sys','{"access":512,"plugin":"mysql_native_password","authentication_string":"","account_locked":true,"password_last_changed":0}'); > > > 2. why access:512 ? It's FILE_ACL, iirc. > > > > > > Because LOAD used in tests and so in reality probably, so we need > > FILE_ACL > > > > We need FILE_ACL, but why mariadb.sys account needs it? > > mysql.user is a read-only view of the mysql.global_priv table, > > its owner doesn't need FILE or INSERT/UPDATE/DELETE. > > As I told we somewhere in test suite (not main, but I don't remember > exactly) test which require the permission otherwise I would not return and > add more rights. Where? The view is clearly not insertable-into, so INSERT privilege is meaningless. Some fields are updateable, and one can delele from it. This was more accidental than intentional, but ok, let's not change it in 10.4. And why does it need FILE privilege? The view owner cannot possibly do any LOAD DATA. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 784cc5970dd: MDEV-19650: Privilege bug on MariaDB 10.4
by Sergei Golubchik 20 Apr '20

20 Apr '20

Hi, Oleksandr! On Apr 20, Oleksandr Byelkin wrote: > > Hi, Oleksandr! > > > > On Apr 20, Oleksandr Byelkin wrote: > > > revision-id: 784cc5970dd (mariadb-10.4.11-68-g784cc5970dd) > > > parent(s): c5e00fea102 > > > author: Oleksandr Byelkin <sanja(a)mariadb.com> > > > committer: Oleksandr Byelkin <sanja(a)mariadb.com> > > > timestamp: 2020-02-20 14:06:09 +0100 > > > message: > > > > > > MDEV-19650: Privilege bug on MariaDB 10.4 > > > > > > diff --git a/scripts/mysql_system_tables.sql > > > index 29f2a4c1ef6..af852444d0c 100644 > > > --- a/scripts/mysql_system_tables.sql > > > +++ b/scripts/mysql_system_tables.sql > > > @@ -33,9 +33,17 @@ CREATE TABLE IF NOT EXISTS db ( Host char(60) > > binary DEFAULT '' NOT NULL, Db c > > > -- Remember for later if db table already existed > > > set @had_db_table= @@warning_count != 0; > > > > > > -CREATE TABLE IF NOT EXISTS global_priv (Host char(60) binary DEFAULT > > '', User char(80) binary DEFAULT '', Priv JSON NOT NULL DEFAULT '{}' > > CHECK(JSON_VALID(Priv)), PRIMARY KEY Host (Host,User)) engine=Aria > > transactional=1 CHARACTER SET utf8 COLLATE utf8_bin comment='Users and > > global privileges'; > > > +CREATE TABLE IF NOT EXISTS global_priv (Host char(60) binary DEFAULT > > '', User char(80) binary DEFAULT '', Priv JSON NOT NULL DEFAULT '{}' > > CHECK(JSON_VALID(Priv)), PRIMARY KEY (Host,User)) engine=Aria > > transactional=1 CHARACTER SET utf8 COLLATE utf8_bin comment='Users and > > global privileges'; > > > > > > -CREATE DEFINER=root@localhost SQL SECURITY DEFINER VIEW IF NOT EXISTS > > user AS SELECT > > > +set @had_sys_user= @@warning_count != 0 OR 0 <> (select count(*) from > > mysql.global_priv where Host="localhost" and User="mariadb.sys"); > > > + > > > +CREATE TEMPORARY TABLE tmp_user_sys LIKE global_priv; > > > +INSERT INTO tmp_user_sys (Host,User,Priv) VALUES > > ('localhost','mariadb.sys','{"access":512,"plugin":"mysql_native_password","authentication_string":"","account_locked":true,"password_last_changed":0}'); > > > +INSERT INTO global_priv SELECT * FROM tmp_user_sys WHERE NOT > > @had_sys_user; > > > +DROP TABLE tmp_user_sys; > > > > 1. This could've been simply INSERT IGNORE, I suspect > > Nope, the idea is do not insert more than needed. Why would INSERT IGNORE insert more than needed? > 2. why access:512 ? It's FILE_ACL, iirc. > > Because LOAD used in tests and so in reality probably, so we need FILE_ACL We need FILE_ACL, but why mariadb.sys account needs it? mysql.user is a read-only view of the mysql.global_priv table, its owner doesn't need FILE or INSERT/UPDATE/DELETE. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 784cc5970dd: MDEV-19650: Privilege bug on MariaDB 10.4
by Sergei Golubchik 20 Apr '20

20 Apr '20

Hi, Oleksandr! On Apr 20, Oleksandr Byelkin wrote: > revision-id: 784cc5970dd (mariadb-10.4.11-68-g784cc5970dd) > parent(s): c5e00fea102 > author: Oleksandr Byelkin <sanja(a)mariadb.com> > committer: Oleksandr Byelkin <sanja(a)mariadb.com> > timestamp: 2020-02-20 14:06:09 +0100 > message: > > MDEV-19650: Privilege bug on MariaDB 10.4 > > Also fixes: > MDEV-21487: Implement option for mysql_upgrade that allows root@localhost to be replaced > MDEV-21486: Implement option for mysql_install_db that allows root@localhost to be replaced > > Add user mariadb.sys to be definer of user view > (and has right on underlying table global_priv for > required operation over global_priv > (SELECT,UPDATE,DELETE,INSERT,FILE)) > > Also changed definer of gis functions in case of creation, > but they work with any definer so upgrade script do not try > to push this change. > > diff --git a/scripts/mysql_system_tables.sql b/scripts/mysql_system_tables.sql > index 29f2a4c1ef6..af852444d0c 100644 > --- a/scripts/mysql_system_tables.sql > +++ b/scripts/mysql_system_tables.sql > @@ -33,9 +33,17 @@ CREATE TABLE IF NOT EXISTS db ( Host char(60) binary DEFAULT '' NOT NULL, Db c > -- Remember for later if db table already existed > set @had_db_table= @@warning_count != 0; > > -CREATE TABLE IF NOT EXISTS global_priv (Host char(60) binary DEFAULT '', User char(80) binary DEFAULT '', Priv JSON NOT NULL DEFAULT '{}' CHECK(JSON_VALID(Priv)), PRIMARY KEY Host (Host,User)) engine=Aria transactional=1 CHARACTER SET utf8 COLLATE utf8_bin comment='Users and global privileges'; > +CREATE TABLE IF NOT EXISTS global_priv (Host char(60) binary DEFAULT '', User char(80) binary DEFAULT '', Priv JSON NOT NULL DEFAULT '{}' CHECK(JSON_VALID(Priv)), PRIMARY KEY (Host,User)) engine=Aria transactional=1 CHARACTER SET utf8 COLLATE utf8_bin comment='Users and global privileges'; > > -CREATE DEFINER=root@localhost SQL SECURITY DEFINER VIEW IF NOT EXISTS user AS SELECT > +set @had_sys_user= @@warning_count != 0 OR 0 <> (select count(*) from mysql.global_priv where Host="localhost" and User="mariadb.sys"); > + > +CREATE TEMPORARY TABLE tmp_user_sys LIKE global_priv; > +INSERT INTO tmp_user_sys (Host,User,Priv) VALUES ('localhost','mariadb.sys','{"access":512,"plugin":"mysql_native_password","authentication_string":"","account_locked":true,"password_last_changed":0}'); > +INSERT INTO global_priv SELECT * FROM tmp_user_sys WHERE NOT @had_sys_user; > +DROP TABLE tmp_user_sys; 1. This could've been simply INSERT IGNORE, I suspect 2. why access:512 ? It's FILE_ACL, iirc. > + > +CREATE DEFINER='mariadb.sys'@'localhost' SQL SECURITY DEFINER VIEW IF NOT EXISTS user AS SELECT > Host, > User, > IF(JSON_VALUE(Priv, '$.plugin') IN ('mysql_native_password', 'mysql_old_password'), IFNULL(JSON_VALUE(Priv, '$.authentication_string'), ''), '') AS Password, > @@ -101,6 +109,11 @@ CREATE TABLE IF NOT EXISTS servers ( Server_name char(64) NOT NULL DEFAULT '', H > > CREATE TABLE IF NOT EXISTS tables_priv ( Host char(60) binary DEFAULT '' NOT NULL, Db char(64) binary DEFAULT '' NOT NULL, User char(80) binary DEFAULT '' NOT NULL, Table_name char(64) binary DEFAULT '' NOT NULL, Grantor char(141) DEFAULT '' NOT NULL, Timestamp timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, Table_priv set('Select','Insert','Update','Delete','Create','Drop','Grant','References','Index','Alter','Create View','Show view','Trigger','Delete versioning rows') COLLATE utf8_general_ci DEFAULT '' NOT NULL, Column_priv set('Select','Insert','Update','References') COLLATE utf8_general_ci DEFAULT '' NOT NULL, PRIMARY KEY (Host,Db,User,Table_name), KEY Grantor (Grantor) ) engine=Aria transactional=1 CHARACTER SET utf8 COLLATE utf8_bin comment='Table privileges'; > > +CREATE TEMPORARY TABLE tmp_user_sys LIKE tables_priv; > +INSERT INTO tmp_user_sys (Host,Db,User,Table_name,Grantor,Timestamp,Table_priv) VALUES ('localhost','mysql','mariadb.sys','global_priv','root@localhost','0','Select,Insert,Update,Delete'); why Insert,Update,Delete ? > +INSERT INTO tables_priv SELECT * FROM tmp_user_sys WHERE NOT @had_sys_user; > +DROP TABLE tmp_user_sys; > + > CREATE TABLE IF NOT EXISTS columns_priv ( Host char(60) binary DEFAULT '' NOT NULL, Db char(64) binary DEFAULT '' NOT NULL, User char(80) binary DEFAULT '' NOT NULL, Table_name char(64) binary DEFAULT '' NOT NULL, Column_name char(64) binary DEFAULT '' NOT NULL, Timestamp timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, Column_priv set('Select','Insert','Update','References') COLLATE utf8_general_ci DEFAULT '' NOT NULL, PRIMARY KEY (Host,Db,User,Table_name,Column_name) ) engine=Aria transactional=1 CHARACTER SET utf8 COLLATE utf8_bin comment='Column privileges'; > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] MDEV-17399 Add support for JSON_TABLE: for ORDINALITY
by Sergey Petrunia 19 Apr '20

19 Apr '20

Hi Alexey, The standard says "An ordinality column provides a sequential numbering of rows. Row numbering is 1-based." Consider this example from the standard: https://gist.github.com/spetrunia/d4d8564a3ed26148ae92035b24e1f294 SELECT jt.rowseq, jt.name, jt.zip FROM bookclub, JSON_TABLE(bookclub.jcol, "lax $" COLUMNS ( rowSeq FOR ORDINALITY, name VARCHAR(30) PATH 'lax $.Name', zip CHAR(5) PATH 'lax $.address.postalCode') ) AS jt +--------+---------+------+ | rowseq | name | zip | +--------+---------+------+ | 0 | John Sm | 1 | | 1 | Peter W | 9 | | 2 | James L | NULL | +--------+---------+------+ The numbering is 0-based, instead of 1-based. What is not entirely clear for me is when the numbering should be reset. For the above example, the SQL standard shows the resultset with rowseq 1-2-3 which hints that the numbering is "global". MySQL-8 produces "1-1-1", which looks like they reset the numbering for every scan of the JSON_TABLE output. What are you thoughts on this? If the numbering should be "global", should it be query-level global, or subselect-level global? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 1

Re: [Maria-developers] 8742d176bc2: Added support for more functions when using partitioned S3 tables
by Sergei Golubchik 18 Apr '20

18 Apr '20

Hi, Michael! On Apr 13, Michael Widenius wrote: > revision-id: 8742d176bc2 (mariadb-10.5.2-127-g8742d176bc2) > parent(s): bf32018be96 > author: Michael Widenius <monty(a)mariadb.com> > committer: Michael Widenius <monty(a)mariadb.com> > timestamp: 2020-04-09 16:41:42 +0300 > message: > > Added support for more functions when using partitioned S3 tables > > MDEV-22088 S3 partitioning support > > All ALTER PARTITION commands should now work on S3 tables except > > REBUILD PARTITION > TRUNCATE PARTITION > REORGANIZE PARTITION > > In addition, PARTIONED S3 TABLES can also be replicated. > This is achived by storing the partition tables .frm and .par file on S3 > for partitioned shared (S3) tables. > > The discovery methods are enchanced by allowing engines that supports > discovery to also support of the partitioned tables .frm and .par file > > Things in more detail > > - The .frm and .par files of partitioned tables are stored in S3 and kept > in sync. > - Added hton callback create_partitioning_metadata to inform handler > that metadata for a partitoned file has changed > - Added back handler::discover_check_version() to be able to check if > a table's or a part table's definition has changed. > - Added handler::check_if_updates_are_ignored(). Needed for partitioning. > - Renamed rebind() -> rebind_psi(), as it was before. > - Changed CHF_xxx hadnler flags to an enum > - Changed some checks from using table->file->ht to use > table->file->partition_ht() to get discovery to work with partitioning. > - If TABLE_SHARE::init_from_binary_frm_image() fails, ensure that we > don't leave any .frm or .par files around. > - Fixed that writefrm() doesn't leave unusable .frm files around > - Appended extension to path for writefrm() to be able to reuse to function > for creating .par files. > - Added DBUG_PUSH("") to a a few functions that caused a lot of not > critical tracing. > > diff --git a/mysql-test/suite/s3/replication_partition.test b/mysql-test/suite/s3/replication_partition.test > new file mode 100644 > index 00000000000..8a177f8f075 > --- /dev/null > +++ b/mysql-test/suite/s3/replication_partition.test > @@ -0,0 +1,170 @@ > +--source include/have_s3.inc > +--source include/have_partition.inc > +--source include/master-slave.inc master-slave should always be included last, after all other have_xxx includes. > +--source include/have_binlog_format_mixed.inc > +--source include/have_innodb.inc > +--source include/have_sequence.inc > +--source create_database.inc > + > +connection slave; > +let $MYSQLD_DATADIR= `select @@datadir`; > +--replace_result $database database > +--eval use $database > +connection master; > + > +--echo # > +--echo # Check replication of parititioned S3 tables > +--echo # > + > +CREATE TABLE t1 ( > + c1 INT DEFAULT NULL > +) ENGINE=Aria > + PARTITION BY HASH (c1) > + PARTITIONS 3; > +INSERT INTO t1 VALUE (1), (2), (101), (102), (201), (202); > +ALTER TABLE t1 ENGINE=S3; > +ALTER TABLE t1 ADD PARTITION PARTITIONS 6; > +select sum(c1) from t1; > +ALTER TABLE t1 ADD COLUMN c INT; > +select sum(c1) from t1; > +sync_slave_with_master; > +show create table t1; > +select sum(c1) from t1; > +connection master; > +drop table t1; > + > +--echo # > +--echo # Checking that the slave is keeping in sync with changed partitions > +--echo # > + > +CREATE TABLE t1 ( > + c1 int primary key, > + c2 int DEFAULT NULL > +) ENGINE=InnoDB > + PARTITION BY RANGE (c1) > + (PARTITION p1 VALUES LESS THAN (200), > + PARTITION p2 VALUES LESS THAN (300), > + PARTITION p3 VALUES LESS THAN (400)); > +insert into t1 select seq*100,seq*100 from seq_1_to_3; > +alter table t1 engine=S3; > +show create table t1; > + > +sync_slave_with_master; > +select sum(c1) from t1; > +--file_exists $MYSQLD_DATADIR/$database/t1.frm > +--file_exists $MYSQLD_DATADIR/$database/t1.par > +stop slave; > +connection master; > +ALTER TABLE t1 ADD PARTITION (PARTITION p4 VALUES LESS THAN (500)); > +connection slave; > +show create table t1; > +select sum(c1) from t1; > +start slave; > +connection master; > +sync_slave_with_master; > +select sum(c1)+0 from t1; > +stop slave; > + > +# .frm amd .par files should not exists on the salve as it has just seen the > +# ALTER TABLE which cased the removal of the .frm and .par files. The table > +# from the above "select sum()" came from table cache and was used as it's > +# id matches the one in S3 > +--error 1 > +--file_exists $MYSQLD_DATADIR/$database/t1.frm > +--error 1 > +--file_exists $MYSQLD_DATADIR/$database/t1.par > +# Flushing the table cache will force the .frm and .par files to be > +# re-generated > +flush tables; > +select sum(c1)+0 from t1; > +--file_exists $MYSQLD_DATADIR/$database/t1.frm > +--file_exists $MYSQLD_DATADIR/$database/t1.par > + > +connection master; > +drop table t1; > +connection slave; > +--file_exists $MYSQLD_DATADIR/$database/t1.par > +--replace_result $database database > +--error ER_NO_SUCH_TABLE > +select sum(c1) from t1; > +--error 1 > +--file_exists $MYSQLD_DATADIR/$database/t1.par > +start slave; > +connection master; > + > +--echo # > +--echo # Check altering partitioned table to S3 and back > +--echo # Checks also rename partitoned table and drop partition > +--echo # > + > +CREATE TABLE t2 ( > + c1 int primary key, > + c2 int DEFAULT NULL > +) ENGINE=InnoDB > + PARTITION BY RANGE (c1) > + (PARTITION p1 VALUES LESS THAN (200), > + PARTITION p2 VALUES LESS THAN (300), > + PARTITION p3 VALUES LESS THAN (400)); > +insert into t2 select seq*100,seq*100 from seq_1_to_3; > +alter table t2 engine=S3; > +rename table t2 to t1; > +alter table t1 drop partition p1; > +sync_slave_with_master; > +select sum(c1) from t1; > +connection master; > +alter table t1 engine=innodb; > +sync_slave_with_master; > +select sum(c1) from t1; > +connection master; > +drop table t1; > + > +--echo # > +--echo # Check that slaves ignores changes to S3 tables. > +--echo # > + > +connection master; > +CREATE TABLE t1 ( > + c1 int primary key, > + c2 int DEFAULT NULL > +) ENGINE=InnoDB > + PARTITION BY RANGE (c1) > + (PARTITION p1 VALUES LESS THAN (200), > + PARTITION p2 VALUES LESS THAN (300), > + PARTITION p3 VALUES LESS THAN (400)); > +insert into t1 select seq*100,seq*100 from seq_1_to_3; > +create table t2 like t1; > +alter table t2 remove partitioning; > +insert into t2 values (450,450); > +sync_slave_with_master; > +stop slave; > +connection master; > +alter table t1 engine=s3; > +alter table t2 engine=s3; > +ALTER TABLE t1 ADD PARTITION (PARTITION p4 VALUES LESS THAN (500)); > +alter table t1 exchange partition p4 with table t2; > +select count(*) from t1; > +drop table t1,t2; > +connection slave; > +start slave; > +connection master; > +sync_slave_with_master; > +--replace_result $database database > +--error ER_NO_SUCH_TABLE > +select sum(c1) from t1; > +connection master; > + > +--echo # > +--echo # Check slave binary log > +--echo # > + > +sync_slave_with_master; > +--let $binlog_database=$database > +--source include/show_binlog_events.inc > +connection master; > + > +--echo # > +--echo # clean up > +--echo # > +--source drop_database.inc > +sync_slave_with_master; > +--source include/rpl_end.inc > diff --git a/mysys/my_symlink.c b/mysys/my_symlink.c > index cbee78a7f5c..323ae69a39c 100644 > --- a/mysys/my_symlink.c > +++ b/mysys/my_symlink.c > @@ -154,7 +154,8 @@ int my_realpath(char *to, const char *filename, myf MyFlags) > original name but will at least be able to resolve paths that starts > with '.'. > */ > - DBUG_PRINT("error",("realpath failed with errno: %d", errno)); > + if (MyFlags) > + DBUG_PRINT("error",("realpath failed with errno: %d", errno)); This is kind of against the concept of dbug. dbug's idea is that one should not edit the code every time, but put DBUG points one and enable/disable them at run-time. In cases like that you can specify precisely what you want to see in the trace with complicated command-line --debug string. But I usually just let it log everything and then use dbug/remove_function_from_trace.pl script. > my_errno=errno; > if (MyFlags & MY_WME) > my_error(EE_REALPATH, MYF(0), filename, my_errno); > diff --git a/sql/discover.cc b/sql/discover.cc > index e49a2a3b0c0..7f3fe73c155 100644 > --- a/sql/discover.cc > +++ b/sql/discover.cc > @@ -99,23 +99,23 @@ int readfrm(const char *name, const uchar **frmdata, size_t *len) > > > /* > - Write the content of a frm data pointer > - to a frm file. > + Write the content of a frm data pointer to a frm or par file. really? writefrm() writes to a .par file? may be rename it to writefile() then? > > - @param path path to table-file "db/name" > - @param frmdata frm data > - @param len length of the frmdata > + @param path full path to table-file "db/name.frm" or .par > + @param db Database name. Only used for my_error() > + @param table Table name. Only used for my_error() > + @param data data to write to file > + @param len length of the data > > @retval > 0 ok > @retval > - 2 Could not write file > + <> 0 Could not write file. In this case the file is not created > */ > > int writefrm(const char *path, const char *db, const char *table, > - bool tmp_table, const uchar *frmdata, size_t len) > + bool tmp_table, const uchar *data, size_t len) > { > - char file_name[FN_REFLEN+1]; > int error; > int create_flags= O_RDWR | O_TRUNC; > DBUG_ENTER("writefrm"); > diff --git a/sql/ha_partition.cc b/sql/ha_partition.cc > index 0ec1f2138ab..b8b5085f389 100644 > --- a/sql/ha_partition.cc > +++ b/sql/ha_partition.cc > @@ -629,7 +629,8 @@ int ha_partition::rename_table(const char *from, const char *to) > > SYNOPSIS > create_partitioning_metadata() > - name Full path of table name > + path Full path of new table name > + old_p Full path of old table name This is a rather meaningless phrase, there cannot be a "path" of a "name" May be, write "A path to the new/old frm file but without the extension" instead? > create_info Create info generated for CREATE TABLE > > RETURN VALUE > @@ -678,6 +681,19 @@ int ha_partition::create_partitioning_metadata(const char *path, > DBUG_RETURN(1); > } > } > + > + /* m_part_info is only NULL when we failed to create a partition table */ How can m_part_info be NULL here? create_handler_file() dereferences m_part_info unconditionally, if m_part_info would've been NULL it'll crash. > + if (m_part_info) > + { > + part= m_part_info->partitions.head(); > + if ((part->engine_type)->create_partitioning_metadata && > + ((part->engine_type)->create_partitioning_metadata)(path, old_path, > + action_flag)) > + { > + my_error(ER_CANT_CREATE_HANDLER_FILE, MYF(0)); > + DBUG_RETURN(1); > + } > + } > DBUG_RETURN(0); > } > > @@ -1604,6 +1620,7 @@ int ha_partition::prepare_new_partition(TABLE *tbl, > > if (!(file->ht->flags & HTON_CAN_READ_CONNECT_STRING_IN_PARTITION)) > tbl->s->connect_string= p_elem->connect_string; > + create_info->options|= HA_CREATE_TMP_ALTER; This looks wrong. The partition isn't created as a temporary file. HA_CREATE_TMP_ALTER is used for autiding and perfschema to distinguish between normal tables, temporary tables and these alter-tmp-tables. I see that you've hijacked this flag and check it inside s3, but it wasn't designed for that, so change s3 instead, not what the flag means. I don't think you need that if inside s3 at all, if hton flag says HTON_TEMPORARY_NOT_SUPPORTED, then s3 won't be asked to create a temporary table and you don't need a check for that in create. > if ((error= file->ha_create(part_name, tbl, create_info))) > { > /* > @@ -2361,14 +2379,20 @@ uint ha_partition::del_ren_table(const char *from, const char *to) > const char *to_path= NULL; > uint i; > handler **file, **abort_file; > + THD *thd= ha_thd(); > DBUG_ENTER("ha_partition::del_ren_table"); > > - if (get_from_handler_file(from, ha_thd()->mem_root, false)) > - DBUG_RETURN(TRUE); > + if (get_from_handler_file(from, thd->mem_root, false)) > + DBUG_RETURN(my_errno ? my_errno : ENOENT); > DBUG_ASSERT(m_file_buffer); > DBUG_PRINT("enter", ("from: (%s) to: (%s)", from, to ? to : "(nil)")); > name_buffer_ptr= m_name_buffer_ptr; > + > file= m_file; > + /* The command should be logged with IF EXISTS if using a shared table */ > + if (m_file[0]->ht->flags & HTON_TABLE_MAY_NOT_EXIST_ON_SLAVE) > + thd->replication_flags|= OPTION_IF_EXISTS; I don't think this and the whole thd->replication_flags is needed, because in the sql_table.cc you can check directly table->file->partition_ht()->flags && HTON_TABLE_MAY_NOT_EXIST_ON_SLAVE > + > if (to == NULL) > { > /* > @@ -2424,7 +2453,35 @@ uint ha_partition::del_ren_table(const char *from, const char *to) > goto rename_error; > } > } > + > + /* Update .par file in the handlers that supports it */ > + if ((*m_file)->ht->create_partitioning_metadata) > + { > + if (to == NULL) > + error= (*m_file)->ht->create_partitioning_metadata(NULL, from, > + CHF_DELETE_FLAG); > + else > + error= (*m_file)->ht->create_partitioning_metadata(to, from, > + CHF_RENAME_FLAG); why not just error= (*m_file)->ht->create_partitioning_metadata(to, from); the engine can distingush CREATE from RENAME from DELETE by looking at whether 'to' or `from` is NULL. > + DBUG_EXECUTE_IF("failed_create_partitioning_metadata", > + { my_message_sql(ER_OUT_OF_RESOURCES,"Simulated crash",MYF(0)); > + error= 1; > + }); > + if (error) > + { > + if (to) > + { > + (void) handler::rename_table(to, from); > + (void) (*m_file)->ht->create_partitioning_metadata(from, to, > + CHF_RENAME_FLAG); > + goto rename_error; > + } > + else > + save_error=error; > + } > + } > DBUG_RETURN(save_error); > + > rename_error: > name_buffer_ptr= m_name_buffer_ptr; > for (abort_file= file, file= m_file; file < abort_file; file++) > @@ -3729,6 +3787,16 @@ int ha_partition::rebind() > #endif /* HAVE_M_PSI_PER_PARTITION */ > > > +/* > + Check if the table definition has changed for the part tables > + We use the first partition for the check. > +*/ > + > +int ha_partition::discover_check_version() > +{ > + return m_file[0]->discover_check_version(); > +} 1. generally, you have to do it for every open partition, not just for the first one. 2. m_file[0] partition is not necessarily open, you need to use the first open partition here (and everywhere) > + > /** > Clone the open and locked partitioning handler. > > @@ -11382,6 +11450,12 @@ int ha_partition::end_bulk_delete() > } > > > +bool ha_partition::check_if_updates_are_ignored(const char *op) const > +{ > + return (handler::check_if_updates_are_ignored(op) || > + ha_check_if_updates_are_ignored(table->in_use, partition_ht(), op)); no you don't need this virtual method at all, simply do - if (ha_check_if_updates_are_ignored(ha_thd(), ht, "DROP")) + if (ha_check_if_updates_are_ignored(ha_thd(), partition_ht(), "DROP")) that's why partition_ht() was created in the first place. > +} > + > /** > Perform initialization for a direct update request. > > diff --git a/sql/handler.h b/sql/handler.h > index 8c45c64bec8..eadbf28229c 100644 > --- a/sql/handler.h > +++ b/sql/handler.h > @@ -3183,7 +3200,10 @@ class handler :public Sql_alloc > > public: > virtual void unbind_psi(); > - virtual int rebind(); > + virtual void rebind_psi(); this doesn't have to be virtual anymore, because you moved the check into the virtual discover_check_version() > + /* Return error if definition doesn't match for already opened table */ > + virtual int discover_check_version() { return 0; } > + > /** > Put the handler in 'batch' mode when collecting > table io instrumented events. > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > index 53ccdb5d4c3..b1756b83056 100644 > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -5906,6 +5906,8 @@ mysql_execute_command(THD *thd) > case SQLCOM_CALL: > case SQLCOM_REVOKE: > case SQLCOM_GRANT: > + if (thd->variables.option_bits & OPTION_IF_EXISTS) > + lex->create_info.set(DDL_options_st::OPT_IF_EXISTS); 1. please remove the same (now redundant) code from Sql_cmd_alter_table::execute() 2. better put it only for the relevant case. Like this case SQLCOM_ALTER_TABLE: if (thd->variables.option_bits & OPTION_IF_EXISTS) lex->create_info.set(DDL_options_st::OPT_IF_EXISTS); /* fall through */ case SQLCOM_ANALYZE_TABLE: ... > DBUG_ASSERT(lex->m_sql_cmd != NULL); > res= lex->m_sql_cmd->execute(thd); > DBUG_PRINT("result", ("res: %d killed: %d is_error: %d", > diff --git a/sql/sql_partition.cc b/sql/sql_partition.cc > index 7385f9059e1..351464939e2 100644 > --- a/sql/sql_partition.cc > +++ b/sql/sql_partition.cc > @@ -7057,6 +7059,10 @@ uint fast_alter_partition_table(THD *thd, TABLE *table, > lpt->pack_frm_data= NULL; > lpt->pack_frm_len= 0; > > + /* Add IF EXISTS to binlog if shared table */ > + if (table->file->partition_ht()->flags & HTON_TABLE_MAY_NOT_EXIST_ON_SLAVE) > + thd->variables.option_bits|= OPTION_IF_EXISTS; > + No need to, you already have that in mysql_alter_table(). Just fix the check there, like this: if (!if_exists && - (table->s->db_type()->flags & HTON_TABLE_MAY_NOT_EXIST_ON_SLAVE)) + (table->file->partition_ht()->flags & HTON_TABLE_MAY_NOT_EXIST_ON_SLAVE)) > if (table->file->alter_table_flags(alter_info->flags) & > HA_PARTITION_ONE_PHASE) > { > diff --git a/sql/sql_partition_admin.cc b/sql/sql_partition_admin.cc > index ed77c0938f3..d29c014bdc2 100644 > --- a/sql/sql_partition_admin.cc > +++ b/sql/sql_partition_admin.cc > @@ -529,14 +530,46 @@ bool Sql_cmd_alter_table_exchange_partition:: > table_list->mdl_request.set_type(MDL_SHARED_NO_WRITE); > if (unlikely(open_tables(thd, &table_list, &table_counter, 0, > &alter_prelocking_strategy))) > + { > + if (thd->lex->if_exists() && > + thd->get_stmt_da()->sql_errno() == ER_NO_SUCH_TABLE) > + { > + /* > + ALTER TABLE IF EXISTS was used on not existing table > + We have to log the query on a slave as the table may be a shared one > + from the master and we need to ensure that the next slave can see > + the statement as this slave may not have the table shared > + */ > + thd->clear_error(); > + if (thd->slave_thread && > + write_bin_log(thd, true, thd->query(), thd->query_length())) > + DBUG_RETURN(true); > + my_ok(thd); > + DBUG_RETURN(false); > + } I don't like that. There are many Sql_cmd_alter_table* classes and lots of duplicated code. Okay, you've put your OPTION_IF_EXISTS check into the big switch in the mysql_execute_command() and thus avoided duplicating that code. But this if() is also in mysql_alter_table() (and may be more?) I'm thinking that doing many Sql_cmd_alter_table* classes was a bad idea which leads to code duplication. How can we fix that? > DBUG_RETURN(true); > + } > > part_table= table_list->table; > swap_table= swap_table_list->table; > > + if (part_table->file->check_if_updates_are_ignored("ALTER")) > + { > + if (thd->slave_thread && > + write_bin_log_with_if_exists(thd, true, false, true)) > + DBUG_RETURN(true); > + my_ok(thd); > + DBUG_RETURN(false); > + } another duplicated one > + > if (unlikely(check_exchange_partition(swap_table, part_table))) > DBUG_RETURN(TRUE); > > + /* Add IF EXISTS to binlog if shared table */ > + if (part_table->file->partition_ht()->flags & > + HTON_TABLE_MAY_NOT_EXIST_ON_SLAVE) > + force_if_exists= 1; and again > + > /* set lock pruning on first table */ > partition_name= alter_info->partition_names.head(); > if (unlikely(table_list->table->part_info-> > @@ -784,16 +821,51 @@ bool Sql_cmd_alter_table_truncate_partition::execute(THD *thd) > #endif /* WITH_WSREP */ > > if (open_tables(thd, &first_table, &table_counter, 0)) > - DBUG_RETURN(true); > + { > + if (thd->lex->if_exists() && > + thd->get_stmt_da()->sql_errno() == ER_NO_SUCH_TABLE) > + { > + /* > + ALTER TABLE IF EXISTS was used on not existing table > + We have to log the query on a slave as the table may be a shared one > + from the master and we need to ensure that the next slave can see > + the statement as this slave may not have the table shared > + */ > + thd->clear_error(); > + if (thd->slave_thread && > + write_bin_log(thd, true, thd->query(), thd->query_length())) > + DBUG_RETURN(TRUE); > + my_ok(thd); > + DBUG_RETURN(FALSE); > + } > + DBUG_RETURN(TRUE); > + } see? > > - if (!first_table->table || first_table->view || > - first_table->table->s->db_type() != partition_hton) > + if (!first_table->table || first_table->view) > { > my_error(ER_PARTITION_MGMT_ON_NONPARTITIONED, MYF(0)); > DBUG_RETURN(TRUE); > } > > - > + if (first_table->table->file->check_if_updates_are_ignored("ALTER")) > + { > + if (thd->slave_thread && > + write_bin_log_with_if_exists(thd, true, false, 1)) > + DBUG_RETURN(true); > + my_ok(thd); > + DBUG_RETURN(false); > + } > + > + if (first_table->table->s->db_type() != partition_hton) > + { > + my_error(ER_PARTITION_MGMT_ON_NONPARTITIONED, MYF(0)); > + DBUG_RETURN(TRUE); > + } why did you split the check for ER_PARTITION_MGMT_ON_NONPARTITIONED? > + > + if (first_table->table->file->partition_ht()->flags & > + HTON_TABLE_MAY_NOT_EXIST_ON_SLAVE) > + force_if_exists= 1; > + > /* > Prune all, but named partitions, > to avoid excessive calls to external_lock(). > diff --git a/sql/table.cc b/sql/table.cc > index fe096835144..c6b42b7ba36 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -25,7 +25,8 @@ > // primary_key_name > #include "sql_parse.h" // free_items > #include "strfunc.h" // unhex_type2 > -#include "sql_partition.h" // mysql_unpack_partition, > +#include "ha_partition.h" // PART_EXT > + // mysql_unpack_partition, also included below > // fix_partition_func, partition_info > #include "sql_base.h" > #include "create_options.h" > @@ -42,6 +43,7 @@ > #include "rpl_filter.h" > #include "sql_cte.h" > #include "ha_sequence.h" > +#include "ha_partition.h" also included above > #include "sql_show.h" > #include "opt_trace.h" > > @@ -620,6 +622,9 @@ enum open_frm_error open_table_def(THD *thd, TABLE_SHARE *share, uint flags) > { > DBUG_ASSERT(flags & GTS_TABLE); > DBUG_ASSERT(flags & GTS_USE_DISCOVERY); > + /* Delete .frm and .par files */ > + mysql_file_delete_with_symlink(key_file_frm, path, "", MYF(0)); > + strxmov(path, share->normalized_path.str, PAR_EXT, NullS); > mysql_file_delete_with_symlink(key_file_frm, path, "", MYF(0)); no need to strxmov, this third "" argument is the file extension, you can directly do mysql_file_delete_with_symlink(key_file_frm, share->normalized_path.str, PAR_EXT, MYF(0)); also it should be key_file_partition_ddl_log not key_file_frm (it's not a ddl log, but sql_table.cc uses key_file_partition_ddl_log for par files) > file= -1; > } > @@ -1679,12 +1687,13 @@ class Field_data_type_info_array > > 42..46 are unused since 5.0 (were for RAID support) > Also, there're few unused bytes in forminfo. > - > */ > > int TABLE_SHARE::init_from_binary_frm_image(THD *thd, bool write, > const uchar *frm_image, > - size_t frm_length) > + size_t frm_length, > + const uchar *par_image, > + size_t par_length) I don't think writing .par file belongs here, why not to do it in the caller? Or, better, don't send .par file to S3 and don't discover it, it'll make everything much simpler. .par can be created from the .frm, see sql_table.cc: file= mysql_create_frm_image(thd, orig_db, orig_table_name, create_info, alter_info, create_table_mode, key_info, key_count, frm); ... if (file->ha_create_partitioning_metadata(path, NULL, CHF_CREATE_FLAG)) goto err; > { > TABLE_SHARE *share= this; > uint new_frm_ver, field_pack_length, new_field_pack_flag; > @@ -1715,24 +1724,31 @@ int TABLE_SHARE::init_from_binary_frm_image(THD *thd, bool write, > uint len; > uint ext_key_parts= 0; > plugin_ref se_plugin= 0; > - bool vers_can_native= false; > + bool vers_can_native= false, frm_created= 0; > Field_data_type_info_array field_data_type_info_array; > - > MEM_ROOT *old_root= thd->mem_root; > Virtual_column_info **table_check_constraints; > extra2_fields extra2; > - > DBUG_ENTER("TABLE_SHARE::init_from_binary_frm_image"); > > keyinfo= &first_keyinfo; > thd->mem_root= &share->mem_root; > > - if (write && write_frm_image(frm_image, frm_length)) > - goto err; > - > if (frm_length < FRM_HEADER_SIZE + FRM_FORMINFO_SIZE) > goto err; > > + if (write) > + { > +#ifdef WITH_PARTITION_STORAGE_ENGINE > + if (par_image) > + if (write_par_image(par_image, par_length)) > + goto err; > +#endif > + frm_created= 1; > + if (write_frm_image(frm_image, frm_length)) > + goto err; > + } why not to create it at the end? then you won't need to delete it on an error. > + > share->frm_version= frm_image[2]; > /* > Check if .frm file created by MySQL 5.0. In this case we want to > diff --git a/storage/maria/ha_maria.cc b/storage/maria/ha_maria.cc > index ed9c5404a7f..1f99fd9d895 100644 > --- a/storage/maria/ha_maria.cc > +++ b/storage/maria/ha_maria.cc > @@ -2724,7 +2724,7 @@ void ha_maria::drop_table(const char *name) > { > DBUG_ASSERT(file->s->temporary); > (void) ha_close(); > - (void) maria_delete_table_files(name, 1, 0); > + (void) maria_delete_table_files(name, 1, MY_WME); MYF(MY_WME) as always? > } > > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 9243921c84b: Make all #sql temporary table names uniform
by Sergei Golubchik 18 Apr '20

18 Apr '20

Hi, Michael! On Apr 13, Michael Widenius wrote: > revision-id: 9243921c84b (mariadb-10.5.2-128-g9243921c84b) > parent(s): 8742d176bc2 > author: Michael Widenius <monty(a)mariadb.com> > committer: Michael Widenius <monty(a)mariadb.com> > timestamp: 2020-04-09 22:02:06 +0300 > message: > > Make all #sql temporary table names uniform > > The reason for this is to make all temporary file names similar and > also to be able to figure out from where a #sql-xxx name orginates. > > New format is for most cases: > #sql-name-current_pid-thread_id[-increment] > Where name is one of subselect, alter, exchange, temptable or backup > > The execptions are: typo, "exceptions" > ALTER PARTITION shadow files: > #sql-shadow-'original_table_name' Please, add a thread_id here at the end. normally MDL should ensure that no two threads can have a shadow for the same table at the same time, but we have enough bugs as it is to introduce another vector when two threads can overwrite each other temp files. > Names from the temp pool: > #sql-name-current_pid-pool_number Would you mind if I drop temp pool completely in 10.5? It was added by Jeremy in January 2001 with the comment Added --temp-pool option to mysqld. This will cause temporary files created to use a small set of filenames, to try and avoid problems in the Linux kernel. And I doubt it's still an issue in 2020. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

3 4

Re: [Maria-developers] faf8de3aa98: Fixed some assert crashes related to keyread.
by Sergei Golubchik 18 Apr '20

18 Apr '20

Hi, Michael! On Apr 13, Michael Widenius wrote: > revision-id: faf8de3aa98 (mariadb-10.5.2-121-gfaf8de3aa98) > parent(s): 3cbe15bd78c > author: Michael Widenius <monty(a)mariadb.com> > committer: Michael Widenius <monty(a)mariadb.com> > timestamp: 2020-04-09 01:37:02 +0300 > message: > > Fixed some assert crashes related to keyread. > > - MDEV-22062 Assertion `!table->file->keyread_enabled()' failed in > close_thread_table() > - MDEV-22077 table->no_keyread .. failed in join_read_first() > > diff --git a/mysql-test/main/keyread.test b/mysql-test/main/keyread.test > index d9d3002d392..76d0f5674dd 100644 > --- a/mysql-test/main/keyread.test > +++ b/mysql-test/main/keyread.test > @@ -8,3 +8,14 @@ create view v1 as select * from t1 where f2 = 1; > select distinct f1 from v1; > drop view v1; > drop table t1; > + > +# > +# MDEV-22062 Assertion `!table->file->keyread_enabled()' failed in > +# close_thread_table > +# > + > +CREATE TABLE t1 (a INT NOT NULL, UNIQUE(a)) ENGINE=InnoDB; > +INSERT INTO t1 VALUES (1),(2); > +DELETE FROM t1 ORDER BY a LIMIT 1; > +SELECT * FROM t1; > +DROP TABLE t1; where's a test case for MDEV-22077? > diff --git a/sql/sql_delete.cc b/sql/sql_delete.cc > index 19eabbb053c..2fc0de4345f 100644 > --- a/sql/sql_delete.cc > +++ b/sql/sql_delete.cc > @@ -547,10 +547,12 @@ bool mysql_delete(THD *thd, TABLE_LIST *table_list, COND *conds, > else > { > ha_rows scanned_limit= query_plan.scanned_rows; > + table->no_keyread= 1; > query_plan.index= get_index_for_order(order, table, select, limit, > &scanned_limit, > &query_plan.using_filesort, > &reverse); > + table->no_keyread= 0; why? > if (!query_plan.using_filesort) > query_plan.scanned_rows= scanned_limit; > } > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index 129dae9eedb..1f05156b96f 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -23584,7 +23584,7 @@ test_if_skip_sort_order(JOIN_TAB *tab,ORDER *order,ha_rows select_limit, > If ref_key used index tree reading only ('Using index' in EXPLAIN), > and best_key doesn't, then revert the decision. > */ > - if (table->covering_keys.is_set(best_key)) > + if (table->covering_keys.is_set(best_key) && !table->no_keyread) > table->file->ha_start_keyread(best_key); > else > table->file->ha_end_keyread(); > @@ -28568,8 +28568,6 @@ test_if_cheaper_ordering(const JOIN_TAB *tab, ORDER *order, TABLE *table, > if (new_used_key_parts != NULL) > *new_used_key_parts= best_key_parts; > table->file->ha_end_keyread(); > - if (is_best_covering && !table->no_keyread) > - table->file->ha_start_keyread(best_key); I find it very confusing that test_if_cheaper_ordering has a side effect of ending keyread. It was ok, when it ended previous keyread and started a new one. But just disabling - it's looks very strange. > DBUG_RETURN(TRUE); > } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 3

Re: [Maria-developers] bc5c062b1d1: Don't try to open temporary tables if there are no temporary tables
by Sergei Golubchik 17 Apr '20

17 Apr '20

Hi, Michael! On Apr 13, Michael Widenius wrote: > revision-id: bc5c062b1d1 (mariadb-10.5.2-124-gbc5c062b1d1) > parent(s): fb29c886701 > author: Michael Widenius <monty(a)mariadb.com> > committer: Michael Widenius <monty(a)mariadb.com> > timestamp: 2020-04-09 01:37:02 +0300 > message: > > Don't try to open temporary tables if there are no temporary tables Why? > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > index 4d7a7606136..be19a2e1d82 100644 > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -3704,9 +3704,9 @@ open_and_process_table(THD *thd, TABLE_LIST *tables, uint *counter, uint flags, > The problem is that since those attributes are not set in merge > children, another round of PREPARE will not help. > */ > - error= thd->open_temporary_table(tables); > - > - if (!error && !tables->table) > + if (!thd->has_temporary_tables() || > + (!(error= thd->open_temporary_table(tables)) && > + !tables->table)) please don't write conditions like that. keep it as before, two statements: if (thd->has_temporary_tables()) error= thd->open_temporary_table(tables); if (!error && !tables->table) > error= open_table(thd, tables, ot_ctx); > > thd->pop_internal_handler(); > @@ -3723,9 +3723,9 @@ open_and_process_table(THD *thd, TABLE_LIST *tables, uint *counter, uint flags, > Repair_mrg_table_error_handler repair_mrg_table_handler; > thd->push_internal_handler(&repair_mrg_table_handler); > > - error= thd->open_temporary_table(tables); > - > - if (!error && !tables->table) > + if (!thd->has_temporary_tables() || > + (!(error= thd->open_temporary_table(tables)) && > + !tables->table)) > error= open_table(thd, tables, ot_ctx); > > thd->pop_internal_handler(); > @@ -3740,7 +3740,8 @@ open_and_process_table(THD *thd, TABLE_LIST *tables, uint *counter, uint flags, > still might need to look for a temporary table if this table > list element corresponds to underlying table of a merge table. > */ > - error= thd->open_temporary_table(tables); > + if (thd->has_temporary_tables()) > + error= thd->open_temporary_table(tables); > } > > if (!error && !tables->table) > diff --git a/sql/temporary_tables.cc b/sql/temporary_tables.cc > index 407072a7b49..553460729a1 100644 > --- a/sql/temporary_tables.cc > +++ b/sql/temporary_tables.cc > @@ -338,9 +338,9 @@ bool THD::open_temporary_table(TABLE_LIST *tl) > have invalid db or table name. > Instead THD::open_tables() should be used. > */ > - DBUG_ASSERT(!tl->derived && !tl->schema_table); > + DBUG_ASSERT(!tl->derived && !tl->schema_table && has_temporary_tables()); please, never do asserts with &&-ed conditions, this should be DBUG_ASSERT(!tl->derived); DBUG_ASSERT(!tl->schema_table); DBUG_ASSERT(has_temporary_tables()); > - if (tl->open_type == OT_BASE_ONLY || !has_temporary_tables()) > + if (tl->open_type == OT_BASE_ONLY) > { > DBUG_PRINT("info", ("skip_temporary is set or no temporary tables")); > DBUG_RETURN(false); Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 2

Re: [Maria-developers] bf32018be96: Added support for VISIBLE attribute for indexes in CREATE TABLE
by Sergei Golubchik 17 Apr '20

17 Apr '20

Hi, Michael! On Apr 13, Michael Widenius wrote: > revision-id: bf32018be96 (mariadb-10.5.2-126-gbf32018be96) > parent(s): 62c2d0f3e1f > author: Michael Widenius <monty(a)mariadb.com> > committer: Michael Widenius <monty(a)mariadb.com> > timestamp: 2020-04-09 01:37:02 +0300 > message: > > Added support for VISIBLE attribute for indexes in CREATE TABLE > > MDEV-22199 Add VISIBLE attribute for indexes in CREATE TABLE > > This was done to make it easier to read in dumps from MySQL 8.0 This is not a "dump from MySQL 8.0". It is an SQL export file generated by MySQL Workbench. And MySQL Workbench has a configuration option whether to put this VISIBLE in the sql file or not. This is a client tool misconfiguration issue. I think the user can trivially tell his tool not to generate MariaDB-incompatible syntax and we don't have to change the server for this. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 2

Re: [Maria-developers] fb29c886701: Handle errors from external_unlock & mysql_unlock_tables
by Sergei Golubchik 16 Apr '20

16 Apr '20

Hi, Michael! On Apr 13, Michael Widenius wrote: > revision-id: fb29c886701 (mariadb-10.5.2-123-gfb29c886701) > parent(s): 22fb7f8995c > author: Michael Widenius <monty(a)mariadb.com> > committer: Michael Widenius <monty(a)mariadb.com> > timestamp: 2020-04-09 01:37:02 +0300 > message: > > Handle errors from external_unlock & mysql_unlock_tables > > Other things: > - Handler errors from ha_maria::implict_commit > - Disable DBUG in safe_mutex_lock to get trace file easier to read > > diff --git a/mysys/thr_mutex.c b/mysys/thr_mutex.c > index 4f495048f63..f32132136b8 100644 > --- a/mysys/thr_mutex.c > +++ b/mysys/thr_mutex.c > @@ -233,6 +233,7 @@ int safe_mutex_lock(safe_mutex_t *mp, myf my_flags, const char *file, > int error; > DBUG_PRINT("mutex", ("%s (0x%lx) locking", mp->name ? mp->name : "Null", > (ulong) mp)); > + DBUG_PUSH(""); > > pthread_mutex_lock(&mp->global); > if (!mp->file) > @@ -283,7 +284,7 @@ int safe_mutex_lock(safe_mutex_t *mp, myf my_flags, const char *file, > { > error= pthread_mutex_trylock(&mp->mutex); > if (error == EBUSY) > - return error; > + goto end; > } > else > error= pthread_mutex_lock(&mp->mutex); > @@ -393,6 +394,8 @@ int safe_mutex_lock(safe_mutex_t *mp, myf my_flags, const char *file, > } > } > > +end: > + DBUG_POP(); > DBUG_PRINT("mutex", ("%s (0x%lx) locked", mp->name, (ulong) mp)); this message is no longer always correct, is it? > return error; > } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 3cbe15bd78c: Fixed core dump in alter table if ADD PARTITION fails
by Sergei Golubchik 16 Apr '20

16 Apr '20

Hi, Michael! On Apr 13, Michael Widenius wrote: > revision-id: 3cbe15bd78c (mariadb-10.5.2-120-g3cbe15bd78c) > parent(s): d4d332d196d > author: Michael Widenius <monty(a)mariadb.com> > committer: Michael Widenius <monty(a)mariadb.com> > timestamp: 2020-04-09 01:37:01 +0300 > message: > > Fixed core dump in alter table if ADD PARTITION fails > > I didn't add a test case as to reproduce this we need to > have a failed write in on of the engines. Bug found and bug fix > verified while debugging S3 and partitions. > > --- > sql/sql_partition.cc | 3 ++- > 1 file changed, 2 insertions(+), 1 deletion(-) > > diff --git a/sql/sql_partition.cc b/sql/sql_partition.cc > index ef8ef5114a8..4e984fa775d 100644 > --- a/sql/sql_partition.cc > +++ b/sql/sql_partition.cc > @@ -6817,7 +6817,8 @@ static void handle_alter_part_error(ALTER_PARTITION_PARAM_TYPE *lpt, > DBUG_ENTER("handle_alter_part_error"); > DBUG_ASSERT(table->m_needs_reopen); > > - if (close_table) > + /* The table may not be open if ha_partition::change_partitions() failed */ > + if (close_table && !table->file->is_open()) This looks *very* confusing, like you close the table only if it's *not* open. But then the whole if() is very confusing, as it closes in both branches, so the meaning of "close_table" is totally not clear. May be instead of this your fix you'd better cherry-pick 9c02b7d6670b069866 from MySQL? It removes close_table completely, so I expect it to work for S3 just as you wanted. > { > /* > All instances of this table needs to be closed. > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 62c2d0f3e1f: Added mariadb-config.1 to .gitignore
by Sergei Golubchik 13 Apr '20

13 Apr '20

Hi, Michael! On Apr 13, Michael Widenius wrote: > revision-id: 62c2d0f3e1f (mariadb-10.5.2-125-g62c2d0f3e1f) > parent(s): bc5c062b1d1 > author: Michael Widenius <monty(a)mariadb.com> > committer: Michael Widenius <monty(a)mariadb.com> > timestamp: 2020-04-09 01:37:02 +0300 > message: > > Added mariadb-config.1 to .gitignore Where does it come from, where is it created? I've just grepped the whole tree, I don't see it ever gets created anywhere, so why to ignore it? > diff --git a/.gitignore b/.gitignore > index 8d3b5245447..8bd46093cbe 100644 > --- a/.gitignore > +++ b/.gitignore > @@ -581,6 +581,7 @@ man/mariadb-check.1 > man/mariadb-client-test.1 > man/mariadb-client-test-embedded.1 > man/mariadb_config.1 > +man/mariadb-config.1 > man/mariadb-convert-table-format.1 > man/mariadbd.8 > man/mariadbd-multi.1 > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 22fb7f8995c: Updated client and server to use new binary names in --debug traces
by Sergei Golubchik 13 Apr '20

13 Apr '20

Hi, Michael! On Apr 13, Michael Widenius wrote: > revision-id: 22fb7f8995c (mariadb-10.5.2-122-g22fb7f8995c) > parent(s): faf8de3aa98 > author: Michael Widenius <monty(a)mariadb.com> > committer: Michael Widenius <monty(a)mariadb.com> > timestamp: 2020-04-09 01:37:02 +0300 > message: > > Updated client and server to use new binary names in --debug traces ok to push Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 9941c6a3179: MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT
by Sergei Golubchik 13 Apr '20

13 Apr '20

Hi, Aleksey! On Apr 07, Aleksey Midenkov wrote: > revision-id: 9941c6a3179 (mariadb-10.5.2-164-g9941c6a3179) > parent(s): 920c3c6b237 > author: Aleksey Midenkov <midenok(a)gmail.com> > committer: Aleksey Midenkov <midenok(a)gmail.com> > timestamp: 2020-04-06 08:05:43 +0300 > message: > > MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT See some comment below, please > diff --git a/mysql-test/suite/versioning/common.inc b/mysql-test/suite/versioning/common.inc > index 355b571e5a0..b35a5138015 100644 > --- a/mysql-test/suite/versioning/common.inc > +++ b/mysql-test/suite/versioning/common.inc > @@ -6,6 +6,7 @@ if (!$TEST_VERSIONING_SO) > source include/have_innodb.inc; > > set @@session.time_zone='+00:00'; > +set @@global.time_zone='+00:00'; Why is that? I understand you might've needed it when a partition was added in a separate thread, but why now? > select ifnull(max(transaction_id), 0) into @start_trx_id from mysql.transaction_registry; > set @test_start=now(6); > > diff --git a/mysql-test/suite/versioning/r/delete_history.result b/mysql-test/suite/versioning/r/delete_history.result > index cb865a835b3..2e4a2bf9974 100644 > --- a/mysql-test/suite/versioning/r/delete_history.result > +++ b/mysql-test/suite/versioning/r/delete_history.result > @@ -154,3 +154,18 @@ select * from t1; > a > 1 > drop table t1; > +# > +# MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT > +# > +# Don't auto-create new partition on DELETE HISTORY: > +create or replace table t (a int) with system versioning > +partition by system_time limit 1000 auto_increment; this looks like a hack, I think we need a dedicated syntax for that. but I couldn't think of anything good now. ok, I see that you allow both AUTO_INCREMENT and AUTO. May be better just to use AUTO? > +delete history from t; > +show create table t; > +Table Create Table > +t CREATE TABLE `t` ( > + `a` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME LIMIT 1000 AUTO_INCREMENT > +PARTITIONS 2 > +drop table t; > diff --git a/mysql-test/suite/versioning/r/partition.result b/mysql-test/suite/versioning/r/partition.result > index a7047cbd11b..660d2c81961 100644 > --- a/mysql-test/suite/versioning/r/partition.result > +++ b/mysql-test/suite/versioning/r/partition.result > @@ -289,11 +291,27 @@ x > 6 > 7 > 8 > -## rotation by INTERVAL > +# Auto-create history partitions > +create or replace table t1 (x int) with system versioning > +partition by system_time limit 1000 auto_increment; > +lock tables t1 write; > +insert into t1 values (1); > +update t1 set x= x + 1; > +unlock tables; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME LIMIT 1000 AUTO_INCREMENT > +PARTITIONS 3 this test needs a comment, I don't understand what's happening here. why lock tables? > +# > +# Rotation by INTERVAL > +# > create or replace table t1 (x int) > with system versioning > partition by system_time interval 0 second partitions 3; > -ERROR HY000: Wrong parameters for partitioned `t1`: wrong value for 'INTERVAL' > +ERROR HY000: Wrong parameters for partitioned `t1`: wrong value for INTERVAL > create table t1 (i int) with system versioning > partition by system_time interval 6 day limit 98; > ERROR 42000: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'limit 98' at line 2 > @@ -302,7 +320,7 @@ partition by system_time interval 10 year partitions 3; > ERROR 22003: TIMESTAMP value is out of range in 'INTERVAL' > # INTERVAL and ALTER TABLE > create or replace table t1 (i int) with system versioning > -partition by system_time interval 1 hour; > +partition by system_time interval 59 minute; why? > set @ts=(select partition_description from information_schema.partitions > where table_schema='test' and table_name='t1' and partition_name='p0'); > alter table t1 add column b int; > @@ -353,28 +371,51 @@ Warning 4114 Versioned table `test`.`t1`: last HISTORY partition (`p1`) is out o > delete from t1; > Warnings: > Warning 4114 Versioned table `test`.`t1`: last HISTORY partition (`p1`) is out of INTERVAL, need more HISTORY partitions > -select subpartition_name,partition_description,table_rows from information_schema.partitions where table_schema='test' and table_name='t1'; > +select subpartition_name,partition_description from information_schema.partitions where table_schema='test' and table_name='t1'; why? > -subpartition_name partition_description table_rows > -p1sp0 2001-02-04 00:00:00 1 > -p1sp1 2001-02-04 00:00:00 1 > -pnsp0 CURRENT 0 > -pnsp1 CURRENT 0 > +subpartition_name partition_description > +p1sp0 2001-02-04 00:00:00 > +p1sp1 2001-02-04 00:00:00 > +pnsp0 CURRENT > +pnsp1 CURRENT > +select * from t1 partition (p1); > +i > +1 > +2 > set timestamp=unix_timestamp('2001-02-04 10:20:55'); > alter table t1 add partition (partition p0 history, partition p2 history); > set timestamp=unix_timestamp('2001-02-04 10:30:00'); > insert t1 values (4),(5); > set timestamp=unix_timestamp('2001-02-04 10:30:10'); > update t1 set i=6 where i=5; > -select subpartition_name,partition_description,table_rows from information_schema.partitions where table_schema='test' and table_name='t1'; > -subpartition_name partition_description table_rows > -p1sp0 2001-02-04 00:00:00 1 > -p1sp1 2001-02-04 00:00:00 0 > -p0sp0 2001-02-05 00:00:00 1 > -p0sp1 2001-02-05 00:00:00 1 > -p2sp0 2001-02-06 00:00:00 0 > -p2sp1 2001-02-06 00:00:00 0 > -pnsp0 CURRENT 0 > -pnsp1 CURRENT 2 > +select subpartition_name, partition_description from information_schema.partitions where table_schema='test' and table_name='t1'; > +subpartition_name partition_description > +p1sp0 2001-02-04 00:00:00 > +p1sp1 2001-02-04 00:00:00 > +p0sp0 2001-02-05 00:00:00 > +p0sp1 2001-02-05 00:00:00 > +p2sp0 2001-02-06 00:00:00 > +p2sp1 2001-02-06 00:00:00 > +pnsp0 CURRENT > +pnsp1 CURRENT > +select * from t1 partition (p1); > +i > +1 > +select * from t1 partition (p0); > +i > +5 > +2 > +select * from t1 partition (p2); > +i > +alter table t1 rebuild partition p0, p1, p2; > +select * from t1 partition (p1); > +i > +1 > +select * from t1 partition (p0); > +i > +5 > +2 > +select * from t1 partition (p2); > +i > ## pruning check > set @ts=(select partition_description from information_schema.partitions > where table_schema='test' and table_name='t1' and partition_name='p0' limit 1); > @@ -1044,3 +1085,568 @@ t1 CREATE TABLE `t1` ( > PARTITION BY SYSTEM_TIME > PARTITIONS 8 > drop tables t1; > +# > +# MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT > +# > +create or replace table t1 (x int) with system versioning > +partition by system_time limit 999 auto_increment; > +ERROR HY000: Wrong parameters for partitioned `t1`: wrong value for LIMIT (< 1000) > +create or replace table t1 (x int) with system versioning > +partition by system_time interval 3599 second auto_increment; > +ERROR HY000: Wrong parameters for partitioned `t1`: wrong value for INTERVAL (< 1 HOUR) no arbitrary limitations, please > +create or replace table t1 (x int) with system versioning > +partition by system_time limit 1000 auto_increment; > +affected rows: 0 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME LIMIT 1000 AUTO_INCREMENT > +PARTITIONS 2 > +affected rows: 1 > +insert into t1 values (1); > +affected rows: 1 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME LIMIT 1000 AUTO_INCREMENT > +PARTITIONS 3 > +affected rows: 1 > +# Increment from 2 to 5 > +create or replace table t1 (x int) with system versioning > +partition by system_time interval 3600 second > +starts '2000-01-01 00:00:00' auto_increment; > +affected rows: 0 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 3600 SECOND STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +PARTITIONS 2 > +affected rows: 1 > +set timestamp= unix_timestamp('2000-01-01 00:00:00'); > +affected rows: 0 > +insert into t1 values (1); > +affected rows: 1 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 3600 SECOND STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +PARTITIONS 3 > +affected rows: 1 > +set timestamp= unix_timestamp('2000-01-01 01:00:00'); > +affected rows: 0 > +update t1 set x= x + 1; > +affected rows: 1 > +info: Rows matched: 1 Changed: 1 Inserted: 1 Warnings: 0 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 3600 SECOND STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +PARTITIONS 4 > +affected rows: 1 > +set timestamp= unix_timestamp('2000-01-01 02:00:00'); > +affected rows: 0 > +update t1 set x= x + 2; > +affected rows: 1 > +info: Rows matched: 1 Changed: 1 Inserted: 1 Warnings: 0 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 3600 SECOND STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +PARTITIONS 5 > +affected rows: 1 > +# Increment from 3 to 6, manual names, LOCK TABLES again, I think this is overcomplication and overengineering. I don't see any need for mixing automatic and manual partition names. > +set timestamp= unix_timestamp('2000-01-01 00:00:00'); > +affected rows: 0 > +create or replace table t1 (x int) with system versioning > +partition by system_time interval 1 hour auto_increment ( > +partition p1 history, > +partition p3 history, > +partition pn current); > +affected rows: 0 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +(PARTITION `p1` HISTORY ENGINE = DEFAULT_ENGINE, > + PARTITION `p3` HISTORY ENGINE = DEFAULT_ENGINE, > + PARTITION `pn` CURRENT ENGINE = DEFAULT_ENGINE) > +affected rows: 1 > +insert into t1 values (1); > +affected rows: 1 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +(PARTITION `p1` HISTORY ENGINE = DEFAULT_ENGINE, > + PARTITION `p3` HISTORY ENGINE = DEFAULT_ENGINE, > + PARTITION `pn` CURRENT ENGINE = DEFAULT_ENGINE) > +affected rows: 1 > +set timestamp= unix_timestamp('2000-01-01 01:00:00'); > +affected rows: 0 > +update t1 set x= x + 3; > +affected rows: 1 > +info: Rows matched: 1 Changed: 1 Inserted: 1 Warnings: 0 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +(PARTITION `p1` HISTORY ENGINE = DEFAULT_ENGINE, > + PARTITION `p3` HISTORY ENGINE = DEFAULT_ENGINE, > + PARTITION `p2` HISTORY ENGINE = DEFAULT_ENGINE, > + PARTITION `pn` CURRENT ENGINE = DEFAULT_ENGINE) > +affected rows: 1 > +set timestamp= unix_timestamp('2000-01-01 02:00:00'); > +affected rows: 0 > +update t1 set x= x + 4; > +affected rows: 1 > +info: Rows matched: 1 Changed: 1 Inserted: 1 Warnings: 0 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +(PARTITION `p1` HISTORY ENGINE = DEFAULT_ENGINE, > + PARTITION `p3` HISTORY ENGINE = DEFAULT_ENGINE, > + PARTITION `p2` HISTORY ENGINE = DEFAULT_ENGINE, > + PARTITION `p4` HISTORY ENGINE = DEFAULT_ENGINE, > + PARTITION `pn` CURRENT ENGINE = DEFAULT_ENGINE) > +affected rows: 1 > +lock tables t1 write; > +affected rows: 0 > +set timestamp= unix_timestamp('2000-01-01 03:00:00'); > +affected rows: 0 > +update t1 set x= x + 5; > +affected rows: 1 > +info: Rows matched: 1 Changed: 1 Inserted: 1 Warnings: 0 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +(PARTITION `p1` HISTORY ENGINE = DEFAULT_ENGINE, > + PARTITION `p3` HISTORY ENGINE = DEFAULT_ENGINE, > + PARTITION `p2` HISTORY ENGINE = DEFAULT_ENGINE, > + PARTITION `p4` HISTORY ENGINE = DEFAULT_ENGINE, > + PARTITION `p5` HISTORY ENGINE = DEFAULT_ENGINE, > + PARTITION `pn` CURRENT ENGINE = DEFAULT_ENGINE) > +affected rows: 1 > +unlock tables; > +affected rows: 0 > +# Test VIEW, LOCK TABLES > +set timestamp= unix_timestamp('2000-01-01 00:00:00'); > +affected rows: 0 > +create or replace table t1 (x int) with system versioning > +partition by system_time interval 1 hour auto_increment; > +affected rows: 0 > +create or replace view v1 as select * from t1; > +affected rows: 0 > +insert into v1 values (1); > +affected rows: 1 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +PARTITIONS 3 > +affected rows: 1 > +lock tables t1 write; > +affected rows: 0 > +set timestamp= unix_timestamp('2000-01-01 01:00:00'); > +affected rows: 0 > +update t1 set x= x + 3; > +affected rows: 1 > +info: Rows matched: 1 Changed: 1 Inserted: 1 Warnings: 0 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +PARTITIONS 4 > +affected rows: 1 > +unlock tables; > +affected rows: 0 > +drop view v1; > +affected rows: 0 > +drop tables t1; > +affected rows: 0 > +# Multiple increments in single command > +set timestamp= unix_timestamp('2000-01-01 00:00:00'); > +affected rows: 0 > +create or replace table t1 (x int) with system versioning > +partition by system_time interval 1 hour auto_increment partitions 3; > +affected rows: 0 > +create or replace table t2 (y int) with system versioning > +partition by system_time interval 1 hour auto_increment partitions 4; > +affected rows: 0 > +insert into t1 values (1); > +affected rows: 1 > +insert into t2 values (2); > +affected rows: 1 > +set timestamp= unix_timestamp('2000-01-01 01:00:00'); > +affected rows: 0 > +update t1, t2 set x= x + 1, y= y + 1; > +affected rows: 2 > +info: Rows matched: 2 Changed: 2 Warnings: 0 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +PARTITIONS 4 > +affected rows: 1 > +show create table t2; > +Table Create Table > +t2 CREATE TABLE `t2` ( > + `y` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +PARTITIONS 4 > +affected rows: 1 > +set timestamp= unix_timestamp('2000-01-01 02:00:00'); > +affected rows: 0 > +update t1, t2 set x= x + 1, y= y + 1; > +affected rows: 2 > +info: Rows matched: 2 Changed: 2 Warnings: 0 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +PARTITIONS 5 > +affected rows: 1 > +show create table t2; > +Table Create Table > +t2 CREATE TABLE `t2` ( > + `y` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +PARTITIONS 5 > +affected rows: 1 > +drop tables t1, t2; > +affected rows: 0 > +# PS, SP, LOCK TABLES > +set timestamp= unix_timestamp('2000-01-01 00:00:00'); > +affected rows: 0 > +create or replace table t1 (x int) with system versioning > +partition by system_time interval 1 hour auto_increment; > +affected rows: 0 > +execute immediate 'insert into t1 values (1)'; > +affected rows: 1 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +PARTITIONS 3 > +affected rows: 1 > +prepare s from 'update t1 set x= x + 6'; > +affected rows: 0 > +info: Statement prepared > +set timestamp= unix_timestamp('2000-01-01 01:00:00'); > +affected rows: 0 > +execute s; > +affected rows: 1 > +info: Rows matched: 1 Changed: 1 Inserted: 1 Warnings: 0 > +execute s; > +affected rows: 1 > +info: Rows matched: 1 Changed: 1 Inserted: 1 Warnings: 0 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +PARTITIONS 4 > +affected rows: 1 > +lock tables t1 write; > +affected rows: 0 > +set timestamp= unix_timestamp('2000-01-01 02:00:00'); > +affected rows: 0 > +execute s; > +affected rows: 1 > +info: Rows matched: 1 Changed: 1 Inserted: 1 Warnings: 0 > +execute s; > +affected rows: 1 > +info: Rows matched: 1 Changed: 1 Inserted: 1 Warnings: 0 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +PARTITIONS 5 add a test where timestamp is incremented by, say, 24 hours, please > +affected rows: 1 > +unlock tables; > +affected rows: 0 > +drop prepare s; > +affected rows: 0 > +create procedure sp() update t1 set x= x + 7; > +affected rows: 0 > +set timestamp= unix_timestamp('2000-01-01 03:00:00'); > +affected rows: 0 > +call sp; > +affected rows: 1 > +call sp; > +affected rows: 1 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +PARTITIONS 6 > +affected rows: 1 > +lock tables t1 write; > +affected rows: 0 > +set timestamp= unix_timestamp('2000-01-01 04:00:00'); > +affected rows: 0 > +call sp; > +affected rows: 1 > +call sp; > +affected rows: 1 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +PARTITIONS 7 > +affected rows: 1 > +unlock tables; > +affected rows: 0 > +drop procedure sp; > +affected rows: 0 > +# Complex table > +set timestamp= unix_timestamp('2000-01-01 00:00:00'); > +affected rows: 0 > +create or replace table t1 ( > +x int primary key auto_increment, > +t timestamp(6) default '2001-11-11 11:11:11', > +b blob(4096) compressed null, > +c varchar(1033) character set utf8 not null, > +u int unique, > +m enum('a', 'b', 'c') not null default 'a' comment 'absolute', > +i1 tinyint, i2 smallint, i3 bigint, > +index three(i1, i2, i3), > +v1 timestamp(6) generated always as (t + interval 1 day), > +v2 timestamp(6) generated always as (t + interval 1 month) stored, > +s timestamp(6) as row start, > +e timestamp(6) as row end, > +period for system_time (s, e), > +ps date, pe date, > +period for app_time (ps, pe), > +constraint check_constr check (u > -1)) > +with system versioning default charset=ucs2 > +partition by system_time interval 1 hour auto_increment ( > +partition p2 history, > +partition pn current); > +affected rows: 0 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) NOT NULL AUTO_INCREMENT, > + `t` timestamp(6) NOT NULL DEFAULT '2001-11-11 11:11:11.000000', > + `b` blob /*!100301 COMPRESSED*/ DEFAULT NULL, > + `c` varchar(1033) CHARACTER SET utf8 NOT NULL, > + `u` int(11) DEFAULT NULL, > + `m` enum('a','b','c') NOT NULL DEFAULT 'a' COMMENT 'absolute', > + `i1` tinyint(4) DEFAULT NULL, > + `i2` smallint(6) DEFAULT NULL, > + `i3` bigint(20) DEFAULT NULL, > + `v1` timestamp(6) GENERATED ALWAYS AS (`t` + interval 1 day) VIRTUAL, > + `v2` timestamp(6) GENERATED ALWAYS AS (`t` + interval 1 month) STORED, > + `s` timestamp(6) GENERATED ALWAYS AS ROW START, > + `e` timestamp(6) GENERATED ALWAYS AS ROW END, > + `ps` date NOT NULL, > + `pe` date NOT NULL, > + PRIMARY KEY (`x`,`e`), > + UNIQUE KEY `u` (`u`,`e`), > + KEY `three` (`i1`,`i2`,`i3`), > + PERIOD FOR SYSTEM_TIME (`s`, `e`), > + PERIOD FOR `app_time` (`ps`, `pe`), > + CONSTRAINT `check_constr` CHECK (`u` > -1) > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=ucs2 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +(PARTITION `p2` HISTORY ENGINE = DEFAULT_ENGINE, > + PARTITION `pn` CURRENT ENGINE = DEFAULT_ENGINE) > +affected rows: 1 > +insert into t1 (x, c, u, i1, i2, i3, ps, pe) > +values (1, 'cc', 0, 1, 2, 3, '1999-01-01', '2000-01-01'); > +affected rows: 1 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) NOT NULL AUTO_INCREMENT, > + `t` timestamp(6) NOT NULL DEFAULT '2001-11-11 11:11:11.000000', > + `b` blob /*!100301 COMPRESSED*/ DEFAULT NULL, > + `c` varchar(1033) CHARACTER SET utf8 NOT NULL, > + `u` int(11) DEFAULT NULL, > + `m` enum('a','b','c') NOT NULL DEFAULT 'a' COMMENT 'absolute', > + `i1` tinyint(4) DEFAULT NULL, > + `i2` smallint(6) DEFAULT NULL, > + `i3` bigint(20) DEFAULT NULL, > + `v1` timestamp(6) GENERATED ALWAYS AS (`t` + interval 1 day) VIRTUAL, > + `v2` timestamp(6) GENERATED ALWAYS AS (`t` + interval 1 month) STORED, > + `s` timestamp(6) GENERATED ALWAYS AS ROW START, > + `e` timestamp(6) GENERATED ALWAYS AS ROW END, > + `ps` date NOT NULL, > + `pe` date NOT NULL, > + PRIMARY KEY (`x`,`e`), > + UNIQUE KEY `u` (`u`,`e`), > + KEY `three` (`i1`,`i2`,`i3`), > + PERIOD FOR SYSTEM_TIME (`s`, `e`), > + PERIOD FOR `app_time` (`ps`, `pe`), > + CONSTRAINT `check_constr` CHECK (`u` > -1) > +) ENGINE=DEFAULT_ENGINE AUTO_INCREMENT=2 DEFAULT CHARSET=ucs2 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +(PARTITION `p2` HISTORY ENGINE = DEFAULT_ENGINE, > + PARTITION `p1` HISTORY ENGINE = DEFAULT_ENGINE, > + PARTITION `pn` CURRENT ENGINE = DEFAULT_ENGINE) > +affected rows: 1 > +set timestamp= unix_timestamp('2000-01-01 01:00:00'); > +affected rows: 0 > +update t1 set x= x + 8; > +affected rows: 1 > +info: Rows matched: 1 Changed: 1 Inserted: 1 Warnings: 0 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) NOT NULL AUTO_INCREMENT, > + `t` timestamp(6) NOT NULL DEFAULT '2001-11-11 11:11:11.000000', > + `b` blob /*!100301 COMPRESSED*/ DEFAULT NULL, > + `c` varchar(1033) CHARACTER SET utf8 NOT NULL, > + `u` int(11) DEFAULT NULL, > + `m` enum('a','b','c') NOT NULL DEFAULT 'a' COMMENT 'absolute', > + `i1` tinyint(4) DEFAULT NULL, > + `i2` smallint(6) DEFAULT NULL, > + `i3` bigint(20) DEFAULT NULL, > + `v1` timestamp(6) GENERATED ALWAYS AS (`t` + interval 1 day) VIRTUAL, > + `v2` timestamp(6) GENERATED ALWAYS AS (`t` + interval 1 month) STORED, > + `s` timestamp(6) GENERATED ALWAYS AS ROW START, > + `e` timestamp(6) GENERATED ALWAYS AS ROW END, > + `ps` date NOT NULL, > + `pe` date NOT NULL, > + PRIMARY KEY (`x`,`e`), > + UNIQUE KEY `u` (`u`,`e`), > + KEY `three` (`i1`,`i2`,`i3`), > + PERIOD FOR SYSTEM_TIME (`s`, `e`), > + PERIOD FOR `app_time` (`ps`, `pe`), > + CONSTRAINT `check_constr` CHECK (`u` > -1) > +) ENGINE=DEFAULT_ENGINE AUTO_INCREMENT=10 DEFAULT CHARSET=ucs2 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +(PARTITION `p2` HISTORY ENGINE = DEFAULT_ENGINE, > + PARTITION `p1` HISTORY ENGINE = DEFAULT_ENGINE, > + PARTITION `p3` HISTORY ENGINE = DEFAULT_ENGINE, > + PARTITION `pn` CURRENT ENGINE = DEFAULT_ENGINE) > +affected rows: 1 > +# Concurrent DML > +set timestamp= unix_timestamp('2000-01-01 00:00:00'); > +create or replace table t1 (x int) with system versioning > +partition by system_time interval 1 hour auto_increment; > +insert into t1 values (1); > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +PARTITIONS 3 > +connect con8, localhost, root; > +connect con7, localhost, root; > +connect con6, localhost, root; > +connect con5, localhost, root; > +connect con4, localhost, root; > +connect con3, localhost, root; > +connect con2, localhost, root; > +connect con1, localhost, root; > +set timestamp= unix_timestamp('2000-01-01 01:00:00'); > +update t1 set x= x + 10; > +connection con2; > +set timestamp= unix_timestamp('2000-01-01 01:00:00'); > +update t1 set x= x + 20; > +connection con3; > +set timestamp= unix_timestamp('2000-01-01 01:00:00'); > +update t1 set x= x + 30; > +connection con4; > +set timestamp= unix_timestamp('2000-01-01 01:00:00'); > +update t1 set x= x + 40; > +connection con5; > +set timestamp= unix_timestamp('2000-01-01 01:00:00'); > +update t1 set x= x + 50; > +connection con6; > +set timestamp= unix_timestamp('2000-01-01 01:00:00'); > +update t1 set x= x + 60; > +connection con7; > +set timestamp= unix_timestamp('2000-01-01 01:00:00'); > +update t1 set x= x + 70; > +connection con8; > +set timestamp= unix_timestamp('2000-01-01 01:00:00'); > +update t1 set x= x + 80; > +connection con1; > +disconnect con1; > +connection con2; > +disconnect con2; > +connection con3; > +disconnect con3; > +connection con4; > +disconnect con4; > +connection con5; > +disconnect con5; > +connection con6; > +disconnect con6; > +connection con7; > +disconnect con7; > +disconnect con8; > +connection default; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +PARTITIONS 4 > +drop tables t1; > +create or replace table t1 (x int) with system versioning engine innodb > +partition by system_time interval 1 hour auto; > +start transaction; > +select * from t1; > +x > +connect con1, localhost, root; > +set lock_wait_timeout= 1; > +insert into t1 values (1); > +Warnings: > +Warning 4114 Versioned table `test`.`t1`: last HISTORY partition (`p0`) is out of INTERVAL, need more HISTORY partitions > +Error 1205 Lock wait timeout exceeded; try restarting transaction > +Warning 4171 Auto-increment history partition: alter partition table failed > +Warning 4171 Versioned table `test`.`t1`: adding HISTORY partition failed with error 0, see error log for details "error 0" is strange and "see error log for details" isn't very user-friendly, a user might not have access to the error log at all > +select * from t1; > +x > +1 I don't understand, there was an error above, why did insert succeed? > +disconnect con1; > +connection default; > +drop table t1; > diff --git a/mysql-test/suite/versioning/r/rpl.result b/mysql-test/suite/versioning/r/rpl.result > index 627f3991499..68113190889 100644 > --- a/mysql-test/suite/versioning/r/rpl.result > +++ b/mysql-test/suite/versioning/r/rpl.result > @@ -164,4 +164,65 @@ update t1 set i = 0; > connection slave; > connection master; > drop table t1; > +# > +# MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT > +# > +set timestamp= unix_timestamp('2000-01-01 00:00:00'); > +create or replace table t1 (x int) with system versioning > +partition by system_time interval 1 hour auto_increment; > +insert t1 values (); > +set timestamp= unix_timestamp('2000-01-01 01:00:00'); > +delete from t1; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +PARTITIONS 4 > +connection slave; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +PARTITIONS 4 > +connection master; > +drop table t1; > +# > +# MENT-685 DML events for auto-partitioned tables are written into binary log twice > +# the test below doesn't seem to match the description above > +create table t1 (x int) partition by hash (x); > +alter table t1 add partition partitions 1 auto_increment; > +ERROR 42000: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use > +drop table t1; > +create table t1 (x int) with system versioning > +partition by system_time limit 1000 auto > +(partition p1 history, partition pn current); > +insert into t1 values (1); > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME LIMIT 1000 AUTO_INCREMENT > +(PARTITION `p1` HISTORY ENGINE = ENGINE, > + PARTITION `p2` HISTORY ENGINE = ENGINE, > + PARTITION `pn` CURRENT ENGINE = ENGINE) > +connection slave; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME LIMIT 1000 AUTO_INCREMENT > +(PARTITION `p1` HISTORY ENGINE = ENGINE, > + PARTITION `p2` HISTORY ENGINE = ENGINE, > + PARTITION `pn` CURRENT ENGINE = ENGINE) > +select * from t1; > +x > +1 > +connection master; > +drop table t1; > include/rpl_end.inc > diff --git a/sql/handler.cc b/sql/handler.cc > index c6ecc9566d8..c86f96b9689 100644 > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -1512,7 +1512,7 @@ int ha_commit_trans(THD *thd, bool all) > DBUG_ASSERT(thd->transaction.stmt.ha_list == NULL || > trans == &thd->transaction.stmt); > > - if (thd->in_sub_stmt) > + if (thd->in_sub_stmt & ~SUB_STMT_AUTO_HIST) 1. why? That is, why do you call ha_commit_trans() when adding a partition? 2. please add a test with insert under start transaction. what should happen in this case? > { > DBUG_ASSERT(0); > /* > diff --git a/sql/partition_info.cc b/sql/partition_info.cc > index f4b7260f8b0..617c839721b 100644 > --- a/sql/partition_info.cc > +++ b/sql/partition_info.cc > @@ -848,29 +850,289 @@ void partition_info::vers_set_hist_part(THD *thd) > else > vers_info->hist_part= next; > } > + } > + else if (vers_info->interval.is_set()) > + { > + if (vers_info->hist_part->range_value <= thd->query_start()) > + { > + partition_element *next= NULL; > + bool error= true; > + List_iterator<partition_element> it(partitions); > + while (next != vers_info->hist_part) > + next= it++; > + > + while ((next= it++) != vers_info->now_part) > + { > + vers_info->hist_part= next; > + if (next->range_value > thd->query_start()) > + { > + error= false; > + break; > + } > + } > + if (error) > + my_error(WARN_VERS_PART_FULL, MYF(ME_WARNING|ME_ERROR_LOG), > + table->s->db.str, table->s->table_name.str, > + vers_info->hist_part->partition_name, "INTERVAL"); > + } > + } > + > + if (!vers_info->auto_inc || > + vers_info->hist_part->id + VERS_MIN_EMPTY < vers_info->now_part->id) > return; > + > + switch (thd->lex->sql_command) > + { > + case SQLCOM_DELETE: > + if (thd->lex->last_table()->vers_conditions.delete_history) > + break; > + /* fallthrough */ > + case SQLCOM_UPDATE: > + case SQLCOM_INSERT: > + case SQLCOM_INSERT_SELECT: > + case SQLCOM_LOAD: > + case SQLCOM_REPLACE: > + case SQLCOM_REPLACE_SELECT: > + case SQLCOM_DELETE_MULTI: > + case SQLCOM_UPDATE_MULTI: it's rather fragile to check for specific sql statements. why not to look at the table lock instead? (with a special check for delete history) Ok, it's a bug. Please add a test with multi-update, where a partitioned table is *not* updated. Like update t1, tpart set t1.x=5 where t1.y=tpart.z; here a new partition should clearly not be created. also, a simpler example (multi-update is difficult): insert t1 select * from tpart; add both, please. > + { > + TABLE_SHARE *share; > + List_iterator_fast<TABLE_SHARE> it(thd->vers_auto_part_tables); > + while ((share= it++)) > + { > + if (table->s == share) > + break; > + } > + if (share) > + break; > + /* Prevent spawning multiple instances of vers_add_auto_parts() */ > + bool altering; > + mysql_mutex_lock(&table->s->LOCK_share); > + altering= table->s->vers_auto_part; > + if (!altering) > + table->s->vers_auto_part= true; > + mysql_mutex_unlock(&table->s->LOCK_share); > + if (altering) > + break; what happens if you're altering already? logically this thread should wait. Where does it do it? > + if (thd->vers_auto_part_tables.push_back(table->s)) > + { > + my_error(ER_OUT_OF_RESOURCES, MYF(0)); > + } > + } > + default:; > } > +} > > - if (vers_info->interval.is_set()) > - { > - if (vers_info->hist_part->range_value > thd->query_start()) > - return; > > - partition_element *next= NULL; > - List_iterator<partition_element> it(partitions); > - while (next != vers_info->hist_part) > - next= it++; > +/** > + @brief Run fast_alter_partition_table() to add new history partitions > + for tables requiring them. > +*/ > +void vers_add_auto_parts(THD *thd) > +{ > + HA_CREATE_INFO create_info; > + Alter_info alter_info; > + String query; > + TABLE_LIST *table_list= NULL; > + partition_info *save_part_info= thd->work_part_info; > + Query_tables_list save_query_tables; > + Reprepare_observer *save_reprepare_observer= thd->m_reprepare_observer; > + Diagnostics_area new_stmt_da(thd->query_id, false, true); > + Diagnostics_area *save_stmt_da= thd->get_stmt_da(); > + bool save_no_write_to_binlog= thd->lex->no_write_to_binlog; > + const CSET_STRING save_query= thd->query_string; > + thd->m_reprepare_observer= NULL; > + thd->lex->reset_n_backup_query_tables_list(&save_query_tables); > + thd->in_sub_stmt|= SUB_STMT_AUTO_HIST; > + thd->lex->no_write_to_binlog= !thd->is_current_stmt_binlog_format_row(); > + TABLE_LIST *tl; > + > + DBUG_ASSERT(!thd->vers_auto_part_tables.is_empty()); > + > + for (TABLE_SHARE &share: thd->vers_auto_part_tables) > + { > + tl= (TABLE_LIST *) thd->alloc(sizeof(TABLE_LIST)); > + tl->init_one_table(&share.db, &share.table_name, NULL, TL_READ_NO_INSERT); > + tl->open_type= OT_BASE_ONLY; > + tl->i_s_requested_object= OPEN_TABLE_ONLY; > + tl->next_global= table_list; > + table_list= tl; > + } > + > + /* NB: mysql_execute_command() can be recursive because of PS/SP. > + Don't duplicate any processing including error messages. */ > + thd->vers_auto_part_tables.empty(); > + > + DBUG_ASSERT(!thd->is_error()); > + /* NB: we have to preserve m_affected_rows, m_row_count_func, m_last_insert_id, etc */ > + thd->set_stmt_da(&new_stmt_da); > + new_stmt_da.set_overwrite_status(true); > + > + DDL_options_st ddl_opts_none; > + ddl_opts_none.init(); > + if (open_and_lock_tables(thd, ddl_opts_none, table_list, false, 0)) > + goto open_err; > + > + for (tl= table_list; tl; tl= tl->next_global) > + { > + TABLE *table= tl->table; > + DBUG_ASSERT(table); > + DBUG_ASSERT(table->s->get_table_ref_type() == TABLE_REF_BASE_TABLE); > + DBUG_ASSERT(table->versioned()); > + DBUG_ASSERT(table->part_info); > + DBUG_ASSERT(table->part_info->vers_info); > + alter_info.reset(); > + alter_info.partition_flags= ALTER_PARTITION_ADD|ALTER_PARTITION_AUTO_HIST; > + create_info.init(); > + create_info.alter_info= &alter_info; > + Alter_table_ctx alter_ctx(thd, tl, 1, &table->s->db, &table->s->table_name); > + > + if (thd->mdl_context.upgrade_shared_lock(table->mdl_ticket, > + MDL_SHARED_NO_WRITE, > + thd->variables.lock_wait_timeout)) > + goto exit; > + > + create_info.db_type= table->s->db_type(); > + create_info.options|= HA_VERSIONED_TABLE; > + DBUG_ASSERT(create_info.db_type); > + > + create_info.vers_info.set_start(table->s->vers_start_field()->field_name); > + create_info.vers_info.set_end(table->s->vers_end_field()->field_name); > + > + partition_info *part_info= new partition_info(); > + if (unlikely(!part_info)) > + { > + my_error(ER_OUT_OF_RESOURCES, MYF(0)); > + goto exit; > + } > + part_info->use_default_num_partitions= false; > + part_info->use_default_num_subpartitions= false; > + part_info->num_parts= 1; > + part_info->num_subparts= table->part_info->num_subparts; > + part_info->subpart_type= table->part_info->subpart_type; > + if (unlikely(part_info->vers_init_info(thd))) > + { > + my_error(ER_OUT_OF_RESOURCES, MYF(0)); > + goto exit; > + } > + /* Choose first non-occupied name suffix */ > + uint32 suffix= table->part_info->num_parts - 1; > + DBUG_ASSERT(suffix > 0); > + char part_name[MAX_PART_NAME_SIZE + 1]; > + if (make_partition_name(part_name, suffix)) > + { > +vers_make_name_err: > + push_warning_printf(thd, Sql_condition::WARN_LEVEL_WARN, > + WARN_VERS_HIST_PART_ERROR, > + "Auto-increment history partition: " > + "name generation failed for suffix %d", suffix); > + my_error(WARN_VERS_HIST_PART_ERROR, MYF(ME_WARNING), > + table->s->db.str, table->s->table_name.str, 0); > + goto exit; > + } > + List_iterator_fast<partition_element> it(table->part_info->partitions); > + partition_element *el; > + while ((el= it++)) > + { > + if (0 == my_strcasecmp(&my_charset_latin1, el->partition_name, part_name)) > + { > + if (make_partition_name(part_name, ++suffix)) > + goto vers_make_name_err; > + it.rewind(); > + } > + } > > - while ((next= it++) != vers_info->now_part) > + // NB: set_ok_status() requires DA_EMPTY > + thd->get_stmt_da()->reset_diagnostics_area(); > + > + thd->work_part_info= part_info; > + if (part_info->set_up_defaults_for_partitioning(thd, table->file, > + NULL, suffix + 1)) > { > - vers_info->hist_part= next; > - if (next->range_value > thd->query_start()) > - return; > + push_warning(thd, Sql_condition::WARN_LEVEL_WARN, > + WARN_VERS_HIST_PART_ERROR, > + "Auto-increment history partition: " > + "setting up defaults failed"); > + goto exit; > + } > + bool partition_changed= false; > + bool fast_alter_partition= false; > + if (prep_alter_part_table(thd, table, &alter_info, &create_info, > + &partition_changed, &fast_alter_partition)) > + { > + push_warning(thd, Sql_condition::WARN_LEVEL_WARN, WARN_VERS_HIST_PART_ERROR, > + "Auto-increment history partition: " > + "alter partitition prepare failed"); > + goto exit; > + } > + if (!fast_alter_partition) > + { > + push_warning(thd, Sql_condition::WARN_LEVEL_WARN, WARN_VERS_HIST_PART_ERROR, > + "Auto-increment history partition: " > + "fast alter partitition is not possible"); > + my_error(WARN_VERS_HIST_PART_ERROR, MYF(ME_WARNING), > + table->s->db.str, table->s->table_name.str, 0); > + goto exit; > + } > + DBUG_ASSERT(partition_changed); > + if (mysql_prepare_alter_table(thd, table, &create_info, &alter_info, > + &alter_ctx)) > + { > + push_warning(thd, Sql_condition::WARN_LEVEL_WARN, WARN_VERS_HIST_PART_ERROR, > + "Auto-increment history partition: " > + "alter prepare failed"); > + goto exit; > + } > + > + // Forge query string for rpl logging > + if (!thd->lex->no_write_to_binlog) > + { > + query.set(STRING_WITH_LEN("ALTER TABLE `"), &my_charset_latin1); > + > + if (query.append(table->s->db) || > + query.append(STRING_WITH_LEN("`.`")) || > + query.append(table->s->table_name) || > + query.append("` ADD PARTITION (PARTITION `") || > + query.append(part_name) || > + query.append("` HISTORY) AUTO_INCREMENT")) > + { > + my_error(ER_OUT_OF_RESOURCES, MYF(ME_ERROR_LOG)); > + goto exit; > + } > + CSET_STRING qs(query.c_ptr(), query.length(), &my_charset_latin1); > + thd->set_query(qs); > + } > + > + if (fast_alter_partition_table(thd, table, &alter_info, &create_info, > + tl, &table->s->db, &table->s->table_name)) > + { > + push_warning(thd, Sql_condition::WARN_LEVEL_WARN, WARN_VERS_HIST_PART_ERROR, > + "Auto-increment history partition: " > + "alter partition table failed"); > + my_error(WARN_VERS_HIST_PART_ERROR, MYF(ME_WARNING), > + tl->db.str, tl->table_name.str, 0); > } > - my_error(WARN_VERS_PART_FULL, MYF(ME_WARNING|ME_ERROR_LOG), > - table->s->db.str, table->s->table_name.str, > - vers_info->hist_part->partition_name, "INTERVAL"); > } > + > + if (!thd->transaction.stmt.is_empty()) > + trans_commit_stmt(thd); > + > +exit: > + // If we failed with error allow non-processed tables to be processed next time > + if (tl) > + while ((tl= tl->next_global)) > + tl->table->s->vers_auto_part= false; > + close_thread_tables(thd); > +open_err: > + thd->work_part_info= save_part_info; > + thd->m_reprepare_observer= save_reprepare_observer; > + thd->lex->restore_backup_query_tables_list(&save_query_tables); > + thd->in_sub_stmt&= ~SUB_STMT_AUTO_HIST; > + if (!new_stmt_da.is_warning_info_empty()) > + save_stmt_da->copy_sql_conditions_from_wi(thd, new_stmt_da.get_warning_info()); > + thd->set_stmt_da(save_stmt_da); > + thd->lex->no_write_to_binlog= save_no_write_to_binlog; > + thd->set_query(save_query); > } > > > diff --git a/sql/partition_info.h b/sql/partition_info.h > index eb8e53a381a..d02eecea073 100644 > --- a/sql/partition_info.h > +++ b/sql/partition_info.h > @@ -34,10 +34,19 @@ typedef bool (*check_constants_func)(THD *thd, partition_info *part_info); > > struct st_ddl_log_memory_entry; > > + > +/* Auto-create history partition configuration */ > +static const uint VERS_MIN_EMPTY= 1; No, this doesn't work. One can easily set @@timestamp to do a multi-hour jump and no fixed value of VERS_MIN_EMPTY will help. You need to add partitions before the statement, as I wrote in my previous review. > +static const uint VERS_MIN_INTERVAL= 3600; // seconds > +static const uint VERS_MIN_LIMIT= 1000; > +static const uint VERS_ERROR_TIMEOUT= 300; // seconds > + > + > struct Vers_part_info : public Sql_alloc > { > Vers_part_info() : > limit(0), > + auto_inc(false), > now_part(NULL), > hist_part(NULL) > { > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index 7cc1faea79b..4be3342e78e 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -7521,14 +7533,19 @@ add_partition_rule: > > add_part_extra: > /* empty */ > - | '(' part_def_list ')' > + | '(' part_def_list ')' opt_vers_auto_inc > { > LEX *lex= Lex; > lex->part_info->num_parts= lex->part_info->partitions.elements; > + if ($4) > + lex->alter_info.partition_flags|= ALTER_PARTITION_AUTO_HIST; > } > - | PARTITIONS_SYM real_ulong_num > + | PARTITIONS_SYM real_ulong_num opt_vers_auto_inc > { > - Lex->part_info->num_parts= $2; > + LEX *lex= Lex; ^^^ this pattern is obsolete for, like, 10 years. It does no harm, but adds no value either. > + lex->part_info->num_parts= $2; > + if ($3) > + lex->alter_info.partition_flags|= ALTER_PARTITION_AUTO_HIST; > } > ; > > diff --git a/storage/mroonga/mrn_table.cpp b/storage/mroonga/mrn_table.cpp > index b10668cfcce..6458402f572 100644 > --- a/storage/mroonga/mrn_table.cpp > +++ b/storage/mroonga/mrn_table.cpp > @@ -932,7 +932,7 @@ MRN_SHARE *mrn_get_share(const char *table_name, TABLE *table, int *error) > share->wrap_key_info = NULL; > share->wrap_primary_key = MAX_KEY; > } > - memcpy(wrap_table_share, table->s, sizeof(*wrap_table_share)); > + memcpy((void *)wrap_table_share, (void *)table->s, sizeof(*wrap_table_share)); why is that? > mrn_init_sql_alloc(current_thd, &(wrap_table_share->mem_root)); > wrap_table_share->keys = share->wrap_keys; > wrap_table_share->key_info = share->wrap_key_info; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 3

Re: [Maria-developers] d4d332d196d: Added error message to --die in mtr
by Sergei Golubchik 13 Apr '20

13 Apr '20

Hi, Michael! On Apr 13, Michael Widenius wrote: > revision-id: d4d332d196d (mariadb-10.5.2-119-gd4d332d196d) > parent(s): c7ab676192d > author: Michael Widenius <monty(a)mariadb.com> > committer: Michael Widenius <monty(a)mariadb.com> > timestamp: 2020-04-09 01:37:01 +0300 > message: > > Added error message to --die in mtr > > --- > client/mysqltest.cc | 3 ++- > 1 file changed, 2 insertions(+), 1 deletion(-) > > diff --git a/client/mysqltest.cc b/client/mysqltest.cc > index 3e08d3f62d7..f593d8e949e 100644 > --- a/client/mysqltest.cc > +++ b/client/mysqltest.cc > @@ -9748,7 +9748,8 @@ int main(int argc, char **argv) > break; > case Q_DIE: > /* Abort test with error code and error message */ > - die("%s", command->first_argument); > + die("%s", command->first_argument[0] ? command->first_argument : > + "Explicite --die command executed"); typo. "Explicit" > break; > case Q_EXIT: > /* Stop processing any more commands */ Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] MDEV-17399 Add support for JSON_TABLE.
by Sergey Petrunia 12 Apr '20

12 Apr '20

Hi Alexey, Please find my review input below. There is one big issue and a number of smaller ones. > commit 654fdfee33e3eafe3b7f25d7e213717c22ea1e18 > Author: Alexey Botchkov <holyfoot(a)askmonty.org> > Date: Mon Mar 30 01:00:28 2020 +0400 > > MDEV-17399 Add support for JSON_TABLE. > > Syntax for JSON_TABLE added. > The ha_json_table handler added. Executing the JSON_TABLE we > create the temporary table of the ha_json_table, add dependencies > of other tables and sometimes conditions to WHERE. > == The big issue == I think some design choices of this patch are questionable: The temporary table has a unique key. What is it for? The key is defined over the field holding the JSON text. What if the JSON text gets bigger than MAX_KEY_LEN? Then, there is an Item_func_eq(json_text, temp_table_field), which is set to be "always equal" with set_always_equal(). This looks like a hack. EXPLAIN shows that JSON_TABLE uses a kind of ref access which I think is counter-intuitive. What is the goal of all this? The JSON_TABLE table is "laterally dependent" on the tables that are referred from its arguments. For the optimizer, this means: 1. The JSON_TABLE table must be put after its dependencies (this is similar to outer joins) 2. The JSON_TABLE can only be read when there are "current records" available for each of the dependencies (an example when they are not available: join buffering. There are many records in the buffer, which one we should produce matches for? For outer joins, join buffering code has a complex system to track this). The above "kludges" make an attempt to trick the optimizer into meeting requrirements #1 and #2 but I think this is approach is hack-ish and has holes. It would be much better if the optimizer was explicitly aware that a table is "laterally dependent" on other tables. "table dependencies" are already there, so the property #1 is already taken care of. We still need to take care of #2 which means disable join buffering in such cases. I think this should be easy to do. == Small bits from trying out the patch == === FOR ORDINALITY doesn't seem to work === select * from json_table('[{"a": 1, "b": [11,111]}, {"a": 2, "b": [22,222]}]', '$[*]' COLUMNS(a for ordinality) ) as tt; +------+ | a | +------+ | | | | +------+ 2 rows in set (0.001 sec) === NULL ON ERROR doesn't seem to work === select * from json_table('[{"a": 1, "b": [11,111]}, {"a": "bbbb", "b": [22,222]}]', '$[*]' COLUMNS( a DECIMAL(6,3) PATH '$.a' NULL ON ERROR)) as tt; +-------+ | a | +-------+ | 1.000 | | 0.000 | +-------+ I would expect the second row to have NULL, not 0. === Lack of test coverage === Other parts, like NESTED PATH, also have no test coverage at all. I think this must be addressed before this is done. === No error on Invalid Json input === If I pass invalid JSON, I get no warning or error or anything: MariaDB [test]> select * from json_table('[{"a": 1, invalid-json', '$[*]' COLUMNS( a INT PATH '$.a')) as tt; +------+ | a | +------+ | 1 | +------+ 1 row in set (0.001 sec) === Recursive ON-rule in the grammar === The json_opt_on_empty_or_error rule in sql_yacc.yy is recursive and causes the following to be accepted: select * from json_table( '[{"a": 1, "b": [11,111]}, {"a": 2, "b": [22,222]}]', '$[*]' COLUMNS(a INT PATH '$.a' NULL ON EMPTY NULL ON EMPTY NULL ON EMPTY) ) as tt; Is this intentional? === No error on invalid LATERAL dependency === create table t1 (item_name varchar(32), item_props varchar(1024)); insert into t1 values ('Laptop', '{"color": "black", "price": 1000}'); insert into t1 values ('Jeans', '{"color": "blue", "price": 50}'); MariaDB [test]> select * from t1 right join json_table(t1.item_props,'$' columns( color varchar(100) path '$.color')) as T on 1; Empty set (0.000 sec) The above query cannot be executed as left join execution requires that T is computed before 1, but t is dependent on t1. We dont get an error for this though. MySQL produces this: ERROR 3668 (HY000): INNER or LEFT JOIN must be used for LATERAL references made by 'T' with left join, there's a demo of how the trickery with the optimizer was successful (and I think one can construct other examples of this): MariaDB [test]> select * from t1 left join json_table(t1.item_props,'$' columns( color varchar(100) path '$.color')) as T on 1; +-----------+-----------------------------------+-------+ | item_name | item_props | color | +-----------+-----------------------------------+-------+ | Laptop | {"color": "black", "price": 1000} | blue | | Jeans | {"color": "blue", "price": 50} | blue | +-----------+-----------------------------------+-------+ 2 rows in set (0.002 sec) MariaDB [test]> explain select * from t1 left join json_table(t1.item_props,'$' columns( color varchar(100) path '$.color')) as T on 1; +------+-------------+-------+------+---------------+------+---------+------+------+-------------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +------+-------------+-------+------+---------------+------+---------+------+------+-------------------------------------------------+ | 1 | SIMPLE | t1 | ALL | NULL | NULL | NULL | NULL | 2 | | | 1 | SIMPLE | T | ALL | NULL | NULL | NULL | NULL | 40 | Using where; Using join buffer (flat, BNL join) | +------+-------------+-------+------+---------------+------+---------+------+------+-------------------------------------------------+ 2 rows in set (0.001 sec) === AS is required with table alias === sql_yacc.yy has this: > +table_function: > + JSON_TABLE_SYM '(' expr ',' TEXT_STRING_sys > + { ... > + } > + json_table_columns_clause ')' AS ident_table_alias Is the 'AS' really required? MySQL-8 doesn't seem to require it. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 1

Re: [Maria-developers] [Commits] c1394ab6b5c: MDEV-22191: Range access is not picked when index_merge_sort_union is turned off
by Sergey Petrunia 09 Apr '20

09 Apr '20

Hi Varun, On Wed, Apr 08, 2020 at 11:48:28PM +0530, Varun wrote: > revision-id: c1394ab6b5c0830ec09f6afdae11fa82bae1a123 (mariadb-5.5.67-9-gc1394ab6b5c) > parent(s): 64b70b09e6ac253b7915f6120ade5e69fa750b18 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2020-04-08 23:47:03 +0530 > message: > > MDEV-22191: Range access is not picked when index_merge_sort_union is turned off > > When index_merge_sort_union is turned off only ror scans were considered for range > scans, which is wrong. > To fix the problem ensure both ror scans and non ror scans are considered for range > access > > --- > mysql-test/r/range.result | 19 +++++++++++++++++++ > mysql-test/r/range_mrr_icp.result | 19 +++++++++++++++++++ > mysql-test/t/range.test | 14 ++++++++++++++ > sql/opt_range.cc | 16 ++++++++++------ > 4 files changed, 62 insertions(+), 6 deletions(-) ... > --- a/sql/opt_range.cc > +++ b/sql/opt_range.cc > @@ -949,7 +949,8 @@ QUICK_RANGE_SELECT *get_quick_select(PARAM *param,uint index, > static TRP_RANGE *get_key_scans_params(PARAM *param, SEL_TREE *tree, > bool index_read_must_be_used, > bool update_tbl_stats, > - double read_time); > + double read_time, > + bool ror_scans_required); > static > TRP_INDEX_INTERSECT *get_best_index_intersect(PARAM *param, SEL_TREE *tree, > double read_time); > @@ -3146,7 +3147,7 @@ int SQL_SELECT::test_quick_select(THD *thd, key_map keys_to_use, > > /* Get best 'range' plan and prepare data for making other plans */ > if ((range_trp= get_key_scans_params(&param, tree, FALSE, TRUE, > - best_read_time))) > + best_read_time, FALSE))) > { > best_trp= range_trp; > best_read_time= best_trp->read_cost; > @@ -4708,7 +4709,8 @@ TABLE_READ_PLAN *get_best_disjunct_quick(PARAM *param, SEL_IMERGE *imerge, > { > DBUG_EXECUTE("info", print_sel_tree(param, *ptree, &(*ptree)->keys_map, > "tree in SEL_IMERGE");); > - if (!(*cur_child= get_key_scans_params(param, *ptree, TRUE, FALSE, read_time))) > + if (!(*cur_child= get_key_scans_params(param, *ptree, TRUE, FALSE, > + read_time, TRUE))) > { > /* > One of index scans in this index_merge is more expensive than entire > @@ -5030,7 +5032,7 @@ TABLE_READ_PLAN *merge_same_index_scans(PARAM *param, SEL_IMERGE *imerge, > index merge retrievals are not well calibrated > */ > trp= get_key_scans_params(param, *imerge->trees, FALSE, TRUE, > - read_time); > + read_time, TRUE); > } As far as I understand, this call constructs range scan, not a portion of index_merge scan. So, it is not correct to require a ROR scan here. > > DBUG_RETURN(trp); > @@ -6747,6 +6749,7 @@ TRP_ROR_INTERSECT *get_best_covering_ror_intersect(PARAM *param, > index_read_must_be_used if TRUE, assume 'index only' option will be set > (except for clustered PK indexes) > read_time don't create read plans with cost > read_time. > + ror_scans_required set to TRUE for index merge > RETURN > Best range read plan > NULL if no plan found or error occurred > @@ -6755,7 +6758,8 @@ TRP_ROR_INTERSECT *get_best_covering_ror_intersect(PARAM *param, > static TRP_RANGE *get_key_scans_params(PARAM *param, SEL_TREE *tree, > bool index_read_must_be_used, > bool update_tbl_stats, > - double read_time) > + double read_time, > + bool ror_scans_required) > { > uint idx; > SEL_ARG **key,**end, **key_to_read= NULL; > @@ -6802,7 +6806,7 @@ static TRP_RANGE *get_key_scans_params(PARAM *param, SEL_TREE *tree, > update_tbl_stats, &mrr_flags, > &buf_size, &cost); > > - if (!param->is_ror_scan && > + if (ror_scans_required && !param->is_ror_scan && > !optimizer_flag(param->thd, OPTIMIZER_SWITCH_INDEX_MERGE_SORT_UNION)) So, the meaning if "ror_scans_required" parameter is actually: " require ror scans if index_merge optimizer settings are such that it cannot use ROR-scans" This is very complicated and non-orthogonal. How about moving the "!optimizer_flag(param->thd, OPTIMIZER_SWITCH_INDEX_MERGE_SORT_UNION)" check up into get_best_disjunct_quick()? That is, get_best_disjunct_quick() will pass ror_scans_required=true iff OPTIMIZER_SWITCH_INDEX_MERGE_SORT_UNION is not set. This way, index_merge switches will be only checked in index_merge code and ror_scans_required will mean what its name says. > { > /* The scan is not a ROR-scan, just skip it */ BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] 383e1a5b71e: MENT-703: Disable rpl.rpl_skip_replication, rpl.rpl_set_statement_default_master, rpl.rpl_extra_col_master_myisam, rpl.rpl_slave_load_tmpdir_not_exist tests in ES 10.2 and above
by sujatha 08 Apr '20

08 Apr '20

revision-id: 383e1a5b71e164f1c1b17a085ba90a8c30cbdf46 (mariadb-10.2.24-716-g383e1a5b71e) parent(s): 2969d0702d56405d1aec8c16a272ac85fef7bd61 author: Sujatha committer: Sujatha timestamp: 2020-04-08 13:31:33 +0530 message: MENT-703: Disable rpl.rpl_skip_replication, rpl.rpl_set_statement_default_master, rpl.rpl_extra_col_master_myisam, rpl.rpl_slave_load_tmpdir_not_exist tests in ES 10.2 and above Problem: ======= Disable following tests in ES 10.2 and above, till MDEV-13258 and MDEV-14203 are fixed. rpl.rpl_skip_replication, rpl.rpl_set_statement_default_master, rpl.rpl_extra_col_master_myisam, rpl.rpl_slave_load_tmpdir_not_exist --- mysql-test/suite/rpl/disabled.def | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/mysql-test/suite/rpl/disabled.def b/mysql-test/suite/rpl/disabled.def index bdefb1660bd..951f8a6aa0f 100644 --- a/mysql-test/suite/rpl/disabled.def +++ b/mysql-test/suite/rpl/disabled.def @@ -16,3 +16,7 @@ rpl_partition_archive : MDEV-5077 2013-09-27 svoj Cannot exchange partition rpl_row_binlog_max_cache_size : MDEV-11092 rpl_row_index_choice : MDEV-11666 rpl_slave_grp_exec: MDEV-10514 +rpl_skip_replication : MDEV-13258 +rpl_set_statement_default_master : MDEV-13258 +rpl_extra_col_master_myisam : MDEV-14203 +rpl_slave_load_tmpdir_not_exist : MDEV-14203

1 0

Re: [Maria-developers] 7665323d02a: MENT-202 Implement system variable that makes slow master shutdown the default behavior
by Sergei Golubchik 07 Apr '20

07 Apr '20

Hi, Andrei! ok to push. one comment below. On Apr 07, Andrei Elkin wrote: > revision-id: 7665323d02a (mariadb-10.3.22-194-g7665323d02a) > parent(s): 04e38549af8 > author: Andrei Elkin <andrei.elkin(a)mariadb.com> > committer: Andrei Elkin <andrei.elkin(a)mariadb.com> > timestamp: 2020-04-07 14:25:25 +0300 > message: > > MENT-202 Implement system variable that makes slow master shutdown the default behavior > > @@global.shutdown_wait_slaves is introduced to hold a desirable > shutdown mode that the SHUTDOWN sql command runs with implicitly > when invoked without any option. > > diff --git a/sql/sys_vars.cc b/sql/sys_vars.cc > index 8bb9f86c8b6..2b23edc7000 100644 > --- a/sql/sys_vars.cc > +++ b/sql/sys_vars.cc > @@ -4270,6 +4270,12 @@ static Sys_var_ulong Sys_profiling_history_size( > VALID_RANGE(0, 100), DEFAULT(15), BLOCK_SIZE(1)); > #endif > > +static Sys_var_mybool Sys_shutdown_wait_for_slaves( > + "shutdown_wait_for_slaves", > + "when ON, SHUTDOWN command runs with implicit WAIT FOR ALL SLAVES option.", > + GLOBAL_VAR(opt_shutdown_wait_for_slaves), CMD_LINE(OPT_ARG), > + DEFAULT(FALSE), NO_MUTEX_GUARD, NOT_IN_BINLOG, ON_CHECK(0), ON_UPDATE(0)); You don't need to write all this tail of the default behavior. The last line could be just DEFAULT(FALSE)); Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] 2888a225214: MDEV-21117: --tc-heuristic-recover=rollback is not replication safe
by sujatha 07 Apr '20

07 Apr '20

revision-id: 2888a22521475f734b28ed65d199cfe2a754aed7 (mariadb-10.1.43-100-g2888a225214) parent(s): 5720db2b43491e5aec9265bce9637e00c72fa0aa author: Sujatha committer: Sujatha timestamp: 2020-04-07 19:21:45 +0530 message: MDEV-21117: --tc-heuristic-recover=rollback is not replication safe Problem: ======= When run after master server crash --tc-heuristic-recover=rollback produces inconsistent server state with binlog still containing transactions that were rolled back by the option. Such way recovered server may not be used for replication. Fix: === During "--tc-heuristic-recover=ROLLBACK", query the storage engine to get binlog file name and position corresponding to the last committed transaction. This marks the consistent binlog state. If last_commit_file is not set then checkpoint binary log file is considered as starting point. Look for first transactional event beyond the consistent binlog state. This will be the starting point for heuristic rollback. Consider this event specific starting point as binlog truncation position. Now traverse the rest of binlogs beyond this point. During this traversal check for the presence of DDL or non transactional operations, as they cannot be safely rolled back. If such events are found the truncation will not happen it will return an error. If only transactional events are found beyond the binlog truncation position it is safe to truncate binlog. The log gets truncated to the identified position and the GTID state is adjusted accordingly. If there are more binary logs beyond the being truncated file they are all removed. --- .../r/binlog_heuristic_rollback_active_log.result | 18 + .../binlog/r/binlog_truncate_multi_log.result | 21 + .../suite/binlog/r/binlog_truncate_none.result | 42 ++ .../t/binlog_heuristic_rollback_active_log.test | 75 ++++ .../suite/binlog/t/binlog_truncate_multi_log.test | 81 ++++ .../suite/binlog/t/binlog_truncate_none.test | 130 ++++++ .../suite/rpl/r/rpl_heuristic_fail_over.result | 53 +++ .../suite/rpl/t/rpl_heuristic_fail_over.test | 160 +++++++ sql/log.cc | 479 ++++++++++++++++++++- sql/log.h | 7 +- storage/innobase/handler/ha_innodb.cc | 3 + storage/xtradb/handler/ha_innodb.cc | 3 + 12 files changed, 1068 insertions(+), 4 deletions(-) diff --git a/mysql-test/suite/binlog/r/binlog_heuristic_rollback_active_log.result b/mysql-test/suite/binlog/r/binlog_heuristic_rollback_active_log.result new file mode 100644 index 00000000000..eff95d05aac --- /dev/null +++ b/mysql-test/suite/binlog/r/binlog_heuristic_rollback_active_log.result @@ -0,0 +1,18 @@ +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); +RESET MASTER; +CREATE TABLE t ( f INT ) ENGINE=INNODB; +INSERT INTO t VALUES (10); +SET DEBUG_SYNC= "commit_before_update_end_pos SIGNAL con1_ready WAIT_FOR con1_go"; +INSERT INTO t VALUES (20); +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; +# Kill the server +"One row should be present in table 't'" +SELECT COUNT(*) FROM t; +COUNT(*) +1 +"gtid_binlog_state should be 0-1-2 +SELECT @@GLOBAL.gtid_binlog_state; +@@GLOBAL.gtid_binlog_state +0-1-2 +DROP TABLE t; diff --git a/mysql-test/suite/binlog/r/binlog_truncate_multi_log.result b/mysql-test/suite/binlog/r/binlog_truncate_multi_log.result new file mode 100644 index 00000000000..9796a480cca --- /dev/null +++ b/mysql-test/suite/binlog/r/binlog_truncate_multi_log.result @@ -0,0 +1,21 @@ +SET @old_max_binlog_size= @@global.max_binlog_size; +SET GLOBAL max_binlog_size= 4096; +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); +RESET MASTER; +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +SET DEBUG_SYNC= "commit_before_update_end_pos SIGNAL con1_ready WAIT_FOR con1_go"; +INSERT INTO t1 VALUES (2, REPEAT("x", 4100)); +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; +# Kill the server +"Zero rows shoule be present in table" +SELECT COUNT(*) FROM t1; +COUNT(*) +0 +SELECT @@GLOBAL.gtid_current_pos; +@@GLOBAL.gtid_current_pos +0-1-1 +DROP TABLE t1; +SELECT @@GLOBAL.gtid_binlog_state; +@@GLOBAL.gtid_binlog_state +0-1-2 diff --git a/mysql-test/suite/binlog/r/binlog_truncate_none.result b/mysql-test/suite/binlog/r/binlog_truncate_none.result new file mode 100644 index 00000000000..bae87985208 --- /dev/null +++ b/mysql-test/suite/binlog/r/binlog_truncate_none.result @@ -0,0 +1,42 @@ +SET @old_max_binlog_size= @@global.max_binlog_size; +SET GLOBAL max_binlog_size= 4096; +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); +RESET MASTER; +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +INSERT INTO t1 VALUES (1, REPEAT("x", 4100)); +CREATE TABLE t2 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +show binary logs; +Log_name File_size +master-bin.000001 # +master-bin.000002 # +# Kill the server +"Zero records should be there." +SELECT COUNT(*) FROM t1; +COUNT(*) +1 +show binary logs; +Log_name File_size +master-bin.000001 # +master-bin.000002 # +master-bin.000003 # +master-bin.000004 # +DROP TABLE t1,t2; +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); +RESET MASTER; +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +CREATE TABLE t2 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +show binary logs; +Log_name File_size +master-bin.000001 # +# Kill the server +"Zero records should be there." +SELECT COUNT(*) FROM t1; +COUNT(*) +0 +SHOW TABLES; +Tables_in_test +t1 +t2 +DROP TABLE t1,t2; diff --git a/mysql-test/suite/binlog/t/binlog_heuristic_rollback_active_log.test b/mysql-test/suite/binlog/t/binlog_heuristic_rollback_active_log.test new file mode 100644 index 00000000000..87af83a7a92 --- /dev/null +++ b/mysql-test/suite/binlog/t/binlog_heuristic_rollback_active_log.test @@ -0,0 +1,75 @@ +# ==== Purpose ==== +# +# Test verifies the truncation of single binary log file. +# +# ==== Implementation ==== +# +# Steps: +# 0 - Create table t1 and insert a row. +# 1 - Insert an another row such that it gets written to binlog but commit +# in engine fails as server crashed at this point. +# 2 - Restart server with --tc-heuristic-recover=ROLLBACK +# 3 - Upon server start 'master-bin.000001' will be truncated to contain +# only the first insert +# +# ==== References ==== +# +# MDEV-21117: --tc-heuristic-recover=rollback is not replication safe + + +--source include/have_innodb.inc +--source include/have_log_bin.inc +--source include/have_debug.inc +--source include/have_binlog_format_statement.inc + +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); + +connect(master,localhost,root,,); +connect(master1,localhost,root,,); + +--connection master +RESET MASTER; +CREATE TABLE t ( f INT ) ENGINE=INNODB; +INSERT INTO t VALUES (10); + +--connection master1 +# Hold insert after write to binlog and before "run_commit_ordered" in engine +SET DEBUG_SYNC= "commit_before_update_end_pos SIGNAL con1_ready WAIT_FOR con1_go"; +send INSERT INTO t VALUES (20); + +--connection master +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; + +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +wait +EOF + +--source include/kill_mysqld.inc +--source include/wait_until_disconnected.inc +# +# Server restart +# +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: --tc-heuristic-recover=ROLLBACK +EOF +--source include/wait_until_disconnected.inc + +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: +EOF + +connection default; +--enable_reconnect +--source include/wait_until_connected_again.inc + +--connection master +--enable_reconnect +--source include/wait_until_connected_again.inc + +--echo "One row should be present in table 't'" +SELECT COUNT(*) FROM t; + +--echo "gtid_binlog_state should be 0-1-2 +SELECT @@GLOBAL.gtid_binlog_state; +DROP TABLE t; diff --git a/mysql-test/suite/binlog/t/binlog_truncate_multi_log.test b/mysql-test/suite/binlog/t/binlog_truncate_multi_log.test new file mode 100644 index 00000000000..976b987d3bc --- /dev/null +++ b/mysql-test/suite/binlog/t/binlog_truncate_multi_log.test @@ -0,0 +1,81 @@ +# ==== Purpose ==== +# +# Test verifies truncation of multiple binary logs. +# +# ==== Implementation ==== +# +# Steps: +# 0 - Set max_binlog_size= 4096. Create a table do an insert such that the +# max_binlog_size is reached and binary log gets rotated. +# 1 - Using debug simulation make the server crash at a point where the DML +# transaction is written to binary log but not committed in engine. +# 2 - At the time of crash two binary logs will be there master-bin.0000001 +# and master-bin.000002. +# 3 - Restart server with --tc-heuristic-recover=ROLLBACK +# 4 - Since the prepared DML in master-bin.000001 is rolled back the first +# binlog will be truncated prior to the DML and master-bin.000002 will be +# removed. +# +# ==== References ==== +# +# MDEV-21117: --tc-heuristic-recover=rollback is not replication safe + + +--source include/have_innodb.inc +--source include/have_log_bin.inc +--source include/have_debug.inc +--source include/have_debug_sync.inc +--source include/have_binlog_format_row.inc + +SET @old_max_binlog_size= @@global.max_binlog_size; +SET GLOBAL max_binlog_size= 4096; + +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); + +RESET MASTER; + +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; + +connect(master1,localhost,root,,); +connect(master2,localhost,root,,); + +--connection master1 +# Hold insert after write to binlog and before "run_commit_ordered" in engine +SET DEBUG_SYNC= "commit_before_update_end_pos SIGNAL con1_ready WAIT_FOR con1_go"; +send INSERT INTO t1 VALUES (2, REPEAT("x", 4100)); + +--connection master2 +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; + +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +wait +EOF + +--source include/kill_mysqld.inc +--source include/wait_until_disconnected.inc + +# +# Server restart +# +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: --tc-heuristic-recover=ROLLBACK --debug-dbug=d,simulate_innodb_forget_commit_pos +EOF +--source include/wait_until_disconnected.inc + +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: +EOF + +connection default; +--enable_reconnect +--source include/wait_until_connected_again.inc + +--echo "Zero rows shoule be present in table" +SELECT COUNT(*) FROM t1; + +SELECT @@GLOBAL.gtid_current_pos; + +DROP TABLE t1; +SELECT @@GLOBAL.gtid_binlog_state; + diff --git a/mysql-test/suite/binlog/t/binlog_truncate_none.test b/mysql-test/suite/binlog/t/binlog_truncate_none.test new file mode 100644 index 00000000000..cc3fd4f0b37 --- /dev/null +++ b/mysql-test/suite/binlog/t/binlog_truncate_none.test @@ -0,0 +1,130 @@ +# ==== Purpose ==== +# +# Test case verifies no binlog truncation happens when non transactional +# events are found in binlog after the last committed transaction. +# +# ==== Implementation ==== +# +# Steps: +# 0 - Set max_binlog_size= 4096. Create a table and do an insert such that +# the max_binlog_size is reached and binary log gets rotated. +# 1 - Create a table in newly created binary log and crash the server +# 2 - Restart server with --tc-heuristic-recover=ROLLBACK +# 3 - Recovery code will get the last committed DML specific postion and +# will try to check if binlog can be truncated upto this position. +# Since a DDL is present beyond this no truncation will happen. +# ==== References ==== +# +# MDEV-21117: --tc-heuristic-recover=rollback is not replication safe + + +--source include/have_innodb.inc +--source include/have_log_bin.inc +--source include/have_debug.inc +--source include/have_binlog_format_row.inc + +SET @old_max_binlog_size= @@global.max_binlog_size; +SET GLOBAL max_binlog_size= 4096; + +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); + +RESET MASTER; + +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +INSERT INTO t1 VALUES (1, REPEAT("x", 4100)); +CREATE TABLE t2 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +--source include/show_binary_logs.inc + +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +wait +EOF + +--source include/kill_mysqld.inc +--source include/wait_until_disconnected.inc + +# +# Server restart +# +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: --tc-heuristic-recover=ROLLBACK +EOF +--source include/wait_until_disconnected.inc + +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: +EOF + +connection default; +--enable_reconnect +--source include/wait_until_connected_again.inc + +--echo "Zero records should be there." +SELECT COUNT(*) FROM t1; +--source include/show_binary_logs.inc +DROP TABLE t1,t2; + +# ==== Purpose ==== +# +# Test case verifies no binlog truncation happens when only DDLs are present in +# the binary log. Since none of the DMLs are performed in storage engine, +# Engine will not have last committed transaction file name or position. +# Truncation code should return success. +# +# ==== Implementation ==== +# +# Steps: +# 0 - Create two table t1, t2 +# 1 - Kill and restart server with --tc-heuristic-recover=ROLLBACK +# 2 - Only DDL statements are present in the binary log. Since +# no DML was performed engine will not have last commited transaction +# specific binary log name and position. Since no transactional events +# are found, truncation code should simply return. +# +# ==== References ==== +# +# MDEV-21117: --tc-heuristic-recover=rollback is not replication safe + + +--source include/have_innodb.inc +--source include/have_log_bin.inc +--source include/have_debug.inc +--source include/have_binlog_format_row.inc + +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); + +RESET MASTER; + +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +CREATE TABLE t2 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +--source include/show_binary_logs.inc + +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +wait +EOF + +--source include/kill_mysqld.inc +--source include/wait_until_disconnected.inc + +# +# Server restart +# +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: --tc-heuristic-recover=ROLLBACK --debug-dbug=d,simulate_innodb_forget_commit_pos +EOF +--source include/wait_until_disconnected.inc + +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: +EOF + +connection default; +--enable_reconnect +--source include/wait_until_connected_again.inc + +--echo "Zero records should be there." +SELECT COUNT(*) FROM t1; +SHOW TABLES; +DROP TABLE t1,t2; + diff --git a/mysql-test/suite/rpl/r/rpl_heuristic_fail_over.result b/mysql-test/suite/rpl/r/rpl_heuristic_fail_over.result new file mode 100644 index 00000000000..8391245ba27 --- /dev/null +++ b/mysql-test/suite/rpl/r/rpl_heuristic_fail_over.result @@ -0,0 +1,53 @@ +include/rpl_init.inc [topology=1->2] +include/stop_slave.inc +set global rpl_semi_sync_slave_enabled = 1; +CHANGE MASTER TO master_use_gtid= current_pos; +include/start_slave.inc +ALTER TABLE mysql.gtid_slave_pos ENGINE=InnoDB; +set global rpl_semi_sync_master_enabled = 1; +set global rpl_semi_sync_master_wait_point=AFTER_SYNC; +SET @old_max_binlog_size= @@global.max_binlog_size; +SET GLOBAL max_binlog_size= 4096; +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +INSERT INTO t1 VALUES (1, 'dummy1'); +SET DEBUG_SYNC= "commit_before_update_end_pos SIGNAL con1_ready WAIT_FOR con1_go"; +INSERT INTO t1 VALUES (2, REPEAT("x", 4100)); +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; +# Kill the server +include/stop_slave.inc +SELECT @@GLOBAL.gtid_current_pos; +@@GLOBAL.gtid_current_pos +0-1-5 +FOUND /Crashed binlog file \.\/master\-bin\.000001 size is [0-9]*, but truncated to */ in mysqld.1.err +set global rpl_semi_sync_master_enabled = 1; +set global rpl_semi_sync_master_wait_point=AFTER_SYNC; +CHANGE MASTER TO master_host='127.0.0.1', master_port=SERVER_MYPORT_2, master_user='root', master_use_gtid=CURRENT_POS; +set global rpl_semi_sync_slave_enabled = 1; +include/start_slave.inc +INSERT INTO t1 VALUES (3, 'dummy3'); +SELECT COUNT(*) FROM t1; +COUNT(*) +2 +SHOW VARIABLES LIKE 'gtid_current_pos'; +Variable_name Value +gtid_current_pos 0-2-6 +SHOW VARIABLES LIKE 'gtid_current_pos'; +Variable_name Value +gtid_current_pos 0-2-6 +DROP TABLE t1; +set global rpl_semi_sync_master_enabled = 0; +set global rpl_semi_sync_slave_enabled = 0; +set global rpl_semi_sync_master_wait_point=default; +set global rpl_semi_sync_master_enabled = 0; +set global rpl_semi_sync_slave_enabled = 0; +set global rpl_semi_sync_master_wait_point=default; +include/stop_slave.inc +RESET MASTER; +RESET SLAVE; +RESET MASTER; +RESET SLAVE; +CHANGE MASTER TO master_host='127.0.0.1', master_port=SERVER_MYPORT_1, master_user='root', master_use_gtid=no; +include/start_slave.inc +include/rpl_end.inc diff --git a/mysql-test/suite/rpl/t/rpl_heuristic_fail_over.test b/mysql-test/suite/rpl/t/rpl_heuristic_fail_over.test new file mode 100644 index 00000000000..5e9273de62d --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_heuristic_fail_over.test @@ -0,0 +1,160 @@ +# ==== Purpose ==== +# +# Test verifies replication failover scenario. +# +# ==== Implementation ==== +# +# Steps: +# 0 - Have two servers with id's 1 and 2. Enable semi-sync based +# replication. Have semi sync master wait point as 'after_sync'. +# 1 - Create a table and insert a row. While inserting second row simulate +# a server crash at once the transaction is written to binlog, flushed +# and synced but the binlog position is not updated. +# 2 - Restart the server using tc-heuristic-recover=ROLLBACK +# 3 - Post restart switch the crashed master to slave. Execute CHANGE MASTER +# TO command to connect to server id 2. +# 4 - Slave should be able to connect to master. +# 5 - Execute some DML on new master with server id 2. Ensure that it gets +# replicated to server id 1. +# 6 - Verify the "gtid_current_pos" for correctness. +# 7 - Clean up +# +# ==== References ==== +# +# MDEV-21117: --tc-heuristic-recover=rollback is not replication safe + + +--source include/have_semisync.inc +--source include/have_innodb.inc +--source include/have_log_bin.inc +--source include/have_debug.inc +--source include/have_debug_sync.inc +--source include/have_binlog_format_row.inc +--let $rpl_topology=1->2 +--source include/rpl_init.inc + +--connection server_2 +--source include/stop_slave.inc +set global rpl_semi_sync_slave_enabled = 1; +CHANGE MASTER TO master_use_gtid= current_pos; +--source include/start_slave.inc + + +--connection server_1 +ALTER TABLE mysql.gtid_slave_pos ENGINE=InnoDB; +set global rpl_semi_sync_master_enabled = 1; +set global rpl_semi_sync_master_wait_point=AFTER_SYNC; +SET @old_max_binlog_size= @@global.max_binlog_size; +SET GLOBAL max_binlog_size= 4096; + +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); + +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +INSERT INTO t1 VALUES (1, 'dummy1'); +--save_master_pos + +--connection server_2 +--sync_with_master + +--connection server_1 +connect(master1,localhost,root,,); +connect(master2,localhost,root,,); + +--connection master1 +# Hold insert after write to binlog and before "run_commit_ordered" in engine +SET DEBUG_SYNC= "commit_before_update_end_pos SIGNAL con1_ready WAIT_FOR con1_go"; +send INSERT INTO t1 VALUES (2, REPEAT("x", 4100)); + +--connection master2 +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +wait +EOF + +--source include/kill_mysqld.inc +--source include/wait_until_disconnected.inc + +--connection server_2 +--error 2003 +--source include/stop_slave.inc +SELECT @@GLOBAL.gtid_current_pos; + +--connection server_1 +# +# Server restart +# +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: --tc-heuristic-recover=ROLLBACK +EOF +--source include/wait_until_disconnected.inc + +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: +EOF + +connection default; +--enable_reconnect +--source include/wait_until_connected_again.inc + +--connection server_1 +--enable_reconnect +--source include/wait_until_connected_again.inc + +# Check error log for correct messages. +let $log_error_= `SELECT @@GLOBAL.log_error`; +if(!$log_error_) +{ + # MySQL Server on windows is started with --console and thus + # does not know the location of its .err log, use default location + let $log_error_ = $MYSQLTEST_VARDIR/log/mysqld.1.err; +} +--let SEARCH_FILE=$log_error_ +--let SEARCH_RANGE=-50000 +--let SEARCH_PATTERN=Crashed binlog file \.\/master\-bin\.000001 size is [0-9]*, but truncated to * +--source include/search_pattern_in_file.inc + +--connection server_2 +set global rpl_semi_sync_master_enabled = 1; +set global rpl_semi_sync_master_wait_point=AFTER_SYNC; + +--connection server_1 +--replace_result $SERVER_MYPORT_2 SERVER_MYPORT_2 +eval CHANGE MASTER TO master_host='127.0.0.1', master_port=$SERVER_MYPORT_2, master_user='root', master_use_gtid=CURRENT_POS; +set global rpl_semi_sync_slave_enabled = 1; +--source include/start_slave.inc + +--connection server_2 +INSERT INTO t1 VALUES (3, 'dummy3'); +--save_master_pos + +--connection server_1 +--sync_with_master +SELECT COUNT(*) FROM t1; +SHOW VARIABLES LIKE 'gtid_current_pos'; + +--connection server_2 +SHOW VARIABLES LIKE 'gtid_current_pos'; +DROP TABLE t1; +set global rpl_semi_sync_master_enabled = 0; +set global rpl_semi_sync_slave_enabled = 0; +set global rpl_semi_sync_master_wait_point=default; +--save_master_pos + +--connection server_1 +--sync_with_master +set global rpl_semi_sync_master_enabled = 0; +set global rpl_semi_sync_slave_enabled = 0; +set global rpl_semi_sync_master_wait_point=default; +--source include/stop_slave.inc +RESET MASTER; +RESET SLAVE; + +--connection server_2 +RESET MASTER; +RESET SLAVE; +--replace_result $SERVER_MYPORT_1 SERVER_MYPORT_1 +eval CHANGE MASTER TO master_host='127.0.0.1', master_port=$SERVER_MYPORT_1, master_user='root', master_use_gtid=no; +--source include/start_slave.inc + +--source include/rpl_end.inc diff --git a/sql/log.cc b/sql/log.cc index 0efef6d1e29..f383033e975 100644 --- a/sql/log.cc +++ b/sql/log.cc @@ -3164,6 +3164,7 @@ MYSQL_BIN_LOG::MYSQL_BIN_LOG(uint *sync_period) checksum_alg_reset(BINLOG_CHECKSUM_ALG_UNDEF), relay_log_checksum_alg(BINLOG_CHECKSUM_ALG_UNDEF), description_event_for_exec(0), description_event_for_queue(0), + last_commit_pos_offset(0), current_binlog_id(0) { /* @@ -3173,6 +3174,7 @@ MYSQL_BIN_LOG::MYSQL_BIN_LOG(uint *sync_period) before main(). */ index_file_name[0] = 0; + last_commit_pos_file[0]= 0; bzero((char*) &index_file, sizeof(index_file)); bzero((char*) &purge_index_file, sizeof(purge_index_file)); } @@ -7877,6 +7879,7 @@ MYSQL_BIN_LOG::trx_group_commit_leader(group_commit_entry *leader) first= false; } + DEBUG_SYNC(leader->thd, "commit_before_update_end_pos"); /* update binlog_end_pos so it can be read by dump thread * * note: must be _after_ the RUN_HOOK(after_flush) or else @@ -8964,7 +8967,7 @@ int TC_LOG_MMAP::open(const char *opt_name) { if (my_errno != ENOENT) goto err; - if (using_heuristic_recover()) + if (using_heuristic_recover(opt_name)) return 1; if ((fd= mysql_file_create(key_file_tclog, logname, CREATE_MODE, O_RDWR | O_CLOEXEC, MYF(MY_WME))) < 0) @@ -9497,14 +9500,42 @@ TC_LOG_MMAP tc_log_mmap; 1 heuristic recovery was performed */ -int TC_LOG::using_heuristic_recover() +int TC_LOG::using_heuristic_recover(const char* opt_name) { + LOG_INFO log_info; + int error; + if (!tc_heuristic_recover) return 0; sql_print_information("Heuristic crash recovery mode"); + if (ha_recover(0)) + { sql_print_error("Heuristic crash recovery failed"); + } + + /* + Check if TC log is referring to binary log not memory map. Ensure that + tc_heuristic_recover being ROLLBACK". If both match initiate binlog + truncation mechanism. + */ + if (!strcmp(opt_name,opt_bin_logname) && + tc_heuristic_recover == TC_HEURISTIC_RECOVER_ROLLBACK) + { + if ((error= mysql_bin_log.find_log_pos(&log_info, NullS, 1))) + { + if (error != LOG_INFO_EOF) + sql_print_error("find_log_pos() failed (error: %d)", error); + } + else + { + if ((error= heuristic_binlog_rollback())) + sql_print_error("Heuristic crash recovery failed to remove " + "rolled back trancations from binary log"); + } + } + sql_print_information("Please restart mysqld without --tc-heuristic-recover"); return 1; } @@ -9512,6 +9543,448 @@ int TC_LOG::using_heuristic_recover() /****** transaction coordinator log for 2pc - binlog() based solution ******/ #define TC_LOG_BINLOG MYSQL_BIN_LOG +/** + Truncates the current binlog to specified position. Removes the rest of binlogs + which are present after this binlog file. + + @param truncate_file Holds the binlog name to be truncated + @param truncate_pos Position within binlog from where it needs to + truncated. + + @retval true ok + @retval false error + +*/ +bool MYSQL_BIN_LOG::truncate_and_remove_binlogs(const char *truncate_file, + my_off_t truncate_pos) +{ + int error= 0; +#ifdef HAVE_REPLICATION + LOG_INFO log_info; + THD *thd= current_thd; + my_off_t index_file_offset=0; + File file= -1; + MY_STAT s; + my_off_t binlog_size; + + if ((error= open_purge_index_file(TRUE))) + { + sql_print_error("MYSQL_BIN_LOG::purge_logs failed to sync the index file."); + goto err; + } + + if ((error=find_log_pos(&log_info, truncate_file, 1))) + goto err; + + while (!(error= find_next_log(&log_info, 1))) + { + if (!index_file_offset) + index_file_offset= log_info.index_file_start_offset; + if((error= register_purge_index_entry(log_info.log_file_name))) + { + sql_print_error("MYSQL_BIN_LOG::purge_logs failed to copy %s to register file.", + log_info.log_file_name); + goto err; + } + } + + if (error != LOG_INFO_EOF) + { + sql_print_error("Error while looking for next binlog file to be truncated. " + "Error:%d", error); + goto err; + } + + if (!index_file_offset) + index_file_offset= log_info.index_file_start_offset; + + if ((error= sync_purge_index_file())) + { + sql_print_error("MYSQL_BIN_LOG::purge_logs failed to flush register file."); + goto err; + } + + DBUG_ASSERT(index_file_offset != 0); + // Trim index file + if (mysql_file_chsize(index_file.file, index_file_offset, '\n', MYF(MY_WME)) || + mysql_file_sync(index_file.file, MYF(MY_WME|MY_SYNC_FILESIZE))) + { + sql_print_error("Failed to trim binlog index file " + "when master server is recovering it."); + mysql_file_close(index_file.file, MYF(MY_WME)); + goto err; + } + + /* Reset data in old index cache */ + reinit_io_cache(&index_file, READ_CACHE, (my_off_t) 0, 0, 1); + + /* Read each entry from purge_index_file and delete the file. */ + if (is_inited_purge_index_file() && + (error= purge_index_entry(thd, NULL, TRUE))) + { + sql_print_error("MYSQL_BIN_LOG::purge_logs failed to process registered files" + " that would be purged."); + goto err; + } + + if (truncate_pos) + { + if ((file= mysql_file_open(key_file_binlog, truncate_file, + O_RDWR | O_BINARY, MYF(MY_WME))) < 0) + { + sql_print_error("Failed to open binlog file:%s for truncation.", + truncate_file); + error= 1; + goto err; + } + my_stat(truncate_file, &s, MYF(0)); + binlog_size= s.st_size; + + /* Change binlog file size to truncate_pos */ + if (mysql_file_chsize(file, truncate_pos, 0, MYF(MY_WME)) || + mysql_file_sync(file, MYF(MY_WME|MY_SYNC_FILESIZE))) + { + sql_print_error("Failed to trim the crashed binlog file:%s to size:%llu", + truncate_file, truncate_pos); + mysql_file_close(file, MYF(MY_WME)); + error= 1; + goto err; + } + else + { + sql_print_information("Crashed binlog file %s size is %llu, " + "but truncated to %llu.", + truncate_file, binlog_size, truncate_pos); + } + mysql_file_close(file, MYF(MY_WME)); + file= -1; + } + +err: + if (file >= 0) + mysql_file_close(file, MYF(MY_WME)); + close_purge_index_file(); +#endif + return error; +} + +/** + Returns the checkpoint binlog file name found in the lastest binlog file. + + @param checkpoint_file Holds the binlog checkpoint file name. + + @retval 0 ok + @retval 1 error + +*/ +int TC_LOG_BINLOG::get_binlog_checkpoint_file(char* checkpoint_file) +{ + Log_event *ev= NULL; + bool binlog_checkpoint_found= false; + LOG_INFO log_info; + const char *errmsg; + IO_CACHE log; + File file; + Format_description_log_event fdle(BINLOG_VERSION); + char log_name[FN_REFLEN]; + int error=1; + + if (!fdle.is_valid()) + return 1; + + if ((error= find_log_pos(&log_info, NullS, 1))) + { + sql_print_error("find_log_pos() failed (error: %d)", error); + return error; + } + + // Move to the last binary log. + do + { + strmake_buf(log_name, log_info.log_file_name); + } while (!(error= find_next_log(&log_info, 1))); + + if (error != LOG_INFO_EOF) + { + sql_print_error("find_log_pos() failed (error: %d)", error); + return error; + } + if ((file= open_binlog(&log, log_name, &errmsg)) < 0) + { + sql_print_error("%s", errmsg); + return error; + } + while (!binlog_checkpoint_found && + (ev= Log_event::read_log_event(&log, 0, &fdle, + opt_master_verify_checksum)) && + ev->is_valid()) + { + enum Log_event_type typ= ev->get_type_code(); + if (typ == BINLOG_CHECKPOINT_EVENT) + { + uint dir_len; + Binlog_checkpoint_log_event *cev= (Binlog_checkpoint_log_event *)ev; + if (cev->binlog_file_len >= FN_REFLEN) + { + sql_print_error("Incorrect binlog checkpoint event with too " + "long file name found."); + delete ev; + ev= NULL; + end_io_cache(&log); + mysql_file_close(file, MYF(MY_WME)); + return 1; + } + else + { + dir_len= dirname_length(log_name); + strmake(strnmov(checkpoint_file, log_name, dir_len), + cev->binlog_file_name, FN_REFLEN - 1 - dir_len); + binlog_checkpoint_found= true; + } + } + delete ev; + ev= NULL; + } // End of while + end_io_cache(&log); + mysql_file_close(file, MYF(MY_WME)); + file= -1; + /* + Old binary log without checkpoint found, binlog truncation is not + possible. Hence return error. + */ + if (!binlog_checkpoint_found) + return 1; + else + { + // Look for binlog checkpoint file in binlog index file. + if (find_log_pos(&log_info, checkpoint_file, 1)) + { + sql_print_error("Binlog file '%s' not found in binlog index, needed " + "for recovery. Aborting.", checkpoint_file); + return 1; + } + } + return 0; +} + + +/** + Truncates the binary log, according to the transactions that got rolled + back from engine, during heuristic-recover=ROLLBACK. Global GTID state is + adjusted as per the truncated binlog. + + Called from @c TC_LOG::using_heuristic_recover(const char* opt_name) + + @param opt_name The base name of binary log. + + @return indicates success or failure of binlog rollback + @retval 0 success + @retval 1 failure + +*/ +int TC_LOG_BINLOG::heuristic_binlog_rollback() +{ + int error=0; +#ifdef HAVE_REPLICATION + Log_event *ev= NULL; + char engine_commit_file[FN_REFLEN]; + char binlog_truncate_file_name[FN_REFLEN]; + char checkpoint_file[FN_REFLEN]; + my_off_t binlog_truncate_pos= 0; + my_off_t engine_commit_pos= 0; + LOG_INFO log_info; + const char *errmsg; + IO_CACHE log; + File file; + Format_description_log_event fdle(BINLOG_VERSION); + bool found_engine_commit_pos= false; + bool found_truncate_pos= false; + bool is_safe= true; + my_off_t tmp_truncate_pos=0; + rpl_gtid last_gtid; + bool last_gtid_standalone= false; + bool last_gtid_valid= false; + + if (!fdle.is_valid()) + return 1; + + DBUG_EXECUTE_IF("simulate_innodb_forget_commit_pos", + { + last_commit_pos_file[0]= 0; + };); + + // Initialize engine_commit_file/pos + if (last_commit_pos_file[0] != 0) + { + strmake_buf(engine_commit_file, last_commit_pos_file); + engine_commit_pos= last_commit_pos_offset; + } + else + { + if ((error= get_binlog_checkpoint_file(checkpoint_file))) + return error; + strmake_buf(engine_commit_file, checkpoint_file); + } + + // Look for last last_commit_pos_file in binlog index file + if ((error= find_log_pos(&log_info, engine_commit_file, 1))) + { + sql_print_error("find_log_pos() failed (error: %d)", error); + return error; + } + if ((file= open_binlog(&log, log_info.log_file_name, &errmsg)) < 0) + { + sql_print_error("Failed to open the binlog:%s for recovery. Error: %s", + binlog_truncate_file_name, errmsg); + goto err; + } + + /* + If there is no engine specific commit file we are doing checkpoint file + based recovery. Hence we mark "found_engine_commit_pos" true. Next start + looking for the first transactional event group with is the candidate for + rollback. + */ + if (engine_commit_pos == 0) + found_engine_commit_pos= true; + + error= read_state_from_file(); + if (error && error != 2) + { + sql_print_error("Failed to load global gtid binlog state from file"); + goto err; + } + + for(;;) + { + while (is_safe && (ev= Log_event::read_log_event(&log, 0, &fdle, + opt_master_verify_checksum)) && + ev->is_valid()) + { + enum Log_event_type typ= ev->get_type_code(); + switch (typ) + { + case XID_EVENT: + if (ev->log_pos == engine_commit_pos) + { + found_engine_commit_pos= true; + } + break; + case GTID_LIST_EVENT: + { + Gtid_list_log_event *glev= (Gtid_list_log_event *)ev; + /* Initialise the binlog state from the Gtid_list event. */ + if (!found_truncate_pos && glev->count > 0 && + rpl_global_gtid_binlog_state.load(glev->list, glev->count)) + goto err; + } + break; + case GTID_EVENT: + { + Gtid_log_event *gev= (Gtid_log_event *)ev; + + /* Update the binlog state with any GTID logged after Gtid_list. */ + last_gtid.domain_id= gev->domain_id; + last_gtid.server_id= gev->server_id; + last_gtid.seq_no= gev->seq_no; + last_gtid_standalone= + ((gev->flags2 & Gtid_log_event::FL_STANDALONE) ? true : false); + last_gtid_valid= true; + if (gev->flags2 & Gtid_log_event::FL_TRANSACTIONAL && + !found_truncate_pos) + { + if ((engine_commit_pos == 0 || found_engine_commit_pos)) + { + found_truncate_pos=true; + strmake_buf(binlog_truncate_file_name, log_info.log_file_name); + binlog_truncate_pos= tmp_truncate_pos; + } + } + else + { + if (found_truncate_pos) + is_safe=false; + } + } + break; + default: + /* Nothing. */ + break; + }// End switch + if (!found_truncate_pos && last_gtid_valid && + ((last_gtid_standalone && !ev->is_part_of_group(typ)) || + (!last_gtid_standalone && + (typ == XID_EVENT || + (typ == QUERY_EVENT && + (((Query_log_event *)ev)->is_commit() || + ((Query_log_event *)ev)->is_rollback())))))) + { + if (rpl_global_gtid_binlog_state.update_nolock(&last_gtid, false)) + goto err; + last_gtid_valid= false; + } + // Used to identify the last group specific end position. + tmp_truncate_pos= ev->log_pos; + delete ev; + ev= NULL; + }// End While + end_io_cache(&log); + mysql_file_close(file, MYF(MY_WME)); + file= -1; + + if (is_safe) + { + if ((error=find_next_log(&log_info, 1))) + { + if (error != LOG_INFO_EOF) + { + sql_print_error("Error reading binlog files during recovery. Aborting."); + goto err; + } + else + break; + } + + if ((file= open_binlog(&log, log_info.log_file_name, &errmsg)) < 0) + { + sql_print_error("%s", errmsg); + goto err; + } + } + } //end of for(;;) + if (!found_truncate_pos) + { + return 0; // Nothing to truncate + } + else + DBUG_ASSERT(binlog_truncate_pos > 0); + + if (is_safe) + { + if (truncate_and_remove_binlogs(binlog_truncate_file_name, + binlog_truncate_pos)) + goto err; + } + if (write_state_to_file()) + { + sql_print_error("Failed to save binlog GTID state during heuristic " + "binlog rollback. "); + goto err; + } + return 0; + +err: + error= 1; + if (file >= 0) + { + end_io_cache(&log); + mysql_file_close(file, MYF(MY_WME)); + } + sql_print_error("Binlog truncation failed"); +#endif + return error; +} + int TC_LOG_BINLOG::open(const char *opt_name) { int error= 1; @@ -9526,7 +9999,7 @@ int TC_LOG_BINLOG::open(const char *opt_name) return 1; } - if (using_heuristic_recover()) + if (using_heuristic_recover(opt_name)) { mysql_mutex_lock(&LOCK_log); /* generate a new binlog to mask a corrupted one */ diff --git a/sql/log.h b/sql/log.h index 277e5c6f69c..e91bdd94083 100644 --- a/sql/log.h +++ b/sql/log.h @@ -41,7 +41,8 @@ bool stmt_has_updated_non_trans_table(const THD* thd); class TC_LOG { public: - int using_heuristic_recover(); + int using_heuristic_recover(const char* opt_name); + virtual int heuristic_binlog_rollback() { return 0; }; TC_LOG() {} virtual ~TC_LOG() {} @@ -694,6 +695,7 @@ class MYSQL_BIN_LOG: public TC_LOG, private MYSQL_LOG void commit_checkpoint_notify(void *cookie); int recover(LOG_INFO *linfo, const char *last_log_name, IO_CACHE *first_log, Format_description_log_event *fdle, bool do_xa); + int heuristic_binlog_rollback(); int do_binlog_recovery(const char *opt_name, bool do_xa_recovery); #if !defined(MYSQL_CLIENT) @@ -794,6 +796,9 @@ class MYSQL_BIN_LOG: public TC_LOG, private MYSQL_LOG int purge_first_log(Relay_log_info* rli, bool included); int set_purge_index_file_name(const char *base_file_name); int open_purge_index_file(bool destroy); + bool truncate_and_remove_binlogs(const char *truncate_file, + my_off_t truncate_pos); + int get_binlog_checkpoint_file(char* checkpoint_file); bool is_inited_purge_index_file(); int close_purge_index_file(); int clean_purge_index_file(); diff --git a/storage/innobase/handler/ha_innodb.cc b/storage/innobase/handler/ha_innodb.cc index 4de2cdbeaec..42d655ee0e8 100644 --- a/storage/innobase/handler/ha_innodb.cc +++ b/storage/innobase/handler/ha_innodb.cc @@ -16493,6 +16493,9 @@ innobase_xa_recover( { DBUG_ASSERT(hton == innodb_hton_ptr); + mysql_bin_log.last_commit_pos_offset= trx_sys_mysql_bin_log_pos; + strmake_buf(mysql_bin_log.last_commit_pos_file, trx_sys_mysql_bin_log_name); + if (len == 0 || xid_list == NULL) { return(0); diff --git a/storage/xtradb/handler/ha_innodb.cc b/storage/xtradb/handler/ha_innodb.cc index 2aafb1a44ee..dec106cac54 100644 --- a/storage/xtradb/handler/ha_innodb.cc +++ b/storage/xtradb/handler/ha_innodb.cc @@ -17165,6 +17165,9 @@ innobase_xa_recover( { DBUG_ASSERT(hton == innodb_hton_ptr); + mysql_bin_log.last_commit_pos_offset= trx_sys_mysql_bin_log_pos; + strmake_buf(mysql_bin_log.last_commit_pos_file, trx_sys_mysql_bin_log_name); + if (len == 0 || xid_list == NULL) { return(0);

1 0

Re: [Maria-developers] 7b03ce7af3d: MDEV-17395 REPLACE/INSERT ODKU: support WITHOUT OVERLAPS
by Sergei Golubchik 06 Apr '20

06 Apr '20

Hi, Nikita! First, about tests: On Mar 06, Nikita Malyavin wrote: > revision-id: 7b03ce7af3d (mariadb-10.4.4-504-g7b03ce7af3d) > parent(s): 5c94cf3bf44 > author: Nikita Malyavin <nikitamalyavin(a)gmail.com> > committer: Nikita Malyavin <nikitamalyavin(a)gmail.com> > timestamp: 2019-11-28 01:38:53 +1000 > message: > > MDEV-17395 REPLACE/INSERT ODKU: support WITHOUT OVERLAPS > > --- > mysql-test/suite/period/r/insert_replace.result | 56 +++ > mysql-test/suite/period/t/insert_replace.test | 37 ++ > sql/handler.cc | 14 + > sql/handler.h | 3 +- > sql/sql_insert.cc | 492 ++++++++++++++---------- > sql/sql_load.cc | 6 + > sql/table.cc | 4 +- > sql/table.h | 5 +- > 8 files changed, 409 insertions(+), 208 deletions(-) > > diff --git a/mysql-test/suite/period/r/insert_replace.result b/mysql-test/suite/period/r/insert_replace.result > --- /dev/null > +++ b/mysql-test/suite/period/r/insert_replace.result > @@ -0,0 +1,56 @@ > +create or replace table t(id int, val int, s date, e date, > +period for p(s,e), > +primary key(id, p without overlaps)) engine=myisam; > +insert into t values (1, 1, '2003-01-01', '2003-03-01'), > +(1, 2, '2003-05-01', '2003-07-01'); > +# This just inserts a row; no rows matched > +insert into t values (2, 3, '2003-01-01', '2003-04-01') > +on duplicate key update val=3; > +# The following command is equivalent to > +# MERGE INTO t USING t > +# ON id = 1 AND s <= 2003-04-01 AND e > 2003-01-01 > +# WHEN MATCHED UPDATE SET val=3 > +# WHEN NOT MATCHED INSERT VALUES (1, 3, '2003-01-01', '2003-04-01'); > +insert into t values (1, 3, '2003-01-01', '2003-04-01') > +on duplicate key update val=3; > +select row_count(); > +row_count() > +2 > +select * from t; > +id val s e > +1 3 2003-01-01 2003-03-01 > +1 2 2003-05-01 2003-07-01 > +2 3 2003-01-01 2003-04-01 It's somewhat misleading, because you've used '3' everywhere. Please rewrite your tests (all of them, also tests below) to identify every operation uniquely. For example: insert into t values (1, 1, '2003-01-01', '2003-03-01'), (1, 2, '2003-05-01', '2003-07-01'); # This just inserts a row; no rows matched insert into t values (2, 3, '2003-01-01', '2003-04-01') on duplicate key update val=4; # The following command is equivalent to # MERGE INTO t USING t # ON id = 1 AND s <= 2003-04-01 AND e > 2003-01-01 # WHEN MATCHED UPDATE SET val=6 # WHEN NOT MATCHED INSERT VALUES (1, 5, '2003-01-01', '2003-04-01'); insert into t values (1, 5, '2003-01-01', '2003-04-01') on duplicate key update val=6; > +insert into t values (1, 3, '2003-01-01', '2003-06-01') > +on duplicate key update val=4; > +select row_count(); > +row_count() > +4 > +select * from t; > +id val s e > +1 4 2003-01-01 2003-03-01 > +1 4 2003-05-01 2003-07-01 > +2 3 2003-01-01 2003-04-01 I don't think IODKU is defined via MERGE. Unfortunately. See, how it works: create table t1 (a int not null, b int not null, c int, unique(a), unique(b)); insert t1 values (1,1,1), (2,2,2); insert t1 values (1,2,3) on duplicate key update c=4; select * from t1; a b c 1 1 4 2 2 2 here only one row was updated. If it would've been defined as MERGE INTO t1 USING t1 ON a=1 OR b=2 WHEN MATCHED UPDATE c=4 WHEN NOT MATCHED INSERT VALUES (1,2,3) then it would've updated both rows. As you can see it literally is defined as "insert, and if there's a duplicate key error, then update the conflicting row instead" That is, in your case it should've updated only one row too. Also, please, add this statement to your test: insert into t values (1, 3, '2003-01-01', '2003-02-01') on duplicate key update val=4; > +# No rows matched > +insert into t values (1, 3, '2003-07-01', '2003-08-01') > +on duplicate key update val=5; > +select row_count(); > +row_count() > +1 > +select * from t; > +id val s e > +1 4 2003-01-01 2003-03-01 > +1 4 2003-05-01 2003-07-01 > +2 3 2003-01-01 2003-04-01 > +1 3 2003-07-01 2003-08-01 > +replace into t values(1, 6, '2003-01-01', '2003-06-01'); > +select row_count(); > +row_count() > +4 > +select * from t; > +id val s e > +1 6 2003-01-01 2003-06-01 > +1 4 2003-06-01 2003-07-01 > +2 3 2003-01-01 2003-04-01 > +1 3 2003-07-01 2003-08-01 Here you do DELETE FOR PERIOD. But above you didn't do UPDATE FOR PERIOD. Add also this: replace into t values(1, 6, '2003-01-01', '2003-02-01'); And tests for INSERT SELECT (also with IGNORE, REPLACE, ODKU) and for LOAD DATA (also with IGNORE and REPLACE). > +drop table t; Now, about semantics. It is very arguable here. One options is to do, literally, "insert, if fails delete/update the conflicting row". No periods involved here. The other option is to use FOR PERIOD implicitly for updates and deletes. Example, in all test cases below I'll assume: insert t1 values (1,1,'2003-01-01','2003-03-01'); insert t1 values (1,2,'2003-05-01','2003-06-01'); -> 1 1 2003-01-01 2003-03-01 1 2 2003-05-01 2003-06-01 So, option one: insert t1 values (1,3,'2003-01-01','2003-02-01') on duplicate key update val=4; -> 1 4 2003-01-01 2003-02-01 1 2 2003-05-01 2003-06-01 Option two: insert t1 values (1,3,'2003-01-01','2003-02-01') on duplicate key update val=4; -> 1 4 2003-01-01 2003-02-01 1 1 2003-02-01 2003-03-01 1 2 2003-05-01 2003-06-01 Overlapping range, option two: insert t1 values (1,3,'2003-01-01','2003-04-01') on duplicate key update val=4; -> 1 4 2003-01-01 2003-03-01 1 3 2003-03-01 2003-04-01 1 2 2003-05-01 2003-06-01 And the same for replace, basically: replace t1 values (1,3,'2003-01-01','2003-02-01'); -> 1 3 2003-01-01 2003-02-01 1 2 2003-05-01 2003-06-01 or -> 1 3 2003-01-01 2003-02-01 1 1 2003-02-01 2003-03-01 1 2 2003-05-01 2003-06-01 Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 3

Re: [Maria-developers] 6daf451415f: Let "FTWRL <table_list>" use extra(HA_EXTRA_FLUSH)
by Sergei Golubchik 02 Apr '20

02 Apr '20

Hi, Sergey! On Apr 01, Sergey Vojtovich wrote: > revision-id: 6daf451415f (mariadb-10.5.0-69-g6daf451415f) > parent(s): c24253d0fa3 > author: Sergey Vojtovich <svoj(a)mariadb.org> > committer: Sergey Vojtovich <svoj(a)mariadb.org> > timestamp: 2019-12-25 20:24:24 +0400 > message: > > Let "FTWRL <table_list>" use extra(HA_EXTRA_FLUSH) > > Rather than flushing caches with tdc_remove_table(TDC_RT_REMOVE_UNUSED) > flush them with extra(HA_EXTRA_FLUSH) instead. This goes inline with > regular FTWRL. Not quite. FTWRL calls flush_tables(thd, FLUSH_ALL), and flush_tables() does extra(HA_EXTRA_FLUSH) + tc_release_table() + tdc_release_share(share) So FTWRL still closes all tables properly. > diff --git a/sql/sql_reload.cc b/sql/sql_reload.cc > --- a/sql/sql_reload.cc > +++ b/sql/sql_reload.cc > @@ -539,16 +538,10 @@ bool flush_tables_with_read_lock(THD *thd, TABLE_LIST *all_tables) > > DEBUG_SYNC(thd,"flush_tables_with_read_lock_after_acquire_locks"); > > - for (table_list= all_tables; table_list; > + /* Reset ticket to satisfy asserts in open_tables(). */ > + for (auto table_list= all_tables; table_list; > table_list= table_list->next_global) > - { > - /* Request removal of table from cache. */ > - tdc_remove_table(thd, TDC_RT_REMOVE_UNUSED, > - table_list->db.str, > - table_list->table_name.str); > - /* Reset ticket to satisfy asserts in open_tables(). */ > table_list->mdl_request.ticket= NULL; > - } > } > > thd->variables.option_bits|= OPTION_TABLE_LOCK; > @@ -583,6 +576,16 @@ bool flush_tables_with_read_lock(THD *thd, TABLE_LIST *all_tables) > } > } > > + if (thd->lex->type & REFRESH_READ_LOCK) > + { > + for (auto table_list= all_tables; table_list; > + table_list= table_list->next_global) > + { > + if (table_list->table->file->extra(HA_EXTRA_FLUSH)) > + goto error_reset_bits; > + } > + } > + > if (thd->locked_tables_list.init_locked_tables(thd)) > goto error_reset_bits; > > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 3

Re: [Maria-developers] a737b71295e: Cleanup mysql_inplace_alter_table()
by Sergei Golubchik 02 Apr '20

02 Apr '20

Hi, Sergey! ok to push On Apr 02, Sergey Vojtovich wrote: > revision-id: a737b71295e (mariadb-10.5.0-71-ga737b71295e) > parent(s): 5c26e91f755 > author: Sergey Vojtovich <svoj(a)mariadb.org> > committer: Sergey Vojtovich <svoj(a)mariadb.org> > timestamp: 2019-12-25 20:24:25 +0400 > message: > > Cleanup mysql_inplace_alter_table() > > Removed redundant tdc_remove_table(TDC_RT_REMOVE_ALL). Share was marked > flushed by preceding wait_while_table_is_used() and eventually flushed by > close_all_tables_for_name(). > > Part of MDEV-17882 - Cleanup refresh version > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 5c26e91f755: Cleanup close_all_tables_for_name()
by Sergei Golubchik 02 Apr '20

02 Apr '20

Hi, Sergey! On Apr 01, Sergey Vojtovich wrote: > revision-id: 5c26e91f755 (mariadb-10.5.0-70-g5c26e91f755) > parent(s): 6daf451415f > author: Sergey Vojtovich <svoj(a)mariadb.org> > committer: Sergey Vojtovich <svoj(a)mariadb.org> > timestamp: 2019-12-25 20:24:25 +0400 > message: > > Cleanup close_all_tables_for_name() > > close_all_tables_for_name() is always preceded by > wait_while_table_is_used(), which makes tdc_remove_table() redundant. > The only (now fixed) exception was close_cached_tables(). > > Part of MDEV-17882 - Cleanup refresh version > > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > index 60e2b9957a4..9494c0b7bd2 100644 > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -391,13 +391,12 @@ bool close_cached_tables(THD *thd, TABLE_LIST *tables, > if (! table) > continue; > > - if (thd->mdl_context.upgrade_shared_lock(table->mdl_ticket, MDL_EXCLUSIVE, > - timeout)) > + if (wait_while_table_is_used(thd, table, > + HA_EXTRA_PREPARE_FOR_FORCED_CLOSE)) the comment just above this block says /* If we are under LOCK TABLES, we need to reopen the tables without opening a door for any concurrent threads to sneak in and get lock on our tables. To achieve this we use exclusive metadata locks. */ how do you achieve it now without the MDL_EXCLUSIVE ? > { > result= true; > break; > } > - table->file->extra(HA_EXTRA_PREPARE_FOR_FORCED_CLOSE); > close_all_tables_for_name(thd, table->s, HA_EXTRA_NOT_USED, NULL); > } the rest is fine Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 2

Re: [Maria-developers] eeba5b2b158: Fixed close_cached_connection_tables() flushing
by Sergei Golubchik 02 Apr '20

02 Apr '20

Hi, Sergey! On Apr 02, Sergey Vojtovich wrote: > revision-id: eeba5b2b158 (mariadb-10.5.0-72-geeba5b2b158) > parent(s): a737b71295e > author: Sergey Vojtovich <svoj(a)mariadb.org> > committer: Sergey Vojtovich <svoj(a)mariadb.org> > timestamp: 2019-12-25 20:24:26 +0400 > message: > > Fixed close_cached_connection_tables() flushing > > Let DROP SERVER and ALTER SERVER perform fair affected tables flushing. > That is acquire MDL_EXCLUSIVE and do tdc_remove_table(TDC_RT_REMOVE_ALL). > > Aim of this patch is elimination of another inconsistent use of > TDC_RT_REMOVE_UNUSED. It fixes (to some extent) a problem described in the > beginning of sql_server.cc, when close_cached_connection_tables() > interferes with concurrent transaction. > > A better fix should probably introduce proper MDL locks for server > objects? > > Part of MDEV-17882 - Cleanup refresh version > > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > index 9494c0b7bd2..5c6d366ce6f 100644 > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -689,33 +689,21 @@ static my_bool close_cached_connection_tables_callback( > Close cached connections Could you, perhaps, move this function to sql_server.cc ? Also it could be made static there. And please clarify the comment, "Close cached connections' was not helpful at all. May be "Close all tables with a given CONNECTION= value" > @return false ok > - @return true If there was an error from closed_cached_connection_tables or > - if there was any open connections that we had to force closed > + @return true error, some tables may keep using old server info > */ > > bool close_cached_connection_tables(THD *thd, LEX_CSTRING *connection) > { > - bool res= false; > - close_cached_connection_tables_arg argument; > + close_cached_connection_tables_arg argument= { thd, connection, 0 }; > DBUG_ENTER("close_cached_connections"); > - DBUG_ASSERT(thd); > - > - argument.thd= thd; > - argument.connection= connection; > - argument.tables= NULL; > > if (tdc_iterate(thd, > (my_hash_walk_action) close_cached_connection_tables_callback, > &argument)) > DBUG_RETURN(true); > > - for (TABLE_LIST *table= argument.tables; table; table= table->next_local) > - res|= tdc_remove_table(thd, TDC_RT_REMOVE_UNUSED, > - table->db.str, > - table->table_name.str); > - > - /* Return true if we found any open connections */ > - DBUG_RETURN(res); > + DBUG_RETURN(close_cached_tables(thd, argument.tables, true, > + thd->variables.lock_wait_timeout)); are you sure you want to do it when argument.tables is empty? It'll do purge_tables() which seems to be a bit extreme for DROP SERVER > } > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] df4db9a1ea1: Yet less TDC hash lookups
by Sergei Golubchik 02 Apr '20

02 Apr '20

Hi, Sergey! ok to push On Apr 02, Sergey Vojtovich wrote: > revision-id: df4db9a1ea1 (mariadb-10.5.0-76-gdf4db9a1ea1) > parent(s): 269793896c4 > author: Sergey Vojtovich <svoj(a)mariadb.org> > committer: Sergey Vojtovich <svoj(a)mariadb.org> > timestamp: 2019-12-27 20:37:11 +0400 > message: > > Yet less TDC hash lookups > > Let auto repair table and truncate table routines flush TABLE_SHARE > directly. > > Part of MDEV-17882 - Cleanup refresh version > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] f92e6520a62: tc_remove_all_unused_tables() cleanup
by Sergei Golubchik 02 Apr '20

02 Apr '20

Hi, Sergey! On Apr 02, Sergey Vojtovich wrote: > revision-id: f92e6520a62 (mariadb-10.5.0-74-gf92e6520a62) > parent(s): df0de9c652d > author: Sergey Vojtovich <svoj(a)mariadb.org> > committer: Sergey Vojtovich <svoj(a)mariadb.org> > timestamp: 2019-12-26 16:54:19 +0400 > message: > > tc_remove_all_unused_tables() cleanup > > As tc_purge() never marks share flushed, let tdc_remove_table() do it > directly. sure Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] df0de9c652d: Proper locking for mysql.gtid_slave_pos truncation
by Sergei Golubchik 02 Apr '20

02 Apr '20

Hi, Sergey! look ok On Apr 02, Sergey Vojtovich wrote: > revision-id: df0de9c652d (mariadb-10.5.0-73-gdf0de9c652d) > parent(s): eeba5b2b158 > author: Sergey Vojtovich <svoj(a)mariadb.org> > committer: Sergey Vojtovich <svoj(a)mariadb.org> > timestamp: 2019-12-26 16:50:51 +0400 > message: > > Proper locking for mysql.gtid_slave_pos truncation > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 269793896c4: Split tdc_remove_table()
by Sergei Golubchik 02 Apr '20

02 Apr '20

Hi, Sergey! ok to push On Apr 02, Sergey Vojtovich wrote: > revision-id: 269793896c4 (mariadb-10.5.0-75-g269793896c4) > parent(s): f92e6520a62 > author: Sergey Vojtovich <svoj(a)mariadb.org> > committer: Sergey Vojtovich <svoj(a)mariadb.org> > timestamp: 2019-12-27 17:14:57 +0400 > message: > > Split tdc_remove_table() > > TDC_RT_REMOVE_ALL -> tdc_remove_table(). Some occurrences replaced with > TDC_element::flush() (whenver TABLE_SHARE is available). > > TDC_RT_REMOVE_NOT_OWN[_KEEP_SHARE] -> TDC_element::flush(). These modes > assume that current thread owns TABLE_SHARE reference, which means we can > avoid hash lookup and flush unused TABLE instances directly. > > TDC_RT_REMOVE_UNUSED -> TDC_element::flush_unused(). Only [ab]used by > mysql_admin_table() currently. Should be removed eventually. > > Part of MDEV-17882 - Cleanup refresh version > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] b62432fc245: MDEV-21117: --tc-heuristic-recover=rollback is not replication safe
by sujatha 01 Apr '20

01 Apr '20

revision-id: b62432fc2459a312aaa0e27c0950de3420a34d53 (mariadb-10.1.43-96-gb62432fc245) parent(s): f9639c2d1a5e24f1a1533b6277fe7eca3aa3c3c0 author: Sujatha committer: Sujatha timestamp: 2020-04-01 23:17:10 +0530 message: MDEV-21117: --tc-heuristic-recover=rollback is not replication safe Problem: ======= When run after master server crash --tc-heuristic-recover=rollback produces inconsistent server state with binlog still containing transactions that were rolled back by the option. Such way recovered server may not be used for replication. Fix: === During --tc-heuristic-recover query the storage engine to get binlog file name and position corrensponding to the last committed transaction. If last_commit_pos_file < checkpoint_file then we need to start looking from checkpoint file. Look for first transactional event as this is the candidate for heuristic rollback. If a transactional event is found then this is used as reference for binlog truncation. Otherwise no truncation is needed. If last_commit_pos_file is equal to checkpoint_file then truncate binlog as per the last_commit_pos. If last_commit_file is not set then checkpoint_file is what we have. Look for first transactional event and consider this as truncate position. Once the binlog file name and position for truncation are identified start reading all the events beyond this position. Check if they are all transactional. If all are transactional truncate the binary log. If any non transactional events are found then truncation will fail. Adjust the global gtid state as per the truncated binlog. --- .../r/binlog_heuristic_rollback_active_log.result | 22 + .../binlog/r/binlog_truncate_multi_log.result | 26 + .../suite/binlog/r/binlog_truncate_none.result | 24 + .../t/binlog_heuristic_rollback_active_log.test | 76 +++ .../suite/binlog/t/binlog_truncate_multi_log.test | 88 ++++ .../suite/binlog/t/binlog_truncate_none.test | 65 +++ sql/log.cc | 565 ++++++++++++++++++++- sql/log.h | 7 +- storage/innobase/handler/ha_innodb.cc | 3 + storage/xtradb/handler/ha_innodb.cc | 3 + 10 files changed, 875 insertions(+), 4 deletions(-) diff --git a/mysql-test/suite/binlog/r/binlog_heuristic_rollback_active_log.result b/mysql-test/suite/binlog/r/binlog_heuristic_rollback_active_log.result new file mode 100644 index 00000000000..512644459f0 --- /dev/null +++ b/mysql-test/suite/binlog/r/binlog_heuristic_rollback_active_log.result @@ -0,0 +1,22 @@ +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); +RESET MASTER; +CREATE TABLE t ( f INT ) ENGINE=INNODB; +INSERT INTO t VALUES (10); +SET GLOBAL DEBUG_DBUG="d,simulate_crash_after_binlog_write"; +INSERT INTO t VALUES (20); +ERROR HY000: Lost connection to MySQL server during query +SELECT * FROM t; +f +10 +include/show_binlog_events.inc +Log_name Pos Event_type Server_id End_log_pos Info +master-bin.000001 # Gtid # # GTID #-#-# +master-bin.000001 # Query # # use `test`; CREATE TABLE t ( f INT ) ENGINE=INNODB +master-bin.000001 # Gtid # # BEGIN GTID #-#-# +master-bin.000001 # Query # # use `test`; INSERT INTO t VALUES (10) +master-bin.000001 # Xid # # COMMIT /* XID */ +DROP TABLE t; +show binlog events in 'master-bin.000003' limit 1,1; +Log_name Pos Event_type Server_id End_log_pos Info +master-bin.000003 # Gtid_list 1 # [0-1-2] diff --git a/mysql-test/suite/binlog/r/binlog_truncate_multi_log.result b/mysql-test/suite/binlog/r/binlog_truncate_multi_log.result new file mode 100644 index 00000000000..b6a3580f916 --- /dev/null +++ b/mysql-test/suite/binlog/r/binlog_truncate_multi_log.result @@ -0,0 +1,26 @@ +SET @old_max_binlog_size= @@global.max_binlog_size; +SET GLOBAL max_binlog_size= 4096; +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); +RESET MASTER; +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +SET DEBUG_SYNC= "commit_after_release_LOCK_log SIGNAL con1_ready WAIT_FOR con1_go"; +INSERT INTO t1 VALUES (1, REPEAT("x", 4100)); +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; +show binary logs; +Log_name File_size +master-bin.000001 # +master-bin.000002 # +SET GLOBAL DEBUG_DBUG="d,simulate_crash_after_binlog_write"; +SET DEBUG_SYNC= "now SIGNAL con1_go"; +ERROR HY000: Lost connection to MySQL server during query +"Only CREATE TABLE t1 should be present in the binary log" +include/show_binlog_events.inc +Log_name Pos Event_type Server_id End_log_pos Info +master-bin.000001 # Gtid # # GTID #-#-# +master-bin.000001 # Query # # use `test`; CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb +"Zero records should be there." +SELECT COUNT(*) FROM t1; +COUNT(*) +0 +DROP TABLE t1; diff --git a/mysql-test/suite/binlog/r/binlog_truncate_none.result b/mysql-test/suite/binlog/r/binlog_truncate_none.result new file mode 100644 index 00000000000..1eee3fb1cd1 --- /dev/null +++ b/mysql-test/suite/binlog/r/binlog_truncate_none.result @@ -0,0 +1,24 @@ +SET @old_max_binlog_size= @@global.max_binlog_size; +SET GLOBAL max_binlog_size= 4096; +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); +RESET MASTER; +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +INSERT INTO t1 VALUES (1, REPEAT("x", 4100)); +CREATE TABLE t2 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +show binary logs; +Log_name File_size +master-bin.000001 # +master-bin.000002 # +# Kill the server +"Zero records should be there." +SELECT COUNT(*) FROM t1; +COUNT(*) +1 +show binary logs; +Log_name File_size +master-bin.000001 # +master-bin.000002 # +master-bin.000003 # +master-bin.000004 # +DROP TABLE t1,t2; diff --git a/mysql-test/suite/binlog/t/binlog_heuristic_rollback_active_log.test b/mysql-test/suite/binlog/t/binlog_heuristic_rollback_active_log.test new file mode 100644 index 00000000000..1abb3b6495a --- /dev/null +++ b/mysql-test/suite/binlog/t/binlog_heuristic_rollback_active_log.test @@ -0,0 +1,76 @@ +# ==== Purpose ==== +# +# Test verifies the truncation of single binary log file. +# +# ==== Implementation ==== +# +# Steps: +# 0 - Create table t1 and insert a row. +# 1 - Insert an another row such that it gets written to binlog but commit +# in engine fails as server crashed at this point. +# 2 - Restart server with --tc-heuristic-recover=ROLLBACK +# 3 - Upon server start 'master-bin.000001' will be truncated to contain +# only the first insert +# +# ==== References ==== +# +# MDEV-21117: --tc-heuristic-recover=rollback is not replication safe + + +--source include/have_innodb.inc +--source include/have_log_bin.inc +--source include/have_debug.inc +--source include/have_binlog_format_statement.inc + +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); + +connect(master,localhost,root,,); +connect(master1,localhost,root,,); + +--connection master +RESET MASTER; +CREATE TABLE t ( f INT ) ENGINE=INNODB; +INSERT INTO t VALUES (10); + +--connection master1 +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +wait +EOF + +SET GLOBAL DEBUG_DBUG="d,simulate_crash_after_binlog_write"; +--error 2013 # CR_SERVER_LOST +INSERT INTO t VALUES (20); +--source include/wait_until_disconnected.inc + +--connection master +--source include/wait_until_disconnected.inc + +# +# Server restart +# +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: --tc-heuristic-recover=ROLLBACK +EOF +--source include/wait_until_disconnected.inc + +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: +EOF + +connection default; +--enable_reconnect +--source include/wait_until_connected_again.inc + +--connection master +--enable_reconnect +--source include/wait_until_connected_again.inc + +--connection master +SELECT * FROM t; +--source include/show_binlog_events.inc + +--connection master +DROP TABLE t; +let $binlog_file= query_get_value(SHOW MASTER STATUS, File, 1); +--source include/show_gtid_list.inc diff --git a/mysql-test/suite/binlog/t/binlog_truncate_multi_log.test b/mysql-test/suite/binlog/t/binlog_truncate_multi_log.test new file mode 100644 index 00000000000..5afa3b782cc --- /dev/null +++ b/mysql-test/suite/binlog/t/binlog_truncate_multi_log.test @@ -0,0 +1,88 @@ +# ==== Purpose ==== +# +# Test verifies truncation of multiple binary logs. +# +# ==== Implementation ==== +# +# Steps: +# 0 - Set max_binlog_size= 4096. Create a table and do an insert such that +# the max_binlog_size is reached and binary log gets rotated. +# 1 - Using debug simulation make the server crash at a point where the DML +# transaction is written to binary log but not committed in engine. +# 2 - At the time of crash two binary logs will be there master-bin.0000001 +# and master-bin.000002. +# 3 - Restart server with --tc-heuristic-recover=ROLLBACK +# 4 - Since the prepared DML in master-bin.000001 is rolled back the first +# binlog will be truncated prior to the DML and master-bin.000002 will be +# removed. +# +# ==== References ==== +# +# MDEV-21117: --tc-heuristic-recover=rollback is not replication safe + + +--source include/have_innodb.inc +--source include/have_log_bin.inc +--source include/have_debug.inc +--source include/have_binlog_format_row.inc + +SET @old_max_binlog_size= @@global.max_binlog_size; +SET GLOBAL max_binlog_size= 4096; + +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); + +RESET MASTER; + +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; + +# Hold insert after write to binlog and before "run_commit_ordered" in engine +connect(master1,localhost,root,,); +connect(master2,localhost,root,,); + +--connection master1 +SET DEBUG_SYNC= "commit_after_release_LOCK_log SIGNAL con1_ready WAIT_FOR con1_go"; +send INSERT INTO t1 VALUES (1, REPEAT("x", 4100)); + +--connection default +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; +--source include/show_binary_logs.inc +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +wait +EOF + +--connection master2 +SET GLOBAL DEBUG_DBUG="d,simulate_crash_after_binlog_write"; +SET DEBUG_SYNC= "now SIGNAL con1_go"; + +--connection master1 +--error 2013 # CR_SERVER_LOST +--reap +--source include/wait_until_disconnected.inc + +--connection master2 +--source include/wait_until_disconnected.inc + +# +# Server restart +# +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: --tc-heuristic-recover=ROLLBACK +EOF +--source include/wait_until_disconnected.inc + +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: +EOF + +connection default; +--enable_reconnect +--source include/wait_until_connected_again.inc +--echo "Only CREATE TABLE t1 should be present in the binary log" +--source include/show_binlog_events.inc + +--echo "Zero records should be there." +SELECT COUNT(*) FROM t1; + +DROP TABLE t1; + diff --git a/mysql-test/suite/binlog/t/binlog_truncate_none.test b/mysql-test/suite/binlog/t/binlog_truncate_none.test new file mode 100644 index 00000000000..9254cfd4d72 --- /dev/null +++ b/mysql-test/suite/binlog/t/binlog_truncate_none.test @@ -0,0 +1,65 @@ +# ==== Purpose ==== +# +# Test case verifies no binlog truncation happens when non transactional +# events are found in binlog after the last committed transaction. +# +# ==== Implementation ==== +# +# Steps: +# 0 - Set max_binlog_size= 4096. Create a table and do an insert such that +# the max_binlog_size is reached and binary log gets rotated. +# 1 - Create a table in newly created binary log and crash the server +# 2 - Restart server with --tc-heuristic-recover=ROLLBACK +# 3 - Recovery code will get the last committed DML specific postion and +# will try to check if binlog can be truncated upto this position. +# Since a DDL is present beyond this no truncation will happen. +# ==== References ==== +# +# MDEV-21117: --tc-heuristic-recover=rollback is not replication safe + + +--source include/have_innodb.inc +--source include/have_log_bin.inc +--source include/have_debug.inc +--source include/have_binlog_format_row.inc + +SET @old_max_binlog_size= @@global.max_binlog_size; +SET GLOBAL max_binlog_size= 4096; + +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); + +RESET MASTER; + +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +INSERT INTO t1 VALUES (1, REPEAT("x", 4100)); +CREATE TABLE t2 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +--source include/show_binary_logs.inc + +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +wait +EOF + +--source include/kill_mysqld.inc +--source include/wait_until_disconnected.inc + +# +# Server restart +# +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: --tc-heuristic-recover=ROLLBACK +EOF +--source include/wait_until_disconnected.inc + +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: +EOF + +connection default; +--enable_reconnect +--source include/wait_until_connected_again.inc + +--echo "Zero records should be there." +SELECT COUNT(*) FROM t1; +--source include/show_binary_logs.inc +DROP TABLE t1,t2; diff --git a/sql/log.cc b/sql/log.cc index 0efef6d1e29..8c997d547da 100644 --- a/sql/log.cc +++ b/sql/log.cc @@ -3164,6 +3164,7 @@ MYSQL_BIN_LOG::MYSQL_BIN_LOG(uint *sync_period) checksum_alg_reset(BINLOG_CHECKSUM_ALG_UNDEF), relay_log_checksum_alg(BINLOG_CHECKSUM_ALG_UNDEF), description_event_for_exec(0), description_event_for_queue(0), + last_commit_pos_offset(0), current_binlog_id(0) { /* @@ -3173,6 +3174,7 @@ MYSQL_BIN_LOG::MYSQL_BIN_LOG(uint *sync_period) before main(). */ index_file_name[0] = 0; + last_commit_pos_file[0]= 0; bzero((char*) &index_file, sizeof(index_file)); bzero((char*) &purge_index_file, sizeof(purge_index_file)); } @@ -7936,6 +7938,8 @@ MYSQL_BIN_LOG::trx_group_commit_leader(group_commit_entry *leader) mysql_mutex_unlock(&LOCK_log); DEBUG_SYNC(leader->thd, "commit_after_release_LOCK_log"); + DBUG_EXECUTE_IF("simulate_crash_after_binlog_write", + DBUG_SUICIDE();); /* Loop through threads and run the binlog_sync hook @@ -8964,7 +8968,7 @@ int TC_LOG_MMAP::open(const char *opt_name) { if (my_errno != ENOENT) goto err; - if (using_heuristic_recover()) + if (using_heuristic_recover(opt_name)) return 1; if ((fd= mysql_file_create(key_file_tclog, logname, CREATE_MODE, O_RDWR | O_CLOEXEC, MYF(MY_WME))) < 0) @@ -9497,14 +9501,40 @@ TC_LOG_MMAP tc_log_mmap; 1 heuristic recovery was performed */ -int TC_LOG::using_heuristic_recover() +int TC_LOG::using_heuristic_recover(const char* opt_name) { + LOG_INFO log_info; + int error; + if (!tc_heuristic_recover) return 0; sql_print_information("Heuristic crash recovery mode"); + if (ha_recover(0)) + { sql_print_error("Heuristic crash recovery failed"); + } + + /* + Check if TC log is referring to binary log not memory map. Ensure that + tc_heuristic_recover being ROLLBACK". If both match initiate binlog + truncation mechanism. + */ + if (!strcmp(opt_name,opt_bin_logname) && + tc_heuristic_recover == TC_HEURISTIC_RECOVER_ROLLBACK) + { + if ((error= mysql_bin_log.find_log_pos(&log_info, NullS, 1))) + { + if (error != LOG_INFO_EOF) + sql_print_error("find_log_pos() failed (error: %d)", error); + else + return 0; + } + if ((error= mysql_bin_log.heuristic_binlog_rollback())) + return error; + } + sql_print_information("Please restart mysqld without --tc-heuristic-recover"); return 1; } @@ -9512,6 +9542,535 @@ int TC_LOG::using_heuristic_recover() /****** transaction coordinator log for 2pc - binlog() based solution ******/ #define TC_LOG_BINLOG MYSQL_BIN_LOG +/** + Truncates the current binlog to specified position. Removes the rest of binlogs + which are present after this binlog file. + + @param truncate_file Holds the binlog name to be truncated + @param truncate_pos Position within binlog from where it needs to + truncated. + + @retval true ok + @retval false error + +*/ +bool MYSQL_BIN_LOG::truncate_and_remove_binlogs(const char *truncate_file, + my_off_t truncate_pos) +{ + int error= 0; +#ifdef HAVE_REPLICATION + LOG_INFO log_info; + THD *thd= current_thd; + my_off_t index_file_offset=0; + File file= -1; + bool included= (!(truncate_pos) ? 1 : 0); + MY_STAT s; + my_off_t binlog_size; + + if ((error= open_purge_index_file(TRUE))) + { + sql_print_error("MYSQL_BIN_LOG::purge_logs failed to sync the index file."); + goto err; + } + + if ((error=find_log_pos(&log_info, truncate_file, 1))) + goto err; + + if (included) + { + index_file_offset= log_info.index_file_start_offset; + if((error= register_purge_index_entry(log_info.log_file_name))) + { + sql_print_error("MYSQL_BIN_LOG::purge_logs failed to copy %s to register file.", + log_info.log_file_name); + goto err; + } + } + while (!(error= find_next_log(&log_info, 1))) + { + if (!index_file_offset) + index_file_offset= log_info.index_file_start_offset; + if((error= register_purge_index_entry(log_info.log_file_name))) + { + sql_print_error("MYSQL_BIN_LOG::purge_logs failed to copy %s to register file.", + log_info.log_file_name); + goto err; + } + } + + if (error != LOG_INFO_EOF) + { + sql_print_error("Error while looking for next binlog file to be truncated. " + "Error:%d", error); + goto err; + } + + if (!index_file_offset) + index_file_offset= log_info.index_file_start_offset; + + if ((error= sync_purge_index_file())) + { + sql_print_error("MYSQL_BIN_LOG::purge_logs failed to flush register file."); + goto err; + } + + DBUG_ASSERT(index_file_offset != 0); + // Trim index file + if (mysql_file_chsize(index_file.file, index_file_offset, '\n', MYF(MY_WME)) || + mysql_file_sync(index_file.file, MYF(MY_WME|MY_SYNC_FILESIZE))) + { + sql_print_error("Failed to trim binlog index file " + "when master server is recovering it."); + mysql_file_close(index_file.file, MYF(MY_WME)); + goto err; + } + + /* Reset data in old index cache */ + reinit_io_cache(&index_file, READ_CACHE, (my_off_t) 0, 0, 1); + + /* Read each entry from purge_index_file and delete the file. */ + if (is_inited_purge_index_file() && + (error= purge_index_entry(thd, NULL, TRUE))) + { + sql_print_error("MYSQL_BIN_LOG::purge_logs failed to process registered files" + " that would be purged."); + goto err; + } + + if (truncate_pos) + { + if ((file= mysql_file_open(key_file_binlog, truncate_file, + O_RDWR | O_BINARY, MYF(MY_WME))) < 0) + { + sql_print_error("Failed to open binlog file:%s for truncation.", + truncate_file); + error= 1; + goto err; + } + my_stat(truncate_file, &s, MYF(0)); + binlog_size= s.st_size; + + /* Change binlog file size to truncate_pos */ + if (mysql_file_chsize(file, truncate_pos, 0, MYF(MY_WME)) || + mysql_file_sync(file, MYF(MY_WME|MY_SYNC_FILESIZE))) + { + sql_print_error("Failed to trim the crashed binlog file:%s to size:%llu", + truncate_file, truncate_pos); + mysql_file_close(file, MYF(MY_WME)); + error= 1; + goto err; + } + else + { + sql_print_information("Crashed binlog file %s size is %llu, " + "but truncated to %llu.", + truncate_file, binlog_size, truncate_pos); + } + clear_inuse_flag_when_closing(file); + mysql_file_close(file, MYF(MY_WME)); + } + +err: + close_purge_index_file(); +#endif + return error; +} + +/** + Returns the checkpoint binlog file name. Iterate though the list of binary + logs. Move to the last one. Read the binlog checkpoint event within the last + log and return it. + + @param checkpoint_file Holds the binlog checkpoint file name. + + @retval 0 ok + @retval 1 error + +*/ +int TC_LOG_BINLOG::get_binlog_checkpoint_file(char* checkpoint_file) +{ + Log_event *ev= NULL; + bool binlog_checkpoint_found= false; + LOG_INFO log_info; + const char *errmsg; + IO_CACHE log; + File file; + Format_description_log_event fdle(BINLOG_VERSION); + char log_name[FN_REFLEN]; + int error=1; + + if (!fdle.is_valid()) + return 1; + + if ((error= find_log_pos(&log_info, NullS, 1))) + { + sql_print_error("find_log_pos() failed (error: %d)", error); + return error; + } + + // Move to the last binary log. + do + { + strmake_buf(log_name, log_info.log_file_name); + } while (!(error= find_next_log(&log_info, 1))); + + if (error != LOG_INFO_EOF) + { + sql_print_error("find_log_pos() failed (error: %d)", error); + return error; + } + if ((file= open_binlog(&log, log_name, &errmsg)) < 0) + { + sql_print_error("%s", errmsg); + return error; + } + while (!binlog_checkpoint_found && + (ev= Log_event::read_log_event(&log, 0, &fdle, + opt_master_verify_checksum)) && + ev->is_valid()) + { + enum Log_event_type typ= ev->get_type_code(); + if (typ == BINLOG_CHECKPOINT_EVENT) + { + uint dir_len; + Binlog_checkpoint_log_event *cev= (Binlog_checkpoint_log_event *)ev; + if (cev->binlog_file_len >= FN_REFLEN) + { + sql_print_error("Incorrect binlog checkpoint event with too " + "long file name found."); + delete ev; + ev= NULL; + end_io_cache(&log); + mysql_file_close(file, MYF(MY_WME)); + return 1; + } + else + { + dir_len= dirname_length(log_name); + strmake(strnmov(checkpoint_file, log_name, dir_len), + cev->binlog_file_name, FN_REFLEN - 1 - dir_len); + binlog_checkpoint_found= true; + } + } + delete ev; + ev= NULL; + } // End of while + end_io_cache(&log); + mysql_file_close(file, MYF(MY_WME)); + file= -1; + /* + Old binary log without checkpoint found, binlog truncation is not + possible. Hence return error. + */ + if (!binlog_checkpoint_found) + return 1; + else + { + // Look for binlog checkpoint file in binlog index file. + if (find_log_pos(&log_info, checkpoint_file, 1)) + { + sql_print_error("Binlog file '%s' not found in binlog index, needed " + "for recovery. Aborting.", checkpoint_file); + return 1; + } + } + return 0; +} + +/** + Given a binlog file name and position the following function will parse the + binlog for the presense of only transactional event groups beyond the + position to be truncated. If all are transactional then truncation is + considered to be safe. If any non transactional or DDL groups are found then + truncation is not safe. + + @param truncate_file Holds the name of binlog to be truncated + + @retval true Safe to truncate + @retval false Truncation is not safe + +*/ +bool MYSQL_BIN_LOG::is_binlog_truncate_safe(const char* truncate_file, my_off_t pos) +{ + Log_event *ev= NULL; + LOG_INFO log_info; + const char *errmsg; + IO_CACHE log; + File file; + Format_description_log_event fdle(BINLOG_VERSION); + int error; + + if (!fdle.is_valid()) + goto err; + + if ((error= find_log_pos(&log_info, truncate_file, 1))) + { + sql_print_error("find_log_pos() failed (error: %d)", error); + goto err; + } + + if ((file= open_binlog(&log, log_info.log_file_name, &errmsg)) < 0) + { + sql_print_error("Failed to open the binlog checkpoint file:%s for recovery." + "Error: %s", truncate_file, errmsg); + goto err; + } + if (pos) + my_b_seek(&log, pos); + + for (;;) + { + while ((ev= Log_event::read_log_event(&log, 0, &fdle, + opt_master_verify_checksum)) && + ev->is_valid()) + { +#ifdef HAVE_REPLICATION + enum Log_event_type typ= ev->get_type_code(); + if (typ == GTID_EVENT) + { + Gtid_log_event *gev= (Gtid_log_event *)ev; + if (!(gev->flags2 & Gtid_log_event::FL_TRANSACTIONAL)) + { + delete ev; + ev= NULL; + goto err; + } + } +#endif + delete ev; + ev= NULL; + }// End While + end_io_cache(&log); + mysql_file_close(file, MYF(MY_WME)); + file= -1; + + if ((error=find_next_log(&log_info, 1))) + { + if (error != LOG_INFO_EOF) + { + sql_print_error("Error reading binlog files during recovery. Aborting."); + goto err; + } + else + break; + } + + if ((file= open_binlog(&log, log_info.log_file_name, &errmsg)) < 0) + { + sql_print_error("%s", errmsg); + goto err; + } + } // End of for(;;) + return true; + +err: + if (file >= 0) + { + end_io_cache(&log); + mysql_file_close(file, MYF(MY_WME)); + file= -1; + } + return false; +} + + +/** + Truncates the binary log, according to the transactions that got rolled + back from engine, during heuristic-recover=ROLLBACK. Global GTID state is + adjusted as per the truncated binlog. + + Called from @c TC_LOG::using_heuristic_recover(const char* opt_name) + + @param opt_name The base name of binary log. + + @return indicates success or failure of binlog rollback + @retval 0 success + @retval 1 failure + +*/ +int TC_LOG_BINLOG::heuristic_binlog_rollback() +{ + int error=0; +#ifdef HAVE_REPLICATION + Log_event *ev= NULL; + char binlog_truncate_file_name[FN_REFLEN]; + char checkpoint_file[FN_REFLEN]; + my_off_t binlog_truncate_pos=0; + LOG_INFO log_info; + const char *errmsg; + IO_CACHE log; + File file; + Format_description_log_event fdle(BINLOG_VERSION); + bool found_truncate_pos= false; + bool is_safe= false; + my_off_t tmp_truncate_pos=0; + rpl_gtid last_gtid; + bool last_gtid_standalone= false; + bool last_gtid_valid= false; + + if (!fdle.is_valid()) + return 1; + + if ((error= get_binlog_checkpoint_file(checkpoint_file))) + return error; + + sql_print_information("Binlog checkpoint_file name:%s", checkpoint_file); + + if (last_commit_pos_file[0] != 0) + { + if (strcmp(last_commit_pos_file, checkpoint_file) > 0) + { + sql_print_error("Engine specific binlog_file:%s cannot be advanced than " + "checkpoint_file:%s", last_commit_pos_file, checkpoint_file); + return 1; + } + else + binlog_truncate_pos= last_commit_pos_offset; + } + /* + If last_commit_pos_file < checkpoint_file then we need to start looking + from checkpoint file. Look for first transactional event as this is the + candidate for heuristic rollback. If a transactional event is found then + this is used as reference for binlog truncation. Otherwise no truncation + is needed. + + If last_commit_pos_file is equal to checkpoint_file then truncate binlog as + per the last_commit_pos. + + If last_commit_file is not set then checkpoint_file is what we have. + Look for first trans event and consider this as truncate position. + */ + strmake_buf(binlog_truncate_file_name, checkpoint_file); + + error= read_state_from_file(); + if (error && error != 2) + { + sql_print_error("Failed to load global gtid binlog state from file"); + return error; + } + + if ((error= find_log_pos(&log_info, checkpoint_file, 1))) + { + sql_print_error("find_log_pos() failed (error: %d)", error); + return error; + } + + if ((file= open_binlog(&log, log_info.log_file_name, &errmsg)) < 0) + { + sql_print_error("Failed to open the binlog checkpoint file:%s for recovery." + "Error: %s", checkpoint_file, errmsg); + return 1; + } + + while (!found_truncate_pos && + ((ev= Log_event::read_log_event(&log, 0, &fdle, + opt_master_verify_checksum)) && + ev->is_valid())) + { + enum Log_event_type typ= ev->get_type_code(); + switch (typ) + { + case XID_EVENT: + if (ev->log_pos == binlog_truncate_pos) + { + found_truncate_pos= true; + } + break; + case GTID_LIST_EVENT: + { + Gtid_list_log_event *glev= (Gtid_list_log_event *)ev; + /* Initialise the binlog state from the Gtid_list event. */ + if (glev->count > 0 && + rpl_global_gtid_binlog_state.load(glev->list, glev->count)) + goto err; + } + break; + case GTID_EVENT: + if (!found_truncate_pos) + { + Gtid_log_event *gev= (Gtid_log_event *)ev; + + /* Update the binlog state with any GTID logged after Gtid_list. */ + last_gtid.domain_id= gev->domain_id; + last_gtid.server_id= gev->server_id; + last_gtid.seq_no= gev->seq_no; + last_gtid_standalone= + ((gev->flags2 & Gtid_log_event::FL_STANDALONE) ? true : false); + last_gtid_valid= true; + if ((binlog_truncate_pos == 0) && gev->flags2 & Gtid_log_event::FL_TRANSACTIONAL) + { + found_truncate_pos=true; + binlog_truncate_pos= tmp_truncate_pos; + } + } + break; + default: + /* Nothing. */ + break; + }// End switch + if (last_gtid_valid && + ((last_gtid_standalone && !ev->is_part_of_group(typ)) || + (!last_gtid_standalone && + (typ == XID_EVENT || + (typ == QUERY_EVENT && + (((Query_log_event *)ev)->is_commit() || + ((Query_log_event *)ev)->is_rollback())))))) + { + if (rpl_global_gtid_binlog_state.update_nolock(&last_gtid, false)) + goto err; + last_gtid_valid= false; + } + // Used to identify the last group specific end position. + tmp_truncate_pos= ev->log_pos; + delete ev; + ev= NULL; + }// End While + end_io_cache(&log); + mysql_file_close(file, MYF(MY_WME)); + file= -1; + + if (!found_truncate_pos) + { + if (binlog_truncate_pos != 0) + { + sql_print_error("Failed to locate event with offset:%llu in binlog:%s" + "Unable to truncate the binary log", binlog_truncate_pos, + binlog_truncate_file_name); + goto err; + } + else + return 0; // Nothing to truncate + } + + is_safe= is_binlog_truncate_safe(binlog_truncate_file_name, + binlog_truncate_pos); + + if (is_safe) + { + if (truncate_and_remove_binlogs(binlog_truncate_file_name, + binlog_truncate_pos)) + goto err; + } + if (write_state_to_file()) + { + sql_print_error("Failed to save binlog GTID state during heuristic " + "binlog rollback. "); + goto err; + } + return 0; + +err: + error= 1; + if (file >= 0) + { + end_io_cache(&log); + mysql_file_close(file, MYF(MY_WME)); + } + sql_print_error("Binlog truncation failed"); +#endif + return error; +} + int TC_LOG_BINLOG::open(const char *opt_name) { int error= 1; @@ -9526,7 +10085,7 @@ int TC_LOG_BINLOG::open(const char *opt_name) return 1; } - if (using_heuristic_recover()) + if (using_heuristic_recover(opt_name)) { mysql_mutex_lock(&LOCK_log); /* generate a new binlog to mask a corrupted one */ diff --git a/sql/log.h b/sql/log.h index 277e5c6f69c..c6179c27d8e 100644 --- a/sql/log.h +++ b/sql/log.h @@ -41,7 +41,7 @@ bool stmt_has_updated_non_trans_table(const THD* thd); class TC_LOG { public: - int using_heuristic_recover(); + int using_heuristic_recover(const char* opt_name); TC_LOG() {} virtual ~TC_LOG() {} @@ -692,6 +692,7 @@ class MYSQL_BIN_LOG: public TC_LOG, private MYSQL_LOG bool need_prepare_ordered, bool need_commit_ordered); int unlog(ulong cookie, my_xid xid); void commit_checkpoint_notify(void *cookie); + int heuristic_binlog_rollback(); int recover(LOG_INFO *linfo, const char *last_log_name, IO_CACHE *first_log, Format_description_log_event *fdle, bool do_xa); int do_binlog_recovery(const char *opt_name, bool do_xa_recovery); @@ -794,6 +795,10 @@ class MYSQL_BIN_LOG: public TC_LOG, private MYSQL_LOG int purge_first_log(Relay_log_info* rli, bool included); int set_purge_index_file_name(const char *base_file_name); int open_purge_index_file(bool destroy); + bool truncate_and_remove_binlogs(const char *truncate_file, + my_off_t truncate_pos); + int get_binlog_checkpoint_file(char* checkpoint_file); + bool is_binlog_truncate_safe(const char* checkpoint_file, my_off_t pos); bool is_inited_purge_index_file(); int close_purge_index_file(); int clean_purge_index_file(); diff --git a/storage/innobase/handler/ha_innodb.cc b/storage/innobase/handler/ha_innodb.cc index 4de2cdbeaec..42d655ee0e8 100644 --- a/storage/innobase/handler/ha_innodb.cc +++ b/storage/innobase/handler/ha_innodb.cc @@ -16493,6 +16493,9 @@ innobase_xa_recover( { DBUG_ASSERT(hton == innodb_hton_ptr); + mysql_bin_log.last_commit_pos_offset= trx_sys_mysql_bin_log_pos; + strmake_buf(mysql_bin_log.last_commit_pos_file, trx_sys_mysql_bin_log_name); + if (len == 0 || xid_list == NULL) { return(0); diff --git a/storage/xtradb/handler/ha_innodb.cc b/storage/xtradb/handler/ha_innodb.cc index 2aafb1a44ee..dec106cac54 100644 --- a/storage/xtradb/handler/ha_innodb.cc +++ b/storage/xtradb/handler/ha_innodb.cc @@ -17165,6 +17165,9 @@ innobase_xa_recover( { DBUG_ASSERT(hton == innodb_hton_ptr); + mysql_bin_log.last_commit_pos_offset= trx_sys_mysql_bin_log_pos; + strmake_buf(mysql_bin_log.last_commit_pos_file, trx_sys_mysql_bin_log_name); + if (len == 0 || xid_list == NULL) { return(0);

1 0

Re: [Maria-developers] 46c66698d63: MENT-566 jUnit xml fixes
by Sergei Golubchik 01 Apr '20

01 Apr '20

Hi, Rasmus! Please, see below On Mar 30, Rasmus Johansson wrote: > revision-id: 46c66698d63 (mariadb-10.2.31-189-g46c66698d63) > parent(s): 065a3d3eed7 > author: Rasmus Johansson <razze(a)iki.fi> > committer: Rasmus Johansson <razze(a)iki.fi> > timestamp: 2020-03-26 10:50:06 +0000 > message: > > MENT-566 jUnit xml fixes > > diff --git a/mysql-test/lib/mtr_report.pm b/mysql-test/lib/mtr_report.pm > index 3701ad79b15..1c6825d8fb7 100644 > --- a/mysql-test/lib/mtr_report.pm > +++ b/mysql-test/lib/mtr_report.pm > @@ -35,6 +37,7 @@ use My::Platform; > use POSIX qw[ _exit ]; > use IO::Handle qw[ flush ]; > use mtr_results; > +use Cwd 'abs_path'; this seems to be unused now > > use Term::ANSIColor; > > @@ -60,6 +63,8 @@ our $verbose; > our $verbose_restart= 0; > our $timer= 1; > > +my $xml_report = ""; this could be moved down inside if ($::opt_xml_report) > + > sub report_option { > my ($opt, $value)= @_; > > @@ -402,6 +409,92 @@ sub mtr_report_stats ($$$$) { > print "All $tot_tests tests were successful.\n\n"; > } > > + if ($::opt_xml_report) { > + my @sorted_tests = sort {$a->{'name'} cmp $b->{'name'}} @$tests; > + my $last_suite = ""; > + my $current_suite = ""; > + my $timest = isotime(time); > + my %suite_totals; > + my %suite_time; > + my %suite_tests; > + my %suite_failed; > + my %suite_disabled; > + my %suite_skipped; > + my $host = hostname; > + my $suiteNo = 0; > + > + # loop through test results to count totals > + foreach my $test ( @sorted_tests ) { > + $current_suite = $test->{'suite'}->{'name'}; > + > + if ($test->{'timer'} eq "") { > + $test->{'timer'} = 0; > + } > + > + # $suite_totals{$current_suite . "_time"} = $suite_totals{$current_suite . "_time"} + $test->{'timer'}; > + # $suite_totals{$current_suite . "_tests"} = $suite_totals{$current_suite . "_tests"} + 1; don't forget to remove all these commented-out lines before pushing > + $suite_time{$current_suite} = $suite_time{$current_suite} + $test->{'timer'}; > + $suite_tests{$current_suite} = $suite_tests{$current_suite} + 1; > + > + if ($test->{'result'} eq "MTR_RES_FAILED") { > + # $suite_totals{$current_suite . "_failed"} = $suite_totals{$current_suite . "_failed"} + 1; > + $suite_failed{$current_suite} = $suite_failed{$current_suite} + 1; > + } elsif ($test->{'result'} eq "MTR_RES_SKIPPED" && $test->{'disable'}) { > + # $suite_totals{$current_suite . "_disabled"} = $suite_totals{$current_suite . "_disabled"} + 1; > + $suite_disabled{$current_suite} = $suite_disabled{$current_suite} + 1; > + } elsif ($test->{'result'} eq "MTR_RES_SKIPPED") { > + $suite_skipped{$current_suite} = $suite_skipped{$current_suite} + 1; > + } > + > + $suite_totals{"all_time"} = $suite_totals{"all_time"} + $test->{'timer'}; > + } > + > + # generate xml > + $xml_report = "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n"; > + $xml_report = $xml_report . "<testsuites disabled=\"" . $tot_disabled . "\" errors=\"\" failures=\"" . $tot_failed . "\" name=\"\" tests=\"" . $tot_tests . "\" time=\"" . $suite_totals{"all_time"} . "\">\n"; I find it more concise to write $xml_report .= "<testsuites ..." instead of $xml_report = $xml_report . "<testsuites ..." and also it's easier to read, because the first form directly means "append to the variable", while the second means "concatenate two strings and store the result" and by using the target variable also as first string one gets the append effect. > + > + foreach my $test ( @sorted_tests ) { > + $current_suite = $test->{'suite'}->{'name'}; > + > + if ($current_suite ne $last_suite) { > + if ($last_suite ne "") { > + $xml_report = $xml_report . "\t</testsuite>\n"; > + $suiteNo++; > + } > + > + $xml_report = $xml_report . "\t<testsuite disabled=\"" . $suite_disabled{$current_suite} . "\" errors=\"\" failures=\"" . $suite_failed{$current_suite} . "\" hostname=\"" . $host . "\" id=\"" . $suiteNo . "\" name=\"" . $current_suite . "\" package=\"\" skipped=\"" . $suite_skipped{$current_suite} . "\" tests=\"" . $suite_tests{$current_suite} . "\" time=\"" . $suite_time{$current_suite} . "\" timestamp=\"" . $timest . "\">\n"; why do you always avoid variable interapolation? The above line could've been written as $xml_report .= qq(\t<testsuite disabled="$suite_disabled{$current_suite}" errors="" failures="$suite_failed{$current_suite}" hostname="$host" id="$suiteNo" name="$current_suite" package="" skipped="$suite_skipped{$current_suite}" tests="$suite_tests{$current_suite}" time="$suite_time{$current_suite}" timestamp="$timest">\n); which is notably shorter and more readable > + $last_suite = $current_suite; > + } > + > + $xml_report = $xml_report . "\t\t<testcase assertions=\"\" classname=\"" . $current_suite . "\" name=\"$test->{'name'}\" status=\"$test->{'result'}\" time=\"" . $test->{timer} . "\""; > + > + if ($test->{'result'} eq "MTR_RES_FAILED") { > + $xml_report = $xml_report . ">\n\t\t\t<failure message=\"\" type=\"" . $test->{'result'} . "\">\n<![CDATA[" . $test->{'logfile'} . "]]>\n\t\t\t</failure>\n\t\t</testcase>\n"; > + } elsif ($test->{'result'} eq "MTR_RES_SKIPPED" && $test->{'disable'}) { > + $xml_report = $xml_report . ">\n\t\t\t<failure message=\"disabled\" type=\"" . $test->{'result'} . "\"/>\n\t\t</testcase>\n"; > + } elsif ($test->{'result'} eq "MTR_RES_SKIPPED") { > + $xml_report = $xml_report . ">\n\t\t\t<failure message=\"skipped\" type=\"" . $test->{'result'} . "\"/>\n\t\t</testcase>\n"; > + } else { > + $xml_report = $xml_report . " />\n"; > + } > + > + # check if last test > + if ($test eq @sorted_tests[$#sorted_tests]) { > + $xml_report = $xml_report . "\t</testsuite>\n"; > + } better to print this after the loop, then to do a check for every test. it will not be wrong, because your condition $test eq @sorted_tests[$#sorted_tests] can only be true once, for the very last test. So you can as well remove the whole if() and put the unconditional $xml_report .= "\t</testsuite>\n"; after the loop. > + } > + > + $xml_report = $xml_report . "</testsuites>\n"; or just combine it with this one: $xml_report .= "\t</testsuite>\n</testsuites>\n"; > + > + # save to file > + # my $xml_file = $::opt_vardir . "/log/" . $::opt_xml_report; > + my $xml_file = $::opt_xml_report; > + > + open XML_FILE, ">" . $xml_file; error checking? Even if it's just open XML_FILE, ">" , $xml_file or die "$!"; and always use the 3-form open(). Where ">" is a separate argument, not concatenated to the file name (see how I wrote it above) > + print XML_FILE $xml_report; > + close XML_FILE; > + } > + > if (@$extra_warnings) > { > print <<MSG; > diff --git a/mysql-test/mysql-test-run.pl b/mysql-test/mysql-test-run.pl > index ef37cb4144d..298ba0015a3 100755 > --- a/mysql-test/mysql-test-run.pl > +++ b/mysql-test/mysql-test-run.pl > @@ -80,6 +80,7 @@ use lib "lib"; > > use Cwd ; > use Cwd 'realpath'; > +use Cwd 'abs_path'; this seems to be unused now > use Getopt::Long; > use My::File::Path; # Patched version of File::Path > use File::Basename; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 2

Re: [Maria-developers] 7740cb24572: Don't use plugin->data for storage engine plugins
by Sergei Golubchik 31 Mar '20

31 Mar '20

Hi, Sergey! On Mar 31, Sergey Vojtovich wrote: > revision-id: 7740cb24572 (mariadb-10.4.4-472-g7740cb24572) > parent(s): da6d7f72b0a > author: Sergey Vojtovich <svoj(a)mariadb.org> > committer: Sergey Vojtovich <svoj(a)mariadb.org> > timestamp: 2019-11-15 15:23:42 +0400 > message: > > Don't use plugin->data for storage engine plugins > > Use plugin->plugin->info->hton instead. > plugin_data() replaced with plugin_hton(). > plugin_hton() must never return NULL anymore and is only good to be called > against plugins in PLUGIN_IS_READY state. Why is that? Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

[Maria-developers] GSOC 2020 - MDEV-11263 Aggregate Window Functions
by Tavneet Sarna 30 Mar '20

30 Mar '20

Hi everyone, As part of GSOC 2020, this is one of the two projects I am interested in pursuing. As guided by last years' comments by Varun and Vicențiu, I have set up a debugger and corresponding breakpoints in the Item_sum_sum::add and Item_sum_sp::add for a custom sum aggregate function to understand the code flow. I had a couple of queries regarding the same: 1. In *do_add *from decimal.c, there are three parts with comments - /* part 1 - MY_MAX(frac) ... min (frac) */, /* part 2 - MY_MIN(frac) ... MY_MIN(intg) */. Can someone please elaborate on what do the comments mean ? 2. In *Item_sum_sum::add_helper*, there is an unlikely branch for variable direct_added. Can someone please give an idea about when will direct added be true ? In fact in all the uses for direct_added, it is always in an unlikely branch in Item_sum.cc. Regards Tavneet

2 1

[Maria-developers] Guidance regarding GSoC Projects.
by Arhant Jain 27 Mar '20

27 Mar '20

Hello Everyone, I'm interested in the work associated with these projects. I tried to explore them on my level. It will be very helpful if the concerned developers can direct some more information regarding these projects. 1. Switching off/on separate tests in mysql-test-run suite 2. Add a bitmap for NULL/empty data 3. Speed up RowGroup data access methods reducing level of indirection Thanks Arhant

1 0

[Maria-developers] Regarding GSoc :ADD FULL OUTER JOIN
by Jeeukrishnan Kayshyap. 26 Mar '20

26 Mar '20

Hi ,I am a GSoC aspirant . I am trying to solve https://jira.mariadb.org/browse/MDEV-13648 . Firstly , I discussed a new algorithm with Igor Babev sir , but it was wrong as it will create space problem. Now , I am planning to add some code in Sql_yacc.yy : /*full join*/ after the right join where /*full join */ will contain : 1.left join implementation (similar to the implementation of left join in sql_yacc.yy (line no. 11956-11950)) The result table will be in a temporary table ->tab1. 2.Use of right join doing the swapping of table (similar to right join in this part ;(line no. 11992-12030)) the result of this will be stored in another temporary table ->tab2 3.Union all clause: implementation of union all clause to combine tab1 and tab2. So , Oleksandr Byelkin sir replied to me that , addition of UNION will make it more complicated.it could be done with less effort than adding full UNION. But at present I am not able to get any idea how to reduce the complexity or any kind of new idea to implement it . Please guide me in the right direction and help me to solve this issue.

2 4

Re: [Maria-developers] fc90a39426c: MDEV-20604: Duplicate key value is silently truncated to 64 characters in print_keydup_error
by Sergei Golubchik 25 Mar '20

25 Mar '20

Hi, Oleksandr! Looks good. Thanks for tests. Ok to push. One comment, see below. On Mar 25, Oleksandr Byelkin wrote: > revision-id: fc90a39426c (mariadb-10.2.31-87-gfc90a39426c) > parent(s): fb74de97287 > author: Oleksandr Byelkin <sanja(a)mariadb.com> > committer: Oleksandr Byelkin <sanja(a)mariadb.com> > timestamp: 2020-03-24 10:34:14 +0100 > message: > > MDEV-20604: Duplicate key value is silently truncated to 64 characters in print_keydup_error > > Added indication of truncated string for "s" and "M" formats > > diff --git a/unittest/mysys/my_vsnprintf-t.c b/unittest/mysys/my_vsnprintf-t.c > index 6ba0a42cf7e..872e88ddd7e 100644 > --- a/unittest/mysys/my_vsnprintf-t.c > +++ b/unittest/mysys/my_vsnprintf-t.c > @@ -168,14 +184,14 @@ int main(void) > test1("M with positional: 0 \"Internal error/check (Not system error)\"", > "M with positional: %1$M", 0); > > - test1("M with width: 0 \"Internal error/ch", > + test1("M with width: 0 \"Internal error...", > "M with width: %.20M", 0); > - test1("M with width positional: 0 \"Internal error/ch", > + test1("M with width positional: 0 \"Internal error...", > "M with width positional: %2$.*1$M", 20, 0); > > - test_w_len("M small buf: 0 \"In", > + test_w_len("M small buf: 0 \"..", > 19, "M small buf: %M", 0); > - test_w_len("M small buf positional: 0 \"In", > + test_w_len("M small buf positional: 0 \"..", > 30, "M small buf positional: %1$M", 0); > > return exit_status(); We didn't discuss truncation when the buffer is too small, only truncation by width. It's not at all clear why truncation by small buffer should also use ellipsis. But it's not clear that it shouldn't either. I don't have a strong opinion in favor of either, so feel free to keep it your way. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] GSOC 2020 - MDEV-383
by Tavneet Sarna 25 Mar '20

25 Mar '20

Hi, I was looking into MDEV-383 <https://jira.mariadb.org/browse/MDEV-383> - "*Evaluate subquery predicates earlier or later depending on their SELECTIVITY*" as a possible GSOC 2020 project and I have a few queries about the same. 1. "There are a lot of subquery conditions out there that are inexpensive to evaluate and have good selectivity." - How is the cost of a subquery predicate calculated ? 2. One of the assumptions in MDEV-83 <https://jira.mariadb.org/browse/MDEV-83> was that "selectivity of a subquery predicate cannot be estimated during optimization." - what has changed since that we are looking into estimating selectivity ( or maybe my understanding of the task is incorrect so I would be grateful if you could just point me to any resources regarding this). 3. Igor had responded on the jira - "for starters, you could introduce an option that would allow not to push correlated subquery predicates to joined tables." - did you mean a command line argument/option and I don't get how this would help with the main task. 4. Also, can anyone please point me to the classes where predicate cost and join plans are evaluated ? Apologies if the queries are a bit rudimentary. Regards Tavneet

1 0

[Maria-developers] Regarding GSoc :ADD FULL OUTER JOIN
by Jeeukrishnan Kayshyap. 24 Mar '20

24 Mar '20

HI , to implement FULL OUTER JOIN between T1 and T2 can we do "T1 Left Outer Join T2 union all T2 Anti Join T1" ?? Is this method already tried ? Please guide me to solve this issue and if possible someone please reply to my last email.

1 0

[Maria-developers] MDEV-14296: Moving wsrep_sst_common to /usr/libexec
by Vicențiu Ciorbaru 23 Mar '20

23 Mar '20

Hi Otto! I am doing a push towards cleaning up as much as possible from our packaging issues. Starting with the older ones: *Background:* wsrep_sst_common.sh is a script that should never be run standalone, it is rather a helper script used for other wsrep_* scripts. As such, it *does not really belong in /usr/bin*. See MDEV-14296 [1] for more details. <https://jira.mariadb.org/browse/MDEV-14296> It looks like rpm packagers - read Fedora - [1a] require us to ship wsrep_sst_common in */usr/libexec* (or alternatively /usr/share, but /usr/libexec makes much more sense semantically to me, after reading the FHS [2]). There is PR #603 [3] from Daniel Black, which has some missing changes, which I've done myself, namely updating the debian/*.install files to put the file in the correct location and a change in path for our apparmor profile [4] [5]. *My question for you is*: Is this acceptable to do in 10.1 (merge PR 603 + commits [4] [5] into 10.1) from a packaging point of view? The debian packaging policy here [6], allows one to follow FSH version 3.0 and does not mention any exception for libexec. <https://www.debian.org/doc/debian-policy/ch-opersys.html> I have tested this and I have not encountered any regressions while upgrading and the script seems to function as intended (although I am not an expert DBA to make use of wsrep_* scripts). Thanks, Vicențiu [1] https://jira.mariadb.org/browse/MDEV-14296 [1a] https://src.fedoraproject.org/rpms/mariadb/blob/ecb40d449c382ea7e4052c75d5d… [2] https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch04s07.html [3] https://github.com/MariaDB/server/pull/603 [4] https://github.com/MariaDB/server/commit/8042265659656c312f39304a5dc60de708… [5] https://github.com/MariaDB/server/commit/dbed51d86e4aae9b8a54d6b5aa31c974fe… [6] https://www.debian.org/doc/debian-policy/ch-opersys.html

1 0

[Maria-developers] Regarding GSoc :ADD FULL OUTER JOIN
by Jeeukrishnan Kayshyap. 23 Mar '20

23 Mar '20

Hi , I studied the join_table part of sql_yacc.yy and the the convert_right_join() part from sql_parse.cc . Sir to add the full outer join : if we add a /*Full outer join variant */ , where first we will do left outer join and then right outer join in that , after that we can use the "UNION" part in /*Full outer join variant */ to join the left outer join and right outer join table to get the full outer join table. Will this approach gonna work Sir? please guide me for this issue.Is it completely wrong or need some modification?

2 1

[Maria-developers] Regarding GSoC : ADD FULL OUTER JOIN
by Jeeukrishnan Kayshyap. 23 Mar '20

23 Mar '20

What are the other thing we need to consider if we include *st_select_lex::convert_full_join() similar to *st_select_lex::convert_right_join() ?

1 0

Re: [Maria-developers] 4b142536a24: MDEV-20604: Duplicate key value is silently truncated to 64 characters in print_keydup_error
by Sergei Golubchik 22 Mar '20

22 Mar '20

Hi, Oleksandr! On Mar 22, Oleksandr Byelkin wrote: > revision-id: 4b142536a24 (mariadb-10.2.31-56-g4b142536a24) > parent(s): ed21202a14e > author: Oleksandr Byelkin <sanja(a)mariadb.com> > committer: Oleksandr Byelkin <sanja(a)mariadb.com> > timestamp: 2020-03-18 08:25:13 +0100 > message: > > MDEV-20604: Duplicate key value is silently truncated to 64 characters in print_keydup_error > > Added indication of truncated string with special width/precission suffix ':'. The comment looks wrong :) > diff --git a/client/mysql_upgrade.c b/client/mysql_upgrade.c > index 0bbb8a149ff..003701514cf 100644 > --- a/client/mysql_upgrade.c > +++ b/client/mysql_upgrade.c > @@ -501,10 +499,10 @@ static void find_tool(char *tool_executable_name, const char *tool_name, > last_fn_libchar -= 6; > } > > - len= (int)(last_fn_libchar - self_name); > - > - my_snprintf(tool_executable_name, FN_REFLEN, "%.*s%c%s", > - len, self_name, FN_LIBCHAR, tool_name); > + last_fn_libchar[0]= 0; > + my_snprintf(tool_executable_name, FN_REFLEN, "%s%c%s", > + self_name, FN_LIBCHAR, tool_name); > + last_fn_libchar[0]= FN_LIBCHAR; Okay, but why not to use %b here? > } > > if (opt_verbose) > diff --git a/client/mysqltest.cc b/client/mysqltest.cc > index 60a203ccedd..045ab566fdb 100644 > --- a/client/mysqltest.cc > +++ b/client/mysqltest.cc > @@ -9534,6 +9533,7 @@ int main(int argc, char **argv) > case Q_LET: do_let(command); break; > case Q_EVAL_RESULT: > die("'eval_result' command is deprecated"); > + break; // never called but keep compiler calm wouldn't it be better to specify __attribute__ ((noreturn)) for die()? > case Q_EVAL: > case Q_EVALP: > case Q_QUERY_VERTICAL: > diff --git a/strings/my_vsnprintf.c b/strings/my_vsnprintf.c > index ad3517e4252..9ae10f337cc 100644 > --- a/strings/my_vsnprintf.c > +++ b/strings/my_vsnprintf.c > @@ -28,7 +28,9 @@ > #define LENGTH_ARG 1 > #define WIDTH_ARG 2 > #define PREZERO_ARG 4 > -#define ESCAPED_ARG 8 > +#define ESCAPED_ARG 8 > +// Show truncation of string argument > +#define SH_TRUNC_ARG 16 you don't really need a flag, because process_str_arg should always do it. By the way, in your code it doesn't. It doesn't print dots for %M and for %1$s. > > typedef struct pos_arg_info ARGS_INFO; > typedef struct print_info PRINT_INFO; > @@ -198,18 +199,42 @@ static char *process_str_arg(CHARSET_INFO *cs, char *to, const char *end, > size_t width, char *par, uint print_type) > { > int well_formed_error; > - size_t plen, left_len= (size_t) (end - to) + 1; > + uint dots= 0; > + size_t plen, left_len= (size_t) (end - to) + 1, slen=0; > if (!par) > par = (char*) "(null)"; > > - plen= strnlen(par, width); > + plen= slen= strnlen(par, width + 1); > + if (plen > width) > + plen= width; > if (left_len <= plen) > plen = left_len - 1; > + if ((slen > plen) && (print_type & SH_TRUNC_ARG)) > + { > + if (plen < 3) > + { > + dots= plen; > + plen= 0; > + } > + else > + { > + plen-= 3; > + dots= 3; > + } > + } > + > plen= my_well_formed_length(cs, par, par + plen, width, &well_formed_error); > if (print_type & ESCAPED_ARG) > to= backtick_string(cs, to, end, par, plen, '`'); > else > to= strnmov(to,par,plen); > + > + if (dots) > + { > + for (; dots; dots--) > + *(to++)= '.'; > + *(to)= 0; > + } This should look strange when truncating a quoted string. See the test case below > return to; > } > > diff --git a/unittest/mysys/my_vsnprintf-t.c b/unittest/mysys/my_vsnprintf-t.c > index 6ba0a42cf7e..e8831f7c1f4 100644 > --- a/unittest/mysys/my_vsnprintf-t.c > +++ b/unittest/mysys/my_vsnprintf-t.c > @@ -99,12 +99,12 @@ int main(void) > test1("Width is ignored for strings <x> <y>", > "Width is ignored for strings <%04s> <%5s>", "x", "y"); > > - test1("Precision works for strings <abcde>", > + test1("Precision works for strings <ab...>", > "Precision works for strings <%.5s>", "abcdef!"); > > - test1("Flag '`' (backtick) works: `abcd` `op``q` (mysql extension)", This > - "Flag '`' (backtick) works: %`s %`.4s (mysql extension)", This > - "abcd", "op`qrst"); > + test1("Flag '`' (backtick) works: `abcd` `op``q`... (mysql extension)", > + "Flag '`' (backtick) works: %`s %`.7s (mysql extension)", > + "abcd", "op`qrstuuuuuuuuu"); Here, the test case with a quoted string. First, you print 10 characters instead of 7. Second, you put dots after the closing backtick, which looks confusing, the truncation happens before the backtick. Third, as you can see, you cannot use strlen() for a string length because there can be backticks inside the string. perhaps it would be simpler to move ... inside backtick_string() > > test1("Length modifiers work: 1 * -1 * 2 * 3", > "Length modifiers work: %d * %ld * %lld * %zd", 1, -1L, 2LL, (size_t)3); > @@ -125,7 +125,7 @@ int main(void) > test1("Asterisk '*' as a width works: < 4>", > "Asterisk '*' as a width works: <%*d>", 5, 4); > > - test1("Asterisk '*' as a precision works: <qwerty>", > + test1("Asterisk '*' as a precision works: <qwe...>", > "Asterisk '*' as a precision works: <%.*s>", 6, "qwertyuiop"); please add tests for ...truncation in %M and %1$s > > test1("Positional arguments for a width: < 4>", > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] cf70893ac9d: MDEV-21303 Make executables MariaDB named
by Sergei Golubchik 18 Mar '20

18 Mar '20

Hi, Rasmus! See my comments/questions below: On Mar 16, Rasmus Johansson wrote: > revision-id: cf70893ac9d (mariadb-10.5.0-391-gcf70893ac9d) > parent(s): 56402e84b5b > author: Rasmus Johansson <razze(a)iki.fi> > committer: Rasmus Johansson <razze(a)iki.fi> > timestamp: 2020-03-16 11:10:25 +0000 > message: > > MDEV-21303 Make executables MariaDB named > > To change all executables to have a mariadb name I had to: > - Do name changes in every CMakeLists.txt that produces executables > - CREATE_MARIADB_SYMLINK was removed and GET_SYMLINK added by Wlad to reuse the function in other places also > - The scripts/CMakeLists.txt could make use of GET_SYMLINK instead of introducing redundant code, but I thought I'll leave that for next release > - A lot of changes to debian/.install and debian/.links files due to swapping of real executable and symlink. I did not however change the name of the manpages, so the real name is still mysql there and mariadb are symlinks. > - The Windows part needed a change now when we made the executables mariadb -named. MSI (and ZIP) do not support symlinks and to not break backward compatibility we had to include mysql named binaries also. Done by Wlad > > diff --git a/cmake/mysql_add_executable.cmake b/cmake/mysql_add_executable.cmake > index eec370d51af..f4d71ae9cef 100644 > --- a/cmake/mysql_add_executable.cmake > +++ b/cmake/mysql_add_executable.cmake > @@ -48,7 +48,7 @@ FUNCTION (MYSQL_ADD_EXECUTABLE) > ENDIF() > > IF (ARG_WIN32) > - SET(WIN32 WIN32) > + SET(WIN32 ARG_WIN32) This looks wrong, see below > ELSE() > UNSET(WIN32) > ENDIF() > @@ -62,6 +62,7 @@ FUNCTION (MYSQL_ADD_EXECUTABLE) > ELSE() > UNSET(EXCLUDE_FROM_ALL) > ENDIF() > + > ADD_EXECUTABLE(${target} ${WIN32} ${MACOSX_BUNDLE} ${EXCLUDE_FROM_ALL} ${sources}) here it'll pass ARG_WIN32 to ADD_EXECUTABLE. But ADD_EXECUTABLE takes an optional WIN32 keyword, not ARG_WIN32. > > # tell CPack where to install > @@ -79,16 +80,49 @@ FUNCTION (MYSQL_ADD_EXECUTABLE) > IF (COMP MATCHES ${SKIP_COMPONENTS}) > RETURN() > ENDIF() > + > IF (WITH_STRIPPED_CLIENT AND NOT target STREQUAL mysqld) this should be `STREQUAL mariadbd` now, I believe. > INSTALL(CODE "SET(CMAKE_INSTALL_DO_STRIP 1)" COMPONENT ${COMP}) > SET(reset_strip ON) > ENDIF() > - MYSQL_INSTALL_TARGETS(${target} DESTINATION ${ARG_DESTINATION} COMPONENT ${COMP}) > + > + IF(NOT ${mariadbname} STREQUAL "") where is mariadbname set? > + MYSQL_INSTALL_TARGETS(${mariadbname} DESTINATION ${ARG_DESTINATION} COMPONENT ${COMP}) > + ELSE() > + MYSQL_INSTALL_TARGETS(${target} DESTINATION ${ARG_DESTINATION} COMPONENT ${COMP}) > + ENDIF() > + > IF (reset_strip) > INSTALL(CODE "SET(CMAKE_INSTALL_DO_STRIP 0)" COMPONENT ${COMP}) > ENDIF() > ENDIF() > > - # create mariadb named symlink > - CREATE_MARIADB_SYMLINK(${target} ${ARG_DESTINATION} ${COMP}) > + # create MySQL named "legacy links" > + > + # Windows note: > + # Here, hardlinks are used, because cmake can't install symlinks. > + # In packages, there are won't be links, just copies. better to put the comment below in the ELSE branch, not here, far from the code that it is supposed to explain. > + GET_SYMLINK(${target} link) > + IF(link) > + IF(UNIX) > + ADD_CUSTOM_COMMAND(TARGET ${target} POST_BUILD > + COMMAND ${CMAKE_COMMAND} -E create_symlink > + ${target} ${link} > + COMMENT "Creating ${link} link" > + WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/${CMAKE_CFG_INTDIR}) > + INSTALL(PROGRAMS > + ${CMAKE_CURRENT_BINARY_DIR}/${CMAKE_CFG_INTDIR}/${link} > + DESTINATION > + ${ARG_DESTINATION} > + COMPONENT ${COMP}) > + ELSE() > + SET(link ${link}.exe) > + ADD_CUSTOM_COMMAND(TARGET ${target} POST_BUILD > + COMMAND cmake -E remove -f ${link} > + COMMAND mklink /H ${link} $<TARGET_FILE_NAME:${target}> > + COMMENT "Creating ${link} link" > + WORKING_DIRECTORY $<TARGET_FILE_DIR:${target}>) > + INSTALL(PROGRAMS $<TARGET_FILE_DIR:${target}>/${link} DESTINATION ${ARG_DESTINATION} COMPONENT ${COMP}) > + ENDIF() > + ENDIF() > ENDFUNCTION() > diff --git a/cmake/symlinks.cmake b/cmake/symlinks.cmake > index ec638bc82de..e040ff19f77 100644 > --- a/cmake/symlinks.cmake > +++ b/cmake/symlinks.cmake > @@ -9,68 +9,46 @@ macro(REGISTER_SYMLINK from to) > endmacro() > > # MariaDB names for executables > -REGISTER_SYMLINK("mysql_client_test_embedded" "mariadb-client-test-embedded") > -REGISTER_SYMLINK("mysql_client_test" "mariadb-client-test") > +REGISTER_SYMLINK("mariadb-client-test-embedded" "mysql_client_test_embedded") > +REGISTER_SYMLINK("mariadb-client-test" "mysql_client_test") ... I suppose you need to make sure that everything works with new names, without symlinks. That is all internal tools and scripts invoke other tools and scripts by their mariadb* names. A way to do it could be to comment out all REGISTER_SYMLINK lines and fix whatever that will break. grep is also a useful tool here, but with rather low signal-to-noise ratio. > + > +MACRO(GET_SYMLINK name out) > + set(${out}) > + list(FIND MARIADB_SYMLINK_FROMS ${name} _index) > if (${_index} GREATER -1) > - list(GET MARIADB_SYMLINK_TOS ${_index} mariadbname) > - endif() > - > - if (mariadbname) > - CREATE_MARIADB_SYMLINK_IN_DIR(${src} ${mariadbname} ${dir} ${comp}) > - endif() > -endmacro(CREATE_MARIADB_SYMLINK) > - > -# Add MariaDB symlinks in directory > -macro(CREATE_MARIADB_SYMLINK_IN_DIR src dest dir comp) > - if(UNIX) > - add_custom_target( > - SYM_${dest} ALL > - DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/${dest} > - ) > - > - add_custom_command(OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/${dest} POST_BUILD > - COMMAND ${CMAKE_COMMAND} -E create_symlink ${src} ${dest} > - COMMENT "mklink ${src} -> ${dest}") > - > - install(FILES ${CMAKE_CURRENT_BINARY_DIR}/${dest} DESTINATION ${dir} COMPONENT ${comp}) > + list(GET MARIADB_SYMLINK_TOS ${_index} ${out}) > endif() > -endmacro(CREATE_MARIADB_SYMLINK_IN_DIR) > +ENDMACRO() > diff --git a/debian/mariadb-backup.install b/debian/mariadb-backup.install > index 2bc61c12079..e1967495b00 100644 > --- a/debian/mariadb-backup.install > +++ b/debian/mariadb-backup.install > @@ -1,5 +1,4 @@ > -usr/bin/mariabackup > +usr/bin/mariadb-backup > usr/bin/mbstream > usr/share/man/man1/mariabackup.1 > -usr/share/man/man1/mariadb-backup.1 this is strange. Why did you remove mariadb-backup.1 and not mariabackup.1 ? > usr/share/man/man1/mbstream.1 > diff --git a/debian/mariadb-client-10.5.install b/debian/mariadb-client-10.5.install > index 67c0c2619c3..216e4015851 100644 > --- a/debian/mariadb-client-10.5.install > +++ b/debian/mariadb-client-10.5.install > @@ -1,29 +1,17 @@ > debian/additions/innotop/innotop usr/bin/ > debian/additions/mysqlreport usr/bin/ > +usr/bin/mariadb-access > +usr/bin/mariadb-admin > usr/bin/mariadb-conv > -usr/bin/mysql_find_rows > -usr/bin/mysql_fix_extensions > -usr/bin/mysql_waitpid > -usr/bin/mysqlaccess > -usr/bin/mysqladmin > -usr/bin/mysqldump > -usr/bin/mysqldumpslow > -usr/bin/mysqlimport > -usr/bin/mysqlshow you removed mysqlshow here, and added mariadb-show to the server package. Intentional? UPD: I see that before your patch mysqlshow was in the client package, while mariadb-show symlink was in the server package. It's a bug, both should be in the client package. > +usr/bin/mariadb-dump > +usr/bin/mariadb-dumpslow > +usr/bin/mariadb-find-rows > +usr/bin/mariadb-fix-extensions > +usr/bin/mariadb-import > +usr/bin/mariadb-slap > +usr/bin/mariadb-waitpid > usr/bin/mysqlslap you did not remove mysqlslap, but added mariadb-slap > usr/bin/mytop > -usr/share/man/man1/mariadb-access.1 > -usr/share/man/man1/mariadb-admin.1 > -usr/share/man/man1/mariadb-binlog.1 > -usr/share/man/man1/mariadb-conv.1 > -usr/share/man/man1/mariadb-dump.1 > -usr/share/man/man1/mariadb-dumpslow.1 > -usr/share/man/man1/mariadb-find-rows.1 > -usr/share/man/man1/mariadb-fix-extensions.1 > -usr/share/man/man1/mariadb-import.1 > -usr/share/man/man1/mariadb-plugin.1 > -usr/share/man/man1/mariadb-slap.1 > -usr/share/man/man1/mariadb-waitpid.1 same weirdness as elsewhere, mariadb*.1 man pages are symlinks to mysql*.1 manpages. Should be vice versa. > usr/share/man/man1/mysql_find_rows.1 > usr/share/man/man1/mysql_fix_extensions.1 > usr/share/man/man1/mysql_waitpid.1 > diff --git a/debian/mariadb-client-10.5.links b/debian/mariadb-client-10.5.links > index 5d966575b76..d1bdfcfa3c8 100644 > --- a/debian/mariadb-client-10.5.links > +++ b/debian/mariadb-client-10.5.links > @@ -1,21 +1,19 @@ > -usr/bin/mysql_find_rows usr/bin/mariadb-find-rows > -usr/bin/mysql_fix_extensions usr/bin/mariadb-fix-extensions > -usr/bin/mysql_plugin usr/bin/mariadb-plugin > -usr/bin/mysql_waitpid usr/bin/mariadb-waitpid > -usr/bin/mysqlaccess usr/bin/mariadb-access > -usr/bin/mysqladmin usr/bin/mariadb-admin > -usr/bin/mysqlbinlog usr/bin/mariadb-binlog you removed mysqlbinlog, but added mariadb-binlog to the server package. Intentional? same for mysql_plugin/mariadb-plugin. UPD: mysqlbinlog was in the server package, while mariadb-binlog symlink was in the client package. So your change is correct, both will be in the server package now. Good. same for mysql_plugin/mariadb-plugin. > -usr/bin/mysqlcheck usr/bin/mariadb-analyze > -usr/bin/mysqlcheck usr/bin/mariadb-optimize > diff --git a/debian/mariadb-plugin-rocksdb.install b/debian/mariadb-plugin-rocksdb.install > index 80987612c30..0fc868a0721 100644 > --- a/debian/mariadb-plugin-rocksdb.install > +++ b/debian/mariadb-plugin-rocksdb.install > @@ -2,6 +2,5 @@ etc/mysql/conf.d/rocksdb.cnf etc/mysql/mariadb.conf.d > usr/bin/myrocks_hotbackup > usr/bin/mysql_ldb you didn't replace mysql_ldb with mariadb-ldb? > usr/lib/mysql/plugin/ha_rocksdb.so > -usr/share/man/man1/mariadb-ldb.1 > usr/share/man/man1/myrocks_hotbackup.1 > usr/share/man/man1/mysql_ldb.1 > diff --git a/debian/mariadb-server-10.5.install b/debian/mariadb-server-10.5.install > index 4a860ff57df..38d9f984276 100644 > --- a/debian/mariadb-server-10.5.install > +++ b/debian/mariadb-server-10.5.install > @@ -14,22 +14,23 @@ usr/bin/aria_pack > usr/bin/aria_read_log > usr/bin/galera_new_cluster > usr/bin/galera_recovery > +usr/bin/mariadb-binlog > +usr/bin/mariadb-convert-table-format > +usr/bin/mariadb-hotcopy > +usr/bin/mariadb-plugin > +usr/bin/mariadb-secure-installation > usr/bin/mariadb-service-convert > +usr/bin/mariadb-setpermission > +usr/bin/mariadb-show > +usr/bin/mariadb-tzinfo-to-sql > +usr/bin/mariadbd-multi > +usr/bin/mariadbd-safe > +usr/bin/mariadbd-safe duplicate line ^^^ > usr/bin/msql2mysql > usr/bin/myisam_ftdump > usr/bin/myisamchk > usr/bin/myisamlog > usr/bin/myisampack > -usr/bin/mysql_convert_table_format > -usr/bin/mysql_plugin > -usr/bin/mysql_secure_installation > -usr/bin/mysql_setpermission > -usr/bin/mysql_tzinfo_to_sql > -usr/bin/mysqlbinlog > -usr/bin/mysqld_multi > -usr/bin/mysqld_safe > -usr/bin/mysqld_safe_helper you don't install mariadbd-safe-helper. Perhaps that's where your duplicate line comes from, it's not a duplicate, it's truncated :) > -usr/bin/mysqlhotcopy > usr/bin/perror > usr/bin/replace > usr/bin/resolve_stack_dump > diff --git a/debian/mariadb-server-10.5.links b/debian/mariadb-server-10.5.links > index f2d97460371..e3a2d68541a 100644 > --- a/debian/mariadb-server-10.5.links > +++ b/debian/mariadb-server-10.5.links > @@ -1,9 +1,20 @@ > -usr/bin/mysql_convert_table_format usr/bin/mariadb-convert-table-format > -usr/bin/mysql_secure_installation usr/bin/mariadb-secure-installation > -usr/bin/mysql_setpermission usr/bin/mariadb-setpermission > -usr/bin/mysql_tzinfo_to_sql usr/bin/mariadb-tzinfo-to-sql > -usr/bin/mysqld_multi usr/bin/mariadbd-multi > -usr/bin/mysqld_safe usr/bin/mariadbd-safe > -usr/bin/mysqld_safe_helper usr/bin/mariadbd-safe-helper > -usr/bin/mysqlhotcopy usr/bin/mariadb-hotcopy > -usr/bin/mysqlshow usr/bin/mariadb-show > +usr/bin/mariadb-binlog usr/bin/mysqlbinlog > +usr/bin/mariadb-convert-table-format usr/bin/mysql_convert_table_format > +usr/bin/mariadb-hotcopy usr/bin/mysqlhotcopy > +usr/bin/mariadb-plugin usr/bin/mysql_plugin > +usr/bin/mariadb-secure-installation usr/bin/mysql_secure_installation > +usr/bin/mariadb-setpermission usr/bin/mysql_setpermission > +usr/bin/mariadb-show /usr/bin/mysqlshow this is the only line in your whole patch where you symlink by the absolute path. Looks like a typo. > +usr/bin/mariadb-tzinfo-to-sql usr/bin/mysql_tzinfo_to_sql > +usr/bin/mariadbd-multi usr/bin/mysqld_multi > +usr/bin/mariadbd-safe usr/bin/mysqld_safe > +usr/bin/mariadbd-safe-helper usr/bin/mysqld_safe_helper > diff --git a/libmariadb b/libmariadb > index 3be5897c334..ca68b114ba7 160000 > --- a/libmariadb > +++ b/libmariadb > @@ -1 +1 @@ > -Subproject commit 3be5897c3346639fa6d7195480d93108798c4917 > +Subproject commit ca68b114ba7e882e8abe8888204d675c4d8751f8 This is clearly wrong, please make sure you've removed it from your commit before pushing! > diff --git a/scripts/CMakeLists.txt b/scripts/CMakeLists.txt > index 7be46ac1985..350fbd14cf6 100644 > --- a/scripts/CMakeLists.txt > +++ b/scripts/CMakeLists.txt > @@ -244,8 +232,20 @@ SET(mysqlaccess_COMPONENT COMPONENT Client) > SET(mysql_find_rows_COMPONENT COMPONENT Client) > SET(mytop_COMPONENT Mytop) > > +MACRO(INSTALL_LINK old new destination component) > + EXECUTE_PROCESS( > + COMMAND ${CMAKE_COMMAND} -E create_symlink ${old} ${new} > + WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR} > + ) > + INSTALL( > + PROGRAMS ${CMAKE_CURRENT_BINARY_DIR}/${new} > + DESTINATION ${destination} > + COMPONENT ${component} > + ) > +ENDMACRO() > + > IF(WIN32) > - # On Windows, some .sh and some .pl.in files are configured > + # On Windows, some .sh and some .pl.in files are configured > # The resulting files will have .pl extension (those are perl scripts) > > # Input files with pl.in extension > @@ -296,60 +296,73 @@ ELSE() > # On Unix, most of the files end up in the bin directory > SET(BIN_SCRIPTS > msql2mysql > - mysql_config > - mysql_setpermission > - mysql_secure_installation > - mysqlaccess > - mysql_convert_table_format > - mysql_find_rows > + mariadb-setpermission > + mariadb-secure-installation > + mariadb-access > + mariadb-convert-table-format > + mariadb-find-rows > mytop > - mysqlhotcopy > + mariadb-hotcopy > + mariadb-install-db > ${SERVER_SCRIPTS} > ${WSREP_SCRIPTS} > ${SYSTEMD_SCRIPTS} > ) > + > FOREACH(file ${BIN_SCRIPTS}) > - IF(EXISTS ${CMAKE_CURRENT_SOURCE_DIR}/${file}.sh) > - CONFIGURE_FILE(${CMAKE_CURRENT_SOURCE_DIR}/${file}.sh > - ${CMAKE_CURRENT_BINARY_DIR}/${file} ESCAPE_QUOTES @ONLY) > - ELSEIF(EXISTS ${CMAKE_CURRENT_SOURCE_DIR}/${file}) > - CONFIGURE_FILE(${CMAKE_CURRENT_SOURCE_DIR}/${file} > - ${CMAKE_CURRENT_BINARY_DIR}/${file} COPYONLY) > - ELSE() > - MESSAGE(FATAL_ERROR "Can not find ${file}.sh or ${file} in " > - "${CMAKE_CURRENT_SOURCE_DIR}" ) > - ENDIF() > - # TODO: The following EXECUTE could be redundant as INSTALL_SCRIPT > - # macro does an INSTALL(PROGRAMS ..) that automatically sets +x on > - # the executable. > - EXECUTE_PROCESS(COMMAND chmod +x ${CMAKE_CURRENT_BINARY_DIR}/${file}) > - IF(NOT ${file}_COMPONENT) > - SET(${file}_COMPONENT Server) > - ENDIF() > - INSTALL_SCRIPT( > - ${CMAKE_CURRENT_BINARY_DIR}/${file} > - DESTINATION ${INSTALL_BINDIR} > - COMPONENT ${${file}_COMPONENT} > - ) > + # set name of executable > + list(FIND MARIADB_SYMLINK_FROMS ${file} _index) > + > + if(${_index} GREATER -1) > + list(GET MARIADB_SYMLINK_TOS ${_index} binname) > + else() > + set(binname ${file}) > + endif() 1. why are you not using GET_SYMLINK ? 2. Why INSTALL_SYMLINK here instead of the old CREATE_MARIADB_SYMLINK ? You avoid creating new targets for script symlinks, but who cares about that? > + > + if(NOT "${binname}" STREQUAL "") how can ${binname} be "" ? > + IF(EXISTS ${CMAKE_CURRENT_SOURCE_DIR}/${binname}.sh) > + CONFIGURE_FILE(${CMAKE_CURRENT_SOURCE_DIR}/${binname}.sh > + ${CMAKE_CURRENT_BINARY_DIR}/${file} ESCAPE_QUOTES @ONLY) > + ELSEIF(EXISTS ${CMAKE_CURRENT_SOURCE_DIR}/${binname}) > + CONFIGURE_FILE(${CMAKE_CURRENT_SOURCE_DIR}/${binname} > + ${CMAKE_CURRENT_BINARY_DIR}/${file} COPYONLY) > + ELSE() > + MESSAGE(FATAL_ERROR "Can not find ${binname}.sh or ${binname} in " > + "${CMAKE_CURRENT_SOURCE_DIR}" ) > + ENDIF() > + > + IF(NOT ${file}_COMPONENT) > + SET(${file}_COMPONENT Server) > + ENDIF() > + INSTALL_SCRIPT( > + ${CMAKE_CURRENT_BINARY_DIR}/${file} > + DESTINATION ${INSTALL_BINDIR} > + COMPONENT ${${file}_COMPONENT} > + ) > + > + # Create symlink > + if (NOT ${binname} STREQUAL ${file}) > + INSTALL_LINK(${file} ${binname} ${INSTALL_BINDIR} ${${file}_COMPONENT}) > + > + # Place mysql_install_db symlink also in scripts > + if (${binname} STREQUAL "mysql_install_db") > + SET(bindir ${bindir1}) > + SET(prefix ${prefix1}) > + SET(sbindir ${sbindir1}) > + SET(scriptdir ${scriptdir1}) > + SET(libexecdir ${libexecdir1}) > + SET(pkgdatadir ${pkgdatadir1}) > + SET(pkgplugindir ${pkgplugindir1}) > + SET(localstatedir ${localstatedir1}) > + CONFIGURE_FILE(${CMAKE_CURRENT_SOURCE_DIR}/mysql_install_db.sh ${CMAKE_CURRENT_BINARY_DIR}/mariadb-install-db ESCAPE_QUOTES @ONLY) > + INSTALL(PROGRAMS ${CMAKE_CURRENT_BINARY_DIR}/${file} DESTINATION ${INSTALL_SCRIPTDIR} COMPONENT Server) > + INSTALL(PROGRAMS ${CMAKE_CURRENT_BINARY_DIR}/${binname} DESTINATION ${INSTALL_SCRIPTDIR} COMPONENT Server) I don't understand that at all. 1. What does the comment "Place mysql_install_db symlink also in scripts" even mean? 2. you CONFIGURE_FILE mariadb-install-db twice. 3. Why two INSTALL(PROGRAM ? 4. You set all bindir/etc variables for mysql_install_db and it will affect CONFIGURE_FILE for all following scripts 5. why do you even compare ${binname} with mysql_install_db and not ${file} with mariadb-install-db ? > + endif() > + endif() > + endif() we customary always use uppercase in all cmake keywords. > ENDFOREACH() > > - SET (wsrep_sst_rsync_wan ${CMAKE_CURRENT_BINARY_DIR}/wsrep_sst_rsync_wan) > - ADD_CUSTOM_COMMAND( > - OUTPUT ${wsrep_sst_rsync_wan} > - COMMAND ${CMAKE_COMMAND} ARGS -E create_symlink > - wsrep_sst_rsync > - wsrep_sst_rsync_wan > - ) > - ADD_CUSTOM_TARGET(symlink_wsrep_sst_rsync > - ALL > - DEPENDS ${wsrep_sst_rsync_wan} > - ) > - INSTALL( > - FILES ${wsrep_sst_rsync_wan} > - DESTINATION ${INSTALL_BINDIR} > - COMPONENT Server > - ) > - > + INSTALL_LINK(wsrep_sst_rsync wsrep_sst_rsync_wan ${INSTALL_BINDIR} Server) > FOREACH(file ${WSREP_SOURCE}) > CONFIGURE_FILE(${CMAKE_CURRENT_SOURCE_DIR}/${file}.sh > ${CMAKE_CURRENT_BINARY_DIR}/${file} ESCAPE_QUOTES @ONLY) > diff --git a/win/packaging/ca/CMakeLists.txt b/win/packaging/ca/CMakeLists.txt > index 99dc7da01fb..326bab47de4 100644 > --- a/win/packaging/ca/CMakeLists.txt > +++ b/win/packaging/ca/CMakeLists.txt I didn't review windows packaging, it's between you and wlad. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 5365db70cb8: Improve update handler (long unique keys on blobs)
by Sergei Golubchik 16 Mar '20

16 Mar '20

Hi, Michael! On Feb 29, Michael Widenius wrote: > commit 5365db70cb8 > Author: Michael Widenius <monty(a)mariadb.com> > Date: Mon Jan 13 18:30:13 2020 +0200 > > Improve update handler (long unique keys on blobs) > > MDEV-21606 Improve update handler (long unique keys on blobs) > MDEV-21470 MyISAM and Aria start_bulk_insert doesn't work with long unique > MDEV-21606 Bug fix for previous version of this code > MDEV-21819 2 Assertion `inited == NONE || update_handler != this' > > - Move update_handler from TABLE to handler > - Move out initialization of update handler from ha_write_row() to > prepare_for_insert() > - Fixed that INSERT DELAYED works with update handler > - Give an error if using long unique with an autoincrement column > - Added handler function to check if table has long unique hash indexes > - Disable write cache in MyISAM and Aria when using update_handler as > if cache is used, the row will not be inserted until end of statement > and update_handler would not find conflicting rows. > - Removed not used handler argument from > check_duplicate_long_entries_update() > - Syntax cleanups > - Indentation fixes > - Don't use single character indentifiers for arguments > > squash! 1ca2f0871ecfb550268ffc1a5cc23d7043d6b855 you may want to remove this line from the commit comment > > diff --git a/mysql-test/main/long_unique.result b/mysql-test/main/long_unique.result > index a4955b3e7b5..fee8e7721bf 100644 > --- a/mysql-test/main/long_unique.result > +++ b/mysql-test/main/long_unique.result > @@ -1477,4 +1477,28 @@ id select_type table type possible_keys key key_len ref rows Extra > SELECT t2.b FROM t1 JOIN t2 ON t1.d = t2.f WHERE t2.pk >= 20; > b > drop table t1,t2; > +# > +# MDEV-21470 MyISAM start_bulk_insert doesn't work with long unique > +# > +CREATE TABLE t1 (a INT, b BLOB) ENGINE=MyISAM; > +INSERT INTO t1 VALUES (1,'foo'),(2,'bar'); > +CREATE TABLE t2 (c BIT, d BLOB, UNIQUE(d)) ENGINE=MyISAM; > +INSERT INTO t2 SELECT * FROM t1; > +Warnings: > +Warning 1264 Out of range value for column 'c' at row 2 > +DROP TABLE t1, t2; > +# > +# MDEV-19338 Using AUTO_INCREMENT with long unique > +# > +CREATE TABLE t1 (pk INT, a TEXT NOT NULL DEFAULT '', PRIMARY KEY (pk), b INT AUTO_INCREMENT, UNIQUE(b), UNIQUE (a,b)) ENGINE=myisam; > +ERROR HY000: Function or expression 'AUTO_INCREMENT' cannot be used in the UNIQUE clause of `a` Quite confusing error message :( I understand where it comes from, but for a user, I'm afraid, it just won't make any sense. A reasonable message could be, for example, AUTO_INCREMENT column %`s cannot be used in the UNIQUE index %`s > +# > +# MDEV-21819 Assertion `inited == NONE || update_handler != this' > +# failed in handler::ha_write_row > +# > +CREATE OR REPLACE TABLE t1 (a INT, b BLOB, s DATE, e DATE, PERIOD FOR app(s,e), UNIQUE(b)) ENGINE=MyISAM PARTITION BY HASH(a) PARTITIONS 2; > +INSERT INTO t1 VALUES (1,'foo','2022-01-01', '2025-01-01'); > +DELETE FROM t1 FOR PORTION OF app FROM '2023-01-01' TO '2024-01-01'; > +ERROR 23000: Duplicate entry 'foo' for key 'b' > +DROP TABLE t1; > set @@GLOBAL.max_allowed_packet= @allowed_packet; > diff --git a/mysql-test/suite/versioning/r/long_unique.result b/mysql-test/suite/versioning/r/long_unique.result > new file mode 100644 > index 00000000000..da07bc66e22 > --- /dev/null > +++ b/mysql-test/suite/versioning/r/long_unique.result > @@ -0,0 +1,8 @@ > +# > +# Assertion `inited == NONE || update_handler != this' failed in > +# handler::ha_write_row > +# > +CREATE TABLE t1 (f VARCHAR(4096), s DATE, e DATE, PERIOD FOR app(s,e), UNIQUE(f)) ENGINE=MyISAM; > +INSERT INTO t1 VALUES ('foo', '2023-08-30', '2025-07-09'),('bar', '2021-01-01', '2021-12-31'); > +DELETE FROM t1 FOR PORTION OF app FROM '2023-08-29' TO '2025-07-01'; > +DROP TABLE t1; you've had almost the same test (MDEV-21819) in main/long_unique.test it's a bit strange to have to very similar tests in different suites I suggest to move MDEV-21819 test from main/long_unique.test to this file. and then move this whole file from versioning suite to the period suite. > diff --git a/sql/ha_partition.cc b/sql/ha_partition.cc > index 73191907aac..c7e61864366 100644 > --- a/sql/ha_partition.cc > +++ b/sql/ha_partition.cc > @@ -4340,6 +4339,10 @@ int ha_partition::write_row(const uchar * buf) > } > m_last_part= part_id; > DBUG_PRINT("info", ("Insert in partition %u", part_id)); > + if (table->s->long_unique_table && > + m_file[part_id]->update_handler == m_file[part_id] && inited == RND) > + m_file[part_id]->prepare_for_insert(0); This looks quite incomprehensible. If the table has a long unique key and the update_handler is the handler itself you _prepare for insert_ ??? if it's an insert it should've been prepared earlier, like, normally for an insert, and it don't have to do anything with long uniques. What you actually do here (as I've found out looking firther in the patch) you prepare update_handler. That is "prepare_for_insert" is VERY confusing here, it sounds like it prepares for a normal insert and should be called at the beginning of mysql_insert(). And it's nothing like that. It would remove half of the questions if you'd call it prepare_update_handler() or, better, prepare_auxiliary_handler() (because it'll be used not only for long uniques) or something like that. Still, some questions are left: 1. why partitioning needs special code for that? 2. when a handler itself is its update_handler AND inited==RND? the handler itself can be the update_handler on INSERT, but then inited is NONE Is it UPDATE with system versioning? > + > start_part_bulk_insert(thd, part_id); > > tmp_disable_binlog(thd); /* Do not replicate the low-level changes. */ > diff --git a/sql/handler.cc b/sql/handler.cc > index 0700b1571e5..1e3f987b4e5 100644 > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -2648,6 +2649,39 @@ handler *handler::clone(const char *name, MEM_ROOT *mem_root) > return NULL; > } > > +/** > + @brief clone of current handler. you generally don't have to write @brief in these cases. Just /** clone of current handler. Creates a clone of handler used in update for unique hash key. */ > + Creates a clone of handler used in update for > + unique hash key. > +*/ > +bool handler::clone_handler_for_update() > +{ > + handler *tmp; > + DBUG_ASSERT(table->s->long_unique_table); > + > + if (update_handler != this) > + return 0; // Already done > + if (!(tmp= clone(table->s->normalized_path.str, table->in_use->mem_root))) > + return 1; > + update_handler= tmp; > + update_handler->ha_external_lock(table->in_use, F_RDLCK); Looks strange. Why F_RDLCK if it's an *update* handler? This definitely needs a comment, why you're using read lock for updates. > + return 0; > +} > + > +/** > + @brief Deletes update handler object > +*/ > +void handler::delete_update_handler() > +{ > + if (update_handler != this) > + { > + update_handler->ha_external_lock(table->in_use, F_UNLCK); > + update_handler->ha_close(); > + delete update_handler; > + } > + update_handler= this; > +} > + > LEX_CSTRING *handler::engine_name() > { > return hton_name(ht); > @@ -6481,6 +6515,8 @@ int handler::ha_reset() > DBUG_ASSERT(inited == NONE); > /* reset the bitmaps to point to defaults */ > table->default_column_bitmaps(); > + if (update_handler != this) > + delete_update_handler(); you already have an if() inside the delete_update_handler(). Either remove it from here or from there. > pushed_cond= NULL; > tracker= NULL; > mark_trx_read_write_done= 0; > @@ -6671,10 +6717,35 @@ static int check_duplicate_long_entries_update(TABLE *table, handler *h, uchar * > } > } > } > - exit: > - return error; > + return 0; > +} > + > + > +/** > + Do all initialization needed for insert > + > + @param force_update_handler Set to TRUE if we should always create an > + update handler. Needed if we don't know if we > + are going to do inserted while a scan is in "going to do inserts" > + progress. > +*/ > + > +int handler::prepare_for_insert(bool force_update_handler) > +{ > + /* Preparation for unique of blob's */ > + if (table->s->long_unique_table && (inited == RND || force_update_handler)) > + { > + /* > + When doing a scan we can't use the same handler to check > + duplicate rows. Create a new temporary one > + */ > + if (clone_handler_for_update()) > + return 1; > + } > + return 0; > } > > + > int handler::ha_write_row(const uchar *buf) > { > int error; > @@ -6977,6 +7044,11 @@ void handler::set_lock_type(enum thr_lock_type lock) > table->reginfo.lock_type= lock; > } > > +bool handler::has_long_unique() > +{ > + return table->s->long_unique_table; > +} I don't see why you need it for Aria. It can always check table->s->long_unique_table without an extra call > + > #ifdef WITH_WSREP > /** > @details > diff --git a/sql/sql_delete.cc b/sql/sql_delete.cc > index 6a4ce266af2..e55ae4f97e6 100644 > --- a/sql/sql_delete.cc > +++ b/sql/sql_delete.cc > @@ -752,6 +752,10 @@ bool mysql_delete(THD *thd, TABLE_LIST *table_list, COND *conds, > && !table->versioned() > && table->file->has_transactions(); > > + if (table->versioned(VERS_TIMESTAMP) || > + (table_list->has_period() && !portion_of_time_through_update)) > + table->file->prepare_for_insert(1); Ok, now I think that your idea of the default update_handler=this is rather fragile. If the default is NULL, than we can put in many places DBUG_ASSERT(update_handler != NULL); to make sure it was initialized when needed. Now we cannot do it. And there are many places in the code and non-trivial conditions when the update_handler has to be initialized and these places and conditions will change in the future. An assert would be helpful here. > + > THD_STAGE_INFO(thd, stage_updating); > while (likely(!(error=info.read_record())) && likely(!thd->killed) && > likely(!thd->is_error())) > diff --git a/sql/table.h b/sql/table.h > index 6ce92ee048e..1a0b9514d23 100644 > --- a/sql/table.h > +++ b/sql/table.h > @@ -1182,6 +1181,7 @@ struct TABLE > /* Map of keys dependent on some constraint */ > key_map constraint_dependent_keys; > KEY *key_info; /* data of keys in database */ > + KEY_PART_INFO *base_key_part; /* Where key parts are stored */ again, this is *only* needed for INSERT DELAYED that very few people ever use. It should be in some INSERT DELAYED specific data structure, not in the common TABLE. By the way, why cannot you get the list of keyparts from table->key_info->key_part ? > > Field **field; /* Pointer to fields */ > Field **vfield; /* Pointer to virtual fields*/ > diff --git a/storage/myisam/ha_myisam.cc b/storage/myisam/ha_myisam.cc > index 38091dae0ba..39147396359 100644 > --- a/storage/myisam/ha_myisam.cc > +++ b/storage/myisam/ha_myisam.cc > @@ -1740,7 +1740,7 @@ void ha_myisam::start_bulk_insert(ha_rows rows, uint flags) > enable_indexes() failed, thus it's essential that indexes are > disabled ONLY for an empty table. > */ > - if (file->state->records == 0 && can_enable_indexes && > + if (file->state->records == 0 && can_enable_indexes && !has_long_unique() && > (!rows || rows >= MI_MIN_ROWS_TO_DISABLE_INDEXES)) why do you disable bulk inserts for long uniques? > { > if (file->open_flag & HA_OPEN_INTERNAL_TABLE) > diff --git a/sql/table.cc b/sql/table.cc > index 718efa5767c..18a942964c3 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -1241,31 +1241,46 @@ bool parse_vcol_defs(THD *thd, MEM_ROOT *mem_root, TABLE *table, > Item *list_item; > KEY *key= 0; > uint key_index, parts= 0; > + KEY_PART_INFO *key_part= table->base_key_part; > + > for (key_index= 0; key_index < table->s->keys; key_index++) > { > - key=table->key_info + key_index; > - parts= key->user_defined_key_parts; > - if (key->key_part[parts].fieldnr == field->field_index + 1) > - break; > + /* > + We have to use key from share as this function may have changed > + table->key_info if it was ever invoked before. This could happen > + in case of INSERT DELAYED. > + */ > + key= table->s->key_info + key_index; > + if (key->algorithm == HA_KEY_ALG_LONG_HASH) > + { > + parts= key->user_defined_key_parts; > + if (key_part[parts].fieldnr == field->field_index + 1) > + break; > + key_part++; > + } Here, again, you've basically duplicated the logic of how long uniques are represented in the keys and keyparts. All for the sake of INSERT DELAYED. I'd really prefer INSERT DELAYED problems to be solved inside the Delayed_insert class. And it can use setup_keyinfo_hash() and re_setup_keyinfo_hash() functions to avoid duplicating the logic. > + key_part+= key->ext_key_parts; > } > - if (!key || key->algorithm != HA_KEY_ALG_LONG_HASH) > + if (key_index == table->s->keys) > goto end; > - KEY_PART_INFO *keypart; > - for (uint i=0; i < parts; i++) > + > + /* Correct the key & key_parts if this function has been called before */ > + key= table->key_info + key_index; > + key->key_part= key_part; > + > + for (uint i=0; i < parts; i++, key_part++) > { > - keypart= key->key_part + i; > - if (keypart->key_part_flag & HA_PART_KEY_SEG) > + if (key_part->key_part_flag & HA_PART_KEY_SEG) > { > - int length= keypart->length/keypart->field->charset()->mbmaxlen; > + int length= key_part->length/key_part->field->charset()->mbmaxlen; > list_item= new (mem_root) Item_func_left(thd, > - new (mem_root) Item_field(thd, keypart->field), > + new (mem_root) Item_field(thd, key_part->field), > new (mem_root) Item_int(thd, length)); > list_item->fix_fields(thd, NULL); > - keypart->field->vcol_info= > - table->field[keypart->field->field_index]->vcol_info; > + key_part->field->vcol_info= > + table->field[key_part->field->field_index]->vcol_info; > } > else > - list_item= new (mem_root) Item_field(thd, keypart->field); > + list_item= new (mem_root) Item_field(thd, key_part->field); > field_list->push_back(list_item, mem_root); > } > Item_func_hash *hash_item= new(mem_root)Item_func_hash(thd, *field_list); Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 3

Re: [Maria-developers] [Commits] 70b0c838fad: MDEV-21922: Allow packing addon fields even if they don't honour max_length_for_sort_data
by Sergey Petrunia 12 Mar '20

12 Mar '20

Hi Varun, * The patch needs a testcase * Please check the comment at https://jira.mariadb.org/browse/MDEV-21922 , and add a comment in filesort_use_addons() about the use of upper bound for computations. * Ok to push after the above is addressed. On Thu, Mar 12, 2020 at 05:11:45PM +0530, Varun wrote: > revision-id: 70b0c838fada8e02d4ce9525611d6f4aeadf56d4 (mariadb-10.5.0-367-g70b0c838fad) > parent(s): 9d7ed94f6a526748eff29dae2939a3fd341f118b > author: Varun Gupta > committer: Varun Gupta > timestamp: 2020-03-12 14:41:03 +0530 > message: > > MDEV-21922: Allow packing addon fields even if they don't honour max_length_for_sort_data > > Addon fields will be packed if the length of addon fields is greater > than max_length_for_sort_data. > > --- > sql/filesort.cc | 2 -- > 1 file changed, 2 deletions(-) > > diff --git a/sql/filesort.cc b/sql/filesort.cc > index 1e19d492220..7b4e0bc31cd 100644 > --- a/sql/filesort.cc > +++ b/sql/filesort.cc > @@ -139,8 +139,6 @@ void Sort_param::try_to_pack_addons(ulong max_length_for_sort_data) > return; > > const uint sz= Addon_fields::size_of_length_field; > - if (rec_length + sz > max_length_for_sort_data) > - return; > > // Heuristic: skip packing if potential savings are less than 10 bytes. > if (m_packable_length < (10 + sz)) > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0