- developers - lists.mariadb.org

Re: [Maria-developers] c01b3d91690: MDEV-22551 Server crashes in skip_trailing_space upon ANALYZE on table with long unique, or Assertion `marked_for_read()' fails
by Sergei Golubchik 03 Jun '20

03 Jun '20

Hi, Sachin! On Jun 03, Sachin wrote: > revision-id: c01b3d91690 (mariadb-10.4.11-224-gc01b3d91690) > parent(s): 0552ed86782 > author: Sachin <sachin.setiya(a)mariadb.com> > committer: Sachin <sachin.setiya(a)mariadb.com> > timestamp: 2020-06-02 08:24:13 +0530 > message: > > MDEV-22551 Server crashes in skip_trailing_space upon ANALYZE on table with long unique, or Assertion `marked_for_read()' fails > > If Blob(or parent) field read_set is not set then Do not calc hash in > that case. No, I think this is wrong. The problem, again, is that read_set bits are set incorrectly, and the fix is, again, to set them correctly, not to perform more read_set checks at run time. Think about what's happening here. The server tries to calculate a generated column value (because gcol is present in read_set) when not all argument columns were read (because they are not in read_set). This is an impossible situation, a gcol can _only_ be in read_set if all its dependencies are also there, you've seen in TABLE::mark_virtual_columns_for_write() that for every marked gcol all dependencies are also recursively marked. So in this case ANALYZE breaks this rule and should be fixed not to do it. There are two possible fixes - either ANALYZE should mark dependencies for all gcols (marking blobs in this bug) or ANALYZE should skip and not mark gcols. Better ask somebody who's more famiiar with EITS about what fix is preferred. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] e2017fc3a5f: MDEV-22441 implement a generic way to change a value of a variable in scope
by Sergei Golubchik 03 Jun '20

03 Jun '20

Hi, Eugene! On Jun 03, Eugene Kosov wrote: > revision-id: e2017fc3a5f (mariadb-10.2.31-219-ge2017fc3a5f) This should go into 10.5, not into old-old GA 10.2. Besides, 10.2 cannot use C++11 features, 10.5 can. > parent(s): 1b3adaab25c > author: Eugene Kosov <claprix(a)yandex.ru> > committer: Eugene Kosov <eugene.kosov(a)mariadb.com> > timestamp: 2020-05-27 12:45:41 +0300 > message: > > MDEV-22441 implement a generic way to change a value of a variable in scope > > Scope_value<T> class changes a variable value to some temporary value while > it exits and restores an old value in it's destructor. > > make_scope_value() is a function targeted for C++11 which allows to omit > a template parameter which is required by Scope_value<T> I think I'd name it backup_and_change_scoped() or something like that. But it's just a suggestion, feel free to keep make_scope_value() name if you like it better. > Sql_mode_save removed and the only usage is replaced with Scope_value<T> > > Sql_mode_save removal should not be merged in newer branches! > > --- > include/scope.h | 46 ++++++++++++++++++++++++++++++++++++++++++++++ > sql/sql_acl.cc | 7 ++++--- > sql/sql_class.h | 16 ---------------- > 3 files changed, 50 insertions(+), 19 deletions(-) > > diff --git a/include/scope.h b/include/scope.h > new file mode 100644 > index 00000000000..81061ceed37 > --- /dev/null > +++ b/include/scope.h > @@ -0,0 +1,46 @@ > +/* > + Copyright (c) 2020, MariaDB Corporation. > + > + This program is free software; you can redistribute it and/or modify > + it under the terms of the GNU General Public License as published by > + the Free Software Foundation; version 2 of the License. > + > + This program is distributed in the hope that it will be useful, > + but WITHOUT ANY WARRANTY; without even the implied warranty of > + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + GNU General Public License for more details. > + > + You should have received a copy of the GNU General Public License > + along with this program; if not, write to the Free Software > + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1335 USA */ > + > +#pragma once use include guards, please, not a pragma > + > +namespace maria and, as you remember, we didn't quite decide on a global server-wide namespace, so it's not the MDEV that should introduce it. > +{ > + > +template <typename T> class Scope_value > +{ > +public: > + Scope_value(T &variable, const T &scope_value) > + : variable_(variable), saved_value_(variable) > + { > + variable= scope_value; > + } > + > + ~Scope_value() { variable_= saved_value_; } > + > +private: > + T &variable_; > + T saved_value_; > +}; > + > +// Starting from C++11 this allows to omit template argument like this: > +// auto _ = make_scope_value(var, tmp_value); > +template <typename T> > +Scope_value<T> make_scope_value(T &variable, const T &scope_value) > +{ > + return Scope_value<T>(variable, scope_value); > +} as you've seen from Sql_mode_save usage, sometimes the value is assigned later, not when it's saved. Please, add another helper that doesn't require to provide a value. Like template <typename T> Scope_value<T> backup_scoped(T &variable) { return Scope_value<T>(variable, variable); } > + > +} // namespace maria Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 0552ed86782: MDEV-21804 Assertion `marked_for_read()' failed upon INSERT into table with long unique blob under binlog_row_image=NOBLOB
by Sergei Golubchik 03 Jun '20

03 Jun '20

Hi, Sachin! ok to push with one change, see below. On Jun 03, Sachin wrote: > revision-id: 0552ed86782 (mariadb-10.4.11-223-g0552ed86782) > parent(s): 61b2cd38d48 > author: Sachin <sachin.setiya(a)mariadb.com> > committer: Sachin <sachin.setiya(a)mariadb.com> > timestamp: 2020-06-02 08:24:02 +0530 > message: > > MDEV-21804 Assertion `marked_for_read()' failed upon INSERT into table with long unique blob under binlog_row_image=NOBLOB > > Problem:- Calling mark_columns_per_binlog_row_image() earlier may change the > result of mark_virtual_columns_for_write() , Since it can set the bitmap on > for virtual column, and henceforth mark_virtual_column_deps(field) will > never be called in mark_virtual_column_with_deps. > > This bug is not specific for long unique, It also fails for this case > create table t2(id int primary key, a blob, b varchar(20) as (LEFT(a,2))); > > diff --git a/sql/table.cc b/sql/table.cc > index bc8ddbc0f9d..8edbbe8d599 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -7040,7 +7040,6 @@ void TABLE::mark_columns_needed_for_update() > DBUG_ENTER("TABLE::mark_columns_needed_for_update"); > bool need_signal= false; > > - mark_columns_per_binlog_row_image(); > > if (triggers) > triggers->mark_fields_used(TRG_EVENT_UPDATE); > @@ -7048,6 +7047,7 @@ void TABLE::mark_columns_needed_for_update() > mark_default_fields_for_write(FALSE); > if (vfield) > need_signal|= mark_virtual_columns_for_write(FALSE); > + mark_columns_per_binlog_row_image(); I think that's too early. In that if() below you can see > if (file->ha_table_flags() & HA_REQUIRES_KEY_COLUMNS_FOR_DELETE) > { > KEY *end= key_info + s->keys; ... if (bitmap_fast_test_and_set(read_set, idx)) continue; if (field[idx]->vcol_info) mark_virtual_col(field[idx]); and mark_columns_per_binlog_row_image() might break that. Better to put mark_columns_per_binlog_row_image() at the very end, just before signalling. > @@ -7132,7 +7132,6 @@ void TABLE::mark_columns_needed_for_update() > void TABLE::mark_columns_needed_for_insert() > { > DBUG_ENTER("mark_columns_needed_for_insert"); > - mark_columns_per_binlog_row_image(); > > if (triggers) > { > @@ -7152,6 +7151,7 @@ void TABLE::mark_columns_needed_for_insert() > /* Mark virtual columns for insert */ > if (vfield) > mark_virtual_columns_for_write(TRUE); > + mark_columns_per_binlog_row_image(); this looks ok. > if (check_constraints) > mark_check_constraint_columns_for_read(); > DBUG_VOID_RETURN; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] Progress report - Week 4
by Mohammed Hammaad Mateen 02 Jun '20

02 Jun '20

Greetings, Hope you are safe and doing great, This post describes the things I've done in my fourth week [25-31 May] of Community Bonding Period under the mentor-ship of Sergei Golubchik and Oleksandr Byelkin for GSoC-20. The tasks taken up for this week was to study and analyze *INSERT SELECT RETURNING* and developing a test suite for *MULTIPLE TABLE DELETE RETURNING.* The following attachment contains the test cases for evaluating Multiple Delete Returning, as they are developed before actual implementation of code, please do notify me of any improvement needed, I'd be glad to resolve. Regards, Mohammed Hammaad Mateen

2 1

Re: [Maria-developers] fab8349af86: MDEV-21804 Assertion `marked_for_read()' failed upon INSERT into table with long unique blob under binlog_row_image=NOBLOB
by Sergei Golubchik 01 Jun '20

01 Jun '20

Hi, Sachin! On May 31, Sachin wrote: > revision-id: fab8349af86 (mariadb-10.4.11-223-gfab8349af86) > parent(s): 61b2cd38d48 > author: Sachin <sachin.setiya(a)mariadb.com> > committer: Sachin <sachin.setiya(a)mariadb.com> > timestamp: 2020-05-29 11:56:45 +0530 > message: > > MDEV-21804 Assertion `marked_for_read()' failed upon INSERT into table with long unique blob under binlog_row_image=NOBLOB > > Problem:- > In the case of BINLOG_ROW_IMAGE_NOBLOB table->readset is PKE + non blob fields > But while updating virtual fields we need to read the blob field, which will > generate assert failure. > Solution:- > If binlog_row_image == NOBLOB, then set read_bit for the blob field. > This bug is not specific for long unique, It also fails for this case > create table t2(id int primary key, a blob, b varchar(20) as (LEFT(a,2))); > > diff --git a/sql/field.cc b/sql/field.cc > index 0a8fdc3d3f5..bcad056e76a 100644 > --- a/sql/field.cc > +++ b/sql/field.cc > @@ -8467,6 +8467,10 @@ longlong Field_blob::val_int(void) > String *Field_blob::val_str(String *val_buffer __attribute__((unused)), > String *val_ptr) > { > + ulong binlog_row_image= table->in_use->variables.binlog_row_image; > + if (table->read_set && !bitmap_is_set(table->read_set, field_index) && > + binlog_row_image == BINLOG_ROW_IMAGE_NOBLOB) > + bitmap_set_bit(table->read_set, field_index); > DBUG_ASSERT(marked_for_read()); > char *blob; > memcpy(&blob, ptr+packlength, sizeof(char*)); A couple of red flags here. First, you don't fix BINLOG_ROW_IMAGE_MINIMAL. And generally you should not modify read/write sets from val_str, if should be done earlier before the execution phase. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] Review for: MDEV-17399 Add support for JSON_TABLE, part #5
by Sergey Petrunia 26 May '20

26 May '20

Hi Alexey, On Fri, May 15, 2020 at 04:26:02PM +0400, Alexey Botchkov wrote: > See the next iteration here > https://github.com/MariaDB/server/commit/692cb566096d61b240ec26e84fc7d3c7d1… > So the ha_json_table;:position() was implemented. > The size of the reference depends on the nested path depth - each level > adds some 9 bytes to the initial 5. > That can be decreased but i decided to keep it simple initially and i doubt > we're going to have really deep > nesting in realistic scenarios. This is ok. I'm not sure if the server has a limit on the rowid size. Perhaps there is, and in that case we just need to limit the depth we allow. > And i'd like to ask you for some testing > queries here. Or just the model how > to produce queries that will be using the ::position() extensively. I've provided a testcase for the position() call in my previous email. https://gist.github.com/spetrunia/a905d51731c58f5439bd9f70c64cdc43 As for rnd_pos(), one case that I'm aware of is when the query does a filesort, and the row being sorted either includes a blob, or has the total length exceeding certain limit. I've tried constructing an example with blobs, and it fails with an assertion before the execution reaches rnd_pos() calls: select * from json_table('[{"color": "blue", "price": 50}, {"color": "red", "price": 100}, {"color": "rojo", "price": 10.0}, {"color": "blanco", "price": 11.0}]', '$[*]' columns( color varchar(100) path '$.color', price text path '$.price', seq for ordinality ) ) as T order by color desc; fails an assertion: mysqld: /home/psergey/dev-git/10.5-json/sql/field.cc:8309: virtual int Field_blob::store(const char*, size_t, CHARSET_INFO*): Assertion `marked_for_write_or_computed()' failed. Once that is fixed, this should use position() and rnd_pos() calls. This will likely expose more issues with rnd_pos(), see my comments to the Json_table_nested_path::set_position below. == A problem with VIEWs == mysql> select * -> from -> json_table( -> '[ {"color": "red", "price": 1}, {"color": "black", "price": 2}]', -> '$[*]' columns( color varchar(100) path '$.color')) as `T_A` where T_A.color<>'azul' ; +-------+ | color | +-------+ | red | | black | +-------+ 2 rows in set (6.34 sec) Good so far. Now, let's try creating a VIEW from this: create view v1 as select * from json_table( '[ {"color": "red", "price": 1}, {"color": "black", "price": 2}]', '$[*]' columns( color varchar(100) path '$.color')) as `T_A` where T_A.color<>'azul' ; select * from v1; ERROR 4041 (HY000): Unexpected end of JSON path in argument 2 to function 'JSON_TABLE' Examining the .frm file, I see something that looks like garbage data: query=select `T_A`.`color` AS `color` from JSON_TABLE(\'[ {"color": "red", "price": 1}, {"color": "black", "price": 2}]\', \'\'\0<D8>7^A<9C><FF>^?\0\0\0\0\0\0\0\0\0\0\0\0^A<A5><A5><A5><A5><A5>^H<D0>kWUU\0\0[ {"color": "red", "price": 1}, {"color": "black", "price": 2}]\0$[*]\'\' COLUMNS (`color` PATH \'^A<9C><FF>^?\0\0\0\0\0\0^A\0\0\0\0\0\0\0<A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5>\0\0\0\0<A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5> <A5><A5><A5><A5><A5><A5> D^A<9C><FF>^?\0\0H:^A<9C><FF>^?\0\0<A5><A5><A5><A5><A5><A5><A5><A5>100\0<8F><8F><8F><8F>$.color\')) T_A where `T_A`.`color` <> \'azul\' EXPLAIN EXTENDED ...; SHOW WARNINGS; - also print something odd. == EXPLAIN [FORMAT=JSON] == I think EXPLAIN output should provide an indication that a table function is used. MySQL does it like so: <TODO> == Unneeded recursive rules in grammar == I've already complained about this: the grammar has recursive ON EMPTY, ON ERROR rules, which cause the following to be accepted (note the two ON EMPTY clauses) : select * from json_table('[{"color": "blue", "price": 50},{"color": "red"}]', '$[*]' columns( price varchar(255) path '$.price' default 'xyz' on empty default 'abc' on empty ) ) as T; > commit 692cb566096d61b240ec26e84fc7d3c7d13f024c > Author: Alexey Botchkov <holyfoot(a)askmonty.org> > Date: Fri May 15 15:25:28 2020 +0400 > > MDEV-17399 Add support for JSON_TABLE. > > Syntax for JSON_TABLE added. > The ha_json_table handler added. Executing the JSON_TABLE we > create the temporary table of the ha_json_table, add dependencies > of other tables and sometimes conditions to WHERE. > and sometimes conditions to WHERE. I think this is not true anymore? ... > diff --git a/include/my_base.h b/include/my_base.h > index 7efa5eb9673..89ef3e8e7c1 100644 > --- a/include/my_base.h > +++ b/include/my_base.h > @@ -523,6 +523,13 @@ enum ha_base_keytype { > #define HA_ERR_TABLESPACE_MISSING 194 /* Missing Tablespace */ > #define HA_ERR_SEQUENCE_INVALID_DATA 195 > #define HA_ERR_SEQUENCE_RUN_OUT 196 > + > +/* > + Share the error code to not increment the HA_ERR_LAST for now, > + as it disturbs some storage engine's tests. > + Probably should be fixed later. > +*/ > +#define HA_ERR_INVALID_JSON HA_ERR_TABLE_IN_FK_CHECK We definitely can't have this in the final patch. We need to either: A. Use an engine-specific error code. Check out MyRocks and ha_rocksdb::get_error_message() for an example of how this is done B. Indeed introduce a generic "Invalid JSON" error code. I'm hesitant about B, let's discuss this with other developers. ... > diff --git a/sql/table_function.cc b/sql/table_function.cc > new file mode 100644 > index 00000000000..71f2378ce7d > --- /dev/null > +++ b/sql/table_function.cc ... > + > +class ha_json_table: public handler > +{ Please add comments about these > +protected: > + Table_function_json_table *m_jt; > + String m_tmps; > + String *m_js; > + uchar *m_cur_pos; > +public: > + ha_json_table(TABLE_SHARE *share_arg, Table_function_json_table *jt): > + handler(&table_function_hton.m_hton, share_arg), m_jt(jt) > + { > + /* > + set the mark_trx_read_write_done to avoid the > + handler::mark_trx_read_write_internal() call. > + It relies on &ha_thd()->ha_data[ht->slot].ha_info[0] to be set. > + But we don't set the ha_data for the ha_json_table, and > + that call makes no sence for ha_json_table. > + */ > + mark_trx_read_write_done= 1; > + ref_length= (jt->m_depth+1) * (4 + 1) + > + jt->m_depth * (sizeof(Json_table_nested_path *)); > + } > + ~ha_json_table() {} > + handler *clone(const char *name, MEM_ROOT *mem_root) { return NULL; } > + const char *index_type(uint inx) { return "NONE"; } > + /* Rows also use a fixed-size format */ > + enum row_type get_row_type() const { return ROW_TYPE_FIXED; } > + ulonglong table_flags() const > + { > + return (HA_FAST_KEY_READ | /*HA_NO_BLOBS |*/ HA_NULL_IN_KEY | > + HA_CAN_SQL_HANDLER | > + HA_REC_NOT_IN_SEQ | HA_NO_TRANSACTIONS | > + HA_HAS_RECORDS | HA_CAN_HASH_KEYS); > + } > + ulong index_flags(uint inx, uint part, bool all_parts) const > + { > + return HA_ONLY_WHOLE_INDEX | HA_KEY_SCAN_NOT_ROR; > + } > + uint max_supported_keys() const { return 1; } > + uint max_supported_key_part_length() const { return MAX_KEY_LENGTH; } > + double scan_time() { return 1000000.0; } > + double read_time(uint index, uint ranges, ha_rows rows) { return 0.0; } The above two functions are never called. Please * add a comment saying that. * add a DBUG_ASSERT() into them to back the point made by the comment :-) > + int open(const char *name, int mode, uint test_if_locked); > + int close(void) { return 0; } > + int rnd_init(bool scan); > + int rnd_next(uchar *buf); > + int rnd_pos(uchar * buf, uchar *pos); > + void position(const uchar *record); > + int can_continue_handler_scan() { return 1; } > + int info(uint); > + int extra(enum ha_extra_function operation); > + THR_LOCK_DATA **store_lock(THD *thd, THR_LOCK_DATA **to, > + enum thr_lock_type lock_type) > + { return NULL; } > + int create(const char *name, TABLE *form, HA_CREATE_INFO *create_info) > + { return 1; } > +private: > + void update_key_stats(); > +}; > + ... > +void Json_table_nested_path::set_position(const char *j_start, const uchar *pos) > +{ * This needs a comment. * I don't see where the value of m_ordinality_counter is restored? * The same about m_null - its value is not restored either? > + if (m_nested) > + { > + memcpy(&m_cur_nested, pos, sizeof(m_cur_nested)); > + pos+= sizeof(m_cur_nested); > + } > + > + m_engine.s.c_str= (uchar *) j_start + sint4korr(pos); > + m_engine.state= (int) pos[4]; > + if (m_cur_nested) > + m_cur_nested->set_position(j_start, pos); > +} > + ... > +int ha_json_table::info(uint) > +{ > + /* We don't want 0 or 1 in stats.records. */ > + stats.records= 4; > + return 0; Does this value matter? As far as I understand it doesn't, as the optimizer will use the estimates obtained from Table_function_json_table::get_estimates. Please add a comment about this. > +} > + ... Please document this function. What's last_column? Why does print() method get it and return it? > +int Json_table_nested_path::print(THD *thd, TABLE_LIST *sql_table, String *str, > + List_iterator_fast<Json_table_column> &it, > + Json_table_column **last_column) > +{ > + Json_table_nested_path *c_path= this; ... > diff --git a/sql/table_function.h b/sql/table_function.h > new file mode 100644 > index 00000000000..49d650cb7b2 > --- /dev/null > +++ b/sql/table_function.h > @@ -0,0 +1,168 @@ ... > +#include <json_lib.h> > + > +/* > + The Json_table_nested_path represents the 'current nesting' level > + for a set of JSON_TABLE columns. > + Each column (Json_table_column instance) is linked with corresponding > + 'nested path' object and gets it's piece of JSON to parse during the computation > + phase. > + The root 'nested_path' is always present as a part of Table_function_json_table, > + then other 'nested_paths' can be created and linked into a tree structure when new > + 'NESTED PATH' is met. The nested 'nested_paths' are linked with 'm_nested', the same-level > + 'nested_paths' are linked with 'm_next_nested'. > + So for instance > + JSON_TABLE( '...', '$[*]' > + COLUMNS( a INT PATH '$.a' , > + NESTED PATH '$.b[*]' COLUMNS (b INT PATH '$', > + NESTED PATH '$.c[*]' COLUMNS(x INT PATH '$')), > + NESTED PATH '$.n[*]' COLUMNS (z INT PAHT '$')) > + results in 4 'nested_path' created: > + root nested_b nested_c nested_n > + m_path '$[*]' '$.b[*]' '$.c[*]' '$.n[*] > + m_nested &nested_b &nested_c NULL NULL > + n_next_nested NULL &nested_n NULL NULL > + > +and 4 columns created: > + a b x z > + m_nest &root &nested_b &nested_c &nested_n > +*/ > + > + > +class Json_table_column; > + > +class Json_table_nested_path : public Sql_alloc > +{ > +public: > + bool m_null; > + json_path_t m_path; > + json_engine_t m_engine; > + json_path_t m_cur_path; > + > + /* Counts the rows produced. Value is set to the FOR ORDINALITY coluns */ > + longlong m_ordinality_counter; > + > + Json_table_nested_path *m_parent; > + Json_table_nested_path *m_nested, *m_next_nested; > + Json_table_nested_path **m_nested_hook; > + Json_table_nested_path *m_cur_nested; Please add documentation about the above members. I see the diagram above, but I think text descriptions are also needed for each member. What's m_nested_hook? > + Json_table_nested_path(Json_table_nested_path *parent_nest): > + m_parent(parent_nest), m_nested(0), m_next_nested(0), > + m_nested_hook(&m_nested) {} > + int set_path(THD *thd, const LEX_CSTRING &path); > + void scan_start(CHARSET_INFO *i_cs, const uchar *str, const uchar *end); > + int scan_next(); > + int print(THD *thd, TABLE_LIST *sql_table, String *str, > + List_iterator_fast<Json_table_column> &it, > + Json_table_column **last_column); > + void get_current_position(const char *j_start, uchar *pos) const; > + void set_position(const char *j_start, const uchar *pos); > +}; ... > +class Table_function_json_table : public Sql_alloc Please document the class and the data members. > +{ > +public: > + Item *m_json; > + Json_table_nested_path m_nested_path; > + List<Json_table_column> m_columns; > + table_map m_dep_tables; > + uint m_depth, m_cur_depth; > + > + Table_function_json_table(Item *json): m_json(json), m_nested_path(0), > + m_depth(0), m_cur_depth(0) {} > + > + /* > + Used in sql_yacc.yy. > + Represents the current NESTED PATH level being parsed. > + */ > + Json_table_nested_path *m_sql_nest; > + void add_nested(Json_table_nested_path *np); > + void leave_nested(); BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] 070df171c1e: MDEV-16937 Strict SQL with system versioned tables causes issues
by Sergei Golubchik 26 May '20

26 May '20

Hi, Aleksey! ok to push On May 26, Aleksey Midenkov wrote: > revision-id: 070df171c1e (mariadb-10.3.21-91-g070df171c1e) > parent(s): ad41da5c934 > author: Aleksey Midenkov <midenok(a)gmail.com> > committer: Aleksey Midenkov <midenok(a)gmail.com> > timestamp: 2020-05-25 14:47:26 +0300 > message: > > MDEV-16937 Strict SQL with system versioned tables causes issues > > Respect system fields in NO_ZERO_DATE mode. > > This is the subject for refactoring in MDEV-19597 > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] cd9cab54aac: MDEV-20015 Assertion `!in_use->is_error()' failed in TABLE::update_virtual_field
by Sergei Golubchik 26 May '20

26 May '20

Hi, Aleksey! Ok to push, thanks! On May 26, Aleksey Midenkov wrote: > revision-id: cd9cab54aac (mariadb-10.2.31-123-gcd9cab54aac) > parent(s): d275ecbd208 > author: Aleksey Midenkov <midenok(a)gmail.com> > committer: Aleksey Midenkov <midenok(a)gmail.com> > timestamp: 2020-05-25 20:56:31 +0300 > message: > > MDEV-20015 Assertion `!in_use->is_error()' failed in TABLE::update_virtual_field > > update_virtual_field() is called as part of index rebuild in > ha_myisam::repair() (MDEV-5800) which is done on bulk INSERT finish. > > Assertion in update_virtual_field() was put as part of MDEV-16222 > because update_virtual_field() returns in_use->is_error(). The idea: > wrongly mixed semantics of error status before update_virtual_field() > and the status returned by update_virtual_field(). The former can > falsely influence the latter. > > diff --git a/sql/table.cc b/sql/table.cc > index d6d86d96016..2429bb12abe 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -7707,15 +7707,17 @@ int TABLE::update_virtual_fields(handler *h, enum_vcol_update_mode update_mode) > > int TABLE::update_virtual_field(Field *vf) > { > - DBUG_ASSERT(!in_use->is_error()); > - Query_arena backup_arena; > DBUG_ENTER("TABLE::update_virtual_field"); > + Query_arena backup_arena; > + Counting_error_handler count_errors; > + in_use->push_internal_handler(&count_errors); > in_use->set_n_backup_active_arena(expr_arena, &backup_arena); > bitmap_clear_all(&tmp_set); > vf->vcol_info->expr->walk(&Item::update_vcol_processor, 0, &tmp_set); > vf->vcol_info->expr->save_in_field(vf, 0); > in_use->restore_active_arena(expr_arena, &backup_arena); > - DBUG_RETURN(in_use->is_error()); > + in_use->pop_internal_handler(); > + DBUG_RETURN(count_errors.errors); > } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 7d593466a22: MDEV-20015 Assertion `!in_use->is_error()' failed in TABLE::update_virtual_field
by Sergei Golubchik 25 May '20

25 May '20

Hi, Aleksey! I don't see what you've changed. We've discussed that fix and that one isn't supposed to swap Diagnostics_area's like that. And in your new patch you do exactly the same. Possible correct approaches: * don't return in_use->is_error(), return the return value of vf->vcol_info->expr->walk() || vf->vcol_info->expr->save_in_field() This means that Item_field::update_vcol_processor() should also do the same, I suspect * Use thd->push_internal_handler() and Counting_error_handler. Or, better, Turn_errors_to_warnings_handler with counting. This is the simplest one. there's a third option: * always return 0, because, looking how it's used, I don't really see how update_virtual_field() can ever get an error. But it's not a particularly future-proof approach. And I just might be wrong about errors. On Apr 23, Aleksey Midenkov wrote: > revision-id: 7d593466a22 (mariadb-10.2.28-4-g7d593466a22) > parent(s): 7bc26de591c > author: Aleksey Midenkov <midenok(a)gmail.com> > committer: Aleksey Midenkov <midenok(a)gmail.com> > timestamp: 2019-11-07 10:45:21 +0300 > message: > > MDEV-20015 Assertion `!in_use->is_error()' failed in TABLE::update_virtual_field > > Preserve and restore statement DA. > > update_virtual_field() is called as part of index rebuild in > ha_myisam::repair() (MDEV-5800) which is done on bulk INSERT finish. > > Assertion in update_virtual_field() was put as part of MDEV-16222 > because update_virtual_field() returns in_use->is_error(). The idea: > wrongly mixed semantics of error status before update_virtual_field() > and the status returned by update_virtual_field(). The former can > falsely influence the latter. > > Preserve global error status and run update_virtual_field() with clear > DA since no matter how SQL command is finished it must update the > index after bulk INSERT. > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

[Maria-developers] Progress Report - Week 3
by Mohammed Hammaad Mateen 25 May '20

25 May '20

Greetings, Hope you are safe and doing great, This post describes the things I've done in my third week [ 18-24 May ] of Community Bonding Period under the mentor-ship of Sergei Golubchik and Oleksandr Byelkin for GSoC-20. The tasks taken up for this week was to study and analyze: *INSERT.... RETURNING:* • *TableList* consists of 1 table instance to use, where each instance is a structure consisting of db argument, table name argument, alias argument and lock type argument. *Referenced this from the table.h file.* • In *sql_parse.cc* <http://sql_parse.cc> under *case SQLCOM_INSERT*: we do a DBUG_ASSERT where first_table == all_table && first_table!=0 // we check if only 1 table is used and the table used is already created. • if lex-> has_returning() we increment the system status var by 1 and perform analyze.. insert.. returning. • we compute result by mysql_insert function by passing the following arguments. - Thread Handler thd - all table instances - list of all the fields used - the values we want to insert - the list of update feilds - and update value list - duplicate flag - ignore - result of select • if Inserting fails due to some reason we equate result to send_explain(thd) • we also update the MYSQL_INSERT_DONE with result of insert and the the row count. *INSERT.. SELECT.. RETURNING:* • We fix the lock on first table. • lock other tables until command is written to the binary log. • the procedure is same as discussed earlier with respect to insert.... returning. • To switch to the second table we traverse from first_table to (->) next_local and we compute select result with the help of select insert function. • we can now unlock the tables and we also need to check if something changed after unlocking, of that happens we should invalidate the table from the query cache using query_cache_invalidate3 function. • Manual cleaning of select result obtained from select insert must be done. Regards, Mohammed Hammaad Mateen

1 0

Re: [Maria-developers] MDEV-22461: JOIN::make_aggr_tables_info(): Assertion `select_options & (1ULL << 17)' failed.
by Sergey Petrunia 24 May '20

24 May '20

Hi Varun, - Please add a testcase to the testsuite - Please address a few cosmetic comments below - Ok to push after addressed. > commit 5e448b77a3812e65623c0a1214049322ace3aacf > Author: Varun Gupta <varun.gupta(a)mariadb.com> > Date: Tue May 5 20:44:43 2020 +0530 > > MDEV-22461: JOIN::make_aggr_tables_info(): Assertion `select_options & (1ULL << 17)' failed. > > A temporary table is needed for window function computation but if only a NAMED WINDOW SPEC > is used and there is no window function, then there is no need to create a temporary > table as there is no stage to compute WINDOW FUNCTION > > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index c601946cfa0..0a1d0c2dbcc 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -2014,11 +2014,16 @@ JOIN::optimize_inner() > } > > need_tmp= test_if_need_tmp_table(); > - //TODO this could probably go in test_if_need_tmp_table. > - if (this->select_lex->window_specs.elements > 0) { > - need_tmp= TRUE; > + > + /* > + If window functions are present then we can't have simple_order set to > + TRUE as the window function needs a temp table for compuatation. typo: compuatation. > + ORDER BY is computed after the window function computation is done, so > + the sort would be done on the temp table. > + */ > + if (this->select_lex->have_window_funcs()) Is there a need to have "this->select_lex"? I think not, please remove, as it only confuses the reader. > simple_order= FALSE; > - } > + > > /* > If the hint FORCE INDEX FOR ORDER BY/GROUP BY is used for the table > diff --git a/sql/sql_select.h b/sql/sql_select.h > index 0e011c9267a..7a892c1af89 100644 > --- a/sql/sql_select.h > +++ b/sql/sql_select.h > @@ -1645,7 +1645,8 @@ class JOIN :public Sql_alloc > ((select_distinct || !simple_order || !simple_group) || > (group_list && order) || > MY_TEST(select_options & OPTION_BUFFER_RESULT))) || > - (rollup.state != ROLLUP::STATE_NONE && select_distinct)); > + (rollup.state != ROLLUP::STATE_NONE && select_distinct) || > + select_lex->have_window_funcs()); Please amend the function comment above this code to cover the new condition as well. > } > bool choose_subquery_plan(table_map join_tables); > void get_partial_cost_and_fanout(int end_tab_idx, > BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] b22a28c2295: fixup! 3fe5cd5e1785e3e8de7add9977a1c2ddd403538b
by Michael Widenius 22 May '20

22 May '20

Hi! On Fri, May 22, 2020 at 3:27 PM Andrei Elkin <andrei.elkin(a)mariadb.com> wrote: <cut> CORNER CASES: read-only, pure myisam, binlog-*, @@skip_log_bin, etc > > Aria just makes yet another previously unknown use case of an engine > that produces THD::ha_info but does not support 2pc, which the assert > implied. > > To explain more, the original block > > #ifndef DBUG_OFF > for (ha_info= thd->transaction.all.ha_list; rw_count > 1 && ha_info; > ha_info= ha_info->next()) > DBUG_ASSERT(ha_info->ht() != binlog_hton); > #endif > > claims there most be no binlog hton in a transaction consisting of more > than 1 hton:s, *when* (at this point) this transaction has not been > binlogged yet. > So combination of binlog + Innodb hton would be raise the assert, to > question "why the heck binlogging has not been done yet?!". > > Aria is different from Innodb in this context in that binlogging > was done at the end of the statement, so to miss > `cache_mngr->need_unlog' flagging (which is at xa prepare time logging). Thanks for the explanation, we should have had that in the code. > I think we should fix the assert rather than to remove. This way: > > cat > assert.diff <<. > diff --git a/sql/log.cc b/sql/log.cc > index 792c6bb1a99..aaf1fae1cd6 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -10124,6 +10124,16 @@ int TC_LOG_BINLOG::unlog_xa_prepare(THD *thd, bool all) > Ha_trx_info *ha_info; > uint rw_count= ha_count_rw_all(thd, &ha_info); > bool rc= false; > +#ifndef DBUG_OFF > + bool no_binlog= true, exist_no_2pc= false; > + for (ha_info= thd->transaction->all.ha_list; rw_count > 1 && ha_info; > + ha_info= ha_info->next()) > + { > + no_binlog= no_binlog && ha_info->ht() != binlog_hton; > + exist_no_2pc= exist_no_2pc || !ha_info->ht()->prepare; > + } > + DBUG_ASSERT(no_binlog || exist_no_2pc); > +#endif > > if (rw_count > 0) > { > . > > I tested it briefly with running XA:s on combination of engines > including. On which tree did you test? bb-10.5-aria? In the end, after discussions on slack, we ended with: #ifndef DBUG_OFF if (rw_count > 1) { /* There must be no binlog_hton used in a transaction consisting of more than 1 engine, *when* (at this point) this transaction has not been binlogged. The one exception is if there is an engine without a prepare method, as in this case the engine doesn't support XA and we have to ignore this check. */ bool binlog= false, exist_hton_without_prepare= false; for (ha_info= thd->transaction->all.ha_list; ha_info; ha_info= ha_info->next()) { if (ha_info->ht() == binlog_hton) binlog= true; if (!ha_info->ht()->prepare) exist_hton_without_prepare= true; } DBUG_ASSERT(!binlog || exist_hton_without_prepare); } #endif > > The reason it was not hit > > is that before Aria was not treated as transactional we could only > > come here in case of errors > > and that code was was apparently not tested, at least with binary logging on. > > > > > With Aria we could come here in case of rollback and we got assert for > > cases that was perfectly ok. > > Well, what 'rollback' do you mean? The function is invoked only > for ha_prepare(). As far as I remember, I come into this code in some edge case when something failed that normally never fails. I think the issue was that Aria doesn't have a prepare handler, which caused issues in ha_prepare(), but not sure. Anyway, we now have a solution for this. Regards, Monty

1 0

Re: [Maria-developers] acbe14b122c: Aria will now register it's transactions
by Sergei Golubchik 22 May '20

22 May '20

Hi, Michael! See below On May 20, Michael Widenius wrote: > revision-id: acbe14b122c (mariadb-10.5.2-255-gacbe14b122c) > parent(s): 48a758696bf > author: Michael Widenius <monty(a)mariadb.com> > committer: Michael Widenius <monty(a)mariadb.com> > timestamp: 2020-05-19 17:52:17 +0300 > message: > > Aria will now register it's transactions > > MDEV-22531 Remove maria::implicit_commit() > diff --git a/sql/sql_class.h b/sql/sql_class.h > index d4a95fa3fd8..a7a071f7cdb 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -5149,6 +5151,40 @@ class THD: public THD_count, /* this must be first */ > > }; > > + > +/* > + Start a new independent transaction for the THD. > + The old one is stored in this object and restored when calling > + restore_old_transaction() or when the object is freed > +*/ > + > +class start_new_trans > +{ > + /* container for handler's private per-connection data */ > + Ha_data old_ha_data[MAX_HA]; > + struct THD::st_transactions *old_transaction, new_transaction; > + Open_tables_backup open_tables_state_backup; > + MDL_savepoint mdl_savepoint; > + PSI_transaction_locker *m_transaction_psi; > + THD *org_thd; > + uint in_sub_stmt; > + uint server_status; > + > +public: > + start_new_trans(THD *thd); > + ~start_new_trans() > + { > + destroy(); > + } > + void destroy() > + { > + if (org_thd) // Safety > + restore_old_transaction(); > + new_transaction.free(); > + } > + void restore_old_transaction(); interesting. You made restore_old_transaction() public and you use it in many places. Why not to use destroy() instead? When one would want to use restore_old_transaction() instead of destroy()? > +}; > + > /** A short cut for thd->get_stmt_da()->set_ok_status(). */ > > inline void > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index dda8e00f6bf..51d7380f622 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -5742,6 +5744,90 @@ void THD::mark_transaction_to_rollback(bool all) > } > > > +/** > + Commit the whole transaction (both statment and all) > + > + This is used mainly to commit an independent transaction, > + like reading system tables. > + > + @return 0 0k > + @return <>0 error code. my_error() has been called() > +*/ > + > +int THD::commit_whole_transaction_and_close_tables() > +{ > + int error, error2; > + DBUG_ENTER("THD::commit_whole_transaction_and_close_tables"); > + > + /* > + This can only happened if we failed to open any table in the > + new transaction > + */ > + if (!open_tables) > + DBUG_RETURN(0); Generally, I think, it should still end the transaction here. Or add an assert that there is no active transaction at the moment. > + > + /* > + Ensure table was locked (opened with open_and_lock_tables()). If not > + the THD can't be part of any transactions and doesn't have to call > + this function. > + */ > + DBUG_ASSERT(lock); > + > + error= ha_commit_trans(this, FALSE); > + /* This will call external_lock to unlock all tables */ > + if ((error2= mysql_unlock_tables(this, lock))) > + { > + my_error(ER_ERROR_DURING_COMMIT, MYF(0), error2); > + error= error2; > + } > + lock= 0; > + if ((error2= ha_commit_trans(this, TRUE))) > + error= error2; > + close_thread_tables(this); I wonder why you're doing it in that specific order. commit(stmt)-unlock-commit(all)-close > + DBUG_RETURN(error); > +} > + > +/** > + Start a new independent transaction > +*/ > + > +start_new_trans::start_new_trans(THD *thd) > +{ > + org_thd= thd; > + mdl_savepoint= thd->mdl_context.mdl_savepoint(); > + memcpy(old_ha_data, thd->ha_data, sizeof(old_ha_data)); > + thd->reset_n_backup_open_tables_state(&open_tables_state_backup); > + bzero(thd->ha_data, sizeof(thd->ha_data)); > + old_transaction= thd->transaction; > + thd->transaction= &new_transaction; > + new_transaction.on= 1; > + in_sub_stmt= thd->in_sub_stmt; > + thd->in_sub_stmt= 0; > + server_status= thd->server_status; > + m_transaction_psi= thd->m_transaction_psi; > + thd->m_transaction_psi= 0; > + thd->server_status&= ~(SERVER_STATUS_IN_TRANS | > + SERVER_STATUS_IN_TRANS_READONLY); > + thd->server_status|= SERVER_STATUS_AUTOCOMMIT; > +} Few thoughts: 1. If you need to save and restore _all that_ then, perhaps, all that should be inside st_transactions ? 2. strictly speaking, ha_data is _per connection_. If you just bzero it, the engine will think it's a new connection, and you cannot just overwrite it on restore without hton->close_connection. A strictly "proper" solution would be to introduce ha_data per transaction in addition to what we have now. But it looks like an overkill. So I'd just add close_system_tables() now into restore_old_transaction() A strictly "proper" solution would be to introduce ha_data per transaction in addition to what we have now. But it looks like an overkill. So I'd just add ha_close_connection() now into restore_old_transaction() and that's all. I see that you've added free_transaction() call, but isn't it redundant? ha_close_connection() does that now. Otherwise very good, thanks, I cannot wait to start using it more for other features. > + > + > +void start_new_trans::restore_old_transaction() > +{ > + org_thd->transaction= old_transaction; > + org_thd->restore_backup_open_tables_state(&open_tables_state_backup); > + ha_free_transactions(org_thd); > + memcpy(org_thd->ha_data, old_ha_data, sizeof(old_ha_data)); > + org_thd->mdl_context.rollback_to_savepoint(mdl_savepoint); > + org_thd->in_sub_stmt= in_sub_stmt; > + org_thd->server_status= server_status; > + if (org_thd->m_transaction_psi) > + MYSQL_COMMIT_TRANSACTION(org_thd->m_transaction_psi); > + org_thd->m_transaction_psi= m_transaction_psi; > + org_thd= 0; > +} > + > + > /** > Decide on logging format to use for the statement and issue errors > or warnings as needed. The decision depends on the following > diff --git a/sql/event_db_repository.cc b/sql/event_db_repository.cc > index af43d92dea7..82b3968de85 100644 > --- a/sql/event_db_repository.cc > +++ b/sql/event_db_repository.cc > @@ -742,7 +748,8 @@ Event_db_repository::create_event(THD *thd, Event_parse_data *parse_data, > ret= 0; > > end: > - close_thread_tables(thd); > + if (table) > + thd->commit_whole_transaction_and_close_tables(); You're checking twice. Here in the caller, and the first thing inside commit_whole_transaction_and_close_tables(). Here and in many places. If you want to check in the caller, there's no need to check inside? You can add an assert instead, can't you? > thd->mdl_context.rollback_to_savepoint(mdl_savepoint); > > thd->variables.sql_mode= saved_mode; > @@ -1117,22 +1124,20 @@ update_timing_fields_for_event(THD *thd, > TABLE *table= NULL; > Field **fields; > int ret= 1; > - enum_binlog_format save_binlog_format; > MYSQL_TIME time; > DBUG_ENTER("Event_db_repository::update_timing_fields_for_event"); > > - /* > - Turn off row binlogging of event timing updates. These are not used > - for RBR of events replicated to the slave. > - */ > - save_binlog_format= thd->set_current_stmt_binlog_format_stmt(); > - > DBUG_ASSERT(thd->security_ctx->master_access & PRIV_IGNORE_READ_ONLY); > > if (open_event_table(thd, TL_WRITE, &table)) > goto end; Why do you reuse a current transaction, instead of creating a new one as everywhere above? > > fields= table->field; > + /* > + Turn off row binlogging of event timing updates. These are not used > + for RBR of events replicated to the slave. > + */ > + table->file->row_logging= 0; > > if (find_named_event(event_db_name, event_name, table)) > goto end; > diff --git a/sql/handler.h b/sql/handler.h > index e4903172c33..ebe23b97062 100644 > --- a/sql/handler.h > +++ b/sql/handler.h > @@ -1765,6 +1766,12 @@ handlerton *ha_default_tmp_handlerton(THD *thd); > */ > #define HTON_TABLE_MAY_NOT_EXIST_ON_SLAVE (1 << 15) > > +/* > + True if handler cannot roolback transactions. If not true, the transaction > + will be put in the transactional binlog cache. > +*/ > +#define HTON_NO_ROLLBACK (1 << 16) How is it different from HA_PERSISTENT_TABLE? > + > class Ha_trx_info; > > struct THD_TRANS > diff --git a/sql/ha_partition.cc b/sql/ha_partition.cc > index 582c9bb110b..0c53bdc5bbe 100644 > --- a/sql/ha_partition.cc > +++ b/sql/ha_partition.cc > @@ -11016,7 +11016,7 @@ int ha_partition::check_misplaced_rows(uint read_part_id, bool do_repair) > If the engine supports transactions, the failure will be > rollbacked. if you're changing it anyway, can you also change rollbacked -> rolled back ? thanks. > */ > - if (!m_file[correct_part_id]->has_transactions()) > + if (!m_file[correct_part_id]->has_transactions_and_rollback()) > { > /* Log this error, so the DBA can notice it and fix it! */ > sql_print_error("Table '%-192s' failed to move/insert a row" > diff --git a/sql/handler.cc b/sql/handler.cc > index 39841cc28d7..e3f5773717d 100644 > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -4514,7 +4515,6 @@ void handler::mark_trx_read_write_internal() > */ > if (ha_info->is_started()) > { > - DBUG_ASSERT(has_transaction_manager()); there should be *some* assert here, I think > /* > table_share can be NULL in ha_delete_table(). See implementation > of standalone function ha_delete_table() in sql_base.cc. > diff --git a/sql/share/errmsg-utf8.txt b/sql/share/errmsg-utf8.txt > index 250da9948a0..1f3bf212d3f 100644 > --- a/sql/share/errmsg-utf8.txt > +++ b/sql/share/errmsg-utf8.txt > @@ -7961,3 +7961,5 @@ ER_KEY_CONTAINS_PERIOD_FIELDS > eng "Key %`s cannot explicitly include column %`s" > ER_KEY_CANT_HAVE_WITHOUT_OVERLAPS > eng "Key %`s cannot have WITHOUT OVERLAPS" > +ER_DATA_WAS_COMMITED_UNDER_ROLLBACK > + eng "Engine %s does not support rollback. Changes where commited during rollback call" A confusing error message. I'd just say "table is not transactional". This is the user point of view difference between a "transactional engine" like InnoDB and "crash safe engine" like Aria. It's very confusing to invent a new category of "transactional engines that cannot roll back" even if internally Aria is treated like one. > diff --git a/sql/sp.cc b/sql/sp.cc > index 51bbeeef368..1629290eb73 100644 > --- a/sql/sp.cc > +++ b/sql/sp.cc > @@ -470,27 +471,29 @@ static Proc_table_intact proc_table_intact; > currently open tables will be saved, and from which will be > restored when we will end work with mysql.proc. > > + NOTES > + On must have a start_new_trans object active when calling this function I think this: DBUG_ASSERT(thd->transcation != &thd->default_transaction); could be a bit safer than a NOTE :) > + > @retval > 0 Error > @retval > \# Pointer to TABLE object of mysql.proc > */ > > -TABLE *open_proc_table_for_read(THD *thd, Open_tables_backup *backup) > +TABLE *open_proc_table_for_read(THD *thd) > { > TABLE_LIST table; > - > DBUG_ENTER("open_proc_table_for_read"); > > table.init_one_table(&MYSQL_SCHEMA_NAME, &MYSQL_PROC_NAME, NULL, TL_READ); > > - if (open_system_tables_for_read(thd, &table, backup)) > + if (open_system_tables_for_read(thd, &table)) > DBUG_RETURN(NULL); > > if (!proc_table_intact.check(table.table, &proc_table_def)) > DBUG_RETURN(table.table); > > - close_system_tables(thd, backup); > + thd->commit_whole_transaction_and_close_tables(); > > DBUG_RETURN(NULL); > } > @@ -517,7 +520,8 @@ static TABLE *open_proc_table_for_update(THD *thd) > MDL_savepoint mdl_savepoint= thd->mdl_context.mdl_savepoint(); > DBUG_ENTER("open_proc_table_for_update"); > same? also need start_new_trans? > > - table_list.init_one_table(&MYSQL_SCHEMA_NAME, &MYSQL_PROC_NAME, NULL, TL_WRITE); > + table_list.init_one_table(&MYSQL_SCHEMA_NAME, &MYSQL_PROC_NAME, NULL, > + TL_WRITE); > > if (!(table= open_system_table_for_update(thd, &table_list))) > DBUG_RETURN(NULL); > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > index 6c938670fdc..325a0f1d41d 100644 > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -4259,7 +4259,7 @@ bool open_tables(THD *thd, const DDL_options_st &options, > list, we still need to call open_and_process_routine() to take > MDL locks on the routines. > */ > - if (thd->locked_tables_mode <= LTM_LOCK_TABLES) > + if (thd->locked_tables_mode <= LTM_LOCK_TABLES && *sroutine_to_open) why? > { > /* > Process elements of the prelocking set which are present there > @@ -8887,17 +8887,16 @@ bool is_equal(const LEX_CSTRING *a, const LEX_CSTRING *b) > open_system_tables_for_read() > thd Thread context. > table_list List of tables to open. > - backup Pointer to Open_tables_state instance where > - information about currently open tables will be > - saved, and from which will be restored when we will > - end work with system tables. > > NOTES > + Caller should have used start_new_trans object to start a new > + transcation when reading system tables. assert, may be? > + > Thanks to restrictions which we put on opening and locking of > system tables for writing, we can open and lock them for reading > - even when we already have some other tables open and locked. One > - must call close_system_tables() to close systems tables opened > - with this call. > + even when we already have some other tables open and locked. > + One should call thd->commit_whole_transaction_and_close_tables() > + to close systems tables opened with this call. > > NOTES > In some situations we use this function to open system tables for > diff --git a/sql/sql_help.cc b/sql/sql_help.cc > index c9307b578fc..3ccab553bfe 100644 > --- a/sql/sql_help.cc > +++ b/sql/sql_help.cc > @@ -709,8 +709,9 @@ static bool mysqld_help_internal(THD *thd, const char *mask) > Reset and backup the current open tables state to > make it possible. > */ > - Open_tables_backup open_tables_state_backup; > - if (open_system_tables_for_read(thd, tables, &open_tables_state_backup)) > + start_new_trans new_trans(thd); I'm starting to think that this (and in some other places) is rather redundant. Transactions can be expensive, the overhead being quite big. It seems like a overkill to wrap read-only system table accesses in a separate transaction, they could as well be performed as a part of the parent transaction. Write accesses has to be wrapped in a dedicated transaction, of course. > + > + if (open_system_tables_for_read(thd, tables)) > goto error2; > > /* > diff --git a/sql/sql_show.cc b/sql/sql_show.cc > index 2528134f4ee..db5b4d1c5fd 100644 > --- a/sql/sql_show.cc > +++ b/sql/sql_show.cc > @@ -6105,7 +6105,7 @@ static my_bool iter_schema_engines(THD *thd, plugin_ref plugin, > table->field[1]->store(option_name, strlen(option_name), scs); > table->field[2]->store(plugin_decl(plugin)->descr, > strlen(plugin_decl(plugin)->descr), scs); > - tmp= &yesno[MY_TEST(hton->commit)]; > + tmp= &yesno[MY_TEST(hton->commit && !(hton->flags & HTON_NO_ROLLBACK))]; Why not to say that hton->rollback==NULL means no rollback? Then you won't need a new flag for that. And you'll remove redundancy, what would it mean if HTON_NO_ROLLBACK is present, but hton->rollback!=NULL? Or the other way around, HTON_NO_ROLLBACK is not present, but hton->rollback==NULL? > table->field[3]->store(tmp->str, tmp->length, scs); > table->field[3]->set_notnull(); > tmp= &yesno[MY_TEST(hton->prepare)]; > diff --git a/sql/sql_statistics.cc b/sql/sql_statistics.cc > index 55e8e52c052..7a817aea97e 100644 > --- a/sql/sql_statistics.cc > +++ b/sql/sql_statistics.cc > @@ -230,17 +230,17 @@ index_stat_def= {INDEX_STAT_N_FIELDS, index_stat_fields, 4, index_stat_pk_col}; > Open all statistical tables and lock them > */ > > -static int open_stat_tables(THD *thd, TABLE_LIST *tables, > - Open_tables_backup *backup, bool for_write) > +static int open_stat_tables(THD *thd, TABLE_LIST *tables, bool for_write) > { > int rc; > - > Dummy_error_handler deh; // suppress errors > + DBUG_ASSERT(thd->transaction != &thd->default_transaction); I see, you already use such an assert :) > + > thd->push_internal_handler(&deh); > init_table_list_for_stat_tables(tables, for_write); > init_mdl_requests(tables); > thd->in_sub_stmt|= SUB_STMT_STAT_TABLES; > - rc= open_system_tables_for_read(thd, tables, backup); > + rc= open_system_tables_for_read(thd, tables); > thd->in_sub_stmt&= ~SUB_STMT_STAT_TABLES; > thd->pop_internal_handler(); > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] b22a28c2295: fixup! 3fe5cd5e1785e3e8de7add9977a1c2ddd403538b
by Sergei Golubchik 22 May '20

22 May '20

Hi, Michael! On May 20, Michael Widenius wrote: > revision-id: b22a28c2295 (mariadb-10.5.2-256-gb22a28c2295) > parent(s): acbe14b122c > author: Michael Widenius <monty(a)mariadb.com> > committer: Michael Widenius <monty(a)mariadb.com> > timestamp: 2020-05-19 17:52:18 +0300 > message: > > fixup! 3fe5cd5e1785e3e8de7add9977a1c2ddd403538b > > MDEV-22607 Assertion `ha_info->ht() != binlog_hton' failed in > MYSQL_BIN_LOG::unlog_xa_prepare > > diff --git a/sql/log.cc b/sql/log.cc > index 7e9e231358a..792c6bb1a99 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -10128,11 +10128,6 @@ int TC_LOG_BINLOG::unlog_xa_prepare(THD *thd, bool all) > if (rw_count > 0) > { > /* an empty XA-prepare event group is logged */ > -#ifndef DBUG_OFF > - for (ha_info= thd->transaction->all.ha_list; rw_count > 1 && ha_info; > - ha_info= ha_info->next()) > - DBUG_ASSERT(ha_info->ht() != binlog_hton); > -#endif > rc= write_empty_xa_prepare(thd, cache_mngr); // normally gains need_unlog > trans_register_ha(thd, true, binlog_hton, 0); // do it for future commmit > } What does this assertion mean and what does it mean that it no longer holds? Regards, Sergei

2 1

Re: [Maria-developers] 20d54b09830: Update galera to work with independent sub transactions
by Sergei Golubchik 22 May '20

22 May '20

Hi, Michael! I see, you basically disable wsrep for these out-of-band transactions. I'm sure it works fine now, with Aria. But will it work with InnoDB? On May 20, Michael Widenius wrote: > revision-id: 20d54b09830 (mariadb-10.5.2-257-g20d54b09830) > parent(s): b22a28c2295 > author: Michael Widenius <monty(a)mariadb.com> > committer: Michael Widenius <monty(a)mariadb.com> > timestamp: 2020-05-19 17:55:00 +0300 > message: > > Update galera to work with independent sub transactions > > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index 51d7380f622..4c87cbeee59 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -5806,6 +5806,8 @@ start_new_trans::start_new_trans(THD *thd) > server_status= thd->server_status; > m_transaction_psi= thd->m_transaction_psi; > thd->m_transaction_psi= 0; > + wsrep_on= thd->variables.wsrep_on; > + thd->variables.wsrep_on= 0; > thd->server_status&= ~(SERVER_STATUS_IN_TRANS | > SERVER_STATUS_IN_TRANS_READONLY); > thd->server_status|= SERVER_STATUS_AUTOCOMMIT; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] c5238d7b3b7: MDEV-22545: my_vsnprintf behaves not as in C standard
by Sergei Golubchik 20 May '20

20 May '20

Hi, Oleksandr! On May 20, Oleksandr Byelkin wrote: > revision-id: c5238d7b3b7 (mariadb-10.5.3-200-gc5238d7b3b7) > parent(s): 8f9c53586bf > author: Oleksandr Byelkin <sanja(a)mariadb.com> > committer: Oleksandr Byelkin <sanja(a)mariadb.com> > timestamp: 2020-05-20 08:35:12 +0200 > message: > > MDEV-22545: my_vsnprintf behaves not as in C standard > > Added parameter %T for string which should be visibly truncated. > > diff --git a/include/mysql/service_my_snprintf.h b/include/mysql/service_my_snprintf.h > index bd1f069c527..94a9e07dcbb 100644 > --- a/include/mysql/service_my_snprintf.h > +++ b/include/mysql/service_my_snprintf.h > @@ -69,6 +70,9 @@ > Format 'M': takes one integer, prints this integer, space, double quote > error message, double quote. In other words > printf("%M", n) === printf("%d \"%s\"", n, strerror(n)) > + > + Format 'T': takes string and print it like s but if the strints shoud be should > + truncated puts "..." at the end. > */ > > #ifdef __cplusplus > diff --git a/mysql-test/main/frm-debug.result b/mysql-test/main/frm-debug.result > index caf344e241d..332d7e00a8f 100644 > --- a/mysql-test/main/frm-debug.result > +++ b/mysql-test/main/frm-debug.result > @@ -13,8 +13,8 @@ CREATE TABLE t1 (c01 INT, c02 CHAR(20), c03 TEXT, c04 DOUBLE); > Warnings: > Note 1105 build_frm_image: Field data type info length: 14 > Note 1105 DBUG: [0] name='c01' type_info='' > -Note 1105 DBUG: [1] name='c02' type_info='xc...' > -Note 1105 DBUG: [2] name='c03' type_info='xb...' > +Note 1105 DBUG: [1] name='c02' type_info='xchar' > +Note 1105 DBUG: [2] name='c03' type_info='xblob' good > Note 1105 DBUG: [3] name='c04' type_info='' > SET SESSION debug_dbug="-d,frm_data_type_info_emulate"; > SET SESSION debug_dbug="-d,frm_data_type_info"; > diff --git a/mysql-test/main/xml.result b/mysql-test/main/xml.result > index efaca961b4a..84dbd68438f 100644 > --- a/mysql-test/main/xml.result > +++ b/mysql-test/main/xml.result > @@ -1007,7 +1007,7 @@ Warning 1292 Truncated incorrect INTEGER value: 'string ' > Warning 1292 Truncated incorrect INTEGER value: 'string ' > DROP PROCEDURE spxml; > select UpdateXML('<a>a</a>',repeat('a b ',1000),''); > -ERROR HY000: XPATH syntax error: 'b a b a b a b a b a b a b a b...' > +ERROR HY000: XPATH syntax error: 'b a b a b a b a b a b a b a b a ' this doesn't look right, the string is truncated, isn't it? > select ExtractValue('<a>a</a>', '/a[@x=@y0123456789_0123456789_0123456789_0123456789]'); > ERROR HY000: XPATH error: comparison of two nodesets is not supported: '=@y0123456789_0123456789_0123...' > select ExtractValue('<a>a</a>', '/a[@x=$y0123456789_0123456789_0123456789_0123456789]'); > @@ -1110,9 +1110,9 @@ NULL > # Bug#57279 updatexml dies with: Assertion failed: str_arg[length] == 0 > # > SELECT UPDATEXML(NULL, (LPAD(0.1111E-15, '2011', 1)), 1); > -ERROR 22007: Illegal double '111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111...' value found during parsing > +ERROR 22007: Illegal double '111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111' value found during parsing same here, lpad appends 2011 characters, so this "double" is clearly truncated. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] b97e45651d1: MDEV-16937 Strict SQL with system versioned tables causes issues
by Sergei Golubchik 19 May '20

19 May '20

Hi, Aleksey! First - is commit b97e45651d1 the one I was supposed to review? On May 17, Aleksey Midenkov wrote: > revision-id: b97e45651d1 (mariadb-10.4.7-33-gb97e45651d1) > parent(s): 7587975bf06 > author: Aleksey Midenkov <midenok(a)gmail.com> > committer: Aleksey Midenkov <midenok(a)gmail.com> > timestamp: 2019-08-19 11:58:07 +0300 > message: > > MDEV-16937 Strict SQL with system versioned tables causes issues > > Vers SQL: respect system fields in NO_ZERO_DATE mode. > > This is the subject for refactoring in tempesta-tech/mariadb#379 > > diff --git a/mysql-test/suite/versioning/engines.combinations b/mysql-test/suite/versioning/engines.combinations > index 26b5bab23f1..57e2af6cd06 100644 > --- a/mysql-test/suite/versioning/engines.combinations > +++ b/mysql-test/suite/versioning/engines.combinations > @@ -7,5 +7,10 @@ default-storage-engine=innodb > [myisam] > default-storage-engine=myisam > > +[traditional] > +default-storage-engine=myisam > +sql-mode=traditional I'm not sure about it. It's, of course, safe to run all tests with every possible settings, but it really blows up testing times. May be it'd make sense to have a more targeted test here? > + > [heap] > default-storage-engine=memory > + > diff --git a/mysql-test/suite/versioning/r/select.result b/mysql-test/suite/versioning/r/select.result > index 3569268ce1d..68df246af6b 100644 > --- a/mysql-test/suite/versioning/r/select.result > +++ b/mysql-test/suite/versioning/r/select.result > @@ -45,7 +45,7 @@ ASOF_x y > 7 107 > 8 108 > 9 109 > -select x as FROMTO_x, y from t1 for system_time from timestamp '0-0-0 0:0:0' to timestamp @t1; > +select x as FROMTO_x, y from t1 for system_time from timestamp '1970-01-01 00:00:00' to timestamp @t1; Does this test use a fixed time zone? If not, is 1970-01-01 00:00:00 a valid timestamp in all time zones? If unsure, better use 1971, that will definitely be fine everywhere. > FROMTO_x y > 0 100 > 1 101 > diff --git a/sql/field.cc b/sql/field.cc > index 969c32a5180..f5580239a9f 100644 > --- a/sql/field.cc > +++ b/sql/field.cc > @@ -11139,6 +11139,13 @@ bool Field::save_in_field_default_value(bool view_error_processing) > { > THD *thd= table->in_use; > > + /* > + TODO: refactor setting the system fields via default_value mechanism. > + This condition will go away as well as other conditions with VERS_SYSTEM_FIELD. > + */ MDEV? There's a pull request, but it is marked as obsolete > + if (flags & VERS_SYSTEM_FIELD) > + return false; > + > if (unlikely(flags & NO_DEFAULT_VALUE_FLAG && > real_type() != MYSQL_TYPE_ENUM)) > { > diff --git a/sql/unireg.cc b/sql/unireg.cc > index d019b5f8a75..484ad0334e4 100644 > --- a/sql/unireg.cc > +++ b/sql/unireg.cc > @@ -1060,7 +1060,8 @@ static bool make_empty_rec(THD *thd, uchar *buff, uint table_options, > !f_bit_as_char(field->pack_flag)) > null_count+= field->length & 7; > > - error= make_empty_rec_store_default(thd, regfield, field->default_value); > + if (!(field->flags & VERS_SYSTEM_FIELD)) > + error= make_empty_rec_store_default(thd, regfield, field->default_value); will the field be initialized at all? It should have something, even bzero will do, but we shouldn't dump uninited memory to frm. > delete regfield; // Avoid memory leaks > if (error) > goto err; > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 3

[Maria-developers] Progress report - Week 2
by Mohammed Hammaad Mateen 18 May '20

18 May '20

Greetings, Hope you are safe and doing great, This post describes the things I've done in my second week [ 11-17 May ] of Community Bonding Period under the mentor-ship of Sergei Golubchik and Oleksandr Byelkin for GSoC-20. Tasks done this week were mostly the continuation of the leftover tasks from the previous week and some related study. Things I've done include: 1:* I was caught up in analysing how was DELETE .... RETURNING implemented, which led me to read from:* • *sp_head.cc* : this made me understand that sp actually meant stored procedures but there is another called PS which I'm yet to figure out. • *sql_parse.cc* : from here I understood that delete return as actually { Analyse->Delete->Return } and all the logic for returning is executed if(lex->has_returning()). 2: Though I've raised the pull request, *MDEV-14558* is not yet completely tested as it does not include tests for *mysql \s*, moreover, I'm facing problems to escape the backslash inorder to write a test-case for *mysql \s* . Regards, Mohammed Hammaad Mateen

1 0

Re: [Maria-developers] 82b6a5cb85f: MDEV-22556: Incorrect result for window function when using encrypt-tmp-files=ON
by Sergei Golubchik 17 May '20

17 May '20

Hi, Varun! Ok to push. One comment below On May 17, Varun Gupta wrote: > revision-id: 82b6a5cb85f (mariadb-10.4.11-192-g82b6a5cb85f) > parent(s): 3bfe305c5cd > author: Varun Gupta <varun.gupta(a)mariadb.com> > committer: Varun Gupta <varun.gupta(a)mariadb.com> > timestamp: 2020-05-15 02:37:16 +0530 > message: > > MDEV-22556: Incorrect result for window function when using encrypt-tmp-files=ON > > diff --git a/mysql-test/suite/encryption/t/tempfiles_encrypted.opt b/mysql-test/suite/encryption/t/tempfiles_encrypted.opt > new file mode 100644 > index 00000000000..f515f7b8689 > --- /dev/null > +++ b/mysql-test/suite/encryption/t/tempfiles_encrypted.opt > @@ -0,0 +1,5 @@ > +--encrypt-tmp_files=ON > +--loose-file-key-management-encryption-algorithm=aes_cbc > +--file-key-management=FORCE > +--loose-file-key-management-filename=$MTR_SUITE_DIR/t/filekeys-data.enc > +--loose-file-key-management-filekey=FILE:$MTR_SUITE_DIR/t/filekeys-data.key this is very redundant, you include have_file_key_management_plugin.inc and it already sets all options for the file_key_management plugin to work. so you only need --encrypt-tmp_files=ON here. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] Progress Report - Week 1-2
by Anton Mikhaylenko 17 May '20

17 May '20

Good day, everyone. In this post, I want to describe the things I've done in the 1-2 weeks of GSoC-20. For begin, I messaged with my mentor Vladislav Vaintroub and I got some tips on where to start. The first step was to run the server and tests for MariaDB on Linux and Windows. At this stage, a little bug was found, which was documented by my mentor https://jira.mariadb.org/browse/MDEV-22479. The next step was to look at the theoretical part of my work. Specifically, what algorithm to use for dynamic concurrency. This is the algorithm "hill-climbing", which is used in .NET ThreadPool https://www.researchgate.net/publication/228977836_Optimizing_concurrency_l… . In the last step, I started to get into the work of the current ThreadPool in MariaDB. At the moment this step is not completed yet. What needs to be done next: Fully understand the theoretic part and how the current thread pool works. Regards, Anton MIkhailenko

1 0

Re: [Maria-developers] Review for MDEV-19751.
by Aleksey Midenkov 14 May '20

14 May '20

Hi, Alexey! Actually the bug is applicable only to primary key and only when default field list is set (i.e. empty KEY() clause). So updated the patch accordingly. Please peek again. On Wed, May 13, 2020 at 2:40 PM Alexey Botchkov <holyfoot(a)mariadb.com> wrote: > > Hello, Aleksey! > > > commit 5ef168e00f7b8c821463e46535a4beccb47ec8ea > + if (key_info->flags & HA_INVISIBLE_KEY) > + continue; > > I'd add || !(key_info->flags & HA_NOSAME). > Can the non-unique index participate in the KEY partitioning? > > + for (uint kp= 0; kp < key_info->user_defined_key_parts; ++kp) > + { > + const KEY_PART_INFO &key_part= key_info->key_part[kp]; > + for (Field **part_field= tab_part_info->part_field_array; > + *part_field; ++part_field) > + { > + if (*part_field == key_part.field) > + { > + *partition_changed= TRUE; > + goto search_finished; > + } > + } // for (part_field) > + } // for (key_part) > > So you check if the key has any of the partitioning fields in it. > In this case we are going to disallow INPLACE with no reason if the > unrelated key is removed just containing the primary key column. > I'd say we should check for the exact match, when the key has > all the part_field_array in that exact order. Disagree? > > And for the test - please add the case for a table with no PRIMARY KEY, bug with the UNIQUE instead. > > Best regards! > HF > -- All the best, Aleksey Midenkov @midenok

1 0

[Maria-developers] Progress Report - Week 1
by Mohammed Hammaad Mateen 12 May '20

12 May '20

Greetings, Hope you are safe and doing great, This post describes the things I've done in my first week [ 4-10 May ] of Community Bonding Period under the mentor-ship of Sergei Golubchik and Oleksandr Byelkin for GSoC-20 and I must acknowledge that it was really an interactive experience and at the same time informative as well. Things I've done include: 1: *Raising my first ever pull request for MariaDB under MDEV-14558 *: Learnt about *strnxmov*, a function of C++ which I had never used earlier. I also came across the procedure of testing my implemented code. Wrote some test cases for the same and generated their corresponding results. 2:* Understand the working of the functions under multi_delete*: Ran it in the debugger and tried to familiarize myself with the internals of how Multiple Table Delete actually works. Things to-do: 1: Though I've raised the pull request, MDEV-14558 is not completed tested as it does not include tests for *mysql \s*, moreover, test cases have to be refactored. 2: Learn more about the working of *multi_delete* so as to develop a parser for Multiple Table Delete Returning. 3: Write some test cases for Multiple Table Delete Returning before hand. Regards, Mohammed Hammaad Mateen

2 1

[Maria-developers] Review for: MDEV-17399 Add support for JSON_TABLE, part #3
by Sergey Petrunia 10 May '20

10 May '20

Hi Alexey, Please find the next batch of input below. (I haven't finished the review but I'm sending it now so you can start addressing it) > commit 68ca18518337443090bdeddf6d612129b6843c21 > Author: Alexey Botchkov <holyfoot(a)askmonty.org> > Date: Fri Apr 17 14:42:40 2020 +0400 > > MDEV-17399 Add support for JSON_TABLE. > > Syntax for JSON_TABLE added. > The ha_json_table handler added. Executing the JSON_TABLE we > create the temporary table of the ha_json_table, add dependencies > of other tables and sometimes conditions to WHERE. > ... > +public: > + ha_json_table(TABLE_SHARE *share_arg, Table_function_json_table *jt): > + handler(&table_function_hton.m_hton, share_arg), m_jt(jt) > + { > + mark_trx_read_write_done= 1; This definitely needs a comment. > + ~ha_json_table() {} > + handler *clone(const char *name, MEM_ROOT *mem_root) { return NULL; } > + const char *index_type(uint inx) { return "HASH"; } Why is ha_json_table still trying to pretend it supports indexes? Is this a left-over from the old way of interfacing with the optimizer that can now be removed? > +int Table_function_json_table::setup(THD *thd, TABLE_LIST *sql_table, > + COND **cond) > +{ > + if (m_json->fix_fields(thd, &m_json)) > + return TRUE; > + > + m_dep_tables= m_json->used_tables(); Why does this function have a 'cond' argument? Please remove it if it's not necessary. == Duplicate column names are allowed == select * from JSON_TABLE('{"color": "black", "price": 1000}', "$" COLUMNS (rowSeq FOR ORDINALITY, color VARCHAR(100) PATH '$.color', color VARCHAR(100) PATH '$.price') ) as T; +--------+-------+-------+ | rowSeq | color | color | +--------+-------+-------+ | 0 | black | 1000 | +--------+-------+-------+ Is it really okay that duplicate column names are allowed? == RIGHT JOIN is allowed == create table t1 (item_name varchar(32), item_props varchar(1024)); insert into t1 values ('Laptop', '{"color": "black", "price": 1000}'); insert into t1 values ('Jeans', '{"color": "blue", "price": 50}'); This query should not be allowed, because JSON_TABLE has a dependency on an inner side of an outer join: select * from t1 right join json_table(t1.item_props, '$' columns(color varchar(100) path '$.color', f2 for ordinality) ) as T on 1; === NATURAL JOIN crashes == create table t10 (color varchar(100), f2 int); select * from t10 natural join json_table(t1.item_props, '$' columns(color varchar(100) path '$.color', f2 for ordinality) ) as JT ; Thread 31 "mysqld" received signal SIGSEGV, Segmentation fault. 0x0000555555d8ac12 in Select_limit_counters::get_select_limit (this=0xa5a5a5a5a5a5ad2d) at /home/psergey/dev-git/10.5-json/sql/sql_limit.h:67 (gdb) wher #0 0x0000555555d8ac12 in Select_limit_counters::get_select_limit (this=0xa5a5a5a5a5a5ad2d) at /home/psergey/dev-git/10.5-json/sql/sql_limit.h:67 #1 0x0000555555e2a4a0 in JOIN::optimize_inner (this=0x7fffac018520) at /home/psergey/dev-git/10.5-json/sql/sql_select.cc:1805 #2 0x0000555555e29b12 in JOIN::optimize (this=0x7fffac018520) at /home/psergey/dev-git/10.5-json/sql/sql_select.cc:1612 BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 1

[Maria-developers] OSX fix need on MariaDB-10.5
by Daniel Black 09 May '20

09 May '20

Mac lovers (who can use C/C++ and a debugger), As your MariaDB developers are hard at work getting 10.5 ready for you, unfortunately a bug has slipped into the 10.5 branch making it unusable on OSX (at least that's what it looks like on the CI infrastructure). The bug report is here: https://jira.mariadb.org/browse/MDEV-22173 As the time of developers is a little short as is the availability of OSX hardware, I'm hoping some of you are able to spend a few hours tackling this task to understand it better, and if possible, create patch. Steps: 1. build environment https://mariadb.com/kb/en/Build_Environment_Setup_for_Mac/ 2. clone the source git clone --branch 10.5 https://github.com/MariaDB/server.git mariadb-server cd mariadb-server git submodule update --recursive --init 3. build instructions https://mariadb.com/kb/en/generic-build-instructions/ (you don't need to install) 4. test (from build directory) mysql-test/mysql-test-run main.select might be useful here: --debugger=NAME or --gdb client-debugger=NAME or client-gdb To collaborative share your findings and to ask for further help MariaDB's community chat stream has been created for you. https://mariadb.zulipchat.com/#narrow/stream/118759-general/topic/Get.20OSX… To paraphrase J.F.Kennedy ask not what your database server community can do for you — ask what you can do for your database server community. Keep the community spirit going, Daniel Black -- Sent as a community member who noticed OSX failures and that everyone is busy

2 3

Re: [Maria-developers] 4312f382b6f: MDEV-22312: Bad error message for SET DEFAULT ROLE when user account is not granted the role
by Sergei Golubchik 08 May '20

08 May '20

Hi, Anel! On May 08, Anel Husakovic wrote: > revision-id: 4312f382b6f (mariadb-10.1.43-138-g4312f382b6f) > parent(s): 8c4b5261210 > author: Anel Husakovic <anel(a)mariadb.org> > committer: Anel Husakovic <anel(a)mariadb.org> > timestamp: 2020-05-07 16:49:33 +0200 > message: > > MDEV-22312: Bad error message for SET DEFAULT ROLE when user account is not granted the role > > diff --git a/mysql-test/suite/roles/set_role-recursive.test b/mysql-test/suite/roles/set_role-recursive.test > index 23d623e1966..dcf3dd5b6fc 100644 > --- a/mysql-test/suite/roles/set_role-recursive.test > +++ b/mysql-test/suite/roles/set_role-recursive.test > @@ -44,7 +44,7 @@ select * from mysql.roles_mapping; > > --sorted_result > show grants; > ---error ER_INVALID_ROLE > +--error ER_SET_DEFAULT_ROLE_FOR_USER > set role test_role2; > select current_user(), current_role(); > --sorted_result > diff --git a/sql/share/errmsg-utf8.txt b/sql/share/errmsg-utf8.txt > index c5e83062f0f..83ecd4735e8 100644 > --- a/sql/share/errmsg-utf8.txt > +++ b/sql/share/errmsg-utf8.txt > @@ -7144,3 +7144,5 @@ ER_WARN_AGGFUNC_DEPENDENCE > ukr "Агрегатна функція '%-.192s)' з SELECTу #%d належить до SELECTу #%d" > WARN_INNODB_PARTITION_OPTION_IGNORED > eng "<%-.64s> option ignored for InnoDB partition" > +ER_SET_DEFAULT_ROLE_FOR_USER > + eng "User '%s' has not been granted a role '%s'." Sorry, this is an absolute no-no. We cannot add new error messages to the old GA versions. Only to the very last RC/GA or to any alpha/beta. When you add an error message, all error messages added after it in newer versions will shift, change their numbers. And error numbers in RC/GA versions cannot change. Instead of a new error message and my_error(ER_SET_DEFAULT_ROLE_FOR_USER, MYF(0), c_usr.ptr(), rolename); you have to write my_printf_error(ER_INVALID_ROLE, "User %`s has not been granted a role %`s", MYF(0), c_usr.c_ptr(), rolename); note other changes: * %`s instead of '%s' for correct identifier quoting * no dot at the end * c_ptr() because printf expects a zero-terminated C string, and ptr() doesn't guarantee that. > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index b6a6f806e50..37141dd2390 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -2041,6 +2041,9 @@ static int check_user_can_set_role(const char *user, const char *host, > ACL_USER_BASE *acl_user_base; > ACL_USER *UNINIT_VAR(acl_user); > bool is_granted= FALSE; > + bool user_can_see_role= FALSE; > + char buf[1024]; > + String c_usr(buf, sizeof(buf), &my_charset_bin); use StringBuffer<1024> c_str; > int result= 0; > > /* clear role privileges */ > @@ -2083,14 +2086,39 @@ static int check_user_can_set_role(const char *user, const char *host, > is_granted= TRUE; > break; > } > + > + for (uint i=0; i < acl_user->role_grants.elements; i++) > + { > + ACL_ROLE *r= *(dynamic_element(&acl_user->role_grants, i, ACL_ROLE**)); > + if (!strcmp(safe_str(r->user.str), safe_str(rolename))) > + { > + size_t len= acl_user->user.length + acl_user->hostname_length + 2; > + c_usr.alloc(len); > + const char c= '@'; > + strmov(strmov(strmov(const_cast<char*>(c_usr.ptr()), > + safe_str(user)), &c), safe_str(host)); > + user_can_see_role= TRUE; > + } > + } No, you repeat this for() block for evert role grant of every role's grantee. And you only use it when there was an error, an uncommon case. Do not do the work that you aren't going to use. And the check doesn't seem correct. You want to use the same rule as for applicable_roles, you need to traverse the graph. Perhaps you can simply use check_role_is_granted() though. And also you need to check whether the current user has read access on mysql.user table - in that case it can see all roles, granted or not. > } > > /* According to SQL standard, the same error message must be presented */ this comment now belongs inside the if(), above ER_INVALID_ROLE > if (!is_granted) > { > - my_error(ER_INVALID_ROLE, MYF(0), rolename); > - result= 1; > - goto end; > + /* Handling of the bad error message - MDEV-22312 */ don't put MDEV numbers into the code, please. > + /* If the current user can see the role generate different type of error. */ > + if(user_can_see_role) > + { > + my_error(ER_SET_DEFAULT_ROLE_FOR_USER, MYF(0), c_usr.ptr(), rolename); > + result= 1; > + goto end; > + } > + else > + { > + my_error(ER_INVALID_ROLE, MYF(0), rolename); > + result= 1; > + goto end; > + } > } > > if (access) Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] Welcome to Google Summer of Code 2020 with MariaDB
by Vicențiu Ciorbaru 04 May '20

04 May '20

Hi! GSoC 2020 accepted proposals have just been announced and you are part of the successful applicants. This year we had a lot of very good proposals and so we have a one of the highest number of students (7) to mentor for GSoC. Congratulations on the effort to learn about the program, get involved and submit successful proposals! The fun part begins now! If you have not done so already, please reach out to your mentor and seek further advice about your project. Each one of your projects can be implemented in multiple ways, some more complex than others. I strongly advise you to create a concrete plan based off of your proposal with your mentor and aim to stick to it. The MariaDB Server codebase is a complex one with lots of different parts, however your mentor can guide you to ensure you have a successful GSoC. Remember that you are here to learn about Open Source, what it means to be part of an Open Source community, to interact with the community, all while becoming better programmers. Please take advantage of this opportunity and keep your technical discussions public, on Zulip https://mariadb.zulipchat.com, and on our developer mailing list https://launchpad.net/~maria-developers. By keeping your discussions public, you get the benefit of not only having your mentor see your questions but all of the MariaDB community. You never know when someone might lend a hand in case your mentor is not yet online. :) The mantra of our community should be that we are always welcoming to new people. Do not be afraid to ask questions. There are no bad or stupid or silly questions. The only key requirement is that you first make an effort to find the answer yourself and only afterwards seek outside help. If you can show a bit of involvement, the community will show you the same. I recommend that you agree with your mentor on some form of weekly progress reporting, via either a blog-post, an email, or something similar that can be publicly shared. I can not stress this enough: we are working on an open source project, our work must be done in the open and it is important to *show the world the good we are doing*. Hope all of you will have a great GSoC! Vicențiu MariaDB Foundation GSoC Admin

1 0

Re: [Maria-developers] d08860b28f3: MDEV-22374: VIEW with security definer require FILE privilege from definer not invoker in case of INTO OUTFILE
by Sergei Golubchik 04 May '20

04 May '20

Hi, Oleksandr! On May 04, Oleksandr Byelkin wrote: > revision-id: d08860b28f3 (mariadb-5.5.67-15-gd08860b28f3) > parent(s): ac2604f923f > author: Oleksandr Byelkin <sanja(a)mariadb.com> > committer: Oleksandr Byelkin <sanja(a)mariadb.com> > timestamp: 2020-04-28 09:16:33 +0200 > message: > > MDEV-22374: VIEW with security definer require FILE privilege from definer not invoker in case of INTO OUTFILE > > Check INTO OUTFILE clause always from invoker. > > --- > mysql-test/r/view_grant.result | 21 +++++++++++++++++++++ > mysql-test/t/view_grant.test | 33 +++++++++++++++++++++++++++++++++ > sql/sql_parse.cc | 18 +++++++++--------- > 3 files changed, 63 insertions(+), 9 deletions(-) > > diff --git a/mysql-test/r/view_grant.result b/mysql-test/r/view_grant.result > index b2d3a0b8ca4..37dc6adc978 100644 > --- a/mysql-test/r/view_grant.result > +++ b/mysql-test/r/view_grant.result > @@ -1587,3 +1587,24 @@ USE test; > DROP DATABASE mysqltest1; > DROP USER 'mysqluser1'@'%'; > DROP USER 'mysqluser2'@'%'; > +# > +# MDEV-22374: VIEW with security definer require FILE privilege from definer > +# not invoker in case of INTO OUTFILE > +# > +create user test@localhost; > +grant select on test.* to test@localhost; > +create table t1 (a int); > +create definer=test@localhost sql security definer view v1 as select * from t1; > +# check that ot works without view typo "it" > +select * INTO OUTFILE '../..//tmp/test_out_txt' from t1; > +# check that ot works without file same why "//" ? > +select * from v1; > +a > +# rights for file should be taken from current user not view > +select * INTO OUTFILE '../../tmp/test_out_txt' from (select count(*) from v1) as dv1; > +select * INTO OUTFILE '../..//tmp/test_out_txt' from (select * from v1) as dv1; > +select * INTO OUTFILE '../..//tmp/test_out_txt' from v1; same > +drop view v1; please add reverse tests too: +create sql security definer view v1 as select * from t1; +connect(con1, localhost, test); +error 1045; +select * INTO OUTFILE '../..//tmp/test_out_txt' from t1; +select * from v1; +error 1045; +select * INTO OUTFILE '../../tmp/test_out_txt' from (select count(*) from v1) as dv1; +error 1045; +select * INTO OUTFILE '../..//tmp/test_out_txt' from (select * from v1) as dv1; +error 1045; +select * INTO OUTFILE '../..//tmp/test_out_txt' from v1; +connection default; > +drop table t1; > +drop user test@localhost; > +# End of 5.5 tests > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > index ae5a6b4cd35..515990c879f 100644 > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -2206,15 +2206,15 @@ mysql_execute_command(THD *thd) > lex->exchange != NULL implies SELECT .. INTO OUTFILE and this > requires FILE_ACL access. > */ > - ulong privileges_requested= lex->exchange ? SELECT_ACL | FILE_ACL : > - SELECT_ACL; > - > - if (all_tables) > - res= check_table_access(thd, > - privileges_requested, > - all_tables, FALSE, UINT_MAX, FALSE); > - else > - res= check_access(thd, privileges_requested, any_db, NULL, NULL, 0, 0); > + if (lex->exchange) > + res= check_access(thd, FILE_ACL, any_db, NULL, NULL, 0, 0); > + if (!res) > + { > + if (all_tables) > + res= check_table_access(thd, SELECT_ACL, all_tables, FALSE, UINT_MAX, FALSE); > + else > + res= check_access(thd, SELECT_ACL, any_db, NULL, NULL, 0, 0); > + } No, I think this special check for a special SELECT ... INTO OUTFILE case is wrong. Why does the server even checks global privileges via views? It should not do it, I think, not for FILE not for any other global or db level privilege. I'd try something like diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc --- a/sql/sql_acl.cc +++ b/sql/sql_acl.cc @@ -4699,7 +4699,7 @@ bool check_grant(THD *thd, ulong want_access, TABLE_LIST > Save a copy of the privileges without the SHOW_VIEW_ACL attribute. It will be checked during making view. */ - tl->grant.orig_want_privilege= (want_access & ~SHOW_VIEW_ACL); + tl->grant.orig_want_privilege= want_access & ~SHOW_VIEW_ACL & TABLE_ACLS; } mysql_rwlock_rdlock(&LOCK_grant); diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc --- a/sql/sql_parse.cc +++ b/sql/sql_parse.cc @@ -5204,7 +5204,8 @@ check_table_access(THD *thd, ulong requirements,TABLE_LIS> Register access for view underlying table. Remove SHOW_VIEW_ACL, because it will be checked during making view */ - table_ref->grant.orig_want_privilege= (want_access & ~SHOW_VIEW_ACL); + table_ref->grant.orig_want_privilege= + want_access & ~SHOW_VIEW_ACL & TABLE_ACLS; if (table_ref->schema_table_reformed) { Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] 3ead2cea95c: MDEV-13266: Race condition in ANALYZE TABLE / statistics collection
by Sergey Petrunia 04 May '20

04 May '20

Hi Varun, On Sun, Apr 12, 2020 at 09:08:37PM +0530, Varun wrote: > revision-id: 3ead2cea95c32b7ceaf6e6ec81f7afbd9137cfe9 (mariadb-10.2.31-103-g3ead2cea95c) > parent(s): d565895bbd8e2a163be48b9bac51fccbf3949c80 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2020-04-12 21:05:36 +0530 > message: > > MDEV-13266: Race condition in ANALYZE TABLE / statistics collection > > Fixing a race condition while collecting the engine independent statistics. > The issue here was when the statistics was collected on specific indexes > then because of some race condition the statistics for indexes was not > collected. The TABLE::keys_in_use_for_query was set to 0 in such cases. > > This happens when the table is opened from TABLE_SHARE instead of the > table share stored in table_cache. It would be nice to write down the race condition that we've caught (in case we need this info later) Is my understanding correct: the error scenario is as follows: Thread1> start running "ANALYZE TABLE t PERISTENT FOR COLUMNS (..) INDEXES ($list)" Thread1> Walk through $list and save it in TABLE::keys_in_use_for_query Thread1> Close/re-open tables Thread2> Make some use of table t. This involves taking table t from Thread2> the table cache, and putting it back (with TABLE::keys_in_use_for_query reset to 0) Thread1> continue collecting EITS stats. Since TABLE::keys_in_use_for_query we will not collect statistics for indexes in $list. Please confirm (and if not, describe the race condition). The patch also introduces this behaviour: analyze table ... persistent for ... indexes(no_such_index); will now cause engine statistics to be still collected. Before the patch it exited with an error. But then, this is consistent with what happens for analyze table ... persistent for columns(no_such_column) ... So I guess this is ok. > --- > sql/sql_admin.cc | 46 +++++++++++++++++++++------------------------- > 1 file changed, 21 insertions(+), 25 deletions(-) > > diff --git a/sql/sql_admin.cc b/sql/sql_admin.cc > index ab95fdc340c..0613495f202 100644 > --- a/sql/sql_admin.cc > +++ b/sql/sql_admin.cc > @@ -769,31 +769,6 @@ static bool mysql_admin_table(THD* thd, TABLE_LIST* tables, > (table->table->s->table_category == TABLE_CATEGORY_USER && > (get_use_stat_tables_mode(thd) > NEVER || > lex->with_persistent_for_clause)); > - > - > - if (!lex->index_list) > - { > - tab->keys_in_use_for_query.init(tab->s->keys); > - } > - else > - { > - int pos; > - LEX_STRING *index_name; > - List_iterator_fast<LEX_STRING> it(*lex->index_list); > - > - tab->keys_in_use_for_query.clear_all(); > - while ((index_name= it++)) > - { > - if (tab->s->keynames.type_names == 0 || > - (pos= find_type(&tab->s->keynames, index_name->str, > - index_name->length, 1)) <= 0) > - { > - compl_result_code= result_code= HA_ADMIN_INVALID; > - break; > - } > - tab->keys_in_use_for_query.set_bit(--pos); > - } > - } > } > > if (result_code == HA_ADMIN_OK) > @@ -878,6 +853,27 @@ static bool mysql_admin_table(THD* thd, TABLE_LIST* tables, > } > tab->file->column_bitmaps_signal(); > } > + if (!lex->index_list) > + tab->keys_in_use_for_query.init(tab->s->keys); > + else > + { > + int pos; > + LEX_STRING *index_name; > + List_iterator_fast<LEX_STRING> it(*lex->index_list); > + > + tab->keys_in_use_for_query.clear_all(); > + while ((index_name= it++)) > + { > + if (tab->s->keynames.type_names == 0 || > + (pos= find_type(&tab->s->keynames, index_name->str, > + index_name->length, 1)) <= 0) > + { > + compl_result_code= result_code= HA_ADMIN_INVALID; > + break; > + } > + tab->keys_in_use_for_query.set_bit(--pos); > + } > + } > if (!(compl_result_code= > alloc_statistics_for_table(thd, table->table)) && > !(compl_result_code= > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

2 2

[Maria-developers] Igor please review: MDEV-22377: Subquery in an [UPDATE] query uses full scan instead of range
by Sergey Petrunia 29 Apr '20

29 Apr '20

Hello Igor, Could you please review the below: commit 793b100cc01228f01ead841d6c45f9e2b07786f9 Author: Sergei Petrunia <psergey(a)askmonty.org> Date: Thu Apr 30 01:25:11 2020 +0300 MDEV-22377: Subquery in an [UPDATE] query uses full scan instead of range When doing IN->EXISTS rewrite, Item_in_subselect::inject_in_to_exists_cond injects equalities into subquery's WHERE condition. The problem is that build_equal_items() has already been called for the subquery's WHERE, and tampering with the WHERE condition can prevent equality propagation from working. If the subquery's WHERE is an Item_cond_and with multiple equalities: - Item_equal objects form a suffix sub-list of the list in WHERE's Item_cond_and::list. The suffix is stored in JOIN::cond_equal->current_level. - Item_cond_and::m_cond_equal must also be preserved. This patch makes inject_in_to_exists_cond() not to break these properties diff --git a/mysql-test/main/subselect2.result b/mysql-test/main/subselect2.result index a3d7fda7abc..c54d635230f 100644 --- a/mysql-test/main/subselect2.result +++ b/mysql-test/main/subselect2.result @@ -262,7 +262,7 @@ id select_type table type possible_keys key key_len ref rows filtered Extra 2 DEPENDENT SUBQUERY t2c ALL NULL NULL NULL NULL 2 100.00 Using where; Using join buffer (incremental, BNL join) Warnings: Note 1276 Field or reference 'test.t1.a' of SELECT #2 was resolved in SELECT #1 -Note 1003 /* select#1 */ select `test`.`t1`.`a` AS `a`,`test`.`t1`.`c1` AS `c1` from `test`.`t1` where !<expr_cache><`test`.`t1`.`c1`,`test`.`t1`.`a`>(<in_optimizer>(`test`.`t1`.`c1`,<exists>(/* select#2 */ select `test`.`t2a`.`c2` from `test`.`t2` `t2a` join `test`.`t2` `t2b` join `test`.`t2` `t2c` where (`test`.`t2b`.`m` <> `test`.`t1`.`a` or `test`.`t2b`.`m` = `test`.`t2a`.`m`) and trigcond(<cache>(`test`.`t1`.`c1`) = `test`.`t2a`.`c2` or `test`.`t2a`.`c2` is null) and `test`.`t2c`.`c2` = `test`.`t2b`.`c2` and `test`.`t2b`.`n` = `test`.`t2a`.`m` having trigcond(`test`.`t2a`.`c2` is null)))) +Note 1003 /* select#1 */ select `test`.`t1`.`a` AS `a`,`test`.`t1`.`c1` AS `c1` from `test`.`t1` where !<expr_cache><`test`.`t1`.`c1`,`test`.`t1`.`a`>(<in_optimizer>(`test`.`t1`.`c1`,<exists>(/* select#2 */ select `test`.`t2a`.`c2` from `test`.`t2` `t2a` join `test`.`t2` `t2b` join `test`.`t2` `t2c` where `test`.`t2c`.`c2` = `test`.`t2b`.`c2` and `test`.`t2b`.`n` = `test`.`t2a`.`m` and (`test`.`t2b`.`m` <> `test`.`t1`.`a` or `test`.`t2b`.`m` = `test`.`t2a`.`m`) and trigcond(<cache>(`test`.`t1`.`c1`) = `test`.`t2a`.`c2` or `test`.`t2a`.`c2` is null) having trigcond(`test`.`t2a`.`c2` is null)))) DROP TABLE t1,t2; # # MDEV-614, also MDEV-536, also LP:1050806: diff --git a/mysql-test/main/subselect3.result b/mysql-test/main/subselect3.result index 5c4544a1b05..0034f61ac23 100644 --- a/mysql-test/main/subselect3.result +++ b/mysql-test/main/subselect3.result @@ -169,7 +169,7 @@ id select_type table type possible_keys key key_len ref rows filtered Extra 2 DEPENDENT SUBQUERY t2 ref a a 5 test.t1.b 1 100.00 Using where Warnings: Note 1276 Field or reference 'test.t3.oref' of SELECT #2 was resolved in SELECT #1 -Note 1003 /* select#1 */ select `test`.`t3`.`a` AS `a`,`test`.`t3`.`oref` AS `oref`,<expr_cache><`test`.`t3`.`a`,`test`.`t3`.`oref`>(<in_optimizer>(`test`.`t3`.`a`,<exists>(/* select#2 */ select `test`.`t1`.`a` from `test`.`t1` join `test`.`t2` where `test`.`t2`.`b` = `test`.`t3`.`oref` and trigcond(<cache>(`test`.`t3`.`a`) = `test`.`t1`.`a` or `test`.`t1`.`a` is null) and `test`.`t2`.`a` = `test`.`t1`.`b` having trigcond(`test`.`t1`.`a` is null)))) AS `Z` from `test`.`t3` +Note 1003 /* select#1 */ select `test`.`t3`.`a` AS `a`,`test`.`t3`.`oref` AS `oref`,<expr_cache><`test`.`t3`.`a`,`test`.`t3`.`oref`>(<in_optimizer>(`test`.`t3`.`a`,<exists>(/* select#2 */ select `test`.`t1`.`a` from `test`.`t1` join `test`.`t2` where `test`.`t2`.`a` = `test`.`t1`.`b` and `test`.`t2`.`b` = `test`.`t3`.`oref` and trigcond(<cache>(`test`.`t3`.`a`) = `test`.`t1`.`a` or `test`.`t1`.`a` is null) having trigcond(`test`.`t1`.`a` is null)))) AS `Z` from `test`.`t3` drop table t1, t2, t3; create table t1 (a int NOT NULL, b int NOT NULL, key(a)); insert into t1 values @@ -197,7 +197,7 @@ id select_type table type possible_keys key key_len ref rows filtered Extra 2 DEPENDENT SUBQUERY t2 ref a a 4 test.t1.b 1 100.00 Using where Warnings: Note 1276 Field or reference 'test.t3.oref' of SELECT #2 was resolved in SELECT #1 -Note 1003 /* select#1 */ select `test`.`t3`.`a` AS `a`,`test`.`t3`.`oref` AS `oref`,<expr_cache><`test`.`t3`.`a`,`test`.`t3`.`oref`>(<in_optimizer>(`test`.`t3`.`a`,<exists>(/* select#2 */ select `test`.`t1`.`a` from `test`.`t1` join `test`.`t2` where `test`.`t2`.`b` = `test`.`t3`.`oref` and trigcond(<cache>(`test`.`t3`.`a`) = `test`.`t1`.`a`) and `test`.`t2`.`a` = `test`.`t1`.`b`))) AS `Z` from `test`.`t3` +Note 1003 /* select#1 */ select `test`.`t3`.`a` AS `a`,`test`.`t3`.`oref` AS `oref`,<expr_cache><`test`.`t3`.`a`,`test`.`t3`.`oref`>(<in_optimizer>(`test`.`t3`.`a`,<exists>(/* select#2 */ select `test`.`t1`.`a` from `test`.`t1` join `test`.`t2` where `test`.`t2`.`a` = `test`.`t1`.`b` and `test`.`t2`.`b` = `test`.`t3`.`oref` and trigcond(<cache>(`test`.`t3`.`a`) = `test`.`t1`.`a`)))) AS `Z` from `test`.`t3` drop table t1,t2,t3; create table t1 (oref int, grp int); insert into t1 (oref, grp) values diff --git a/mysql-test/main/subselect3_jcl6.result b/mysql-test/main/subselect3_jcl6.result index 4260676cc37..b7b18bf80e0 100644 --- a/mysql-test/main/subselect3_jcl6.result +++ b/mysql-test/main/subselect3_jcl6.result @@ -172,7 +172,7 @@ id select_type table type possible_keys key key_len ref rows filtered Extra 2 DEPENDENT SUBQUERY t2 ref a a 5 test.t1.b 1 100.00 Using where; Using join buffer (flat, BKA join); Key-ordered Rowid-ordered scan Warnings: Note 1276 Field or reference 'test.t3.oref' of SELECT #2 was resolved in SELECT #1 -Note 1003 /* select#1 */ select `test`.`t3`.`a` AS `a`,`test`.`t3`.`oref` AS `oref`,<expr_cache><`test`.`t3`.`a`,`test`.`t3`.`oref`>(<in_optimizer>(`test`.`t3`.`a`,<exists>(/* select#2 */ select `test`.`t1`.`a` from `test`.`t1` join `test`.`t2` where `test`.`t2`.`b` = `test`.`t3`.`oref` and trigcond(<cache>(`test`.`t3`.`a`) = `test`.`t1`.`a` or `test`.`t1`.`a` is null) and `test`.`t2`.`a` = `test`.`t1`.`b` having trigcond(`test`.`t1`.`a` is null)))) AS `Z` from `test`.`t3` +Note 1003 /* select#1 */ select `test`.`t3`.`a` AS `a`,`test`.`t3`.`oref` AS `oref`,<expr_cache><`test`.`t3`.`a`,`test`.`t3`.`oref`>(<in_optimizer>(`test`.`t3`.`a`,<exists>(/* select#2 */ select `test`.`t1`.`a` from `test`.`t1` join `test`.`t2` where `test`.`t2`.`a` = `test`.`t1`.`b` and `test`.`t2`.`b` = `test`.`t3`.`oref` and trigcond(<cache>(`test`.`t3`.`a`) = `test`.`t1`.`a` or `test`.`t1`.`a` is null) having trigcond(`test`.`t1`.`a` is null)))) AS `Z` from `test`.`t3` drop table t1, t2, t3; create table t1 (a int NOT NULL, b int NOT NULL, key(a)); insert into t1 values @@ -200,7 +200,7 @@ id select_type table type possible_keys key key_len ref rows filtered Extra 2 DEPENDENT SUBQUERY t2 ref a a 4 test.t1.b 1 100.00 Using where; Using join buffer (flat, BKA join); Key-ordered Rowid-ordered scan Warnings: Note 1276 Field or reference 'test.t3.oref' of SELECT #2 was resolved in SELECT #1 -Note 1003 /* select#1 */ select `test`.`t3`.`a` AS `a`,`test`.`t3`.`oref` AS `oref`,<expr_cache><`test`.`t3`.`a`,`test`.`t3`.`oref`>(<in_optimizer>(`test`.`t3`.`a`,<exists>(/* select#2 */ select `test`.`t1`.`a` from `test`.`t1` join `test`.`t2` where `test`.`t2`.`b` = `test`.`t3`.`oref` and trigcond(<cache>(`test`.`t3`.`a`) = `test`.`t1`.`a`) and `test`.`t2`.`a` = `test`.`t1`.`b`))) AS `Z` from `test`.`t3` +Note 1003 /* select#1 */ select `test`.`t3`.`a` AS `a`,`test`.`t3`.`oref` AS `oref`,<expr_cache><`test`.`t3`.`a`,`test`.`t3`.`oref`>(<in_optimizer>(`test`.`t3`.`a`,<exists>(/* select#2 */ select `test`.`t1`.`a` from `test`.`t1` join `test`.`t2` where `test`.`t2`.`a` = `test`.`t1`.`b` and `test`.`t2`.`b` = `test`.`t3`.`oref` and trigcond(<cache>(`test`.`t3`.`a`) = `test`.`t1`.`a`)))) AS `Z` from `test`.`t3` drop table t1,t2,t3; create table t1 (oref int, grp int); insert into t1 (oref, grp) values diff --git a/mysql-test/main/subselect4.result b/mysql-test/main/subselect4.result index 3dc019cc6c1..e355cfc60f0 100644 --- a/mysql-test/main/subselect4.result +++ b/mysql-test/main/subselect4.result @@ -2349,7 +2349,7 @@ id select_type table type possible_keys key key_len ref rows filtered Extra 1 PRIMARY t2 system NULL NULL NULL NULL 1 100.00 2 DEPENDENT SUBQUERY t1 ALL NULL NULL NULL NULL 2 100.00 Using where Warnings: -Note 1003 /* select#1 */ select 3 AS `f` from dual where !<expr_cache><3>(<in_optimizer>(3,<exists>(/* select#2 */ select `test`.`t1`.`b` from `test`.`t1` where (`test`.`t1`.`c` = 'USA' or `test`.`t1`.`c` <> 'USA') and trigcond(<cache>(3) = `test`.`t1`.`b` or `test`.`t1`.`b` is null) and `test`.`t1`.`b` = `test`.`t1`.`a` having trigcond(`test`.`t1`.`b` is null)))) +Note 1003 /* select#1 */ select 3 AS `f` from dual where !<expr_cache><3>(<in_optimizer>(3,<exists>(/* select#2 */ select `test`.`t1`.`b` from `test`.`t1` where `test`.`t1`.`b` = `test`.`t1`.`a` and (`test`.`t1`.`c` = 'USA' or `test`.`t1`.`c` <> 'USA') and trigcond(<cache>(3) = `test`.`t1`.`b` or `test`.`t1`.`b` is null) having trigcond(`test`.`t1`.`b` is null)))) SELECT * FROM t2 WHERE f NOT IN (SELECT b FROM t1 WHERE 0 OR (c IN ('USA') OR c NOT IN ('USA')) AND a = b); @@ -2582,3 +2582,29 @@ id select_type table type possible_keys key key_len ref rows Extra 1 PRIMARY <subquery2> eq_ref distinct_key distinct_key 4 func 1 2 MATERIALIZED t2 ALL NULL NULL NULL NULL 100 drop table t0, t1, t2; +# +# MDEV-22377: Subquery in an UPDATE query uses full scan instead of range +# +create table t0(a int); +insert into t0 values (0),(1),(2),(3),(4),(5),(6),(7),(8),(9); +create table t1(nkey1 int, col1 int); +insert into t1 select A.a+10*B.a+100*C.a, A.a+10*B.a+100*C.a from t0 A, t0 B, t0 C; +create table t2(key1 int, col1 int, key(key1)); +insert into t2 select A.a+10*B.a, A.a+10*B.a from t0 A, t0 B; +alter table t0 add b int; +# This must use the join order of t0,t2,t1 +# And table t2 must use range access, not full scan: +explain +select * +from t0 +where +a not in (SELECT t1.col1 +FROM t1 JOIN t2 ON t2.key1 = t1.nkey1 +WHERE +t1.nkey1 IN (1,2,3,4,5) and t2.col1>t0.b +); +id select_type table type possible_keys key key_len ref rows Extra +1 PRIMARY t0 ALL NULL NULL NULL NULL 10 Using where +2 DEPENDENT SUBQUERY t2 range key1 key1 5 NULL 5 Using index condition; Using where +2 DEPENDENT SUBQUERY t1 ALL NULL NULL NULL NULL 1000 Using where; Using join buffer (flat, BNL join) +drop table t0, t1, t2; diff --git a/mysql-test/main/subselect4.test b/mysql-test/main/subselect4.test index bc8db41c505..e3d8b5be832 100644 --- a/mysql-test/main/subselect4.test +++ b/mysql-test/main/subselect4.test @@ -2115,6 +2115,32 @@ where exists (select * from t2 where t2.a=t1.a order by t2.b limit 2,3); explain select * from t1 where t1.a in (select t2.a from t2 order by t2.b); +drop table t0, t1, t2; + +--echo # +--echo # MDEV-22377: Subquery in an UPDATE query uses full scan instead of range +--echo # +create table t0(a int); +insert into t0 values (0),(1),(2),(3),(4),(5),(6),(7),(8),(9); +create table t1(nkey1 int, col1 int); +insert into t1 select A.a+10*B.a+100*C.a, A.a+10*B.a+100*C.a from t0 A, t0 B, t0 C; + +create table t2(key1 int, col1 int, key(key1)); +insert into t2 select A.a+10*B.a, A.a+10*B.a from t0 A, t0 B; + +alter table t0 add b int; +--echo # This must use the join order of t0,t2,t1 +--echo # And table t2 must use range access, not full scan: +explain +select * +from t0 +where + a not in (SELECT t1.col1 + FROM t1 JOIN t2 ON t2.key1 = t1.nkey1 + WHERE + t1.nkey1 IN (1,2,3,4,5) and t2.col1>t0.b + ); drop table t0, t1, t2; + diff --git a/mysql-test/main/subselect_mat_cost_bugs.result b/mysql-test/main/subselect_mat_cost_bugs.result index a18c5e608f1..755f1e2ad13 100644 --- a/mysql-test/main/subselect_mat_cost_bugs.result +++ b/mysql-test/main/subselect_mat_cost_bugs.result @@ -100,7 +100,7 @@ id select_type table type possible_keys key key_len ref rows filtered Extra 2 DEPENDENT SUBQUERY t2 index c3 c3 9 NULL 2 100.00 Using where; Using index; Using join buffer (flat, BNL join) Warnings: Note 1276 Field or reference 'test.t1.pk' of SELECT #2 was resolved in SELECT #1 -Note 1003 /* select#1 */ select `test`.`t1`.`pk` AS `pk` from `test`.`t1` where <expr_cache><`test`.`t1`.`c1`,`test`.`t1`.`pk`>(<in_optimizer>(`test`.`t1`.`c1`,<exists>(/* select#2 */ select `test`.`t1a`.`c1` from `test`.`t1b` join `test`.`t2` left join `test`.`t1a` on(`test`.`t1a`.`c2` = `test`.`t1b`.`pk` and 2) where `test`.`t1`.`pk` <> 0 and <cache>(`test`.`t1`.`c1`) = `test`.`t1a`.`c1` and `test`.`t2`.`c3` = `test`.`t1b`.`c4`))) +Note 1003 /* select#1 */ select `test`.`t1`.`pk` AS `pk` from `test`.`t1` where <expr_cache><`test`.`t1`.`c1`,`test`.`t1`.`pk`>(<in_optimizer>(`test`.`t1`.`c1`,<exists>(/* select#2 */ select `test`.`t1a`.`c1` from `test`.`t1b` join `test`.`t2` left join `test`.`t1a` on(`test`.`t1a`.`c2` = `test`.`t1b`.`pk` and 2) where `test`.`t2`.`c3` = `test`.`t1b`.`c4` and `test`.`t1`.`pk` <> 0 and <cache>(`test`.`t1`.`c1`) = `test`.`t1a`.`c1`))) SELECT pk FROM t1 WHERE c1 IN @@ -364,7 +364,7 @@ AND a = SOME (SELECT b FROM t5)); id select_type table type possible_keys key key_len ref rows Extra 1 PRIMARY t3 ALL NULL NULL NULL NULL 2 Using where 2 DEPENDENT SUBQUERY t5 index c c 10 NULL 2 Using where; Using index; Start temporary -2 DEPENDENT SUBQUERY t4 eq_ref PRIMARY PRIMARY 4 test.t5.b 1 Using index condition; Using where; End temporary +2 DEPENDENT SUBQUERY t4 eq_ref PRIMARY PRIMARY 4 test.t5.b 1 Using where; End temporary SELECT * FROM t3 WHERE t3.b > ALL ( diff --git a/sql/item_subselect.cc b/sql/item_subselect.cc index 736dfbd33b4..726c58e49a2 100644 --- a/sql/item_subselect.cc +++ b/sql/item_subselect.cc @@ -2704,9 +2704,28 @@ bool Item_in_subselect::inject_in_to_exists_cond(JOIN *join_arg) and_args= ((Item_cond*) join_arg->conds)->argument_list(); if (join_arg->cond_equal) and_args->disjoin((List<Item> *) &join_arg->cond_equal->current_level); + + /* + If subquery's WHERE is an Item_cond_and, add the injected conditions + into it manually. If we just called and_items(...) that could + potentially replace the top-level Item_cond_and and current top-level + Item_cond_and::m_cond_equal will be lost. + */ + if (where_item->type() == Item::COND_ITEM && + ((Item_cond*)where_item)->functype() == Item_func::COND_AND_FUNC) + { + List_iterator<Item> it(*((Item_cond*)where_item)->argument_list()); + while (Item *item=it++) + and_args->push_back(item); + } + else + and_args->push_back(where_item); + + where_item= join_arg->conds; } + else + where_item= and_items(thd, join_arg->conds, where_item); - where_item= and_items(thd, join_arg->conds, where_item); if (where_item->fix_fields_if_needed(thd, 0)) DBUG_RETURN(true); // TIMOUR TODO: call optimize_cond() for the new where clause @@ -2719,12 +2738,15 @@ bool Item_in_subselect::inject_in_to_exists_cond(JOIN *join_arg) { /* The argument list of the top-level AND may change after fix fields. */ and_args= ((Item_cond*) join_arg->conds)->argument_list(); - List_iterator<Item_equal> li(join_arg->cond_equal->current_level); - Item_equal *elem; - while ((elem= li++)) - { - and_args->push_back(elem, thd->mem_root); - } + + /* + Note that Item_equal objects must be present in this form: + Item_cond::conds list must have the end of the list to be the + same linked list as join_arg->cond_equal->current_level. + The code in substitute_for_best_equal_field depends on it (see the + disjoin() call). + */ + and_args->append((List<Item> *)&join_arg->cond_equal->current_level); } } BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] 96aee3229a2: MENT-703: Disable rpl.rpl_skip_replication, rpl.rpl_set_statement_default_master, rpl.rpl_extra_col_master_myisam, rpl.rpl_slave_load_tmpdir_not_exist tests in ES 10.2 and above
by sujatha 29 Apr '20

29 Apr '20

revision-id: 96aee3229a288dae52c457ca808966092da6d821 (mariadb-10.4.5-669-g96aee3229a2) parent(s): a284ea1909d66e4284a964a4ca7d45b75c450ed7 author: Sujatha committer: Sujatha timestamp: 2020-04-29 13:28:23 +0530 message: MENT-703: Disable rpl.rpl_skip_replication, rpl.rpl_set_statement_default_master, rpl.rpl_extra_col_master_myisam, rpl.rpl_slave_load_tmpdir_not_exist tests in ES 10.2 and above Problem: ======= Disable following tests in ES 10.2 and above, till MDEV-13258 and MDEV-14203 are fixed. rpl.rpl_skip_replication, rpl.rpl_set_statement_default_master, rpl.rpl_extra_col_master_myisam, rpl.rpl_slave_load_tmpdir_not_exist --- mysql-test/suite/rpl/disabled.def | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/mysql-test/suite/rpl/disabled.def b/mysql-test/suite/rpl/disabled.def index 77b01bd8d1d..5a04f4ccd15 100644 --- a/mysql-test/suite/rpl/disabled.def +++ b/mysql-test/suite/rpl/disabled.def @@ -19,3 +19,7 @@ rpl_parallel2 : fails after MDEV-16172 rpl_semi_sync_after_sync : fails after MDEV-16172 rpl_slave_grp_exec: MDEV-10514 rpl_auto_increment_update_failure : disabled for now +rpl_skip_replication : MDEV-13258 +rpl_set_statement_default_master : MDEV-13258 +rpl_extra_col_master_myisam : MDEV-14203 +rpl_slave_load_tmpdir_not_exist : MDEV-14203

1 0

[Maria-developers] 7f8dd7b2044: MENT-703: Disable rpl.rpl_skip_replication, rpl.rpl_set_statement_default_master, rpl.rpl_extra_col_master_myisam, rpl.rpl_slave_load_tmpdir_not_exist tests in ES 10.2 and above
by sujatha 29 Apr '20

29 Apr '20

revision-id: 7f8dd7b20443e0247ca478cbfd98c43749acdae0 (mariadb-10.2.24-722-g7f8dd7b2044) parent(s): 52cb9e87e1d23aeb0958966807050a6a843c181c author: Sujatha committer: Sujatha timestamp: 2020-04-29 11:33:41 +0530 message: MENT-703: Disable rpl.rpl_skip_replication, rpl.rpl_set_statement_default_master, rpl.rpl_extra_col_master_myisam, rpl.rpl_slave_load_tmpdir_not_exist tests in ES 10.2 and above Problem: ======= Disable following tests in ES 10.2 and above, till MDEV-13258 and MDEV-14203 are fixed. rpl.rpl_skip_replication, rpl.rpl_set_statement_default_master, rpl.rpl_extra_col_master_myisam, rpl.rpl_slave_load_tmpdir_not_exist --- mysql-test/suite/rpl/disabled.def | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/mysql-test/suite/rpl/disabled.def b/mysql-test/suite/rpl/disabled.def index bdefb1660bd..951f8a6aa0f 100644 --- a/mysql-test/suite/rpl/disabled.def +++ b/mysql-test/suite/rpl/disabled.def @@ -16,3 +16,7 @@ rpl_partition_archive : MDEV-5077 2013-09-27 svoj Cannot exchange partition rpl_row_binlog_max_cache_size : MDEV-11092 rpl_row_index_choice : MDEV-11666 rpl_slave_grp_exec: MDEV-10514 +rpl_skip_replication : MDEV-13258 +rpl_set_statement_default_master : MDEV-13258 +rpl_extra_col_master_myisam : MDEV-14203 +rpl_slave_load_tmpdir_not_exist : MDEV-14203

1 0

Re: [Maria-developers] [Commits] 85b116d1ab9: MDEV-22317: SIGSEGV in my_free/delete_dynamic in optimized builds (ARIA)
by sujatha 28 Apr '20

28 Apr '20

Hello Andrei, Good morning. Thank you for the review comments. Please find my replies inline. On 27/04/20 11:26 pm, Andrei Elkin wrote: > Sujatha, howdy. > > Thanks for a prompt checking the case! > The patch looks good, only > >> - if (!wild_do_table.elements) >> + if (status && wild_do_table_inited) > I would suggest to reorder the && arg:s to be then > in temporal wrt their evaluation order. Sure Andrei. I have refined changes as shown below. diff --git a/sql/rpl_filter.cc b/sql/rpl_filter.cc index 635a0f4e2d6..8035763bf35 100644 --- a/sql/rpl_filter.cc +++ b/sql/rpl_filter.cc @@ -416,10 +416,13 @@ Rpl_filter::set_wild_do_table(const char* table_spec) status= parse_filter_rule(table_spec, &Rpl_filter::add_wild_do_table); - if (!wild_do_table.elements) + if (wild_do_table_inited && status) { - delete_dynamic(&wild_do_table); - wild_do_table_inited= 0; + if (!wild_do_table.elements) + { + delete_dynamic(&wild_do_table); + wild_do_table_inited= 0; + } } > >> revision-id: 85b116d1ab9ea85dcef63d259b8f6366466e2750 (mariadb-10.5.2-185-g85b116d1ab9) >> parent(s): fbe2712705d464bf8488df249c36115e2c1f63f7 >> author: Sujatha >> committer: Sujatha >> timestamp: 2020-04-27 17:43:51 +0530 >> message: >> >> MDEV-22317: SIGSEGV in my_free/delete_dynamic in optimized builds (ARIA) >> >> Problem: >> ======= >> SET @@GLOBAL.replicate_wild_ignore_table=''; >> SET @@GLOBAL.replicate_wild_do_table=''; >> >> Reports following valgrind error. >> >> Conditional jump or move depends on uninitialised value(s) >> Rpl_filter::set_wild_ignore_table(char const*) (rpl_filter.cc:439) >> >> Conditional jump or move depends on uninitialised value(s) >> at 0xF60390: delete_dynamic (array.c:304) >> by 0x74F3F2: Rpl_filter::set_wild_do_table(char const*) (rpl_filter.cc:421) >> >> Analysis: >> ======== >> List of values provided for options "wild_do_table" and "wild_ignore_table" are >> stored in DYNAMIC_ARRAYS. When an empty list is provided these dynamic arrays >> are not initialized. > Correct. And that's 'cos > > Rpl_filter::Rpl_filter() : > parallel_mode(SLAVE_PARALLEL_CONSERVATIVE), > table_rules_on(0), > do_table_inited(0), ignore_table_inited(0), > wild_do_table_inited(0), wild_ignore_table_inited(0) > { > do_db.empty(); > ignore_db.empty(); > rewrite_db.empty(); > } > > does not initialize the two members of your patch's interest with > empty()-ing. > I wonder if > > + wild_do_table.empty() > + wild_ignore_table.empty() > ? > > > that could work out as well which looks as more consistent solution to > me. [It feels like binlog_filter object might object this idea though]. 'do_db' and 'ignore_db' are of type list iterators. Hence they have 'empty' defined for them. 'empty' is not a member of DYNAMIC_ARRAY. I_List<i_string> do_db; I_List<i_string> ignore_db; /home/sujatha/bug_repo/MDEV-22317-10.5/sql/rpl_filter.cc:36:17: error: ‘DYNAMIC_ARRAY {aka struct st_dynamic_array}’ has no member named ‘empty’ wild_do_table.empty(); ^~~~~ /home/sujatha/bug_repo/MDEV-22317-10.5/sql/rpl_filter.cc:37:21: error: ‘DYNAMIC_ARRAY {aka struct st_dynamic_array}’ has no member named ‘empty’ wild_ignore_table.empty(); ^~~~~ "wild_do_table" and "wild_ignore_table" arrays get initialized as part of "add_wild_do_table" and "add_wild_ignore_table". The initialization is done through following calls. status= parse_filter_rule(table_spec, &Rpl_filter::add_wild_do_table); status= parse_filter_rule(table_spec, &Rpl_filter::add_wild_ignore_table); When default empty string is provided for replicate_wild_ignore_table='' and replicate_wild_do_table='' the dynamic array is not initialized. 'wild_do_table.elements' will be '0'. No cleanup is required in this case. Hence we should do the cleanup only when 'wild_do_table_inited' is true. Please let me know your thoughts. Thank you S.Sujatha > > Cheers, > > Andrei > > >> Existing code treats empty element list as an error and >> tries to clean the uninitialized list. This results in above valgrind issue. >> >> Fix: >> === >> The clean up should be initiated only when there is an error while parsing the >> 'wild_do_table' or 'wild_ignore_table' list and the dynamic_array is in >> initialized state. Otherwise for empty list it should simply return success. >> >> --- >> mysql-test/suite/rpl/r/rpl_filter_wild_tables_dynamic.result | 2 ++ >> mysql-test/suite/rpl/t/rpl_filter_wild_tables_dynamic.test | 2 ++ >> sql/rpl_filter.cc | 4 ++-- >> 3 files changed, 6 insertions(+), 2 deletions(-) >> >> diff --git a/mysql-test/suite/rpl/r/rpl_filter_wild_tables_dynamic.result b/mysql-test/suite/rpl/r/rpl_filter_wild_tables_dynamic.result >> index 47cd362a549..fe0b283fc7c 100644 >> --- a/mysql-test/suite/rpl/r/rpl_filter_wild_tables_dynamic.result >> +++ b/mysql-test/suite/rpl/r/rpl_filter_wild_tables_dynamic.result >> @@ -7,6 +7,8 @@ SET @@GLOBAL.replicate_wild_ignore_table="test.b%"; >> ERROR HY000: This operation cannot be performed as you have a running slave ''; run STOP SLAVE '' first >> connection slave; >> include/stop_slave.inc >> +SET @@GLOBAL.replicate_wild_do_table=""; >> +SET @@GLOBAL.replicate_wild_ignore_table=""; >> SET @@GLOBAL.replicate_wild_do_table="test.a%"; >> SET @@GLOBAL.replicate_wild_ignore_table="test.b%"; >> include/start_slave.inc >> diff --git a/mysql-test/suite/rpl/t/rpl_filter_wild_tables_dynamic.test b/mysql-test/suite/rpl/t/rpl_filter_wild_tables_dynamic.test >> index 6db61927eec..657a95cec15 100644 >> --- a/mysql-test/suite/rpl/t/rpl_filter_wild_tables_dynamic.test >> +++ b/mysql-test/suite/rpl/t/rpl_filter_wild_tables_dynamic.test >> @@ -13,6 +13,8 @@ SET @@GLOBAL.replicate_wild_ignore_table="test.b%"; >> >> connection slave; >> source include/stop_slave.inc; >> +SET @@GLOBAL.replicate_wild_do_table=""; >> +SET @@GLOBAL.replicate_wild_ignore_table=""; >> SET @@GLOBAL.replicate_wild_do_table="test.a%"; >> SET @@GLOBAL.replicate_wild_ignore_table="test.b%"; >> source include/start_slave.inc; >> diff --git a/sql/rpl_filter.cc b/sql/rpl_filter.cc >> index 635a0f4e2d6..49b498d3568 100644 >> --- a/sql/rpl_filter.cc >> +++ b/sql/rpl_filter.cc >> @@ -416,7 +416,7 @@ Rpl_filter::set_wild_do_table(const char* table_spec) >> >> status= parse_filter_rule(table_spec, &Rpl_filter::add_wild_do_table); >> >> - if (!wild_do_table.elements) >> + if (status && wild_do_table_inited) >> { >> delete_dynamic(&wild_do_table); >> wild_do_table_inited= 0; >> @@ -436,7 +436,7 @@ Rpl_filter::set_wild_ignore_table(const char* table_spec) >> >> status= parse_filter_rule(table_spec, &Rpl_filter::add_wild_ignore_table); >> >> - if (!wild_ignore_table.elements) >> + if (status && wild_ignore_table_inited) >> { >> delete_dynamic(&wild_ignore_table); >> wild_ignore_table_inited= 0; >> _______________________________________________ >> commits mailing list >> commits(a)mariadb.org >> https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

[Maria-developers] 85b116d1ab9: MDEV-22317: SIGSEGV in my_free/delete_dynamic in optimized builds (ARIA)
by sujatha 27 Apr '20

27 Apr '20

revision-id: 85b116d1ab9ea85dcef63d259b8f6366466e2750 (mariadb-10.5.2-185-g85b116d1ab9) parent(s): fbe2712705d464bf8488df249c36115e2c1f63f7 author: Sujatha committer: Sujatha timestamp: 2020-04-27 17:43:51 +0530 message: MDEV-22317: SIGSEGV in my_free/delete_dynamic in optimized builds (ARIA) Problem: ======= SET @@GLOBAL.replicate_wild_ignore_table=''; SET @@GLOBAL.replicate_wild_do_table=''; Reports following valgrind error. Conditional jump or move depends on uninitialised value(s) Rpl_filter::set_wild_ignore_table(char const*) (rpl_filter.cc:439) Conditional jump or move depends on uninitialised value(s) at 0xF60390: delete_dynamic (array.c:304) by 0x74F3F2: Rpl_filter::set_wild_do_table(char const*) (rpl_filter.cc:421) Analysis: ======== List of values provided for options "wild_do_table" and "wild_ignore_table" are stored in DYNAMIC_ARRAYS. When an empty list is provided these dynamic arrays are not initialized. Existing code treats empty element list as an error and tries to clean the uninitialized list. This results in above valgrind issue. Fix: === The clean up should be initiated only when there is an error while parsing the 'wild_do_table' or 'wild_ignore_table' list and the dynamic_array is in initialized state. Otherwise for empty list it should simply return success. --- mysql-test/suite/rpl/r/rpl_filter_wild_tables_dynamic.result | 2 ++ mysql-test/suite/rpl/t/rpl_filter_wild_tables_dynamic.test | 2 ++ sql/rpl_filter.cc | 4 ++-- 3 files changed, 6 insertions(+), 2 deletions(-) diff --git a/mysql-test/suite/rpl/r/rpl_filter_wild_tables_dynamic.result b/mysql-test/suite/rpl/r/rpl_filter_wild_tables_dynamic.result index 47cd362a549..fe0b283fc7c 100644 --- a/mysql-test/suite/rpl/r/rpl_filter_wild_tables_dynamic.result +++ b/mysql-test/suite/rpl/r/rpl_filter_wild_tables_dynamic.result @@ -7,6 +7,8 @@ SET @@GLOBAL.replicate_wild_ignore_table="test.b%"; ERROR HY000: This operation cannot be performed as you have a running slave ''; run STOP SLAVE '' first connection slave; include/stop_slave.inc +SET @@GLOBAL.replicate_wild_do_table=""; +SET @@GLOBAL.replicate_wild_ignore_table=""; SET @@GLOBAL.replicate_wild_do_table="test.a%"; SET @@GLOBAL.replicate_wild_ignore_table="test.b%"; include/start_slave.inc diff --git a/mysql-test/suite/rpl/t/rpl_filter_wild_tables_dynamic.test b/mysql-test/suite/rpl/t/rpl_filter_wild_tables_dynamic.test index 6db61927eec..657a95cec15 100644 --- a/mysql-test/suite/rpl/t/rpl_filter_wild_tables_dynamic.test +++ b/mysql-test/suite/rpl/t/rpl_filter_wild_tables_dynamic.test @@ -13,6 +13,8 @@ SET @@GLOBAL.replicate_wild_ignore_table="test.b%"; connection slave; source include/stop_slave.inc; +SET @@GLOBAL.replicate_wild_do_table=""; +SET @@GLOBAL.replicate_wild_ignore_table=""; SET @@GLOBAL.replicate_wild_do_table="test.a%"; SET @@GLOBAL.replicate_wild_ignore_table="test.b%"; source include/start_slave.inc; diff --git a/sql/rpl_filter.cc b/sql/rpl_filter.cc index 635a0f4e2d6..49b498d3568 100644 --- a/sql/rpl_filter.cc +++ b/sql/rpl_filter.cc @@ -416,7 +416,7 @@ Rpl_filter::set_wild_do_table(const char* table_spec) status= parse_filter_rule(table_spec, &Rpl_filter::add_wild_do_table); - if (!wild_do_table.elements) + if (status && wild_do_table_inited) { delete_dynamic(&wild_do_table); wild_do_table_inited= 0; @@ -436,7 +436,7 @@ Rpl_filter::set_wild_ignore_table(const char* table_spec) status= parse_filter_rule(table_spec, &Rpl_filter::add_wild_ignore_table); - if (!wild_ignore_table.elements) + if (status && wild_ignore_table_inited) { delete_dynamic(&wild_ignore_table); wild_ignore_table_inited= 0;

1 0

Re: [Maria-developers] 8a990ad1774: MDEV-18319 BIGINT UNSIGNED Performance issue
by Sergei Golubchik 27 Apr '20

27 Apr '20

Hi, Alexander! On Apr 26, Alexander Barkov wrote: > revision-id: 8a990ad1774 > author: Alexander Barkov <bar(a)mariadb.com> > committer: Alexander Barkov <bar(a)mariadb.com> > timestamp: 2019-03-25 19:19:48 +0400 > message: MDEV-18319 BIGINT UNSIGNED Performance issue > > diff --git a/mysql-test/main/errors.result b/mysql-test/main/errors.result > index ba05a2b37d45..96ad96395aab 100644 > --- a/mysql-test/main/errors.result > +++ b/mysql-test/main/errors.result > @@ -32,16 +32,21 @@ set sql_mode=default; > CREATE TABLE t1 (a INT); > SELECT a FROM t1 WHERE a IN(1, (SELECT IF(1=0,1,2/0))); > a > +Warnings: > +Warning 1365 Division by 0 Interesting. There was no warning before. > INSERT INTO t1 VALUES(1); > SELECT a FROM t1 WHERE a IN(1, (SELECT IF(1=0,1,2/0))); > a > 1 > diff --git a/mysql-test/main/func_in.result b/mysql-test/main/func_in.result > index 65313148bf80..59822ae1049c 100644 > --- a/mysql-test/main/func_in.result > +++ b/mysql-test/main/func_in.result > @@ -489,6 +489,7 @@ SELECT id FROM t1 WHERE id IN(4564, (SELECT IF(1=0,1,1/0)) ); > id > Warnings: > Warning 1365 Division by 0 > +Warning 1365 Division by 0 This is not very nice. May be we should use Item_cache in these cases? > DROP TABLE t1; > End of 5.0 tests > create table t1(f1 char(1)); > @@ -909,3 +910,71 @@ Warnings: > Warning 1292 Truncated incorrect time value: '' > Warning 1292 Truncated incorrect time value: '' > Warning 1292 Truncated incorrect time value: '' > +# > +# MDEV-18319 BIGINT UNSIGNED Performance issue > +# > +CREATE TABLE t1 ( > +id bigint(20) unsigned NOT NULL AUTO_INCREMENT PRIMARY KEY > +); > +FOR i IN 0..255 > +DO > +INSERT INTO t1 VALUES (); > +END FOR > +$$ > +SELECT MIN(id), MAX(id), COUNT(*) FROM t1; > +MIN(id) MAX(id) COUNT(*) > +1 256 256 interesting. why it's min=1 and max=255, if your loop is in 0..255? > +EXPLAIN SELECT id FROM t1 WHERE id IN (1,2); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 range PRIMARY PRIMARY 8 NULL 2 Using where; Using index > +EXPLAIN SELECT id FROM t1 WHERE id IN (9223372036854775806, 9223372036854775807); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 range PRIMARY PRIMARY 8 NULL 2 Using where; Using index > +EXPLAIN SELECT id FROM t1 WHERE id IN (9223372036854775807, 9223372036854775808); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 range PRIMARY PRIMARY 8 NULL 2 Using where; Using index > +EXPLAIN SELECT id FROM t1 WHERE id IN (1.0,2.0); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 range PRIMARY PRIMARY 8 NULL 2 Using where; Using index > +EXPLAIN SELECT id FROM t1 WHERE id IN (9223372036854775806.0, 9223372036854775807.0); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 range PRIMARY PRIMARY 8 NULL 2 Using where; Using index > +# Cannot compare this as INT (yet) > +EXPLAIN SELECT id FROM t1 WHERE id IN (9223372036854775807.0, 9223372036854775808.0); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 index PRIMARY PRIMARY 8 NULL 256 Using where; Using index > +DROP TABLE t1; > +CREATE TABLE t1 ( > +id bigint(20) NOT NULL AUTO_INCREMENT PRIMARY KEY > +); > +FOR i IN 0..255 > +DO > +INSERT INTO t1 VALUES (); > +END FOR > +$$ > +SELECT MIN(id), MAX(id), COUNT(*) FROM t1; > +MIN(id) MAX(id) COUNT(*) > +1 256 256 > +EXPLAIN SELECT id FROM t1 WHERE id IN (-1,-2); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 range PRIMARY PRIMARY 8 NULL 2 Using where; Using index > +EXPLAIN SELECT id FROM t1 WHERE id IN (-9223372036854775806, -9223372036854775807); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 range PRIMARY PRIMARY 8 NULL 2 Using where; Using index > +EXPLAIN SELECT id FROM t1 WHERE id IN (-9223372036854775807, -9223372036854775808); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 range PRIMARY PRIMARY 8 NULL 2 Using where; Using index > +EXPLAIN SELECT id FROM t1 WHERE id IN (-1.0,-2.0); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 range PRIMARY PRIMARY 8 NULL 2 Using where; Using index > +EXPLAIN SELECT id FROM t1 WHERE id IN (-9223372036854775806.0, -9223372036854775807.0); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 range PRIMARY PRIMARY 8 NULL 2 Using where; Using index > +EXPLAIN SELECT id FROM t1 WHERE id IN (-9223372036854775807.0, -9223372036854775808.0); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 range PRIMARY PRIMARY 8 NULL 2 Using where; Using index > +# Cannot compare this as INT (yet) > +EXPLAIN SELECT id FROM t1 WHERE id IN (-9223372036854775808.0, -9223372036854775809.0); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 index PRIMARY PRIMARY 8 NULL 256 Using where; Using index > +DROP TABLE t1; > diff --git a/mysql-test/main/func_in.test b/mysql-test/main/func_in.test > index b99fad159c22..04b0b328c27b 100644 > --- a/mysql-test/main/func_in.test > +++ b/mysql-test/main/func_in.test > @@ -690,3 +690,51 @@ SELECT > TIME'00:00:00'='' AS c1_true, > TIME'00:00:00' IN ('', TIME'10:20:30') AS c2_true, > TIME'00:00:00' NOT IN ('', TIME'10:20:30') AS c3_false; > + > +--echo # > +--echo # MDEV-18319 BIGINT UNSIGNED Performance issue > +--echo # > + > +CREATE TABLE t1 ( > + id bigint(20) unsigned NOT NULL AUTO_INCREMENT PRIMARY KEY > +); > +DELIMITER $$; > +FOR i IN 0..255 > +DO > + INSERT INTO t1 VALUES (); > +END FOR > +$$ > +DELIMITER ;$$ A couple of one-liners instead: insert t1 select seq from seq_1_to_256; or, in pure SQL insert t1 with recursive g(i) as (select 1 union select i+1 from g where i < 257) select * from g; > +SELECT MIN(id), MAX(id), COUNT(*) FROM t1; > +EXPLAIN SELECT id FROM t1 WHERE id IN (1,2); > +EXPLAIN SELECT id FROM t1 WHERE id IN (9223372036854775806, 9223372036854775807); > +EXPLAIN SELECT id FROM t1 WHERE id IN (9223372036854775807, 9223372036854775808); > + > +EXPLAIN SELECT id FROM t1 WHERE id IN (1.0,2.0); > +EXPLAIN SELECT id FROM t1 WHERE id IN (9223372036854775806.0, 9223372036854775807.0); > +--echo # Cannot compare this as INT (yet) > +EXPLAIN SELECT id FROM t1 WHERE id IN (9223372036854775807.0, 9223372036854775808.0); > +DROP TABLE t1; > + > + > +CREATE TABLE t1 ( > + id bigint(20) NOT NULL AUTO_INCREMENT PRIMARY KEY > +); > +DELIMITER $$; > +FOR i IN 0..255 > +DO > + INSERT INTO t1 VALUES (); > +END FOR > +$$ > +DELIMITER ;$$ > +SELECT MIN(id), MAX(id), COUNT(*) FROM t1; > +EXPLAIN SELECT id FROM t1 WHERE id IN (-1,-2); > +EXPLAIN SELECT id FROM t1 WHERE id IN (-9223372036854775806, -9223372036854775807); > +EXPLAIN SELECT id FROM t1 WHERE id IN (-9223372036854775807, -9223372036854775808); > + > +EXPLAIN SELECT id FROM t1 WHERE id IN (-1.0,-2.0); > +EXPLAIN SELECT id FROM t1 WHERE id IN (-9223372036854775806.0, -9223372036854775807.0); > +EXPLAIN SELECT id FROM t1 WHERE id IN (-9223372036854775807.0, -9223372036854775808.0); > +--echo # Cannot compare this as INT (yet) > +EXPLAIN SELECT id FROM t1 WHERE id IN (-9223372036854775808.0, -9223372036854775809.0); > +DROP TABLE t1; > diff --git a/sql/item.h b/sql/item.h > index 883cc791f384..9f215c74864b 100644 > --- a/sql/item.h > +++ b/sql/item.h > @@ -70,6 +70,11 @@ class Value: public st_value > bool is_temporal() const { return m_type == DYN_COL_DATETIME; } > bool is_string() const { return m_type == DYN_COL_STRING; } > bool is_decimal() const { return m_type == DYN_COL_DECIMAL; } > + Longlong_hybrid to_longlong_hybrid_native() const > + { > + DBUG_ASSERT(is_longlong()); > + return Longlong_hybrid(value.m_longlong, m_type == DYN_COL_UINT); > + } > }; > > > diff --git a/sql/item_cmpfunc.cc b/sql/item_cmpfunc.cc > index b2fa753f2bd2..c7176df43d6a 100644 > --- a/sql/item_cmpfunc.cc > +++ b/sql/item_cmpfunc.cc > @@ -3747,11 +3748,45 @@ bool Predicant_to_list_comparator::add_value(const char *funcname, > Item *tmpargs[2]; > tmpargs[0]= args->arguments()[m_predicant_index]; > tmpargs[1]= args->arguments()[value_index]; > - if (tmp.aggregate_for_comparison(funcname, tmpargs, 2, true)) > + if (tmp.aggregate_for_comparison(funcname, tmpargs, 2, false)) How would that work? Are you going to compare signed to unsigned as longlongs? > { > DBUG_ASSERT(current_thd->is_error()); > return true; > } > + /* > + Try to compare using type handler of the predicant when possible, > + as this can use indexes for conditions like: > + WHERE field IN (const1, .. constN) > + */ > + if (prefer_predicant_type_handler && > + args->arguments()[value_index]->const_item() && > + !args->arguments()[value_index]->is_expensive() && > + tmp.type_handler()->cmp_type() == DECIMAL_RESULT && > + args->arguments()[m_predicant_index]->cmp_type() == INT_RESULT) > + { > + /* > + For now we catch only one special case: > + an INT predicant can be compared to a DECIMAL constant > + using Longlong_hybrid comparison, when the DECIMAL constant: > + a. has no significant fractional digits, and > + b. is within the signed longlong range > + For DECIMAL constants in the range LONGLONG_MAX..ULONGLONG_MAX > + we cannot call to_longlong_hybrid() safely, because DECIMAL type > + Items usually set their "unsigned_flag" to "false", so val_int() > + will truncate such constants to LONGLONG_MAX (instead of ULONGLONG_MAX). > + TODO: fix to_longlong_hybrid() for DECIMAL LONGLONG_MAX..ULONGLONG_MAX > + TODO: move this code to Type_handler (will be done by MDEV-18898) > + TODO: skip constants outside of LONGLONG_MIN..ULONGLONG_MAX, as > + such conditon can never be true, e.g.: > + WHERE int_expr IN (.. -9223372036854775809 ..) > + WHERE int_expr IN (.. 18446744073709551616 ..) > + */ > + my_decimal buf, *dec= args->arguments()[value_index]->val_decimal(&buf); > + longlong res; > + if (dec && decimal_actual_fraction(dec) == 0 && > + my_decimal2int(0, dec, false /*SIGNED*/, &res) == 0) > + tmp.set_handler(&type_handler_longlong); why not to replace the item with Item_int or Item_cache_int here? Then you'll be able to handle decimals up to ULONGLONG_MAX, that's your first TODO. For the last TODO, you can simply remove the always-false element from the arguments[] array. > + } > m_comparators[m_comparator_count].m_handler= tmp.type_handler(); > m_comparators[m_comparator_count].m_arg_index= value_index; > m_comparator_count++; > diff --git a/sql/item_cmpfunc.h b/sql/item_cmpfunc.h > index 06f15503258a..4e84c3008578 100644 > --- a/sql/item_cmpfunc.h > +++ b/sql/item_cmpfunc.h > @@ -1588,29 +1588,28 @@ class cmp_item_sort_string :public cmp_item_string > > class cmp_item_int : public cmp_item_scalar > { > - longlong value; > + Longlong_hybrid m_value; > public: > - cmp_item_int() {} /* Remove gcc warning */ > + cmp_item_int(): m_value(0, false) {} > void store_value(Item *item) > { > - value= item->val_int(); > + m_value= item->to_longlong_hybrid(); I wonder if to_longlong_hybrid() returns a Longlong_hybrid like that, where the compiler will allocate it? Could you check it, please? > m_null_value= item->null_value; > } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 00f9df29224: MDEV-20787: Script dgcov.pl does not work
by Sergei Golubchik 23 Apr '20

23 Apr '20

Hi, Anel! On Apr 23, Anel Husakovic wrote: > revision-id: 00f9df29224 (mariadb-10.2.29-23-g00f9df29224) > parent(s): 6718d3bc324 > author: Anel Husakovic <anel(a)mariadb.org> > committer: Anel Husakovic <anel(a)mariadb.org> > timestamp: 2019-11-19 15:00:48 +0100 > message: > > MDEV-20787: Script dgcov.pl does not work > > Let's change CMakeList with `--coverage` flag as an alias for > `-fprofile-arcs -ftest-coverage -lgcov` in addition. > When the server is compiled with `-DENABLE_GCOV=ON`, from object files are generated > `.gcno` and `.gcda` files. > `./mtr --gcov is_check_constraint` is invoking the script calls > `./dgcov.pl --purge`, `./mtr is_check_constraint`, > `./dgcov.pl --generate>/var/last_changes.dgcov`. > The `purge` flag is clearing `.gcda` files (and others extensions), > while running the test new `.gcda` files are obtained. > With `generate` flag, `gcov -i` (`intermediate format`) is called > on obtained `<object-files-name>.gcda` files (`dbug.c.gcda` e.g.). > The patch is tested on `gcov 6.3` and `gcov 7.4` versions > and can be seen that resulting `.gcov` file for `6.3` creates > `<full path>.gcov` (`dbug.c.gcda.gcov` e.g.) file, > where `gcov 7.4` is still creating `object-file-names.gcov`(`dbug.c.gcov`) files > as `gcov` in general is doing. > > diff --git a/CMakeLists.txt b/CMakeLists.txt > index 0dcc2a75587..af025a0312f 100644 > --- a/CMakeLists.txt > +++ b/CMakeLists.txt > @@ -249,7 +249,7 @@ SET(CMAKE_C_FLAGS_DEBUG "${CMAKE_C_FLAGS_DEBUG} -DENABLED_DEBUG_SYNC") > > OPTION(ENABLE_GCOV "Enable gcov (debug, Linux builds only)" OFF) > IF (ENABLE_GCOV) > - MY_CHECK_AND_SET_COMPILER_FLAG("-fprofile-arcs -ftest-coverage -lgcov" DEBUG) > + MY_CHECK_AND_SET_COMPILER_FLAG("--coverage" DEBUG) Why? Since what version does gcc support it? > ENDIF() > > MY_CHECK_AND_SET_COMPILER_FLAG(-ggdb3 DEBUG) > diff --git a/mysql-test/dgcov.pl b/mysql-test/dgcov.pl > index fbc5540e697..47ffaca04ef 100755 > --- a/mysql-test/dgcov.pl > +++ b/mysql-test/dgcov.pl > @@ -68,8 +68,11 @@ if ($opt_purge) > system($cmd)==0 or die "system($cmd): $? $!"; > exit 0; > } > - > +my $gcov_vers= `gcov -v`; > +$gcov_vers=~ s/\D//g; > +$gcov_vers= substr($gcov_vers, 0, 1); my gcov is 9.3.0. gcc version 10 is already out. You get just the first digit, 10 will become 1. Better to something like $gcov_vers ~= s/^.*(\d+\.\d+\.\d+).*$/\1/s; > find(\&gcov_one_file, $root); > +undef $gcov_vers; why? > find(\&write_coverage, $root) if $opt_generate; > exit 0 if $opt_only_gcov; > > @@ -162,7 +165,16 @@ sub gcov_one_file { > } > > # now, read the generated file > - open FH, '<', "$_.gcov" or die "open(<$_.gcov): $!"; > + if($gcov_vers<7) > + { > + open FH, '<', "$_.gcov" or die "open(<$_.gcov): $!"; > + } > + else > + { > + my $f=substr $_, 0, -5; > + open FH, '<', "$f.gcov" or die "open(<$f.gcov): $!"; > + undef $f; > + } I'd rather write it more compact, in two lines like my $f = $gcov_vers < 7 ? $_ : substr $_, 0, -5; open FH, '<', "$f.gcov" or die "open(<$f.gcov): $!"; > my $fname; > while (<FH>) { > chomp; > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] fb963a7fd3b: MDEV-21117: --tc-heuristic-recover=rollback is not replication safe
by sujatha 21 Apr '20

21 Apr '20

revision-id: fb963a7fd3b3e0733fb7ad9ece6256aebd8c176c (mariadb-10.1.43-112-gfb963a7fd3b) parent(s): ad4b70562bb94dd063eebde5189c6e730d3120a2 author: Sujatha committer: Sujatha timestamp: 2020-04-21 18:19:34 +0530 message: MDEV-21117: --tc-heuristic-recover=rollback is not replication safe Problem: ======= When run after master server crash --tc-heuristic-recover=rollback produces inconsistent server state with binlog still containing transactions that were rolled back by the option. Such way recovered server may not be used for replication. Fix: === During "--tc-heuristic-recover=ROLLBACK", query the storage engine to get binlog file name and position corresponding to the last committed transaction. This marks the consistent binlog state. If last_commit_file is not set then checkpoint binary log file is considered as starting point. Look for first transactional event beyond the consistent binlog state. This will be the starting point for heuristic rollback. Consider this event specific starting point as binlog truncation position. Now traverse the rest of binlogs beyond this point. During this traversal check for the presence of DDL or non transactional operations, as they cannot be safely rolled back. If such events are found the truncation will not happen it will return an error. If only transactional events are found beyond the binlog truncation position it is safe to truncate binlog. The log gets truncated to the identified position and the GTID state is adjusted accordingly. If there are more binary logs beyond the being truncated file they are all removed. If recovery is initiated based on last committed transaction specific binary log name and position, and there is failure during recovery, then recovery may be retried by remove the source of the failure. i.e --tc-heuristic-recover=rollback may be retried. Retry will not be successful if recovery is based on checkpoint file. As at the end of tc-heuristic-recover a new binary log file is created which will change the checkpoint file. Appropriate error message is written to error log for each case. --- .../r/binlog_heuristic_rollback_active_log.result | 18 + .../binlog/r/binlog_truncate_multi_log.result | 33 ++ .../r/binlog_truncate_multi_log_unsafe.result | 30 ++ .../suite/binlog/r/binlog_truncate_none.result | 42 ++ .../binlog/r/binlog_truncate_retry_success.result | 28 + .../t/binlog_heuristic_rollback_active_log.test | 75 +++ .../suite/binlog/t/binlog_truncate_multi_log.test | 87 +++ .../binlog/t/binlog_truncate_multi_log_unsafe.test | 106 ++++ .../suite/binlog/t/binlog_truncate_none.test | 130 +++++ .../binlog/t/binlog_truncate_retry_success.test | 84 +++ .../suite/rpl/r/rpl_heuristic_fail_over.result | 53 ++ .../suite/rpl/t/rpl_heuristic_fail_over.test | 160 ++++++ sql/log.cc | 589 ++++++++++++++++++++- sql/log.h | 7 +- sql/mysqld.h | 3 +- storage/innobase/handler/ha_innodb.cc | 7 + storage/innobase/log/log0recv.cc | 5 +- storage/xtradb/handler/ha_innodb.cc | 7 + storage/xtradb/log/log0recv.cc | 5 +- 19 files changed, 1457 insertions(+), 12 deletions(-) diff --git a/mysql-test/suite/binlog/r/binlog_heuristic_rollback_active_log.result b/mysql-test/suite/binlog/r/binlog_heuristic_rollback_active_log.result new file mode 100644 index 00000000000..eff95d05aac --- /dev/null +++ b/mysql-test/suite/binlog/r/binlog_heuristic_rollback_active_log.result @@ -0,0 +1,18 @@ +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); +RESET MASTER; +CREATE TABLE t ( f INT ) ENGINE=INNODB; +INSERT INTO t VALUES (10); +SET DEBUG_SYNC= "commit_before_update_end_pos SIGNAL con1_ready WAIT_FOR con1_go"; +INSERT INTO t VALUES (20); +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; +# Kill the server +"One row should be present in table 't'" +SELECT COUNT(*) FROM t; +COUNT(*) +1 +"gtid_binlog_state should be 0-1-2 +SELECT @@GLOBAL.gtid_binlog_state; +@@GLOBAL.gtid_binlog_state +0-1-2 +DROP TABLE t; diff --git a/mysql-test/suite/binlog/r/binlog_truncate_multi_log.result b/mysql-test/suite/binlog/r/binlog_truncate_multi_log.result new file mode 100644 index 00000000000..1a3d49b5463 --- /dev/null +++ b/mysql-test/suite/binlog/r/binlog_truncate_multi_log.result @@ -0,0 +1,33 @@ +SET @old_max_binlog_size= @@GLOBAL.max_binlog_size; +SET GLOBAL max_binlog_size= 4096; +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); +RESET MASTER; +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +FLUSH LOGS; +"List of binary logs before rotation" +show binary logs; +Log_name File_size +master-bin.000001 # +master-bin.000002 # +SET DEBUG_SYNC= "commit_after_release_LOCK_log SIGNAL con1_ready WAIT_FOR con1_go"; +INSERT INTO t1 VALUES (2, REPEAT("x", 4100)); +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; +"List of binary logs after rotation" +show binary logs; +Log_name File_size +master-bin.000001 # +master-bin.000002 # +master-bin.000003 # +# Kill the server +"Zero rows shoule be present in table" +SELECT COUNT(*) FROM t1; +COUNT(*) +0 +SELECT @@GLOBAL.gtid_current_pos; +@@GLOBAL.gtid_current_pos +0-1-1 +DROP TABLE t1; +SELECT @@GLOBAL.gtid_binlog_state; +@@GLOBAL.gtid_binlog_state +0-1-2 diff --git a/mysql-test/suite/binlog/r/binlog_truncate_multi_log_unsafe.result b/mysql-test/suite/binlog/r/binlog_truncate_multi_log_unsafe.result new file mode 100644 index 00000000000..5b7d45d81de --- /dev/null +++ b/mysql-test/suite/binlog/r/binlog_truncate_multi_log_unsafe.result @@ -0,0 +1,30 @@ +SET @old_max_binlog_size= @@global.max_binlog_size; +SET GLOBAL max_binlog_size= 4096; +call mtr.add_suppression("Table '.*tm' is marked as crashed and should be repaired"); +call mtr.add_suppression("Got an error from unknown thread"); +call mtr.add_suppression("Checking table: '.*tm'"); +call mtr.add_suppression("Recovering table: '.*tm'"); +call mtr.add_suppression("tc-heuristic-recover cannot trim the binary log to"); +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); +RESET MASTER; +CREATE TABLE ti (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +CREATE TABLE tm (f INT) ENGINE=MYISAM; +SET DEBUG_SYNC= "commit_before_get_LOCK_commit_ordered SIGNAL con1_ready WAIT_FOR con1_go"; +INSERT INTO ti VALUES (2, REPEAT("x", 4100)); +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; +INSERT INTO tm VALUES (30);; +# Kill the server +FOUND /tc-heuristic-recover cannot trim the binary log to File/ in mysqld.1.err +"Zero rows shoule be present in 'ti' table." +SELECT COUNT(*) FROM ti; +COUNT(*) +0 +"One row must be present in 'tm' table." +SELECT COUNT(*) FROM tm; +COUNT(*) +1 +SELECT @@GLOBAL.gtid_current_pos; +@@GLOBAL.gtid_current_pos +0-1-4 +DROP TABLE ti, tm; diff --git a/mysql-test/suite/binlog/r/binlog_truncate_none.result b/mysql-test/suite/binlog/r/binlog_truncate_none.result new file mode 100644 index 00000000000..bae87985208 --- /dev/null +++ b/mysql-test/suite/binlog/r/binlog_truncate_none.result @@ -0,0 +1,42 @@ +SET @old_max_binlog_size= @@global.max_binlog_size; +SET GLOBAL max_binlog_size= 4096; +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); +RESET MASTER; +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +INSERT INTO t1 VALUES (1, REPEAT("x", 4100)); +CREATE TABLE t2 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +show binary logs; +Log_name File_size +master-bin.000001 # +master-bin.000002 # +# Kill the server +"Zero records should be there." +SELECT COUNT(*) FROM t1; +COUNT(*) +1 +show binary logs; +Log_name File_size +master-bin.000001 # +master-bin.000002 # +master-bin.000003 # +master-bin.000004 # +DROP TABLE t1,t2; +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); +RESET MASTER; +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +CREATE TABLE t2 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +show binary logs; +Log_name File_size +master-bin.000001 # +# Kill the server +"Zero records should be there." +SELECT COUNT(*) FROM t1; +COUNT(*) +0 +SHOW TABLES; +Tables_in_test +t1 +t2 +DROP TABLE t1,t2; diff --git a/mysql-test/suite/binlog/r/binlog_truncate_retry_success.result b/mysql-test/suite/binlog/r/binlog_truncate_retry_success.result new file mode 100644 index 00000000000..cd0c4c23d05 --- /dev/null +++ b/mysql-test/suite/binlog/r/binlog_truncate_retry_success.result @@ -0,0 +1,28 @@ +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); +RESET MASTER; +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +INSERT INTO t1 VALUES (1,'dummy'); +SET DEBUG_SYNC= "commit_after_release_LOCK_log SIGNAL con1_ready WAIT_FOR con1_go"; +INSERT INTO t1 VALUES (2,'dummy'); +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; +"List of binary logs" +show binary logs; +Log_name File_size +master-bin.000001 # +# Kill the server +TEST 1: Recovery with error +TEST 2: Recovery without error +"One row shoule be present in table" +SELECT COUNT(*) FROM t1; +COUNT(*) +1 +"Two gtids should be present 0-1-2" +SELECT @@GLOBAL.gtid_current_pos; +@@GLOBAL.gtid_current_pos +0-1-2 +"Two gtids should be present 0-1-3" +DROP TABLE t1; +SELECT @@GLOBAL.gtid_binlog_state; +@@GLOBAL.gtid_binlog_state +0-1-3 diff --git a/mysql-test/suite/binlog/t/binlog_heuristic_rollback_active_log.test b/mysql-test/suite/binlog/t/binlog_heuristic_rollback_active_log.test new file mode 100644 index 00000000000..87af83a7a92 --- /dev/null +++ b/mysql-test/suite/binlog/t/binlog_heuristic_rollback_active_log.test @@ -0,0 +1,75 @@ +# ==== Purpose ==== +# +# Test verifies the truncation of single binary log file. +# +# ==== Implementation ==== +# +# Steps: +# 0 - Create table t1 and insert a row. +# 1 - Insert an another row such that it gets written to binlog but commit +# in engine fails as server crashed at this point. +# 2 - Restart server with --tc-heuristic-recover=ROLLBACK +# 3 - Upon server start 'master-bin.000001' will be truncated to contain +# only the first insert +# +# ==== References ==== +# +# MDEV-21117: --tc-heuristic-recover=rollback is not replication safe + + +--source include/have_innodb.inc +--source include/have_log_bin.inc +--source include/have_debug.inc +--source include/have_binlog_format_statement.inc + +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); + +connect(master,localhost,root,,); +connect(master1,localhost,root,,); + +--connection master +RESET MASTER; +CREATE TABLE t ( f INT ) ENGINE=INNODB; +INSERT INTO t VALUES (10); + +--connection master1 +# Hold insert after write to binlog and before "run_commit_ordered" in engine +SET DEBUG_SYNC= "commit_before_update_end_pos SIGNAL con1_ready WAIT_FOR con1_go"; +send INSERT INTO t VALUES (20); + +--connection master +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; + +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +wait +EOF + +--source include/kill_mysqld.inc +--source include/wait_until_disconnected.inc +# +# Server restart +# +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: --tc-heuristic-recover=ROLLBACK +EOF +--source include/wait_until_disconnected.inc + +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: +EOF + +connection default; +--enable_reconnect +--source include/wait_until_connected_again.inc + +--connection master +--enable_reconnect +--source include/wait_until_connected_again.inc + +--echo "One row should be present in table 't'" +SELECT COUNT(*) FROM t; + +--echo "gtid_binlog_state should be 0-1-2 +SELECT @@GLOBAL.gtid_binlog_state; +DROP TABLE t; diff --git a/mysql-test/suite/binlog/t/binlog_truncate_multi_log.test b/mysql-test/suite/binlog/t/binlog_truncate_multi_log.test new file mode 100644 index 00000000000..0a6e011627e --- /dev/null +++ b/mysql-test/suite/binlog/t/binlog_truncate_multi_log.test @@ -0,0 +1,87 @@ +# ==== Purpose ==== +# +# Test verifies truncation of multiple binary logs. +# +# ==== Implementation ==== +# +# Steps: +# 0 - Create a table in innodb engine and execute FLUSH LOGS command to +# generate a new binary log. +# 1 - Set max_binlog_size= 4096. Insert a row such that the max_binlog_size +# is reached and binary log gets rotated. +# 2 - Using debug simulation make the server crash at a point where the DML +# transaction is written to binary log but not committed in engine. +# 3 - At the time of crash three binary logs will be there +# master-bin.0000001, master-bin.000002 and master-bin.000003. +# 4 - Restart server with --tc-heuristic-recover=ROLLBACK +# 5 - Since the prepared DML in master-bin.000002 the binary log will be +# truncated and master-bin.000003 will be removed. +# +# ==== References ==== +# +# MDEV-21117: --tc-heuristic-recover=rollback is not replication safe + + +--source include/have_innodb.inc +--source include/have_log_bin.inc +--source include/have_debug.inc +--source include/have_debug_sync.inc +--source include/have_binlog_format_row.inc + +SET @old_max_binlog_size= @@GLOBAL.max_binlog_size; +SET GLOBAL max_binlog_size= 4096; +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); + +RESET MASTER; + +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +FLUSH LOGS; +connect(master1,localhost,root,,); +connect(master2,localhost,root,,); + +--connection master1 +# Hold insert after write to binlog and before "run_commit_ordered" in engine. +# Use "commit_after_release_LOCK_log" sync point. This point is reached after +# the binary log end position is updated which actually triggers binlog to be +# rotated. +--echo "List of binary logs before rotation" +--source include/show_binary_logs.inc +SET DEBUG_SYNC= "commit_after_release_LOCK_log SIGNAL con1_ready WAIT_FOR con1_go"; +send INSERT INTO t1 VALUES (2, REPEAT("x", 4100)); + +--connection master2 +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; +--echo "List of binary logs after rotation" +--source include/show_binary_logs.inc +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +wait +EOF + +--source include/kill_mysqld.inc +--source include/wait_until_disconnected.inc + +# +# Server restart +# +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: --tc-heuristic-recover=ROLLBACK --debug-dbug=d,simulate_innodb_forget_commit_pos +EOF +--source include/wait_until_disconnected.inc + +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: +EOF + +connection default; +--enable_reconnect +--source include/wait_until_connected_again.inc + +--echo "Zero rows shoule be present in table" +SELECT COUNT(*) FROM t1; + +SELECT @@GLOBAL.gtid_current_pos; + +DROP TABLE t1; +SELECT @@GLOBAL.gtid_binlog_state; + diff --git a/mysql-test/suite/binlog/t/binlog_truncate_multi_log_unsafe.test b/mysql-test/suite/binlog/t/binlog_truncate_multi_log_unsafe.test new file mode 100644 index 00000000000..cf4f3d34a24 --- /dev/null +++ b/mysql-test/suite/binlog/t/binlog_truncate_multi_log_unsafe.test @@ -0,0 +1,106 @@ +# ==== Purpose ==== +# +# Test verifies truncation of multiple binary logs will report an error on +# unsafe scenario. +# +# ==== Implementation ==== +# +# Steps: +# 0 - Set max_binlog_size= 4096. Create a table 'ti' using transactional +# storage engine. Do an insert such that the max_binlog_size is reached +# and binary log gets rotated. Hold this thread at a position where +# transaction is written to binary log but not committed in engine. +# 1 - Create a table named 'tm' using non transactional storage engine. +# 2 - Insert a row in 'tm' table. The DML will reach the engine and it is +# also written to binary log. +# 3 - Kill and restart the server with --tc-heuristic-recover=ROLLBACK +# 4 - Check for binary log truncation unsafe message in error log. +# 5 - No rows should be present in 'ti' table. One row should be present in +# 'tm' table. +# +# ==== References ==== +# +# MDEV-21117: --tc-heuristic-recover=rollback is not replication safe + + +--source include/have_innodb.inc +--source include/have_log_bin.inc +--source include/have_debug.inc +--source include/have_debug_sync.inc +--source include/have_binlog_format_row.inc + +SET @old_max_binlog_size= @@global.max_binlog_size; +SET GLOBAL max_binlog_size= 4096; + +call mtr.add_suppression("Table '.*tm' is marked as crashed and should be repaired"); +call mtr.add_suppression("Got an error from unknown thread"); +call mtr.add_suppression("Checking table: '.*tm'"); +call mtr.add_suppression("Recovering table: '.*tm'"); +call mtr.add_suppression("tc-heuristic-recover cannot trim the binary log to"); +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); + +RESET MASTER; + +CREATE TABLE ti (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +CREATE TABLE tm (f INT) ENGINE=MYISAM; + +connect(master1,localhost,root,,); +connect(master2,localhost,root,,); + +--connection master1 +# Hold insert after write to binlog and before "run_commit_ordered" in engine +SET DEBUG_SYNC= "commit_before_get_LOCK_commit_ordered SIGNAL con1_ready WAIT_FOR con1_go"; +--send INSERT INTO ti VALUES (2, REPEAT("x", 4100)) + +--connection master2 +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; +--send INSERT INTO tm VALUES (30); + +--connection default +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +wait +EOF + +--source include/kill_mysqld.inc +--source include/wait_until_disconnected.inc + +# +# Server restart +# +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: --tc-heuristic-recover=ROLLBACK --debug-dbug=d,simulate_innodb_forget_commit_pos +EOF +--source include/wait_until_disconnected.inc + +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: +EOF + +connection default; +--enable_reconnect +--source include/wait_until_connected_again.inc + +# Check error log for correct messages. +let $log_error_= `SELECT @@GLOBAL.log_error`; +if(!$log_error_) +{ + # MySQL Server on windows is started with --console and thus + # does not know the location of its .err log, use default location + let $log_error_ = $MYSQLTEST_VARDIR/log/mysqld.1.err; +} +--let SEARCH_FILE=$log_error_ +--let SEARCH_RANGE=-50000 +--let SEARCH_PATTERN=tc-heuristic-recover cannot trim the binary log to File +--source include/search_pattern_in_file.inc + +--echo "Zero rows shoule be present in 'ti' table." +SELECT COUNT(*) FROM ti; +--echo "One row must be present in 'tm' table." +--replace_regex /Table '.*tm/Table 'tm/ +--disable_warnings +SELECT COUNT(*) FROM tm; +--enable_warnings +SELECT @@GLOBAL.gtid_current_pos; + +DROP TABLE ti, tm; diff --git a/mysql-test/suite/binlog/t/binlog_truncate_none.test b/mysql-test/suite/binlog/t/binlog_truncate_none.test new file mode 100644 index 00000000000..cc3fd4f0b37 --- /dev/null +++ b/mysql-test/suite/binlog/t/binlog_truncate_none.test @@ -0,0 +1,130 @@ +# ==== Purpose ==== +# +# Test case verifies no binlog truncation happens when non transactional +# events are found in binlog after the last committed transaction. +# +# ==== Implementation ==== +# +# Steps: +# 0 - Set max_binlog_size= 4096. Create a table and do an insert such that +# the max_binlog_size is reached and binary log gets rotated. +# 1 - Create a table in newly created binary log and crash the server +# 2 - Restart server with --tc-heuristic-recover=ROLLBACK +# 3 - Recovery code will get the last committed DML specific postion and +# will try to check if binlog can be truncated upto this position. +# Since a DDL is present beyond this no truncation will happen. +# ==== References ==== +# +# MDEV-21117: --tc-heuristic-recover=rollback is not replication safe + + +--source include/have_innodb.inc +--source include/have_log_bin.inc +--source include/have_debug.inc +--source include/have_binlog_format_row.inc + +SET @old_max_binlog_size= @@global.max_binlog_size; +SET GLOBAL max_binlog_size= 4096; + +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); + +RESET MASTER; + +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +INSERT INTO t1 VALUES (1, REPEAT("x", 4100)); +CREATE TABLE t2 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +--source include/show_binary_logs.inc + +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +wait +EOF + +--source include/kill_mysqld.inc +--source include/wait_until_disconnected.inc + +# +# Server restart +# +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: --tc-heuristic-recover=ROLLBACK +EOF +--source include/wait_until_disconnected.inc + +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: +EOF + +connection default; +--enable_reconnect +--source include/wait_until_connected_again.inc + +--echo "Zero records should be there." +SELECT COUNT(*) FROM t1; +--source include/show_binary_logs.inc +DROP TABLE t1,t2; + +# ==== Purpose ==== +# +# Test case verifies no binlog truncation happens when only DDLs are present in +# the binary log. Since none of the DMLs are performed in storage engine, +# Engine will not have last committed transaction file name or position. +# Truncation code should return success. +# +# ==== Implementation ==== +# +# Steps: +# 0 - Create two table t1, t2 +# 1 - Kill and restart server with --tc-heuristic-recover=ROLLBACK +# 2 - Only DDL statements are present in the binary log. Since +# no DML was performed engine will not have last commited transaction +# specific binary log name and position. Since no transactional events +# are found, truncation code should simply return. +# +# ==== References ==== +# +# MDEV-21117: --tc-heuristic-recover=rollback is not replication safe + + +--source include/have_innodb.inc +--source include/have_log_bin.inc +--source include/have_debug.inc +--source include/have_binlog_format_row.inc + +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); + +RESET MASTER; + +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +CREATE TABLE t2 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +--source include/show_binary_logs.inc + +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +wait +EOF + +--source include/kill_mysqld.inc +--source include/wait_until_disconnected.inc + +# +# Server restart +# +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: --tc-heuristic-recover=ROLLBACK --debug-dbug=d,simulate_innodb_forget_commit_pos +EOF +--source include/wait_until_disconnected.inc + +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: +EOF + +connection default; +--enable_reconnect +--source include/wait_until_connected_again.inc + +--echo "Zero records should be there." +SELECT COUNT(*) FROM t1; +SHOW TABLES; +DROP TABLE t1,t2; + diff --git a/mysql-test/suite/binlog/t/binlog_truncate_retry_success.test b/mysql-test/suite/binlog/t/binlog_truncate_retry_success.test new file mode 100644 index 00000000000..6e92667c9b4 --- /dev/null +++ b/mysql-test/suite/binlog/t/binlog_truncate_retry_success.test @@ -0,0 +1,84 @@ +# ==== Purpose ==== +# +# Test verifies that tc-heuristic-recover=ROLLBACK can be retried. +# +# ==== Implementation ==== +# +# Steps: +# 0 - Create a table in Innodb storage engine. Insert a row into the table. +# During the commit of this DML engine will persist last committed +# transaction specific binary log file name and position. +# 1 - Do an another DML into the table, and simulate a crash in the middle +# of DML commit, so that DML is present in binary log but not committed in +# the storage engine. +# 2 - Restart the server using --tc-heuristic-recover=ROLLBACK. Simulate an +# error during the binary log rollback operation. Verify appropriate +# error is reported in the error log. +# 3 - Retry the --tc-heuristic-recver=ROLLBACK operation. Verify that +# binary log gets truncated as expected. +# 4 - Verify that global gtid state is according to the rolled back +# transaction. +# ==== References ==== +# +# MDEV-21117: --tc-heuristic-recover=rollback is not replication safe + + +--source include/have_innodb.inc +--source include/not_embedded.inc +--source include/have_log_bin.inc +--source include/have_debug.inc +--source include/have_debug_sync.inc +--source include/have_binlog_format_row.inc + +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); + +RESET MASTER; + +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +INSERT INTO t1 VALUES (1,'dummy'); +connect(master1,localhost,root,,); +connect(master2,localhost,root,,); + +--connection master1 +SET DEBUG_SYNC= "commit_after_release_LOCK_log SIGNAL con1_ready WAIT_FOR con1_go"; +send INSERT INTO t1 VALUES (2,'dummy'); + +--connection master2 +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; +--echo "List of binary logs" +--source include/show_binary_logs.inc +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +wait +EOF + +--source include/kill_mysqld.inc +--source include/wait_until_disconnected.inc + +# +# Server restart +# +--echo TEST 1: Recovery with error +--error 1 +--exec $MYSQLD_LAST_CMD --tc-heuristic-recover=ROLLBACK --debug-dbug=d,fault_injection_opening_binlog > $MYSQLTEST_VARDIR/log/mysqld.1.err 2>&1 +--source include/wait_until_disconnected.inc + +--echo TEST 2: Recovery without error +--error 1 +--exec $MYSQLD_LAST_CMD --tc-heuristic-recover=ROLLBACK > $MYSQLTEST_VARDIR/log/mysqld.1.err 2>&1 +--source include/wait_until_disconnected.inc + +--source include/start_mysqld.inc + +--echo "One row shoule be present in table" +SELECT COUNT(*) FROM t1; + +--echo "Two gtids should be present 0-1-2" +SELECT @@GLOBAL.gtid_current_pos; + +--echo "Two gtids should be present 0-1-3" +DROP TABLE t1; +SELECT @@GLOBAL.gtid_binlog_state; + +--disconnect master1 +--disconnect master2 diff --git a/mysql-test/suite/rpl/r/rpl_heuristic_fail_over.result b/mysql-test/suite/rpl/r/rpl_heuristic_fail_over.result new file mode 100644 index 00000000000..02fc47def41 --- /dev/null +++ b/mysql-test/suite/rpl/r/rpl_heuristic_fail_over.result @@ -0,0 +1,53 @@ +include/rpl_init.inc [topology=1->2] +include/stop_slave.inc +set global rpl_semi_sync_slave_enabled = 1; +CHANGE MASTER TO master_use_gtid= current_pos; +include/start_slave.inc +ALTER TABLE mysql.gtid_slave_pos ENGINE=InnoDB; +set global rpl_semi_sync_master_enabled = 1; +set global rpl_semi_sync_master_wait_point=AFTER_SYNC; +SET @old_max_binlog_size= @@global.max_binlog_size; +SET GLOBAL max_binlog_size= 4096; +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +INSERT INTO t1 VALUES (1, 'dummy1'); +SET DEBUG_SYNC= "commit_before_update_end_pos SIGNAL con1_ready WAIT_FOR con1_go"; +INSERT INTO t1 VALUES (2, REPEAT("x", 4100)); +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; +# Kill the server +include/stop_slave.inc +SELECT @@GLOBAL.gtid_current_pos; +@@GLOBAL.gtid_current_pos +0-1-5 +FOUND /tc-heuristic-recover: Truncated binlog File: \.\/master\-bin\.000001 of Size:[0-9]*, to Position */ in mysqld.1.err +set global rpl_semi_sync_master_enabled = 1; +set global rpl_semi_sync_master_wait_point=AFTER_SYNC; +CHANGE MASTER TO master_host='127.0.0.1', master_port=SERVER_MYPORT_2, master_user='root', master_use_gtid=CURRENT_POS; +set global rpl_semi_sync_slave_enabled = 1; +include/start_slave.inc +INSERT INTO t1 VALUES (3, 'dummy3'); +SELECT COUNT(*) FROM t1; +COUNT(*) +2 +SHOW VARIABLES LIKE 'gtid_current_pos'; +Variable_name Value +gtid_current_pos 0-2-6 +SHOW VARIABLES LIKE 'gtid_current_pos'; +Variable_name Value +gtid_current_pos 0-2-6 +DROP TABLE t1; +set global rpl_semi_sync_master_enabled = 0; +set global rpl_semi_sync_slave_enabled = 0; +set global rpl_semi_sync_master_wait_point=default; +set global rpl_semi_sync_master_enabled = 0; +set global rpl_semi_sync_slave_enabled = 0; +set global rpl_semi_sync_master_wait_point=default; +include/stop_slave.inc +RESET MASTER; +RESET SLAVE; +RESET MASTER; +RESET SLAVE; +CHANGE MASTER TO master_host='127.0.0.1', master_port=SERVER_MYPORT_1, master_user='root', master_use_gtid=no; +include/start_slave.inc +include/rpl_end.inc diff --git a/mysql-test/suite/rpl/t/rpl_heuristic_fail_over.test b/mysql-test/suite/rpl/t/rpl_heuristic_fail_over.test new file mode 100644 index 00000000000..39b1f7545d0 --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_heuristic_fail_over.test @@ -0,0 +1,160 @@ +# ==== Purpose ==== +# +# Test verifies replication failover scenario. +# +# ==== Implementation ==== +# +# Steps: +# 0 - Have two servers with id's 1 and 2. Enable semi-sync based +# replication. Have semi sync master wait point as 'after_sync'. +# 1 - Create a table and insert a row. While inserting second row simulate +# a server crash at once the transaction is written to binlog, flushed +# and synced but the binlog position is not updated. +# 2 - Restart the server using tc-heuristic-recover=ROLLBACK +# 3 - Post restart switch the crashed master to slave. Execute CHANGE MASTER +# TO command to connect to server id 2. +# 4 - Slave should be able to connect to master. +# 5 - Execute some DML on new master with server id 2. Ensure that it gets +# replicated to server id 1. +# 6 - Verify the "gtid_current_pos" for correctness. +# 7 - Clean up +# +# ==== References ==== +# +# MDEV-21117: --tc-heuristic-recover=rollback is not replication safe + + +--source include/have_semisync.inc +--source include/have_innodb.inc +--source include/have_log_bin.inc +--source include/have_debug.inc +--source include/have_debug_sync.inc +--source include/have_binlog_format_row.inc +--let $rpl_topology=1->2 +--source include/rpl_init.inc + +--connection server_2 +--source include/stop_slave.inc +set global rpl_semi_sync_slave_enabled = 1; +CHANGE MASTER TO master_use_gtid= current_pos; +--source include/start_slave.inc + + +--connection server_1 +ALTER TABLE mysql.gtid_slave_pos ENGINE=InnoDB; +set global rpl_semi_sync_master_enabled = 1; +set global rpl_semi_sync_master_wait_point=AFTER_SYNC; +SET @old_max_binlog_size= @@global.max_binlog_size; +SET GLOBAL max_binlog_size= 4096; + +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); + +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +INSERT INTO t1 VALUES (1, 'dummy1'); +--save_master_pos + +--connection server_2 +--sync_with_master + +--connection server_1 +connect(master1,localhost,root,,); +connect(master2,localhost,root,,); + +--connection master1 +# Hold insert after write to binlog and before "run_commit_ordered" in engine +SET DEBUG_SYNC= "commit_before_update_end_pos SIGNAL con1_ready WAIT_FOR con1_go"; +send INSERT INTO t1 VALUES (2, REPEAT("x", 4100)); + +--connection master2 +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +wait +EOF + +--source include/kill_mysqld.inc +--source include/wait_until_disconnected.inc + +--connection server_2 +--error 2003 +--source include/stop_slave.inc +SELECT @@GLOBAL.gtid_current_pos; + +--connection server_1 +# +# Server restart +# +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: --tc-heuristic-recover=ROLLBACK +EOF +--source include/wait_until_disconnected.inc + +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: +EOF + +connection default; +--enable_reconnect +--source include/wait_until_connected_again.inc + +--connection server_1 +--enable_reconnect +--source include/wait_until_connected_again.inc + +# Check error log for correct messages. +let $log_error_= `SELECT @@GLOBAL.log_error`; +if(!$log_error_) +{ + # MySQL Server on windows is started with --console and thus + # does not know the location of its .err log, use default location + let $log_error_ = $MYSQLTEST_VARDIR/log/mysqld.1.err; +} +--let SEARCH_FILE=$log_error_ +--let SEARCH_RANGE=-50000 +--let SEARCH_PATTERN=tc-heuristic-recover: Truncated binlog File: \.\/master\-bin\.000001 of Size:[0-9]*, to Position * +--source include/search_pattern_in_file.inc + +--connection server_2 +set global rpl_semi_sync_master_enabled = 1; +set global rpl_semi_sync_master_wait_point=AFTER_SYNC; + +--connection server_1 +--replace_result $SERVER_MYPORT_2 SERVER_MYPORT_2 +eval CHANGE MASTER TO master_host='127.0.0.1', master_port=$SERVER_MYPORT_2, master_user='root', master_use_gtid=CURRENT_POS; +set global rpl_semi_sync_slave_enabled = 1; +--source include/start_slave.inc + +--connection server_2 +INSERT INTO t1 VALUES (3, 'dummy3'); +--save_master_pos + +--connection server_1 +--sync_with_master +SELECT COUNT(*) FROM t1; +SHOW VARIABLES LIKE 'gtid_current_pos'; + +--connection server_2 +SHOW VARIABLES LIKE 'gtid_current_pos'; +DROP TABLE t1; +set global rpl_semi_sync_master_enabled = 0; +set global rpl_semi_sync_slave_enabled = 0; +set global rpl_semi_sync_master_wait_point=default; +--save_master_pos + +--connection server_1 +--sync_with_master +set global rpl_semi_sync_master_enabled = 0; +set global rpl_semi_sync_slave_enabled = 0; +set global rpl_semi_sync_master_wait_point=default; +--source include/stop_slave.inc +RESET MASTER; +RESET SLAVE; + +--connection server_2 +RESET MASTER; +RESET SLAVE; +--replace_result $SERVER_MYPORT_1 SERVER_MYPORT_1 +eval CHANGE MASTER TO master_host='127.0.0.1', master_port=$SERVER_MYPORT_1, master_user='root', master_use_gtid=no; +--source include/start_slave.inc + +--source include/rpl_end.inc diff --git a/sql/log.cc b/sql/log.cc index 0efef6d1e29..66445944b40 100644 --- a/sql/log.cc +++ b/sql/log.cc @@ -3164,6 +3164,7 @@ MYSQL_BIN_LOG::MYSQL_BIN_LOG(uint *sync_period) checksum_alg_reset(BINLOG_CHECKSUM_ALG_UNDEF), relay_log_checksum_alg(BINLOG_CHECKSUM_ALG_UNDEF), description_event_for_exec(0), description_event_for_queue(0), + last_commit_pos_offset(0), current_binlog_id(0) { /* @@ -3173,6 +3174,7 @@ MYSQL_BIN_LOG::MYSQL_BIN_LOG(uint *sync_period) before main(). */ index_file_name[0] = 0; + last_commit_pos_file[0]= 0; bzero((char*) &index_file, sizeof(index_file)); bzero((char*) &purge_index_file, sizeof(purge_index_file)); } @@ -4576,8 +4578,8 @@ int MYSQL_BIN_LOG::open_purge_index_file(bool destroy) 0, 0, MYF(MY_WME | MY_NABP | MY_WAIT_IF_FULL))) { error= 1; - sql_print_error("MYSQL_BIN_LOG::open_purge_index_file failed to open register " - " file."); + sql_print_error("MYSQL_BIN_LOG::open_purge_index_file failed to open " + "register file."); } } DBUG_RETURN(error); @@ -4742,7 +4744,7 @@ int MYSQL_BIN_LOG::purge_index_entry(THD *thd, ulonglong *reclaimed_space, } goto err; } - + error= 0; DBUG_PRINT("info",("purging %s",log_info.log_file_name)); @@ -7877,6 +7879,7 @@ MYSQL_BIN_LOG::trx_group_commit_leader(group_commit_entry *leader) first= false; } + DEBUG_SYNC(leader->thd, "commit_before_update_end_pos"); /* update binlog_end_pos so it can be read by dump thread * * note: must be _after_ the RUN_HOOK(after_flush) or else @@ -8964,7 +8967,7 @@ int TC_LOG_MMAP::open(const char *opt_name) { if (my_errno != ENOENT) goto err; - if (using_heuristic_recover()) + if (using_heuristic_recover(opt_name)) return 1; if ((fd= mysql_file_create(key_file_tclog, logname, CREATE_MODE, O_RDWR | O_CLOEXEC, MYF(MY_WME))) < 0) @@ -9497,14 +9500,40 @@ TC_LOG_MMAP tc_log_mmap; 1 heuristic recovery was performed */ -int TC_LOG::using_heuristic_recover() +int TC_LOG::using_heuristic_recover(const char* opt_name) { + LOG_INFO log_info; + int error; + if (!tc_heuristic_recover) return 0; sql_print_information("Heuristic crash recovery mode"); + if (ha_recover(0)) + { sql_print_error("Heuristic crash recovery failed"); + } + + if (!strcmp(opt_name, opt_bin_logname) && + tc_heuristic_recover == TC_HEURISTIC_RECOVER_ROLLBACK) + { + if ((error= mysql_bin_log.find_log_pos(&log_info, NullS, 1))) + { + if (error != LOG_INFO_EOF) + sql_print_error("tc-heuristic-recover failed in find_log_pos(). " + "Error: %d", error); + } + else + { + if ((error= heuristic_binlog_rollback())) + { + sql_print_error("Heuristic crash recovery of binary log failed."); + return 1; + } + } + } + sql_print_information("Please restart mysqld without --tc-heuristic-recover"); return 1; } @@ -9512,6 +9541,554 @@ int TC_LOG::using_heuristic_recover() /****** transaction coordinator log for 2pc - binlog() based solution ******/ #define TC_LOG_BINLOG MYSQL_BIN_LOG +/** + Truncates the current binlog to specified position. Removes the rest of binlogs + which are present after this binlog file. + + @param truncate_file Holds the binlog name to be truncated + @param truncate_pos Position within binlog from where it needs to + truncated. + + @retval true ok + @retval false error + +*/ +bool MYSQL_BIN_LOG::truncate_and_remove_binlogs(const char *truncate_file, + my_off_t truncate_pos) +{ + int error= 0; +#ifdef HAVE_REPLICATION + LOG_INFO log_info; + THD *thd= current_thd; + my_off_t index_file_offset= 0; + File file= -1; + IO_CACHE cache; + MY_STAT s; + my_off_t binlog_size; + + if ((error= find_log_pos(&log_info, truncate_file, 1))) + { + sql_print_error("tc-heuristic-recover: Failed to locate binary log file:%s." + "Error:%d", truncate_file, error); + goto end; + } + + while (!(error= find_next_log(&log_info, 1))) + { + if (!index_file_offset) + { + index_file_offset= log_info.index_file_start_offset; + if ((error= open_purge_index_file(TRUE))) + { + sql_print_error("tc-heuristic-recover: Failed to open purge index " + "file:%s. Error:%d", purge_index_file_name, error); + goto end; + } + } + if ((error= register_purge_index_entry(log_info.log_file_name))) + { + sql_print_error("tc-heuristic-recover: Failed to copy %s to purge index" + " file. Error:%d", log_info.log_file_name, error); + goto end; + } + } + + if (error != LOG_INFO_EOF) + { + sql_print_error("tc-heuristic-recover: Failed to find the next binlog to " + "add to purge index register. Error:%d", error); + goto end; + } + + if (is_inited_purge_index_file()) + { + if (!index_file_offset) + index_file_offset= log_info.index_file_start_offset; + + if ((error= sync_purge_index_file())) + { + sql_print_error("tc-heuristic-recover: Failed to flush purge index " + "file. Error:%d", error); + goto end; + } + + // Trim index file + if ((error= + mysql_file_chsize(index_file.file, index_file_offset, '\n', + MYF(MY_WME))) || + (error= + mysql_file_sync(index_file.file, MYF(MY_WME|MY_SYNC_FILESIZE)))) + { + sql_print_error("tc-heuristic-recover: Failed to trim binlog index " + "file:%s to offset:%llu. Error:%d", index_file_name, + index_file_offset); + mysql_file_close(index_file.file, MYF(MY_WME)); + goto end; + } + + /* Reset data in old index cache */ + if ((error= reinit_io_cache(&index_file, READ_CACHE, (my_off_t) 0, 0, 1))) + { + sql_print_error("tc-heuristic-recover: Failed to reinit binlog index " + "file. Error:%d", error); + mysql_file_close(index_file.file, MYF(MY_WME)); + goto end; + } + + /* Read each entry from purge_index_file and delete the file. */ + if ((error= purge_index_entry(thd, NULL, TRUE))) + { + sql_print_error("tc-heuristic-recover: Failed to process registered " + "files that would be purged."); + goto end; + } + } + + DBUG_ASSERT(truncate_pos); + + if ((file= mysql_file_open(key_file_binlog, truncate_file, + O_RDWR | O_BINARY, MYF(MY_WME))) < 0) + { + error= 1; + sql_print_error("tc-heuristic-recover: Failed to open binlog file:%s for " + "truncation.", truncate_file); + goto end; + } + my_stat(truncate_file, &s, MYF(0)); + binlog_size= s.st_size; + + /* Change binlog file size to truncate_pos */ + if ((error= + mysql_file_chsize(file, truncate_pos, 0, MYF(MY_WME))) || + (error= mysql_file_sync(file, MYF(MY_WME|MY_SYNC_FILESIZE)))) + { + sql_print_error("tc-heuristic-recover: Failed to trim the " + "binlog file:%s to size:%llu. Error:%d", + truncate_file, truncate_pos, error); + goto end; + } + else + { + sql_print_information("tc-heuristic-recover: Truncated binlog " + "File: %s of Size:%llu, to Position:%llu.", + truncate_file, binlog_size, truncate_pos); + } + if (!(error= init_io_cache(&cache, file, IO_SIZE, WRITE_CACHE, + (my_off_t) truncate_pos, 0, MYF(MY_WME|MY_NABP)))) + { + /* + Write Stop_log_event to ensure clean end point for the binary log being + truncated. + */ + Stop_log_event se; + se.checksum_alg= (enum_binlog_checksum_alg) binlog_checksum_options; + if ((error= write_event(&se, &cache))) + { + sql_print_error("tc-heuristic-recover: Failed to write stop event to " + "binary log. Errno:%d", + (cache.error == -1) ? my_errno : error); + goto end; + } + if ((error= flush_io_cache(&cache)) || + (error= mysql_file_sync(file, MYF(MY_WME|MY_SYNC_FILESIZE)))) + { + sql_print_error("tc-heuristic-recover: Faild to write stop event to " + "binary log. Errno:%d", + (cache.error == -1) ? my_errno : error); + } + } + else + sql_print_error("tc-heuristic-recover: Failed to initialize binary log " + "cache for writing stop event. Errno:%d", + (cache.error == -1) ? my_errno : error); + +end: + if (file >= 0) + { + end_io_cache(&cache); + mysql_file_close(file, MYF(MY_WME)); + } + error= error || close_purge_index_file(); +#endif + return error > 0; +} + +/** + Returns the checkpoint binlog file name found in the lastest binlog file. + + @param checkpoint_file Holds the binlog checkpoint file name. + + @retval 0 ok + @retval 1 error + +*/ +int TC_LOG_BINLOG::get_binlog_checkpoint_file(char* checkpoint_file) +{ + Log_event *ev= NULL; + bool binlog_checkpoint_found= false; + LOG_INFO log_info; + const char *errmsg; + IO_CACHE log; + File file; + Format_description_log_event fdle(BINLOG_VERSION); + char log_name[FN_REFLEN]; + int error=1; + + if (!fdle.is_valid()) + return 1; + + if ((error= find_log_pos(&log_info, NullS, 1))) + { + sql_print_error("tc-heuristic-recover: find_log_pos() failed to read first " + "binary log entry from index file.(error: %d)", error); + return error; + } + + // Move to the last binary log. + do + { + strmake_buf(log_name, log_info.log_file_name); + } while (!(error= find_next_log(&log_info, 1))); + + if (error != LOG_INFO_EOF) + { + sql_print_error("tc-heuristic-recover: find_next_log() failed " + "(error: %d)", error); + return error; + } + if ((file= open_binlog(&log, log_name, &errmsg)) < 0) + { + sql_print_error("tc-heuristic-recover failed to open the binlog:%s for " + "reading checkpoint file name. Error: %s", + log_info.log_file_name, errmsg); + return error; + } + while (!binlog_checkpoint_found && + (ev= + Log_event::read_log_event(&log, 0, &fdle, opt_master_verify_checksum)) + && ev->is_valid()) + { + enum Log_event_type typ= ev->get_type_code(); + if (typ == BINLOG_CHECKPOINT_EVENT) + { + size_t dir_len; + Binlog_checkpoint_log_event *cev= (Binlog_checkpoint_log_event *)ev; + if (cev->binlog_file_len >= FN_REFLEN) + { + sql_print_error("tc-heuristic-recover: Incorrect binlog checkpoint " + "event with too long file name found."); + delete ev; + ev= NULL; + end_io_cache(&log); + mysql_file_close(file, MYF(MY_WME)); + return 1; + } + else + { + dir_len= dirname_length(log_name); + strmake(strnmov(checkpoint_file, log_name, dir_len), + cev->binlog_file_name, FN_REFLEN - 1 - dir_len); + binlog_checkpoint_found= true; + } + } + delete ev; + ev= NULL; + } // End of while + end_io_cache(&log); + mysql_file_close(file, MYF(MY_WME)); + file= -1; + /* + Old binary log without checkpoint found, binlog truncation is not + possible. Hence return error. + */ + if (!binlog_checkpoint_found) + return 1; + + return 0; +} + + +/** + Truncates the binary log, according to the transactions that got rolled + back from engine, during heuristic-recover=ROLLBACK. Global GTID state is + adjusted as per the truncated binlog. + + Called from @c TC_LOG::using_heuristic_recover(const char* opt_name) + + @param opt_name The base name of binary log. + + @return indicates success or failure of binlog rollback + @retval 0 success + @retval 1 failure + +*/ +int TC_LOG_BINLOG::heuristic_binlog_rollback() +{ + int error=0; +#ifdef HAVE_REPLICATION + Log_event *ev= NULL; + char engine_commit_file[FN_REFLEN]; + char binlog_truncate_file_name[FN_REFLEN]; + char checkpoint_file[FN_REFLEN]; + my_off_t binlog_truncate_pos= 0; + my_off_t engine_commit_pos= 0; + LOG_INFO log_info; + const char *errmsg; + IO_CACHE log; + File file=-1; + Format_description_log_event fdle(BINLOG_VERSION); + bool found_engine_commit_pos= false; + bool found_truncate_pos= false; + bool is_safe= true; + my_off_t tmp_truncate_pos=0; + rpl_gtid last_gtid; + bool last_gtid_standalone= false; + bool last_gtid_valid= false; + bool is_checkpoint_based_recovery= false; + + + DBUG_EXECUTE_IF("simulate_innodb_forget_commit_pos", + { + last_commit_pos_file[0]= 0; + };); + + // Initialize engine_commit_file/pos + if (last_commit_pos_file[0] != 0) + { + strmake_buf(engine_commit_file, last_commit_pos_file); + engine_commit_pos= last_commit_pos_offset; + sql_print_information("tc-heuristic-recover: Initialising heuristic " + "rollback of binary log using last committed " + "transaction specific binary log name:%s and " + "position:%llu", last_commit_pos_file, + last_commit_pos_offset); + } + else + { + if ((error= get_binlog_checkpoint_file(checkpoint_file))) + { + is_checkpoint_based_recovery= true; + sql_print_error("tc-heuristic-recover: Failed to read latest checkpoint " + "binary log name."); + goto end; + } + strmake_buf(engine_commit_file, checkpoint_file); + sql_print_information("tc-heuristic-recover: Initialising heuristic " + "rollback of binary log using last checkpoint " + "file:%s.", engine_commit_file); + /* + If there is no engine specific commit file we are doing checkpoint file + based recovery. Hence we mark "found_engine_commit_pos" true, and start + looking for the first transactional event group with is the candidate for + rollback. + */ + found_engine_commit_pos= true; + } + + if ((error= find_log_pos(&log_info, engine_commit_file, 1))) + { + sql_print_error("tc-heuristic-recover: Failed to locate binary log file:%s " + "in index file. Error:%d", engine_commit_file, error); + goto end; + } + if ((file= open_binlog(&log, log_info.log_file_name, &errmsg)) < 0 || + DBUG_EVALUATE_IF("fault_injection_opening_binlog", (errmsg="Unknown"), + FALSE)) + { + error= 1; + sql_print_error("tc-heuristic-recover: Failed to open the binlog:%s for " + "recovery. Error:%s", log_info.log_file_name, errmsg); + goto end; + } + + + error= read_state_from_file(); + if (error && error != 2) + { + sql_print_error("tc-heuristic-recover: Failed to load global gtid binlog " + "state from file"); + goto end; + } + if (!fdle.is_valid()) + { + error= 1; + sql_print_error("tc-heuristic-recover: Failed due to invalid format " + "description log event."); + goto end; + } + + for(;;) + { + while (is_safe && + (ev= Log_event::read_log_event(&log, 0, &fdle, + opt_master_verify_checksum)) && ev->is_valid()) + { + enum Log_event_type typ= ev->get_type_code(); + switch (typ) + { + case XID_EVENT: + if (ev->log_pos == engine_commit_pos) + { + found_engine_commit_pos= true; + } + break; + case GTID_LIST_EVENT: + { + Gtid_list_log_event *glev= (Gtid_list_log_event *)ev; + /* Initialise the binlog state from the Gtid_list event. */ + if (!found_truncate_pos && glev->count > 0 && + rpl_global_gtid_binlog_state.load(glev->list, glev->count)) + { + error= 1; + sql_print_error("tc-heuristic-recover: Failed to read GTID List " + "event."); + goto end; + } + } + break; + case GTID_EVENT: + { + Gtid_log_event *gev= (Gtid_log_event *)ev; + + /* Update the binlog state with any GTID logged after Gtid_list. */ + last_gtid.domain_id= gev->domain_id; + last_gtid.server_id= gev->server_id; + last_gtid.seq_no= gev->seq_no; + last_gtid_standalone= + ((gev->flags2 & Gtid_log_event::FL_STANDALONE) ? true : false); + last_gtid_valid= true; + if (gev->flags2 & Gtid_log_event::FL_TRANSACTIONAL && + !found_truncate_pos) + { + if ((engine_commit_pos == 0 || found_engine_commit_pos)) + { + found_truncate_pos= true; + strmake_buf(binlog_truncate_file_name, log_info.log_file_name); + binlog_truncate_pos= tmp_truncate_pos; + } + } + else + { + if (found_truncate_pos) + is_safe= false; + } + } + break; + default: + /* Nothing. */ + break; + }// End switch + if (!found_truncate_pos && last_gtid_valid && + ((last_gtid_standalone && !ev->is_part_of_group(typ)) || + (!last_gtid_standalone && + (typ == XID_EVENT || + (typ == QUERY_EVENT && + (((Query_log_event *)ev)->is_commit() || + ((Query_log_event *)ev)->is_rollback())))))) + { + if ((error= rpl_global_gtid_binlog_state.update_nolock(&last_gtid, + false))) + { + sql_print_error("tc-heuristic-recover: Failed to update GTID within " + "global gtid state."); + goto end; + } + last_gtid_valid= false; + } + // Used to identify the last group specific end position. + tmp_truncate_pos= ev->log_pos; + delete ev; + ev= NULL; + }// End While + if (file >= 0) + { + end_io_cache(&log); + mysql_file_close(file, MYF(MY_WME)); + file= -1; + } + if (is_safe) + { + if ((error= find_next_log(&log_info, 1))) + { + if (error != LOG_INFO_EOF) + { + sql_print_error("tc-heuristic-recover: Failed to read next binary " + "log during recovery."); + goto end; + } + else + { + error= 0; // LOG_INFO_EOF= -1 is not an error. + break; + } + } + if ((file= open_binlog(&log, log_info.log_file_name, &errmsg)) < 0) + { + error= 1; + sql_print_error("tc-heuristic-recover: Failed to open the binlog:%s for " + "recovery. Error:%s", log_info.log_file_name, errmsg); + goto end; + } + } + else + break; + } //end of for(;;) + sql_print_information("tc-heuristic-recover: Binary log to be truncated " + "File:%s Pos:%llu.", binlog_truncate_file_name, + binlog_truncate_pos); + if (!found_truncate_pos) + goto end; // Nothing to truncate + else + DBUG_ASSERT(binlog_truncate_pos > 0); + + if (is_safe) + { + if ((error= truncate_and_remove_binlogs(binlog_truncate_file_name, + binlog_truncate_pos))) + { + sql_print_error("tc-heuristic-recover: Failed to trim the binary log to " + "File:%s Pos:%llu.", binlog_truncate_file_name, + binlog_truncate_pos); + goto end; + } + } + else + { + sql_print_error("tc-heuristic-recover cannot trim the binary log to " + "File:%s Pos:%llu as unsafe statements (non-trans/DDL) " + "statements are found beyond the truncation position.", + binlog_truncate_file_name, binlog_truncate_pos); + } + if ((error= write_state_to_file())) + { + sql_print_error("tc-heuristic-recover: Failed to write global gtid state " + "to file"); + goto end; + } + +end: + if (file >= 0) + { + end_io_cache(&log); + mysql_file_close(file, MYF(MY_WME)); + } + if (error) + { + if (is_checkpoint_based_recovery) + { + sql_print_error("tc-heuristic-recover failed during binary log rollback." + "Further retry attemps won't succeed."); + } + else + sql_print_error("tc-heuristic-recover failed during binary log rollback. " + "After removing the source of the failure " + "--tc-heuristic-recover=rollback may be retried."); + } + +#endif + return error; +} + int TC_LOG_BINLOG::open(const char *opt_name) { int error= 1; @@ -9526,7 +10103,7 @@ int TC_LOG_BINLOG::open(const char *opt_name) return 1; } - if (using_heuristic_recover()) + if (using_heuristic_recover(opt_name)) { mysql_mutex_lock(&LOCK_log); /* generate a new binlog to mask a corrupted one */ diff --git a/sql/log.h b/sql/log.h index 277e5c6f69c..84c0a457b52 100644 --- a/sql/log.h +++ b/sql/log.h @@ -41,7 +41,8 @@ bool stmt_has_updated_non_trans_table(const THD* thd); class TC_LOG { public: - int using_heuristic_recover(); + int using_heuristic_recover(const char* opt_name); + virtual int heuristic_binlog_rollback() { return 0; }; TC_LOG() {} virtual ~TC_LOG() {} @@ -694,6 +695,7 @@ class MYSQL_BIN_LOG: public TC_LOG, private MYSQL_LOG void commit_checkpoint_notify(void *cookie); int recover(LOG_INFO *linfo, const char *last_log_name, IO_CACHE *first_log, Format_description_log_event *fdle, bool do_xa); + int heuristic_binlog_rollback(); int do_binlog_recovery(const char *opt_name, bool do_xa_recovery); #if !defined(MYSQL_CLIENT) @@ -794,6 +796,9 @@ class MYSQL_BIN_LOG: public TC_LOG, private MYSQL_LOG int purge_first_log(Relay_log_info* rli, bool included); int set_purge_index_file_name(const char *base_file_name); int open_purge_index_file(bool destroy); + bool truncate_and_remove_binlogs(const char *truncate_file, + my_off_t truncate_pos); + int get_binlog_checkpoint_file(char* checkpoint_file); bool is_inited_purge_index_file(); int close_purge_index_file(); int clean_purge_index_file(); diff --git a/sql/mysqld.h b/sql/mysqld.h index e939524dbff..515d5f39c3d 100644 --- a/sql/mysqld.h +++ b/sql/mysqld.h @@ -94,6 +94,7 @@ extern "C" MYSQL_PLUGIN_IMPORT CHARSET_INFO *system_charset_info; extern MYSQL_PLUGIN_IMPORT CHARSET_INFO *files_charset_info ; extern MYSQL_PLUGIN_IMPORT CHARSET_INFO *national_charset_info; extern MYSQL_PLUGIN_IMPORT CHARSET_INFO *table_alias_charset; +extern MYSQL_PLUGIN_IMPORT bool opt_bin_log; /** Character set of the buildin error messages loaded from errmsg.sys. @@ -104,7 +105,7 @@ extern CHARSET_INFO *character_set_filesystem; extern MY_BITMAP temp_pool; extern bool opt_large_files, server_id_supplied; -extern bool opt_update_log, opt_bin_log, opt_error_log; +extern bool opt_update_log, opt_error_log; extern my_bool opt_log, opt_bootstrap; extern my_bool opt_backup_history_log; extern my_bool opt_backup_progress_log; diff --git a/storage/innobase/handler/ha_innodb.cc b/storage/innobase/handler/ha_innodb.cc index 4de2cdbeaec..552348ee2c5 100644 --- a/storage/innobase/handler/ha_innodb.cc +++ b/storage/innobase/handler/ha_innodb.cc @@ -16493,6 +16493,13 @@ innobase_xa_recover( { DBUG_ASSERT(hton == innodb_hton_ptr); + if (opt_bin_log) + { + mysql_bin_log.last_commit_pos_offset= trx_sys_mysql_bin_log_pos; + strmake_buf(mysql_bin_log.last_commit_pos_file, + trx_sys_mysql_bin_log_name); + } + if (len == 0 || xid_list == NULL) { return(0); diff --git a/storage/innobase/log/log0recv.cc b/storage/innobase/log/log0recv.cc index 3b3c7c23224..c8a742d95e9 100644 --- a/storage/innobase/log/log0recv.cc +++ b/storage/innobase/log/log0recv.cc @@ -2,7 +2,7 @@ Copyright (c) 1997, 2017, Oracle and/or its affiliates. All Rights Reserved. Copyright (c) 2012, Facebook Inc. -Copyright (c) 2013, 2019, MariaDB Corporation. +Copyright (c) 2013, 2020, MariaDB Corporation. This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software @@ -3301,7 +3301,8 @@ void recv_recovery_from_checkpoint_finish(void) /*======================================*/ { - if (recv_needed_recovery) { + extern MYSQL_PLUGIN_IMPORT bool opt_bin_log; + if (opt_bin_log) { trx_sys_print_mysql_master_log_pos(); trx_sys_print_mysql_binlog_offset(); } diff --git a/storage/xtradb/handler/ha_innodb.cc b/storage/xtradb/handler/ha_innodb.cc index 2aafb1a44ee..a966f726a03 100644 --- a/storage/xtradb/handler/ha_innodb.cc +++ b/storage/xtradb/handler/ha_innodb.cc @@ -17165,6 +17165,13 @@ innobase_xa_recover( { DBUG_ASSERT(hton == innodb_hton_ptr); + if (opt_bin_log) + { + mysql_bin_log.last_commit_pos_offset= trx_sys_mysql_bin_log_pos; + strmake_buf(mysql_bin_log.last_commit_pos_file, + trx_sys_mysql_bin_log_name); + } + if (len == 0 || xid_list == NULL) { return(0); diff --git a/storage/xtradb/log/log0recv.cc b/storage/xtradb/log/log0recv.cc index dd55d31218a..a4844507328 100644 --- a/storage/xtradb/log/log0recv.cc +++ b/storage/xtradb/log/log0recv.cc @@ -2,7 +2,7 @@ Copyright (c) 1997, 2017, Oracle and/or its affiliates. All Rights Reserved. Copyright (c) 2012, Facebook Inc. -Copyright (c) 2013, 2019, MariaDB Corporation. +Copyright (c) 2013, 2020, MariaDB Corporation. This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software @@ -3397,7 +3397,8 @@ void recv_recovery_from_checkpoint_finish(void) /*======================================*/ { - if (recv_needed_recovery) { + extern MYSQL_PLUGIN_IMPORT bool opt_bin_log; + if (opt_bin_log) { trx_sys_print_mysql_master_log_pos(); trx_sys_print_mysql_binlog_offset(); }

1 0

Re: [Maria-developers] 784cc5970dd: MDEV-19650: Privilege bug on MariaDB 10.4
by Sergei Golubchik 21 Apr '20

21 Apr '20

Hi, Oleksandr! On Apr 21, Oleksandr Byelkin wrote: > > > OK, INSERT probably is not needed, but we have tons (more than 40 I think) > of DELETE in our tests (and so UPDATE should work). Yes, the view is delete-able and some columns are update-able. So, let's keep these privileges. INSERT and LOAD never worked. > Also we have not LOAD but this: > > rpl.rpl_current_user 'stmt' w7 [ fail ] > Test ended at 2020-04-21 09:34:44 > > CURRENT_TEST: rpl.rpl_current_user > mysqltest: In included file "./include/diff_tables.inc": > included from > /home/sanja/maria/git/10.4/mysql-test/suite/rpl/t/rpl_current_user.test at > line 65: > At line 170: query 'SELECT * INTO OUTFILE '$_dt_outfile' FROM > $_dt_database.$_dt_table ORDER BY `$_dt_column_list`' failed: 1356: View > 'test.v_user' references invalid table(s) or column(s) or function(s) or > definer/invoker of view lack rights to use them This must be a bug. If one does SELECT * INTO OUTFILE FROM some.view than the current user should have FILE, not the view definer. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 784cc5970dd: MDEV-19650: Privilege bug on MariaDB 10.4
by Sergei Golubchik 20 Apr '20

20 Apr '20

Hi, Oleksandr! On Apr 20, Oleksandr Byelkin wrote: > > > > > + > > > > > +CREATE TEMPORARY TABLE tmp_user_sys LIKE global_priv; > > > > > +INSERT INTO tmp_user_sys (Host,User,Priv) VALUES ('localhost','mariadb.sys','{"access":512,"plugin":"mysql_native_password","authentication_string":"","account_locked":true,"password_last_changed":0}'); > > > > > +INSERT INTO global_priv SELECT * FROM tmp_user_sys WHERE NOT @had_sys_user; > > > > > +DROP TABLE tmp_user_sys; > > > > > > > > 1. This could've been simply INSERT IGNORE, I suspect > > > > > > Nope, the idea is do not insert more than needed. > > > > Why would INSERT IGNORE insert more than needed? > > I thought you propose insert ignore instead of where clause, in any case I > do not see why it should help. I mean, instead of four lines above, I think, one can simply do INSERT IGNORE global_priv (Host,User,Priv) VALUES ('localhost','mariadb.sys','{"access":512,"plugin":"mysql_native_password","authentication_string":"","account_locked":true,"password_last_changed":0}'); > > > 2. why access:512 ? It's FILE_ACL, iirc. > > > > > > Because LOAD used in tests and so in reality probably, so we need > > FILE_ACL > > > > We need FILE_ACL, but why mariadb.sys account needs it? > > mysql.user is a read-only view of the mysql.global_priv table, > > its owner doesn't need FILE or INSERT/UPDATE/DELETE. > > As I told we somewhere in test suite (not main, but I don't remember > exactly) test which require the permission otherwise I would not return and > add more rights. Where? The view is clearly not insertable-into, so INSERT privilege is meaningless. Some fields are updateable, and one can delele from it. This was more accidental than intentional, but ok, let's not change it in 10.4. And why does it need FILE privilege? The view owner cannot possibly do any LOAD DATA. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 784cc5970dd: MDEV-19650: Privilege bug on MariaDB 10.4
by Sergei Golubchik 20 Apr '20

20 Apr '20

Hi, Oleksandr! On Apr 20, Oleksandr Byelkin wrote: > > Hi, Oleksandr! > > > > On Apr 20, Oleksandr Byelkin wrote: > > > revision-id: 784cc5970dd (mariadb-10.4.11-68-g784cc5970dd) > > > parent(s): c5e00fea102 > > > author: Oleksandr Byelkin <sanja(a)mariadb.com> > > > committer: Oleksandr Byelkin <sanja(a)mariadb.com> > > > timestamp: 2020-02-20 14:06:09 +0100 > > > message: > > > > > > MDEV-19650: Privilege bug on MariaDB 10.4 > > > > > > diff --git a/scripts/mysql_system_tables.sql > > > index 29f2a4c1ef6..af852444d0c 100644 > > > --- a/scripts/mysql_system_tables.sql > > > +++ b/scripts/mysql_system_tables.sql > > > @@ -33,9 +33,17 @@ CREATE TABLE IF NOT EXISTS db ( Host char(60) > > binary DEFAULT '' NOT NULL, Db c > > > -- Remember for later if db table already existed > > > set @had_db_table= @@warning_count != 0; > > > > > > -CREATE TABLE IF NOT EXISTS global_priv (Host char(60) binary DEFAULT > > '', User char(80) binary DEFAULT '', Priv JSON NOT NULL DEFAULT '{}' > > CHECK(JSON_VALID(Priv)), PRIMARY KEY Host (Host,User)) engine=Aria > > transactional=1 CHARACTER SET utf8 COLLATE utf8_bin comment='Users and > > global privileges'; > > > +CREATE TABLE IF NOT EXISTS global_priv (Host char(60) binary DEFAULT > > '', User char(80) binary DEFAULT '', Priv JSON NOT NULL DEFAULT '{}' > > CHECK(JSON_VALID(Priv)), PRIMARY KEY (Host,User)) engine=Aria > > transactional=1 CHARACTER SET utf8 COLLATE utf8_bin comment='Users and > > global privileges'; > > > > > > -CREATE DEFINER=root@localhost SQL SECURITY DEFINER VIEW IF NOT EXISTS > > user AS SELECT > > > +set @had_sys_user= @@warning_count != 0 OR 0 <> (select count(*) from > > mysql.global_priv where Host="localhost" and User="mariadb.sys"); > > > + > > > +CREATE TEMPORARY TABLE tmp_user_sys LIKE global_priv; > > > +INSERT INTO tmp_user_sys (Host,User,Priv) VALUES > > ('localhost','mariadb.sys','{"access":512,"plugin":"mysql_native_password","authentication_string":"","account_locked":true,"password_last_changed":0}'); > > > +INSERT INTO global_priv SELECT * FROM tmp_user_sys WHERE NOT > > @had_sys_user; > > > +DROP TABLE tmp_user_sys; > > > > 1. This could've been simply INSERT IGNORE, I suspect > > Nope, the idea is do not insert more than needed. Why would INSERT IGNORE insert more than needed? > 2. why access:512 ? It's FILE_ACL, iirc. > > Because LOAD used in tests and so in reality probably, so we need FILE_ACL We need FILE_ACL, but why mariadb.sys account needs it? mysql.user is a read-only view of the mysql.global_priv table, its owner doesn't need FILE or INSERT/UPDATE/DELETE. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 784cc5970dd: MDEV-19650: Privilege bug on MariaDB 10.4
by Sergei Golubchik 20 Apr '20

20 Apr '20

Hi, Oleksandr! On Apr 20, Oleksandr Byelkin wrote: > revision-id: 784cc5970dd (mariadb-10.4.11-68-g784cc5970dd) > parent(s): c5e00fea102 > author: Oleksandr Byelkin <sanja(a)mariadb.com> > committer: Oleksandr Byelkin <sanja(a)mariadb.com> > timestamp: 2020-02-20 14:06:09 +0100 > message: > > MDEV-19650: Privilege bug on MariaDB 10.4 > > Also fixes: > MDEV-21487: Implement option for mysql_upgrade that allows root@localhost to be replaced > MDEV-21486: Implement option for mysql_install_db that allows root@localhost to be replaced > > Add user mariadb.sys to be definer of user view > (and has right on underlying table global_priv for > required operation over global_priv > (SELECT,UPDATE,DELETE,INSERT,FILE)) > > Also changed definer of gis functions in case of creation, > but they work with any definer so upgrade script do not try > to push this change. > > diff --git a/scripts/mysql_system_tables.sql b/scripts/mysql_system_tables.sql > index 29f2a4c1ef6..af852444d0c 100644 > --- a/scripts/mysql_system_tables.sql > +++ b/scripts/mysql_system_tables.sql > @@ -33,9 +33,17 @@ CREATE TABLE IF NOT EXISTS db ( Host char(60) binary DEFAULT '' NOT NULL, Db c > -- Remember for later if db table already existed > set @had_db_table= @@warning_count != 0; > > -CREATE TABLE IF NOT EXISTS global_priv (Host char(60) binary DEFAULT '', User char(80) binary DEFAULT '', Priv JSON NOT NULL DEFAULT '{}' CHECK(JSON_VALID(Priv)), PRIMARY KEY Host (Host,User)) engine=Aria transactional=1 CHARACTER SET utf8 COLLATE utf8_bin comment='Users and global privileges'; > +CREATE TABLE IF NOT EXISTS global_priv (Host char(60) binary DEFAULT '', User char(80) binary DEFAULT '', Priv JSON NOT NULL DEFAULT '{}' CHECK(JSON_VALID(Priv)), PRIMARY KEY (Host,User)) engine=Aria transactional=1 CHARACTER SET utf8 COLLATE utf8_bin comment='Users and global privileges'; > > -CREATE DEFINER=root@localhost SQL SECURITY DEFINER VIEW IF NOT EXISTS user AS SELECT > +set @had_sys_user= @@warning_count != 0 OR 0 <> (select count(*) from mysql.global_priv where Host="localhost" and User="mariadb.sys"); > + > +CREATE TEMPORARY TABLE tmp_user_sys LIKE global_priv; > +INSERT INTO tmp_user_sys (Host,User,Priv) VALUES ('localhost','mariadb.sys','{"access":512,"plugin":"mysql_native_password","authentication_string":"","account_locked":true,"password_last_changed":0}'); > +INSERT INTO global_priv SELECT * FROM tmp_user_sys WHERE NOT @had_sys_user; > +DROP TABLE tmp_user_sys; 1. This could've been simply INSERT IGNORE, I suspect 2. why access:512 ? It's FILE_ACL, iirc. > + > +CREATE DEFINER='mariadb.sys'@'localhost' SQL SECURITY DEFINER VIEW IF NOT EXISTS user AS SELECT > Host, > User, > IF(JSON_VALUE(Priv, '$.plugin') IN ('mysql_native_password', 'mysql_old_password'), IFNULL(JSON_VALUE(Priv, '$.authentication_string'), ''), '') AS Password, > @@ -101,6 +109,11 @@ CREATE TABLE IF NOT EXISTS servers ( Server_name char(64) NOT NULL DEFAULT '', H > > CREATE TABLE IF NOT EXISTS tables_priv ( Host char(60) binary DEFAULT '' NOT NULL, Db char(64) binary DEFAULT '' NOT NULL, User char(80) binary DEFAULT '' NOT NULL, Table_name char(64) binary DEFAULT '' NOT NULL, Grantor char(141) DEFAULT '' NOT NULL, Timestamp timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, Table_priv set('Select','Insert','Update','Delete','Create','Drop','Grant','References','Index','Alter','Create View','Show view','Trigger','Delete versioning rows') COLLATE utf8_general_ci DEFAULT '' NOT NULL, Column_priv set('Select','Insert','Update','References') COLLATE utf8_general_ci DEFAULT '' NOT NULL, PRIMARY KEY (Host,Db,User,Table_name), KEY Grantor (Grantor) ) engine=Aria transactional=1 CHARACTER SET utf8 COLLATE utf8_bin comment='Table privileges'; > > +CREATE TEMPORARY TABLE tmp_user_sys LIKE tables_priv; > +INSERT INTO tmp_user_sys (Host,Db,User,Table_name,Grantor,Timestamp,Table_priv) VALUES ('localhost','mysql','mariadb.sys','global_priv','root@localhost','0','Select,Insert,Update,Delete'); why Insert,Update,Delete ? > +INSERT INTO tables_priv SELECT * FROM tmp_user_sys WHERE NOT @had_sys_user; > +DROP TABLE tmp_user_sys; > + > CREATE TABLE IF NOT EXISTS columns_priv ( Host char(60) binary DEFAULT '' NOT NULL, Db char(64) binary DEFAULT '' NOT NULL, User char(80) binary DEFAULT '' NOT NULL, Table_name char(64) binary DEFAULT '' NOT NULL, Column_name char(64) binary DEFAULT '' NOT NULL, Timestamp timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, Column_priv set('Select','Insert','Update','References') COLLATE utf8_general_ci DEFAULT '' NOT NULL, PRIMARY KEY (Host,Db,User,Table_name,Column_name) ) engine=Aria transactional=1 CHARACTER SET utf8 COLLATE utf8_bin comment='Column privileges'; > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] MDEV-17399 Add support for JSON_TABLE: for ORDINALITY
by Sergey Petrunia 19 Apr '20

19 Apr '20

Hi Alexey, The standard says "An ordinality column provides a sequential numbering of rows. Row numbering is 1-based." Consider this example from the standard: https://gist.github.com/spetrunia/d4d8564a3ed26148ae92035b24e1f294 SELECT jt.rowseq, jt.name, jt.zip FROM bookclub, JSON_TABLE(bookclub.jcol, "lax $" COLUMNS ( rowSeq FOR ORDINALITY, name VARCHAR(30) PATH 'lax $.Name', zip CHAR(5) PATH 'lax $.address.postalCode') ) AS jt +--------+---------+------+ | rowseq | name | zip | +--------+---------+------+ | 0 | John Sm | 1 | | 1 | Peter W | 9 | | 2 | James L | NULL | +--------+---------+------+ The numbering is 0-based, instead of 1-based. What is not entirely clear for me is when the numbering should be reset. For the above example, the SQL standard shows the resultset with rowseq 1-2-3 which hints that the numbering is "global". MySQL-8 produces "1-1-1", which looks like they reset the numbering for every scan of the JSON_TABLE output. What are you thoughts on this? If the numbering should be "global", should it be query-level global, or subselect-level global? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 1

Re: [Maria-developers] 8742d176bc2: Added support for more functions when using partitioned S3 tables
by Sergei Golubchik 18 Apr '20

18 Apr '20

Hi, Michael! On Apr 13, Michael Widenius wrote: > revision-id: 8742d176bc2 (mariadb-10.5.2-127-g8742d176bc2) > parent(s): bf32018be96 > author: Michael Widenius <monty(a)mariadb.com> > committer: Michael Widenius <monty(a)mariadb.com> > timestamp: 2020-04-09 16:41:42 +0300 > message: > > Added support for more functions when using partitioned S3 tables > > MDEV-22088 S3 partitioning support > > All ALTER PARTITION commands should now work on S3 tables except > > REBUILD PARTITION > TRUNCATE PARTITION > REORGANIZE PARTITION > > In addition, PARTIONED S3 TABLES can also be replicated. > This is achived by storing the partition tables .frm and .par file on S3 > for partitioned shared (S3) tables. > > The discovery methods are enchanced by allowing engines that supports > discovery to also support of the partitioned tables .frm and .par file > > Things in more detail > > - The .frm and .par files of partitioned tables are stored in S3 and kept > in sync. > - Added hton callback create_partitioning_metadata to inform handler > that metadata for a partitoned file has changed > - Added back handler::discover_check_version() to be able to check if > a table's or a part table's definition has changed. > - Added handler::check_if_updates_are_ignored(). Needed for partitioning. > - Renamed rebind() -> rebind_psi(), as it was before. > - Changed CHF_xxx hadnler flags to an enum > - Changed some checks from using table->file->ht to use > table->file->partition_ht() to get discovery to work with partitioning. > - If TABLE_SHARE::init_from_binary_frm_image() fails, ensure that we > don't leave any .frm or .par files around. > - Fixed that writefrm() doesn't leave unusable .frm files around > - Appended extension to path for writefrm() to be able to reuse to function > for creating .par files. > - Added DBUG_PUSH("") to a a few functions that caused a lot of not > critical tracing. > > diff --git a/mysql-test/suite/s3/replication_partition.test b/mysql-test/suite/s3/replication_partition.test > new file mode 100644 > index 00000000000..8a177f8f075 > --- /dev/null > +++ b/mysql-test/suite/s3/replication_partition.test > @@ -0,0 +1,170 @@ > +--source include/have_s3.inc > +--source include/have_partition.inc > +--source include/master-slave.inc master-slave should always be included last, after all other have_xxx includes. > +--source include/have_binlog_format_mixed.inc > +--source include/have_innodb.inc > +--source include/have_sequence.inc > +--source create_database.inc > + > +connection slave; > +let $MYSQLD_DATADIR= `select @@datadir`; > +--replace_result $database database > +--eval use $database > +connection master; > + > +--echo # > +--echo # Check replication of parititioned S3 tables > +--echo # > + > +CREATE TABLE t1 ( > + c1 INT DEFAULT NULL > +) ENGINE=Aria > + PARTITION BY HASH (c1) > + PARTITIONS 3; > +INSERT INTO t1 VALUE (1), (2), (101), (102), (201), (202); > +ALTER TABLE t1 ENGINE=S3; > +ALTER TABLE t1 ADD PARTITION PARTITIONS 6; > +select sum(c1) from t1; > +ALTER TABLE t1 ADD COLUMN c INT; > +select sum(c1) from t1; > +sync_slave_with_master; > +show create table t1; > +select sum(c1) from t1; > +connection master; > +drop table t1; > + > +--echo # > +--echo # Checking that the slave is keeping in sync with changed partitions > +--echo # > + > +CREATE TABLE t1 ( > + c1 int primary key, > + c2 int DEFAULT NULL > +) ENGINE=InnoDB > + PARTITION BY RANGE (c1) > + (PARTITION p1 VALUES LESS THAN (200), > + PARTITION p2 VALUES LESS THAN (300), > + PARTITION p3 VALUES LESS THAN (400)); > +insert into t1 select seq*100,seq*100 from seq_1_to_3; > +alter table t1 engine=S3; > +show create table t1; > + > +sync_slave_with_master; > +select sum(c1) from t1; > +--file_exists $MYSQLD_DATADIR/$database/t1.frm > +--file_exists $MYSQLD_DATADIR/$database/t1.par > +stop slave; > +connection master; > +ALTER TABLE t1 ADD PARTITION (PARTITION p4 VALUES LESS THAN (500)); > +connection slave; > +show create table t1; > +select sum(c1) from t1; > +start slave; > +connection master; > +sync_slave_with_master; > +select sum(c1)+0 from t1; > +stop slave; > + > +# .frm amd .par files should not exists on the salve as it has just seen the > +# ALTER TABLE which cased the removal of the .frm and .par files. The table > +# from the above "select sum()" came from table cache and was used as it's > +# id matches the one in S3 > +--error 1 > +--file_exists $MYSQLD_DATADIR/$database/t1.frm > +--error 1 > +--file_exists $MYSQLD_DATADIR/$database/t1.par > +# Flushing the table cache will force the .frm and .par files to be > +# re-generated > +flush tables; > +select sum(c1)+0 from t1; > +--file_exists $MYSQLD_DATADIR/$database/t1.frm > +--file_exists $MYSQLD_DATADIR/$database/t1.par > + > +connection master; > +drop table t1; > +connection slave; > +--file_exists $MYSQLD_DATADIR/$database/t1.par > +--replace_result $database database > +--error ER_NO_SUCH_TABLE > +select sum(c1) from t1; > +--error 1 > +--file_exists $MYSQLD_DATADIR/$database/t1.par > +start slave; > +connection master; > + > +--echo # > +--echo # Check altering partitioned table to S3 and back > +--echo # Checks also rename partitoned table and drop partition > +--echo # > + > +CREATE TABLE t2 ( > + c1 int primary key, > + c2 int DEFAULT NULL > +) ENGINE=InnoDB > + PARTITION BY RANGE (c1) > + (PARTITION p1 VALUES LESS THAN (200), > + PARTITION p2 VALUES LESS THAN (300), > + PARTITION p3 VALUES LESS THAN (400)); > +insert into t2 select seq*100,seq*100 from seq_1_to_3; > +alter table t2 engine=S3; > +rename table t2 to t1; > +alter table t1 drop partition p1; > +sync_slave_with_master; > +select sum(c1) from t1; > +connection master; > +alter table t1 engine=innodb; > +sync_slave_with_master; > +select sum(c1) from t1; > +connection master; > +drop table t1; > + > +--echo # > +--echo # Check that slaves ignores changes to S3 tables. > +--echo # > + > +connection master; > +CREATE TABLE t1 ( > + c1 int primary key, > + c2 int DEFAULT NULL > +) ENGINE=InnoDB > + PARTITION BY RANGE (c1) > + (PARTITION p1 VALUES LESS THAN (200), > + PARTITION p2 VALUES LESS THAN (300), > + PARTITION p3 VALUES LESS THAN (400)); > +insert into t1 select seq*100,seq*100 from seq_1_to_3; > +create table t2 like t1; > +alter table t2 remove partitioning; > +insert into t2 values (450,450); > +sync_slave_with_master; > +stop slave; > +connection master; > +alter table t1 engine=s3; > +alter table t2 engine=s3; > +ALTER TABLE t1 ADD PARTITION (PARTITION p4 VALUES LESS THAN (500)); > +alter table t1 exchange partition p4 with table t2; > +select count(*) from t1; > +drop table t1,t2; > +connection slave; > +start slave; > +connection master; > +sync_slave_with_master; > +--replace_result $database database > +--error ER_NO_SUCH_TABLE > +select sum(c1) from t1; > +connection master; > + > +--echo # > +--echo # Check slave binary log > +--echo # > + > +sync_slave_with_master; > +--let $binlog_database=$database > +--source include/show_binlog_events.inc > +connection master; > + > +--echo # > +--echo # clean up > +--echo # > +--source drop_database.inc > +sync_slave_with_master; > +--source include/rpl_end.inc > diff --git a/mysys/my_symlink.c b/mysys/my_symlink.c > index cbee78a7f5c..323ae69a39c 100644 > --- a/mysys/my_symlink.c > +++ b/mysys/my_symlink.c > @@ -154,7 +154,8 @@ int my_realpath(char *to, const char *filename, myf MyFlags) > original name but will at least be able to resolve paths that starts > with '.'. > */ > - DBUG_PRINT("error",("realpath failed with errno: %d", errno)); > + if (MyFlags) > + DBUG_PRINT("error",("realpath failed with errno: %d", errno)); This is kind of against the concept of dbug. dbug's idea is that one should not edit the code every time, but put DBUG points one and enable/disable them at run-time. In cases like that you can specify precisely what you want to see in the trace with complicated command-line --debug string. But I usually just let it log everything and then use dbug/remove_function_from_trace.pl script. > my_errno=errno; > if (MyFlags & MY_WME) > my_error(EE_REALPATH, MYF(0), filename, my_errno); > diff --git a/sql/discover.cc b/sql/discover.cc > index e49a2a3b0c0..7f3fe73c155 100644 > --- a/sql/discover.cc > +++ b/sql/discover.cc > @@ -99,23 +99,23 @@ int readfrm(const char *name, const uchar **frmdata, size_t *len) > > > /* > - Write the content of a frm data pointer > - to a frm file. > + Write the content of a frm data pointer to a frm or par file. really? writefrm() writes to a .par file? may be rename it to writefile() then? > > - @param path path to table-file "db/name" > - @param frmdata frm data > - @param len length of the frmdata > + @param path full path to table-file "db/name.frm" or .par > + @param db Database name. Only used for my_error() > + @param table Table name. Only used for my_error() > + @param data data to write to file > + @param len length of the data > > @retval > 0 ok > @retval > - 2 Could not write file > + <> 0 Could not write file. In this case the file is not created > */ > > int writefrm(const char *path, const char *db, const char *table, > - bool tmp_table, const uchar *frmdata, size_t len) > + bool tmp_table, const uchar *data, size_t len) > { > - char file_name[FN_REFLEN+1]; > int error; > int create_flags= O_RDWR | O_TRUNC; > DBUG_ENTER("writefrm"); > diff --git a/sql/ha_partition.cc b/sql/ha_partition.cc > index 0ec1f2138ab..b8b5085f389 100644 > --- a/sql/ha_partition.cc > +++ b/sql/ha_partition.cc > @@ -629,7 +629,8 @@ int ha_partition::rename_table(const char *from, const char *to) > > SYNOPSIS > create_partitioning_metadata() > - name Full path of table name > + path Full path of new table name > + old_p Full path of old table name This is a rather meaningless phrase, there cannot be a "path" of a "name" May be, write "A path to the new/old frm file but without the extension" instead? > create_info Create info generated for CREATE TABLE > > RETURN VALUE > @@ -678,6 +681,19 @@ int ha_partition::create_partitioning_metadata(const char *path, > DBUG_RETURN(1); > } > } > + > + /* m_part_info is only NULL when we failed to create a partition table */ How can m_part_info be NULL here? create_handler_file() dereferences m_part_info unconditionally, if m_part_info would've been NULL it'll crash. > + if (m_part_info) > + { > + part= m_part_info->partitions.head(); > + if ((part->engine_type)->create_partitioning_metadata && > + ((part->engine_type)->create_partitioning_metadata)(path, old_path, > + action_flag)) > + { > + my_error(ER_CANT_CREATE_HANDLER_FILE, MYF(0)); > + DBUG_RETURN(1); > + } > + } > DBUG_RETURN(0); > } > > @@ -1604,6 +1620,7 @@ int ha_partition::prepare_new_partition(TABLE *tbl, > > if (!(file->ht->flags & HTON_CAN_READ_CONNECT_STRING_IN_PARTITION)) > tbl->s->connect_string= p_elem->connect_string; > + create_info->options|= HA_CREATE_TMP_ALTER; This looks wrong. The partition isn't created as a temporary file. HA_CREATE_TMP_ALTER is used for autiding and perfschema to distinguish between normal tables, temporary tables and these alter-tmp-tables. I see that you've hijacked this flag and check it inside s3, but it wasn't designed for that, so change s3 instead, not what the flag means. I don't think you need that if inside s3 at all, if hton flag says HTON_TEMPORARY_NOT_SUPPORTED, then s3 won't be asked to create a temporary table and you don't need a check for that in create. > if ((error= file->ha_create(part_name, tbl, create_info))) > { > /* > @@ -2361,14 +2379,20 @@ uint ha_partition::del_ren_table(const char *from, const char *to) > const char *to_path= NULL; > uint i; > handler **file, **abort_file; > + THD *thd= ha_thd(); > DBUG_ENTER("ha_partition::del_ren_table"); > > - if (get_from_handler_file(from, ha_thd()->mem_root, false)) > - DBUG_RETURN(TRUE); > + if (get_from_handler_file(from, thd->mem_root, false)) > + DBUG_RETURN(my_errno ? my_errno : ENOENT); > DBUG_ASSERT(m_file_buffer); > DBUG_PRINT("enter", ("from: (%s) to: (%s)", from, to ? to : "(nil)")); > name_buffer_ptr= m_name_buffer_ptr; > + > file= m_file; > + /* The command should be logged with IF EXISTS if using a shared table */ > + if (m_file[0]->ht->flags & HTON_TABLE_MAY_NOT_EXIST_ON_SLAVE) > + thd->replication_flags|= OPTION_IF_EXISTS; I don't think this and the whole thd->replication_flags is needed, because in the sql_table.cc you can check directly table->file->partition_ht()->flags && HTON_TABLE_MAY_NOT_EXIST_ON_SLAVE > + > if (to == NULL) > { > /* > @@ -2424,7 +2453,35 @@ uint ha_partition::del_ren_table(const char *from, const char *to) > goto rename_error; > } > } > + > + /* Update .par file in the handlers that supports it */ > + if ((*m_file)->ht->create_partitioning_metadata) > + { > + if (to == NULL) > + error= (*m_file)->ht->create_partitioning_metadata(NULL, from, > + CHF_DELETE_FLAG); > + else > + error= (*m_file)->ht->create_partitioning_metadata(to, from, > + CHF_RENAME_FLAG); why not just error= (*m_file)->ht->create_partitioning_metadata(to, from); the engine can distingush CREATE from RENAME from DELETE by looking at whether 'to' or `from` is NULL. > + DBUG_EXECUTE_IF("failed_create_partitioning_metadata", > + { my_message_sql(ER_OUT_OF_RESOURCES,"Simulated crash",MYF(0)); > + error= 1; > + }); > + if (error) > + { > + if (to) > + { > + (void) handler::rename_table(to, from); > + (void) (*m_file)->ht->create_partitioning_metadata(from, to, > + CHF_RENAME_FLAG); > + goto rename_error; > + } > + else > + save_error=error; > + } > + } > DBUG_RETURN(save_error); > + > rename_error: > name_buffer_ptr= m_name_buffer_ptr; > for (abort_file= file, file= m_file; file < abort_file; file++) > @@ -3729,6 +3787,16 @@ int ha_partition::rebind() > #endif /* HAVE_M_PSI_PER_PARTITION */ > > > +/* > + Check if the table definition has changed for the part tables > + We use the first partition for the check. > +*/ > + > +int ha_partition::discover_check_version() > +{ > + return m_file[0]->discover_check_version(); > +} 1. generally, you have to do it for every open partition, not just for the first one. 2. m_file[0] partition is not necessarily open, you need to use the first open partition here (and everywhere) > + > /** > Clone the open and locked partitioning handler. > > @@ -11382,6 +11450,12 @@ int ha_partition::end_bulk_delete() > } > > > +bool ha_partition::check_if_updates_are_ignored(const char *op) const > +{ > + return (handler::check_if_updates_are_ignored(op) || > + ha_check_if_updates_are_ignored(table->in_use, partition_ht(), op)); no you don't need this virtual method at all, simply do - if (ha_check_if_updates_are_ignored(ha_thd(), ht, "DROP")) + if (ha_check_if_updates_are_ignored(ha_thd(), partition_ht(), "DROP")) that's why partition_ht() was created in the first place. > +} > + > /** > Perform initialization for a direct update request. > > diff --git a/sql/handler.h b/sql/handler.h > index 8c45c64bec8..eadbf28229c 100644 > --- a/sql/handler.h > +++ b/sql/handler.h > @@ -3183,7 +3200,10 @@ class handler :public Sql_alloc > > public: > virtual void unbind_psi(); > - virtual int rebind(); > + virtual void rebind_psi(); this doesn't have to be virtual anymore, because you moved the check into the virtual discover_check_version() > + /* Return error if definition doesn't match for already opened table */ > + virtual int discover_check_version() { return 0; } > + > /** > Put the handler in 'batch' mode when collecting > table io instrumented events. > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > index 53ccdb5d4c3..b1756b83056 100644 > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -5906,6 +5906,8 @@ mysql_execute_command(THD *thd) > case SQLCOM_CALL: > case SQLCOM_REVOKE: > case SQLCOM_GRANT: > + if (thd->variables.option_bits & OPTION_IF_EXISTS) > + lex->create_info.set(DDL_options_st::OPT_IF_EXISTS); 1. please remove the same (now redundant) code from Sql_cmd_alter_table::execute() 2. better put it only for the relevant case. Like this case SQLCOM_ALTER_TABLE: if (thd->variables.option_bits & OPTION_IF_EXISTS) lex->create_info.set(DDL_options_st::OPT_IF_EXISTS); /* fall through */ case SQLCOM_ANALYZE_TABLE: ... > DBUG_ASSERT(lex->m_sql_cmd != NULL); > res= lex->m_sql_cmd->execute(thd); > DBUG_PRINT("result", ("res: %d killed: %d is_error: %d", > diff --git a/sql/sql_partition.cc b/sql/sql_partition.cc > index 7385f9059e1..351464939e2 100644 > --- a/sql/sql_partition.cc > +++ b/sql/sql_partition.cc > @@ -7057,6 +7059,10 @@ uint fast_alter_partition_table(THD *thd, TABLE *table, > lpt->pack_frm_data= NULL; > lpt->pack_frm_len= 0; > > + /* Add IF EXISTS to binlog if shared table */ > + if (table->file->partition_ht()->flags & HTON_TABLE_MAY_NOT_EXIST_ON_SLAVE) > + thd->variables.option_bits|= OPTION_IF_EXISTS; > + No need to, you already have that in mysql_alter_table(). Just fix the check there, like this: if (!if_exists && - (table->s->db_type()->flags & HTON_TABLE_MAY_NOT_EXIST_ON_SLAVE)) + (table->file->partition_ht()->flags & HTON_TABLE_MAY_NOT_EXIST_ON_SLAVE)) > if (table->file->alter_table_flags(alter_info->flags) & > HA_PARTITION_ONE_PHASE) > { > diff --git a/sql/sql_partition_admin.cc b/sql/sql_partition_admin.cc > index ed77c0938f3..d29c014bdc2 100644 > --- a/sql/sql_partition_admin.cc > +++ b/sql/sql_partition_admin.cc > @@ -529,14 +530,46 @@ bool Sql_cmd_alter_table_exchange_partition:: > table_list->mdl_request.set_type(MDL_SHARED_NO_WRITE); > if (unlikely(open_tables(thd, &table_list, &table_counter, 0, > &alter_prelocking_strategy))) > + { > + if (thd->lex->if_exists() && > + thd->get_stmt_da()->sql_errno() == ER_NO_SUCH_TABLE) > + { > + /* > + ALTER TABLE IF EXISTS was used on not existing table > + We have to log the query on a slave as the table may be a shared one > + from the master and we need to ensure that the next slave can see > + the statement as this slave may not have the table shared > + */ > + thd->clear_error(); > + if (thd->slave_thread && > + write_bin_log(thd, true, thd->query(), thd->query_length())) > + DBUG_RETURN(true); > + my_ok(thd); > + DBUG_RETURN(false); > + } I don't like that. There are many Sql_cmd_alter_table* classes and lots of duplicated code. Okay, you've put your OPTION_IF_EXISTS check into the big switch in the mysql_execute_command() and thus avoided duplicating that code. But this if() is also in mysql_alter_table() (and may be more?) I'm thinking that doing many Sql_cmd_alter_table* classes was a bad idea which leads to code duplication. How can we fix that? > DBUG_RETURN(true); > + } > > part_table= table_list->table; > swap_table= swap_table_list->table; > > + if (part_table->file->check_if_updates_are_ignored("ALTER")) > + { > + if (thd->slave_thread && > + write_bin_log_with_if_exists(thd, true, false, true)) > + DBUG_RETURN(true); > + my_ok(thd); > + DBUG_RETURN(false); > + } another duplicated one > + > if (unlikely(check_exchange_partition(swap_table, part_table))) > DBUG_RETURN(TRUE); > > + /* Add IF EXISTS to binlog if shared table */ > + if (part_table->file->partition_ht()->flags & > + HTON_TABLE_MAY_NOT_EXIST_ON_SLAVE) > + force_if_exists= 1; and again > + > /* set lock pruning on first table */ > partition_name= alter_info->partition_names.head(); > if (unlikely(table_list->table->part_info-> > @@ -784,16 +821,51 @@ bool Sql_cmd_alter_table_truncate_partition::execute(THD *thd) > #endif /* WITH_WSREP */ > > if (open_tables(thd, &first_table, &table_counter, 0)) > - DBUG_RETURN(true); > + { > + if (thd->lex->if_exists() && > + thd->get_stmt_da()->sql_errno() == ER_NO_SUCH_TABLE) > + { > + /* > + ALTER TABLE IF EXISTS was used on not existing table > + We have to log the query on a slave as the table may be a shared one > + from the master and we need to ensure that the next slave can see > + the statement as this slave may not have the table shared > + */ > + thd->clear_error(); > + if (thd->slave_thread && > + write_bin_log(thd, true, thd->query(), thd->query_length())) > + DBUG_RETURN(TRUE); > + my_ok(thd); > + DBUG_RETURN(FALSE); > + } > + DBUG_RETURN(TRUE); > + } see? > > - if (!first_table->table || first_table->view || > - first_table->table->s->db_type() != partition_hton) > + if (!first_table->table || first_table->view) > { > my_error(ER_PARTITION_MGMT_ON_NONPARTITIONED, MYF(0)); > DBUG_RETURN(TRUE); > } > > - > + if (first_table->table->file->check_if_updates_are_ignored("ALTER")) > + { > + if (thd->slave_thread && > + write_bin_log_with_if_exists(thd, true, false, 1)) > + DBUG_RETURN(true); > + my_ok(thd); > + DBUG_RETURN(false); > + } > + > + if (first_table->table->s->db_type() != partition_hton) > + { > + my_error(ER_PARTITION_MGMT_ON_NONPARTITIONED, MYF(0)); > + DBUG_RETURN(TRUE); > + } why did you split the check for ER_PARTITION_MGMT_ON_NONPARTITIONED? > + > + if (first_table->table->file->partition_ht()->flags & > + HTON_TABLE_MAY_NOT_EXIST_ON_SLAVE) > + force_if_exists= 1; > + > /* > Prune all, but named partitions, > to avoid excessive calls to external_lock(). > diff --git a/sql/table.cc b/sql/table.cc > index fe096835144..c6b42b7ba36 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -25,7 +25,8 @@ > // primary_key_name > #include "sql_parse.h" // free_items > #include "strfunc.h" // unhex_type2 > -#include "sql_partition.h" // mysql_unpack_partition, > +#include "ha_partition.h" // PART_EXT > + // mysql_unpack_partition, also included below > // fix_partition_func, partition_info > #include "sql_base.h" > #include "create_options.h" > @@ -42,6 +43,7 @@ > #include "rpl_filter.h" > #include "sql_cte.h" > #include "ha_sequence.h" > +#include "ha_partition.h" also included above > #include "sql_show.h" > #include "opt_trace.h" > > @@ -620,6 +622,9 @@ enum open_frm_error open_table_def(THD *thd, TABLE_SHARE *share, uint flags) > { > DBUG_ASSERT(flags & GTS_TABLE); > DBUG_ASSERT(flags & GTS_USE_DISCOVERY); > + /* Delete .frm and .par files */ > + mysql_file_delete_with_symlink(key_file_frm, path, "", MYF(0)); > + strxmov(path, share->normalized_path.str, PAR_EXT, NullS); > mysql_file_delete_with_symlink(key_file_frm, path, "", MYF(0)); no need to strxmov, this third "" argument is the file extension, you can directly do mysql_file_delete_with_symlink(key_file_frm, share->normalized_path.str, PAR_EXT, MYF(0)); also it should be key_file_partition_ddl_log not key_file_frm (it's not a ddl log, but sql_table.cc uses key_file_partition_ddl_log for par files) > file= -1; > } > @@ -1679,12 +1687,13 @@ class Field_data_type_info_array > > 42..46 are unused since 5.0 (were for RAID support) > Also, there're few unused bytes in forminfo. > - > */ > > int TABLE_SHARE::init_from_binary_frm_image(THD *thd, bool write, > const uchar *frm_image, > - size_t frm_length) > + size_t frm_length, > + const uchar *par_image, > + size_t par_length) I don't think writing .par file belongs here, why not to do it in the caller? Or, better, don't send .par file to S3 and don't discover it, it'll make everything much simpler. .par can be created from the .frm, see sql_table.cc: file= mysql_create_frm_image(thd, orig_db, orig_table_name, create_info, alter_info, create_table_mode, key_info, key_count, frm); ... if (file->ha_create_partitioning_metadata(path, NULL, CHF_CREATE_FLAG)) goto err; > { > TABLE_SHARE *share= this; > uint new_frm_ver, field_pack_length, new_field_pack_flag; > @@ -1715,24 +1724,31 @@ int TABLE_SHARE::init_from_binary_frm_image(THD *thd, bool write, > uint len; > uint ext_key_parts= 0; > plugin_ref se_plugin= 0; > - bool vers_can_native= false; > + bool vers_can_native= false, frm_created= 0; > Field_data_type_info_array field_data_type_info_array; > - > MEM_ROOT *old_root= thd->mem_root; > Virtual_column_info **table_check_constraints; > extra2_fields extra2; > - > DBUG_ENTER("TABLE_SHARE::init_from_binary_frm_image"); > > keyinfo= &first_keyinfo; > thd->mem_root= &share->mem_root; > > - if (write && write_frm_image(frm_image, frm_length)) > - goto err; > - > if (frm_length < FRM_HEADER_SIZE + FRM_FORMINFO_SIZE) > goto err; > > + if (write) > + { > +#ifdef WITH_PARTITION_STORAGE_ENGINE > + if (par_image) > + if (write_par_image(par_image, par_length)) > + goto err; > +#endif > + frm_created= 1; > + if (write_frm_image(frm_image, frm_length)) > + goto err; > + } why not to create it at the end? then you won't need to delete it on an error. > + > share->frm_version= frm_image[2]; > /* > Check if .frm file created by MySQL 5.0. In this case we want to > diff --git a/storage/maria/ha_maria.cc b/storage/maria/ha_maria.cc > index ed9c5404a7f..1f99fd9d895 100644 > --- a/storage/maria/ha_maria.cc > +++ b/storage/maria/ha_maria.cc > @@ -2724,7 +2724,7 @@ void ha_maria::drop_table(const char *name) > { > DBUG_ASSERT(file->s->temporary); > (void) ha_close(); > - (void) maria_delete_table_files(name, 1, 0); > + (void) maria_delete_table_files(name, 1, MY_WME); MYF(MY_WME) as always? > } > > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 9243921c84b: Make all #sql temporary table names uniform
by Sergei Golubchik 18 Apr '20

18 Apr '20

Hi, Michael! On Apr 13, Michael Widenius wrote: > revision-id: 9243921c84b (mariadb-10.5.2-128-g9243921c84b) > parent(s): 8742d176bc2 > author: Michael Widenius <monty(a)mariadb.com> > committer: Michael Widenius <monty(a)mariadb.com> > timestamp: 2020-04-09 22:02:06 +0300 > message: > > Make all #sql temporary table names uniform > > The reason for this is to make all temporary file names similar and > also to be able to figure out from where a #sql-xxx name orginates. > > New format is for most cases: > #sql-name-current_pid-thread_id[-increment] > Where name is one of subselect, alter, exchange, temptable or backup > > The execptions are: typo, "exceptions" > ALTER PARTITION shadow files: > #sql-shadow-'original_table_name' Please, add a thread_id here at the end. normally MDL should ensure that no two threads can have a shadow for the same table at the same time, but we have enough bugs as it is to introduce another vector when two threads can overwrite each other temp files. > Names from the temp pool: > #sql-name-current_pid-pool_number Would you mind if I drop temp pool completely in 10.5? It was added by Jeremy in January 2001 with the comment Added --temp-pool option to mysqld. This will cause temporary files created to use a small set of filenames, to try and avoid problems in the Linux kernel. And I doubt it's still an issue in 2020. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

3 4

Re: [Maria-developers] faf8de3aa98: Fixed some assert crashes related to keyread.
by Sergei Golubchik 18 Apr '20

18 Apr '20

Hi, Michael! On Apr 13, Michael Widenius wrote: > revision-id: faf8de3aa98 (mariadb-10.5.2-121-gfaf8de3aa98) > parent(s): 3cbe15bd78c > author: Michael Widenius <monty(a)mariadb.com> > committer: Michael Widenius <monty(a)mariadb.com> > timestamp: 2020-04-09 01:37:02 +0300 > message: > > Fixed some assert crashes related to keyread. > > - MDEV-22062 Assertion `!table->file->keyread_enabled()' failed in > close_thread_table() > - MDEV-22077 table->no_keyread .. failed in join_read_first() > > diff --git a/mysql-test/main/keyread.test b/mysql-test/main/keyread.test > index d9d3002d392..76d0f5674dd 100644 > --- a/mysql-test/main/keyread.test > +++ b/mysql-test/main/keyread.test > @@ -8,3 +8,14 @@ create view v1 as select * from t1 where f2 = 1; > select distinct f1 from v1; > drop view v1; > drop table t1; > + > +# > +# MDEV-22062 Assertion `!table->file->keyread_enabled()' failed in > +# close_thread_table > +# > + > +CREATE TABLE t1 (a INT NOT NULL, UNIQUE(a)) ENGINE=InnoDB; > +INSERT INTO t1 VALUES (1),(2); > +DELETE FROM t1 ORDER BY a LIMIT 1; > +SELECT * FROM t1; > +DROP TABLE t1; where's a test case for MDEV-22077? > diff --git a/sql/sql_delete.cc b/sql/sql_delete.cc > index 19eabbb053c..2fc0de4345f 100644 > --- a/sql/sql_delete.cc > +++ b/sql/sql_delete.cc > @@ -547,10 +547,12 @@ bool mysql_delete(THD *thd, TABLE_LIST *table_list, COND *conds, > else > { > ha_rows scanned_limit= query_plan.scanned_rows; > + table->no_keyread= 1; > query_plan.index= get_index_for_order(order, table, select, limit, > &scanned_limit, > &query_plan.using_filesort, > &reverse); > + table->no_keyread= 0; why? > if (!query_plan.using_filesort) > query_plan.scanned_rows= scanned_limit; > } > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index 129dae9eedb..1f05156b96f 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -23584,7 +23584,7 @@ test_if_skip_sort_order(JOIN_TAB *tab,ORDER *order,ha_rows select_limit, > If ref_key used index tree reading only ('Using index' in EXPLAIN), > and best_key doesn't, then revert the decision. > */ > - if (table->covering_keys.is_set(best_key)) > + if (table->covering_keys.is_set(best_key) && !table->no_keyread) > table->file->ha_start_keyread(best_key); > else > table->file->ha_end_keyread(); > @@ -28568,8 +28568,6 @@ test_if_cheaper_ordering(const JOIN_TAB *tab, ORDER *order, TABLE *table, > if (new_used_key_parts != NULL) > *new_used_key_parts= best_key_parts; > table->file->ha_end_keyread(); > - if (is_best_covering && !table->no_keyread) > - table->file->ha_start_keyread(best_key); I find it very confusing that test_if_cheaper_ordering has a side effect of ending keyread. It was ok, when it ended previous keyread and started a new one. But just disabling - it's looks very strange. > DBUG_RETURN(TRUE); > } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 3

Re: [Maria-developers] bc5c062b1d1: Don't try to open temporary tables if there are no temporary tables
by Sergei Golubchik 17 Apr '20

17 Apr '20

Hi, Michael! On Apr 13, Michael Widenius wrote: > revision-id: bc5c062b1d1 (mariadb-10.5.2-124-gbc5c062b1d1) > parent(s): fb29c886701 > author: Michael Widenius <monty(a)mariadb.com> > committer: Michael Widenius <monty(a)mariadb.com> > timestamp: 2020-04-09 01:37:02 +0300 > message: > > Don't try to open temporary tables if there are no temporary tables Why? > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > index 4d7a7606136..be19a2e1d82 100644 > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -3704,9 +3704,9 @@ open_and_process_table(THD *thd, TABLE_LIST *tables, uint *counter, uint flags, > The problem is that since those attributes are not set in merge > children, another round of PREPARE will not help. > */ > - error= thd->open_temporary_table(tables); > - > - if (!error && !tables->table) > + if (!thd->has_temporary_tables() || > + (!(error= thd->open_temporary_table(tables)) && > + !tables->table)) please don't write conditions like that. keep it as before, two statements: if (thd->has_temporary_tables()) error= thd->open_temporary_table(tables); if (!error && !tables->table) > error= open_table(thd, tables, ot_ctx); > > thd->pop_internal_handler(); > @@ -3723,9 +3723,9 @@ open_and_process_table(THD *thd, TABLE_LIST *tables, uint *counter, uint flags, > Repair_mrg_table_error_handler repair_mrg_table_handler; > thd->push_internal_handler(&repair_mrg_table_handler); > > - error= thd->open_temporary_table(tables); > - > - if (!error && !tables->table) > + if (!thd->has_temporary_tables() || > + (!(error= thd->open_temporary_table(tables)) && > + !tables->table)) > error= open_table(thd, tables, ot_ctx); > > thd->pop_internal_handler(); > @@ -3740,7 +3740,8 @@ open_and_process_table(THD *thd, TABLE_LIST *tables, uint *counter, uint flags, > still might need to look for a temporary table if this table > list element corresponds to underlying table of a merge table. > */ > - error= thd->open_temporary_table(tables); > + if (thd->has_temporary_tables()) > + error= thd->open_temporary_table(tables); > } > > if (!error && !tables->table) > diff --git a/sql/temporary_tables.cc b/sql/temporary_tables.cc > index 407072a7b49..553460729a1 100644 > --- a/sql/temporary_tables.cc > +++ b/sql/temporary_tables.cc > @@ -338,9 +338,9 @@ bool THD::open_temporary_table(TABLE_LIST *tl) > have invalid db or table name. > Instead THD::open_tables() should be used. > */ > - DBUG_ASSERT(!tl->derived && !tl->schema_table); > + DBUG_ASSERT(!tl->derived && !tl->schema_table && has_temporary_tables()); please, never do asserts with &&-ed conditions, this should be DBUG_ASSERT(!tl->derived); DBUG_ASSERT(!tl->schema_table); DBUG_ASSERT(has_temporary_tables()); > - if (tl->open_type == OT_BASE_ONLY || !has_temporary_tables()) > + if (tl->open_type == OT_BASE_ONLY) > { > DBUG_PRINT("info", ("skip_temporary is set or no temporary tables")); > DBUG_RETURN(false); Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 2

Re: [Maria-developers] bf32018be96: Added support for VISIBLE attribute for indexes in CREATE TABLE
by Sergei Golubchik 17 Apr '20

17 Apr '20

Hi, Michael! On Apr 13, Michael Widenius wrote: > revision-id: bf32018be96 (mariadb-10.5.2-126-gbf32018be96) > parent(s): 62c2d0f3e1f > author: Michael Widenius <monty(a)mariadb.com> > committer: Michael Widenius <monty(a)mariadb.com> > timestamp: 2020-04-09 01:37:02 +0300 > message: > > Added support for VISIBLE attribute for indexes in CREATE TABLE > > MDEV-22199 Add VISIBLE attribute for indexes in CREATE TABLE > > This was done to make it easier to read in dumps from MySQL 8.0 This is not a "dump from MySQL 8.0". It is an SQL export file generated by MySQL Workbench. And MySQL Workbench has a configuration option whether to put this VISIBLE in the sql file or not. This is a client tool misconfiguration issue. I think the user can trivially tell his tool not to generate MariaDB-incompatible syntax and we don't have to change the server for this. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 2

Re: [Maria-developers] fb29c886701: Handle errors from external_unlock & mysql_unlock_tables
by Sergei Golubchik 16 Apr '20

16 Apr '20

Hi, Michael! On Apr 13, Michael Widenius wrote: > revision-id: fb29c886701 (mariadb-10.5.2-123-gfb29c886701) > parent(s): 22fb7f8995c > author: Michael Widenius <monty(a)mariadb.com> > committer: Michael Widenius <monty(a)mariadb.com> > timestamp: 2020-04-09 01:37:02 +0300 > message: > > Handle errors from external_unlock & mysql_unlock_tables > > Other things: > - Handler errors from ha_maria::implict_commit > - Disable DBUG in safe_mutex_lock to get trace file easier to read > > diff --git a/mysys/thr_mutex.c b/mysys/thr_mutex.c > index 4f495048f63..f32132136b8 100644 > --- a/mysys/thr_mutex.c > +++ b/mysys/thr_mutex.c > @@ -233,6 +233,7 @@ int safe_mutex_lock(safe_mutex_t *mp, myf my_flags, const char *file, > int error; > DBUG_PRINT("mutex", ("%s (0x%lx) locking", mp->name ? mp->name : "Null", > (ulong) mp)); > + DBUG_PUSH(""); > > pthread_mutex_lock(&mp->global); > if (!mp->file) > @@ -283,7 +284,7 @@ int safe_mutex_lock(safe_mutex_t *mp, myf my_flags, const char *file, > { > error= pthread_mutex_trylock(&mp->mutex); > if (error == EBUSY) > - return error; > + goto end; > } > else > error= pthread_mutex_lock(&mp->mutex); > @@ -393,6 +394,8 @@ int safe_mutex_lock(safe_mutex_t *mp, myf my_flags, const char *file, > } > } > > +end: > + DBUG_POP(); > DBUG_PRINT("mutex", ("%s (0x%lx) locked", mp->name, (ulong) mp)); this message is no longer always correct, is it? > return error; > } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 3cbe15bd78c: Fixed core dump in alter table if ADD PARTITION fails
by Sergei Golubchik 16 Apr '20

16 Apr '20

Hi, Michael! On Apr 13, Michael Widenius wrote: > revision-id: 3cbe15bd78c (mariadb-10.5.2-120-g3cbe15bd78c) > parent(s): d4d332d196d > author: Michael Widenius <monty(a)mariadb.com> > committer: Michael Widenius <monty(a)mariadb.com> > timestamp: 2020-04-09 01:37:01 +0300 > message: > > Fixed core dump in alter table if ADD PARTITION fails > > I didn't add a test case as to reproduce this we need to > have a failed write in on of the engines. Bug found and bug fix > verified while debugging S3 and partitions. > > --- > sql/sql_partition.cc | 3 ++- > 1 file changed, 2 insertions(+), 1 deletion(-) > > diff --git a/sql/sql_partition.cc b/sql/sql_partition.cc > index ef8ef5114a8..4e984fa775d 100644 > --- a/sql/sql_partition.cc > +++ b/sql/sql_partition.cc > @@ -6817,7 +6817,8 @@ static void handle_alter_part_error(ALTER_PARTITION_PARAM_TYPE *lpt, > DBUG_ENTER("handle_alter_part_error"); > DBUG_ASSERT(table->m_needs_reopen); > > - if (close_table) > + /* The table may not be open if ha_partition::change_partitions() failed */ > + if (close_table && !table->file->is_open()) This looks *very* confusing, like you close the table only if it's *not* open. But then the whole if() is very confusing, as it closes in both branches, so the meaning of "close_table" is totally not clear. May be instead of this your fix you'd better cherry-pick 9c02b7d6670b069866 from MySQL? It removes close_table completely, so I expect it to work for S3 just as you wanted. > { > /* > All instances of this table needs to be closed. > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 62c2d0f3e1f: Added mariadb-config.1 to .gitignore
by Sergei Golubchik 13 Apr '20

13 Apr '20

Hi, Michael! On Apr 13, Michael Widenius wrote: > revision-id: 62c2d0f3e1f (mariadb-10.5.2-125-g62c2d0f3e1f) > parent(s): bc5c062b1d1 > author: Michael Widenius <monty(a)mariadb.com> > committer: Michael Widenius <monty(a)mariadb.com> > timestamp: 2020-04-09 01:37:02 +0300 > message: > > Added mariadb-config.1 to .gitignore Where does it come from, where is it created? I've just grepped the whole tree, I don't see it ever gets created anywhere, so why to ignore it? > diff --git a/.gitignore b/.gitignore > index 8d3b5245447..8bd46093cbe 100644 > --- a/.gitignore > +++ b/.gitignore > @@ -581,6 +581,7 @@ man/mariadb-check.1 > man/mariadb-client-test.1 > man/mariadb-client-test-embedded.1 > man/mariadb_config.1 > +man/mariadb-config.1 > man/mariadb-convert-table-format.1 > man/mariadbd.8 > man/mariadbd-multi.1 > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 22fb7f8995c: Updated client and server to use new binary names in --debug traces
by Sergei Golubchik 13 Apr '20

13 Apr '20

Hi, Michael! On Apr 13, Michael Widenius wrote: > revision-id: 22fb7f8995c (mariadb-10.5.2-122-g22fb7f8995c) > parent(s): faf8de3aa98 > author: Michael Widenius <monty(a)mariadb.com> > committer: Michael Widenius <monty(a)mariadb.com> > timestamp: 2020-04-09 01:37:02 +0300 > message: > > Updated client and server to use new binary names in --debug traces ok to push Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 9941c6a3179: MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT
by Sergei Golubchik 13 Apr '20

13 Apr '20

Hi, Aleksey! On Apr 07, Aleksey Midenkov wrote: > revision-id: 9941c6a3179 (mariadb-10.5.2-164-g9941c6a3179) > parent(s): 920c3c6b237 > author: Aleksey Midenkov <midenok(a)gmail.com> > committer: Aleksey Midenkov <midenok(a)gmail.com> > timestamp: 2020-04-06 08:05:43 +0300 > message: > > MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT See some comment below, please > diff --git a/mysql-test/suite/versioning/common.inc b/mysql-test/suite/versioning/common.inc > index 355b571e5a0..b35a5138015 100644 > --- a/mysql-test/suite/versioning/common.inc > +++ b/mysql-test/suite/versioning/common.inc > @@ -6,6 +6,7 @@ if (!$TEST_VERSIONING_SO) > source include/have_innodb.inc; > > set @@session.time_zone='+00:00'; > +set @@global.time_zone='+00:00'; Why is that? I understand you might've needed it when a partition was added in a separate thread, but why now? > select ifnull(max(transaction_id), 0) into @start_trx_id from mysql.transaction_registry; > set @test_start=now(6); > > diff --git a/mysql-test/suite/versioning/r/delete_history.result b/mysql-test/suite/versioning/r/delete_history.result > index cb865a835b3..2e4a2bf9974 100644 > --- a/mysql-test/suite/versioning/r/delete_history.result > +++ b/mysql-test/suite/versioning/r/delete_history.result > @@ -154,3 +154,18 @@ select * from t1; > a > 1 > drop table t1; > +# > +# MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT > +# > +# Don't auto-create new partition on DELETE HISTORY: > +create or replace table t (a int) with system versioning > +partition by system_time limit 1000 auto_increment; this looks like a hack, I think we need a dedicated syntax for that. but I couldn't think of anything good now. ok, I see that you allow both AUTO_INCREMENT and AUTO. May be better just to use AUTO? > +delete history from t; > +show create table t; > +Table Create Table > +t CREATE TABLE `t` ( > + `a` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME LIMIT 1000 AUTO_INCREMENT > +PARTITIONS 2 > +drop table t; > diff --git a/mysql-test/suite/versioning/r/partition.result b/mysql-test/suite/versioning/r/partition.result > index a7047cbd11b..660d2c81961 100644 > --- a/mysql-test/suite/versioning/r/partition.result > +++ b/mysql-test/suite/versioning/r/partition.result > @@ -289,11 +291,27 @@ x > 6 > 7 > 8 > -## rotation by INTERVAL > +# Auto-create history partitions > +create or replace table t1 (x int) with system versioning > +partition by system_time limit 1000 auto_increment; > +lock tables t1 write; > +insert into t1 values (1); > +update t1 set x= x + 1; > +unlock tables; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME LIMIT 1000 AUTO_INCREMENT > +PARTITIONS 3 this test needs a comment, I don't understand what's happening here. why lock tables? > +# > +# Rotation by INTERVAL > +# > create or replace table t1 (x int) > with system versioning > partition by system_time interval 0 second partitions 3; > -ERROR HY000: Wrong parameters for partitioned `t1`: wrong value for 'INTERVAL' > +ERROR HY000: Wrong parameters for partitioned `t1`: wrong value for INTERVAL > create table t1 (i int) with system versioning > partition by system_time interval 6 day limit 98; > ERROR 42000: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'limit 98' at line 2 > @@ -302,7 +320,7 @@ partition by system_time interval 10 year partitions 3; > ERROR 22003: TIMESTAMP value is out of range in 'INTERVAL' > # INTERVAL and ALTER TABLE > create or replace table t1 (i int) with system versioning > -partition by system_time interval 1 hour; > +partition by system_time interval 59 minute; why? > set @ts=(select partition_description from information_schema.partitions > where table_schema='test' and table_name='t1' and partition_name='p0'); > alter table t1 add column b int; > @@ -353,28 +371,51 @@ Warning 4114 Versioned table `test`.`t1`: last HISTORY partition (`p1`) is out o > delete from t1; > Warnings: > Warning 4114 Versioned table `test`.`t1`: last HISTORY partition (`p1`) is out of INTERVAL, need more HISTORY partitions > -select subpartition_name,partition_description,table_rows from information_schema.partitions where table_schema='test' and table_name='t1'; > +select subpartition_name,partition_description from information_schema.partitions where table_schema='test' and table_name='t1'; why? > -subpartition_name partition_description table_rows > -p1sp0 2001-02-04 00:00:00 1 > -p1sp1 2001-02-04 00:00:00 1 > -pnsp0 CURRENT 0 > -pnsp1 CURRENT 0 > +subpartition_name partition_description > +p1sp0 2001-02-04 00:00:00 > +p1sp1 2001-02-04 00:00:00 > +pnsp0 CURRENT > +pnsp1 CURRENT > +select * from t1 partition (p1); > +i > +1 > +2 > set timestamp=unix_timestamp('2001-02-04 10:20:55'); > alter table t1 add partition (partition p0 history, partition p2 history); > set timestamp=unix_timestamp('2001-02-04 10:30:00'); > insert t1 values (4),(5); > set timestamp=unix_timestamp('2001-02-04 10:30:10'); > update t1 set i=6 where i=5; > -select subpartition_name,partition_description,table_rows from information_schema.partitions where table_schema='test' and table_name='t1'; > -subpartition_name partition_description table_rows > -p1sp0 2001-02-04 00:00:00 1 > -p1sp1 2001-02-04 00:00:00 0 > -p0sp0 2001-02-05 00:00:00 1 > -p0sp1 2001-02-05 00:00:00 1 > -p2sp0 2001-02-06 00:00:00 0 > -p2sp1 2001-02-06 00:00:00 0 > -pnsp0 CURRENT 0 > -pnsp1 CURRENT 2 > +select subpartition_name, partition_description from information_schema.partitions where table_schema='test' and table_name='t1'; > +subpartition_name partition_description > +p1sp0 2001-02-04 00:00:00 > +p1sp1 2001-02-04 00:00:00 > +p0sp0 2001-02-05 00:00:00 > +p0sp1 2001-02-05 00:00:00 > +p2sp0 2001-02-06 00:00:00 > +p2sp1 2001-02-06 00:00:00 > +pnsp0 CURRENT > +pnsp1 CURRENT > +select * from t1 partition (p1); > +i > +1 > +select * from t1 partition (p0); > +i > +5 > +2 > +select * from t1 partition (p2); > +i > +alter table t1 rebuild partition p0, p1, p2; > +select * from t1 partition (p1); > +i > +1 > +select * from t1 partition (p0); > +i > +5 > +2 > +select * from t1 partition (p2); > +i > ## pruning check > set @ts=(select partition_description from information_schema.partitions > where table_schema='test' and table_name='t1' and partition_name='p0' limit 1); > @@ -1044,3 +1085,568 @@ t1 CREATE TABLE `t1` ( > PARTITION BY SYSTEM_TIME > PARTITIONS 8 > drop tables t1; > +# > +# MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT > +# > +create or replace table t1 (x int) with system versioning > +partition by system_time limit 999 auto_increment; > +ERROR HY000: Wrong parameters for partitioned `t1`: wrong value for LIMIT (< 1000) > +create or replace table t1 (x int) with system versioning > +partition by system_time interval 3599 second auto_increment; > +ERROR HY000: Wrong parameters for partitioned `t1`: wrong value for INTERVAL (< 1 HOUR) no arbitrary limitations, please > +create or replace table t1 (x int) with system versioning > +partition by system_time limit 1000 auto_increment; > +affected rows: 0 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME LIMIT 1000 AUTO_INCREMENT > +PARTITIONS 2 > +affected rows: 1 > +insert into t1 values (1); > +affected rows: 1 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME LIMIT 1000 AUTO_INCREMENT > +PARTITIONS 3 > +affected rows: 1 > +# Increment from 2 to 5 > +create or replace table t1 (x int) with system versioning > +partition by system_time interval 3600 second > +starts '2000-01-01 00:00:00' auto_increment; > +affected rows: 0 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 3600 SECOND STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +PARTITIONS 2 > +affected rows: 1 > +set timestamp= unix_timestamp('2000-01-01 00:00:00'); > +affected rows: 0 > +insert into t1 values (1); > +affected rows: 1 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 3600 SECOND STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +PARTITIONS 3 > +affected rows: 1 > +set timestamp= unix_timestamp('2000-01-01 01:00:00'); > +affected rows: 0 > +update t1 set x= x + 1; > +affected rows: 1 > +info: Rows matched: 1 Changed: 1 Inserted: 1 Warnings: 0 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 3600 SECOND STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +PARTITIONS 4 > +affected rows: 1 > +set timestamp= unix_timestamp('2000-01-01 02:00:00'); > +affected rows: 0 > +update t1 set x= x + 2; > +affected rows: 1 > +info: Rows matched: 1 Changed: 1 Inserted: 1 Warnings: 0 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 3600 SECOND STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +PARTITIONS 5 > +affected rows: 1 > +# Increment from 3 to 6, manual names, LOCK TABLES again, I think this is overcomplication and overengineering. I don't see any need for mixing automatic and manual partition names. > +set timestamp= unix_timestamp('2000-01-01 00:00:00'); > +affected rows: 0 > +create or replace table t1 (x int) with system versioning > +partition by system_time interval 1 hour auto_increment ( > +partition p1 history, > +partition p3 history, > +partition pn current); > +affected rows: 0 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +(PARTITION `p1` HISTORY ENGINE = DEFAULT_ENGINE, > + PARTITION `p3` HISTORY ENGINE = DEFAULT_ENGINE, > + PARTITION `pn` CURRENT ENGINE = DEFAULT_ENGINE) > +affected rows: 1 > +insert into t1 values (1); > +affected rows: 1 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +(PARTITION `p1` HISTORY ENGINE = DEFAULT_ENGINE, > + PARTITION `p3` HISTORY ENGINE = DEFAULT_ENGINE, > + PARTITION `pn` CURRENT ENGINE = DEFAULT_ENGINE) > +affected rows: 1 > +set timestamp= unix_timestamp('2000-01-01 01:00:00'); > +affected rows: 0 > +update t1 set x= x + 3; > +affected rows: 1 > +info: Rows matched: 1 Changed: 1 Inserted: 1 Warnings: 0 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +(PARTITION `p1` HISTORY ENGINE = DEFAULT_ENGINE, > + PARTITION `p3` HISTORY ENGINE = DEFAULT_ENGINE, > + PARTITION `p2` HISTORY ENGINE = DEFAULT_ENGINE, > + PARTITION `pn` CURRENT ENGINE = DEFAULT_ENGINE) > +affected rows: 1 > +set timestamp= unix_timestamp('2000-01-01 02:00:00'); > +affected rows: 0 > +update t1 set x= x + 4; > +affected rows: 1 > +info: Rows matched: 1 Changed: 1 Inserted: 1 Warnings: 0 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +(PARTITION `p1` HISTORY ENGINE = DEFAULT_ENGINE, > + PARTITION `p3` HISTORY ENGINE = DEFAULT_ENGINE, > + PARTITION `p2` HISTORY ENGINE = DEFAULT_ENGINE, > + PARTITION `p4` HISTORY ENGINE = DEFAULT_ENGINE, > + PARTITION `pn` CURRENT ENGINE = DEFAULT_ENGINE) > +affected rows: 1 > +lock tables t1 write; > +affected rows: 0 > +set timestamp= unix_timestamp('2000-01-01 03:00:00'); > +affected rows: 0 > +update t1 set x= x + 5; > +affected rows: 1 > +info: Rows matched: 1 Changed: 1 Inserted: 1 Warnings: 0 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +(PARTITION `p1` HISTORY ENGINE = DEFAULT_ENGINE, > + PARTITION `p3` HISTORY ENGINE = DEFAULT_ENGINE, > + PARTITION `p2` HISTORY ENGINE = DEFAULT_ENGINE, > + PARTITION `p4` HISTORY ENGINE = DEFAULT_ENGINE, > + PARTITION `p5` HISTORY ENGINE = DEFAULT_ENGINE, > + PARTITION `pn` CURRENT ENGINE = DEFAULT_ENGINE) > +affected rows: 1 > +unlock tables; > +affected rows: 0 > +# Test VIEW, LOCK TABLES > +set timestamp= unix_timestamp('2000-01-01 00:00:00'); > +affected rows: 0 > +create or replace table t1 (x int) with system versioning > +partition by system_time interval 1 hour auto_increment; > +affected rows: 0 > +create or replace view v1 as select * from t1; > +affected rows: 0 > +insert into v1 values (1); > +affected rows: 1 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +PARTITIONS 3 > +affected rows: 1 > +lock tables t1 write; > +affected rows: 0 > +set timestamp= unix_timestamp('2000-01-01 01:00:00'); > +affected rows: 0 > +update t1 set x= x + 3; > +affected rows: 1 > +info: Rows matched: 1 Changed: 1 Inserted: 1 Warnings: 0 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +PARTITIONS 4 > +affected rows: 1 > +unlock tables; > +affected rows: 0 > +drop view v1; > +affected rows: 0 > +drop tables t1; > +affected rows: 0 > +# Multiple increments in single command > +set timestamp= unix_timestamp('2000-01-01 00:00:00'); > +affected rows: 0 > +create or replace table t1 (x int) with system versioning > +partition by system_time interval 1 hour auto_increment partitions 3; > +affected rows: 0 > +create or replace table t2 (y int) with system versioning > +partition by system_time interval 1 hour auto_increment partitions 4; > +affected rows: 0 > +insert into t1 values (1); > +affected rows: 1 > +insert into t2 values (2); > +affected rows: 1 > +set timestamp= unix_timestamp('2000-01-01 01:00:00'); > +affected rows: 0 > +update t1, t2 set x= x + 1, y= y + 1; > +affected rows: 2 > +info: Rows matched: 2 Changed: 2 Warnings: 0 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +PARTITIONS 4 > +affected rows: 1 > +show create table t2; > +Table Create Table > +t2 CREATE TABLE `t2` ( > + `y` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +PARTITIONS 4 > +affected rows: 1 > +set timestamp= unix_timestamp('2000-01-01 02:00:00'); > +affected rows: 0 > +update t1, t2 set x= x + 1, y= y + 1; > +affected rows: 2 > +info: Rows matched: 2 Changed: 2 Warnings: 0 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +PARTITIONS 5 > +affected rows: 1 > +show create table t2; > +Table Create Table > +t2 CREATE TABLE `t2` ( > + `y` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +PARTITIONS 5 > +affected rows: 1 > +drop tables t1, t2; > +affected rows: 0 > +# PS, SP, LOCK TABLES > +set timestamp= unix_timestamp('2000-01-01 00:00:00'); > +affected rows: 0 > +create or replace table t1 (x int) with system versioning > +partition by system_time interval 1 hour auto_increment; > +affected rows: 0 > +execute immediate 'insert into t1 values (1)'; > +affected rows: 1 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +PARTITIONS 3 > +affected rows: 1 > +prepare s from 'update t1 set x= x + 6'; > +affected rows: 0 > +info: Statement prepared > +set timestamp= unix_timestamp('2000-01-01 01:00:00'); > +affected rows: 0 > +execute s; > +affected rows: 1 > +info: Rows matched: 1 Changed: 1 Inserted: 1 Warnings: 0 > +execute s; > +affected rows: 1 > +info: Rows matched: 1 Changed: 1 Inserted: 1 Warnings: 0 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +PARTITIONS 4 > +affected rows: 1 > +lock tables t1 write; > +affected rows: 0 > +set timestamp= unix_timestamp('2000-01-01 02:00:00'); > +affected rows: 0 > +execute s; > +affected rows: 1 > +info: Rows matched: 1 Changed: 1 Inserted: 1 Warnings: 0 > +execute s; > +affected rows: 1 > +info: Rows matched: 1 Changed: 1 Inserted: 1 Warnings: 0 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +PARTITIONS 5 add a test where timestamp is incremented by, say, 24 hours, please > +affected rows: 1 > +unlock tables; > +affected rows: 0 > +drop prepare s; > +affected rows: 0 > +create procedure sp() update t1 set x= x + 7; > +affected rows: 0 > +set timestamp= unix_timestamp('2000-01-01 03:00:00'); > +affected rows: 0 > +call sp; > +affected rows: 1 > +call sp; > +affected rows: 1 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +PARTITIONS 6 > +affected rows: 1 > +lock tables t1 write; > +affected rows: 0 > +set timestamp= unix_timestamp('2000-01-01 04:00:00'); > +affected rows: 0 > +call sp; > +affected rows: 1 > +call sp; > +affected rows: 1 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +PARTITIONS 7 > +affected rows: 1 > +unlock tables; > +affected rows: 0 > +drop procedure sp; > +affected rows: 0 > +# Complex table > +set timestamp= unix_timestamp('2000-01-01 00:00:00'); > +affected rows: 0 > +create or replace table t1 ( > +x int primary key auto_increment, > +t timestamp(6) default '2001-11-11 11:11:11', > +b blob(4096) compressed null, > +c varchar(1033) character set utf8 not null, > +u int unique, > +m enum('a', 'b', 'c') not null default 'a' comment 'absolute', > +i1 tinyint, i2 smallint, i3 bigint, > +index three(i1, i2, i3), > +v1 timestamp(6) generated always as (t + interval 1 day), > +v2 timestamp(6) generated always as (t + interval 1 month) stored, > +s timestamp(6) as row start, > +e timestamp(6) as row end, > +period for system_time (s, e), > +ps date, pe date, > +period for app_time (ps, pe), > +constraint check_constr check (u > -1)) > +with system versioning default charset=ucs2 > +partition by system_time interval 1 hour auto_increment ( > +partition p2 history, > +partition pn current); > +affected rows: 0 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) NOT NULL AUTO_INCREMENT, > + `t` timestamp(6) NOT NULL DEFAULT '2001-11-11 11:11:11.000000', > + `b` blob /*!100301 COMPRESSED*/ DEFAULT NULL, > + `c` varchar(1033) CHARACTER SET utf8 NOT NULL, > + `u` int(11) DEFAULT NULL, > + `m` enum('a','b','c') NOT NULL DEFAULT 'a' COMMENT 'absolute', > + `i1` tinyint(4) DEFAULT NULL, > + `i2` smallint(6) DEFAULT NULL, > + `i3` bigint(20) DEFAULT NULL, > + `v1` timestamp(6) GENERATED ALWAYS AS (`t` + interval 1 day) VIRTUAL, > + `v2` timestamp(6) GENERATED ALWAYS AS (`t` + interval 1 month) STORED, > + `s` timestamp(6) GENERATED ALWAYS AS ROW START, > + `e` timestamp(6) GENERATED ALWAYS AS ROW END, > + `ps` date NOT NULL, > + `pe` date NOT NULL, > + PRIMARY KEY (`x`,`e`), > + UNIQUE KEY `u` (`u`,`e`), > + KEY `three` (`i1`,`i2`,`i3`), > + PERIOD FOR SYSTEM_TIME (`s`, `e`), > + PERIOD FOR `app_time` (`ps`, `pe`), > + CONSTRAINT `check_constr` CHECK (`u` > -1) > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=ucs2 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +(PARTITION `p2` HISTORY ENGINE = DEFAULT_ENGINE, > + PARTITION `pn` CURRENT ENGINE = DEFAULT_ENGINE) > +affected rows: 1 > +insert into t1 (x, c, u, i1, i2, i3, ps, pe) > +values (1, 'cc', 0, 1, 2, 3, '1999-01-01', '2000-01-01'); > +affected rows: 1 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) NOT NULL AUTO_INCREMENT, > + `t` timestamp(6) NOT NULL DEFAULT '2001-11-11 11:11:11.000000', > + `b` blob /*!100301 COMPRESSED*/ DEFAULT NULL, > + `c` varchar(1033) CHARACTER SET utf8 NOT NULL, > + `u` int(11) DEFAULT NULL, > + `m` enum('a','b','c') NOT NULL DEFAULT 'a' COMMENT 'absolute', > + `i1` tinyint(4) DEFAULT NULL, > + `i2` smallint(6) DEFAULT NULL, > + `i3` bigint(20) DEFAULT NULL, > + `v1` timestamp(6) GENERATED ALWAYS AS (`t` + interval 1 day) VIRTUAL, > + `v2` timestamp(6) GENERATED ALWAYS AS (`t` + interval 1 month) STORED, > + `s` timestamp(6) GENERATED ALWAYS AS ROW START, > + `e` timestamp(6) GENERATED ALWAYS AS ROW END, > + `ps` date NOT NULL, > + `pe` date NOT NULL, > + PRIMARY KEY (`x`,`e`), > + UNIQUE KEY `u` (`u`,`e`), > + KEY `three` (`i1`,`i2`,`i3`), > + PERIOD FOR SYSTEM_TIME (`s`, `e`), > + PERIOD FOR `app_time` (`ps`, `pe`), > + CONSTRAINT `check_constr` CHECK (`u` > -1) > +) ENGINE=DEFAULT_ENGINE AUTO_INCREMENT=2 DEFAULT CHARSET=ucs2 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +(PARTITION `p2` HISTORY ENGINE = DEFAULT_ENGINE, > + PARTITION `p1` HISTORY ENGINE = DEFAULT_ENGINE, > + PARTITION `pn` CURRENT ENGINE = DEFAULT_ENGINE) > +affected rows: 1 > +set timestamp= unix_timestamp('2000-01-01 01:00:00'); > +affected rows: 0 > +update t1 set x= x + 8; > +affected rows: 1 > +info: Rows matched: 1 Changed: 1 Inserted: 1 Warnings: 0 > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) NOT NULL AUTO_INCREMENT, > + `t` timestamp(6) NOT NULL DEFAULT '2001-11-11 11:11:11.000000', > + `b` blob /*!100301 COMPRESSED*/ DEFAULT NULL, > + `c` varchar(1033) CHARACTER SET utf8 NOT NULL, > + `u` int(11) DEFAULT NULL, > + `m` enum('a','b','c') NOT NULL DEFAULT 'a' COMMENT 'absolute', > + `i1` tinyint(4) DEFAULT NULL, > + `i2` smallint(6) DEFAULT NULL, > + `i3` bigint(20) DEFAULT NULL, > + `v1` timestamp(6) GENERATED ALWAYS AS (`t` + interval 1 day) VIRTUAL, > + `v2` timestamp(6) GENERATED ALWAYS AS (`t` + interval 1 month) STORED, > + `s` timestamp(6) GENERATED ALWAYS AS ROW START, > + `e` timestamp(6) GENERATED ALWAYS AS ROW END, > + `ps` date NOT NULL, > + `pe` date NOT NULL, > + PRIMARY KEY (`x`,`e`), > + UNIQUE KEY `u` (`u`,`e`), > + KEY `three` (`i1`,`i2`,`i3`), > + PERIOD FOR SYSTEM_TIME (`s`, `e`), > + PERIOD FOR `app_time` (`ps`, `pe`), > + CONSTRAINT `check_constr` CHECK (`u` > -1) > +) ENGINE=DEFAULT_ENGINE AUTO_INCREMENT=10 DEFAULT CHARSET=ucs2 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +(PARTITION `p2` HISTORY ENGINE = DEFAULT_ENGINE, > + PARTITION `p1` HISTORY ENGINE = DEFAULT_ENGINE, > + PARTITION `p3` HISTORY ENGINE = DEFAULT_ENGINE, > + PARTITION `pn` CURRENT ENGINE = DEFAULT_ENGINE) > +affected rows: 1 > +# Concurrent DML > +set timestamp= unix_timestamp('2000-01-01 00:00:00'); > +create or replace table t1 (x int) with system versioning > +partition by system_time interval 1 hour auto_increment; > +insert into t1 values (1); > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +PARTITIONS 3 > +connect con8, localhost, root; > +connect con7, localhost, root; > +connect con6, localhost, root; > +connect con5, localhost, root; > +connect con4, localhost, root; > +connect con3, localhost, root; > +connect con2, localhost, root; > +connect con1, localhost, root; > +set timestamp= unix_timestamp('2000-01-01 01:00:00'); > +update t1 set x= x + 10; > +connection con2; > +set timestamp= unix_timestamp('2000-01-01 01:00:00'); > +update t1 set x= x + 20; > +connection con3; > +set timestamp= unix_timestamp('2000-01-01 01:00:00'); > +update t1 set x= x + 30; > +connection con4; > +set timestamp= unix_timestamp('2000-01-01 01:00:00'); > +update t1 set x= x + 40; > +connection con5; > +set timestamp= unix_timestamp('2000-01-01 01:00:00'); > +update t1 set x= x + 50; > +connection con6; > +set timestamp= unix_timestamp('2000-01-01 01:00:00'); > +update t1 set x= x + 60; > +connection con7; > +set timestamp= unix_timestamp('2000-01-01 01:00:00'); > +update t1 set x= x + 70; > +connection con8; > +set timestamp= unix_timestamp('2000-01-01 01:00:00'); > +update t1 set x= x + 80; > +connection con1; > +disconnect con1; > +connection con2; > +disconnect con2; > +connection con3; > +disconnect con3; > +connection con4; > +disconnect con4; > +connection con5; > +disconnect con5; > +connection con6; > +disconnect con6; > +connection con7; > +disconnect con7; > +disconnect con8; > +connection default; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +PARTITIONS 4 > +drop tables t1; > +create or replace table t1 (x int) with system versioning engine innodb > +partition by system_time interval 1 hour auto; > +start transaction; > +select * from t1; > +x > +connect con1, localhost, root; > +set lock_wait_timeout= 1; > +insert into t1 values (1); > +Warnings: > +Warning 4114 Versioned table `test`.`t1`: last HISTORY partition (`p0`) is out of INTERVAL, need more HISTORY partitions > +Error 1205 Lock wait timeout exceeded; try restarting transaction > +Warning 4171 Auto-increment history partition: alter partition table failed > +Warning 4171 Versioned table `test`.`t1`: adding HISTORY partition failed with error 0, see error log for details "error 0" is strange and "see error log for details" isn't very user-friendly, a user might not have access to the error log at all > +select * from t1; > +x > +1 I don't understand, there was an error above, why did insert succeed? > +disconnect con1; > +connection default; > +drop table t1; > diff --git a/mysql-test/suite/versioning/r/rpl.result b/mysql-test/suite/versioning/r/rpl.result > index 627f3991499..68113190889 100644 > --- a/mysql-test/suite/versioning/r/rpl.result > +++ b/mysql-test/suite/versioning/r/rpl.result > @@ -164,4 +164,65 @@ update t1 set i = 0; > connection slave; > connection master; > drop table t1; > +# > +# MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT > +# > +set timestamp= unix_timestamp('2000-01-01 00:00:00'); > +create or replace table t1 (x int) with system versioning > +partition by system_time interval 1 hour auto_increment; > +insert t1 values (); > +set timestamp= unix_timestamp('2000-01-01 01:00:00'); > +delete from t1; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +PARTITIONS 4 > +connection slave; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME INTERVAL 1 HOUR STARTS TIMESTAMP'2000-01-01 00:00:00' AUTO_INCREMENT > +PARTITIONS 4 > +connection master; > +drop table t1; > +# > +# MENT-685 DML events for auto-partitioned tables are written into binary log twice > +# the test below doesn't seem to match the description above > +create table t1 (x int) partition by hash (x); > +alter table t1 add partition partitions 1 auto_increment; > +ERROR 42000: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use > +drop table t1; > +create table t1 (x int) with system versioning > +partition by system_time limit 1000 auto > +(partition p1 history, partition pn current); > +insert into t1 values (1); > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME LIMIT 1000 AUTO_INCREMENT > +(PARTITION `p1` HISTORY ENGINE = ENGINE, > + PARTITION `p2` HISTORY ENGINE = ENGINE, > + PARTITION `pn` CURRENT ENGINE = ENGINE) > +connection slave; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL > +) ENGINE=ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > + PARTITION BY SYSTEM_TIME LIMIT 1000 AUTO_INCREMENT > +(PARTITION `p1` HISTORY ENGINE = ENGINE, > + PARTITION `p2` HISTORY ENGINE = ENGINE, > + PARTITION `pn` CURRENT ENGINE = ENGINE) > +select * from t1; > +x > +1 > +connection master; > +drop table t1; > include/rpl_end.inc > diff --git a/sql/handler.cc b/sql/handler.cc > index c6ecc9566d8..c86f96b9689 100644 > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -1512,7 +1512,7 @@ int ha_commit_trans(THD *thd, bool all) > DBUG_ASSERT(thd->transaction.stmt.ha_list == NULL || > trans == &thd->transaction.stmt); > > - if (thd->in_sub_stmt) > + if (thd->in_sub_stmt & ~SUB_STMT_AUTO_HIST) 1. why? That is, why do you call ha_commit_trans() when adding a partition? 2. please add a test with insert under start transaction. what should happen in this case? > { > DBUG_ASSERT(0); > /* > diff --git a/sql/partition_info.cc b/sql/partition_info.cc > index f4b7260f8b0..617c839721b 100644 > --- a/sql/partition_info.cc > +++ b/sql/partition_info.cc > @@ -848,29 +850,289 @@ void partition_info::vers_set_hist_part(THD *thd) > else > vers_info->hist_part= next; > } > + } > + else if (vers_info->interval.is_set()) > + { > + if (vers_info->hist_part->range_value <= thd->query_start()) > + { > + partition_element *next= NULL; > + bool error= true; > + List_iterator<partition_element> it(partitions); > + while (next != vers_info->hist_part) > + next= it++; > + > + while ((next= it++) != vers_info->now_part) > + { > + vers_info->hist_part= next; > + if (next->range_value > thd->query_start()) > + { > + error= false; > + break; > + } > + } > + if (error) > + my_error(WARN_VERS_PART_FULL, MYF(ME_WARNING|ME_ERROR_LOG), > + table->s->db.str, table->s->table_name.str, > + vers_info->hist_part->partition_name, "INTERVAL"); > + } > + } > + > + if (!vers_info->auto_inc || > + vers_info->hist_part->id + VERS_MIN_EMPTY < vers_info->now_part->id) > return; > + > + switch (thd->lex->sql_command) > + { > + case SQLCOM_DELETE: > + if (thd->lex->last_table()->vers_conditions.delete_history) > + break; > + /* fallthrough */ > + case SQLCOM_UPDATE: > + case SQLCOM_INSERT: > + case SQLCOM_INSERT_SELECT: > + case SQLCOM_LOAD: > + case SQLCOM_REPLACE: > + case SQLCOM_REPLACE_SELECT: > + case SQLCOM_DELETE_MULTI: > + case SQLCOM_UPDATE_MULTI: it's rather fragile to check for specific sql statements. why not to look at the table lock instead? (with a special check for delete history) Ok, it's a bug. Please add a test with multi-update, where a partitioned table is *not* updated. Like update t1, tpart set t1.x=5 where t1.y=tpart.z; here a new partition should clearly not be created. also, a simpler example (multi-update is difficult): insert t1 select * from tpart; add both, please. > + { > + TABLE_SHARE *share; > + List_iterator_fast<TABLE_SHARE> it(thd->vers_auto_part_tables); > + while ((share= it++)) > + { > + if (table->s == share) > + break; > + } > + if (share) > + break; > + /* Prevent spawning multiple instances of vers_add_auto_parts() */ > + bool altering; > + mysql_mutex_lock(&table->s->LOCK_share); > + altering= table->s->vers_auto_part; > + if (!altering) > + table->s->vers_auto_part= true; > + mysql_mutex_unlock(&table->s->LOCK_share); > + if (altering) > + break; what happens if you're altering already? logically this thread should wait. Where does it do it? > + if (thd->vers_auto_part_tables.push_back(table->s)) > + { > + my_error(ER_OUT_OF_RESOURCES, MYF(0)); > + } > + } > + default:; > } > +} > > - if (vers_info->interval.is_set()) > - { > - if (vers_info->hist_part->range_value > thd->query_start()) > - return; > > - partition_element *next= NULL; > - List_iterator<partition_element> it(partitions); > - while (next != vers_info->hist_part) > - next= it++; > +/** > + @brief Run fast_alter_partition_table() to add new history partitions > + for tables requiring them. > +*/ > +void vers_add_auto_parts(THD *thd) > +{ > + HA_CREATE_INFO create_info; > + Alter_info alter_info; > + String query; > + TABLE_LIST *table_list= NULL; > + partition_info *save_part_info= thd->work_part_info; > + Query_tables_list save_query_tables; > + Reprepare_observer *save_reprepare_observer= thd->m_reprepare_observer; > + Diagnostics_area new_stmt_da(thd->query_id, false, true); > + Diagnostics_area *save_stmt_da= thd->get_stmt_da(); > + bool save_no_write_to_binlog= thd->lex->no_write_to_binlog; > + const CSET_STRING save_query= thd->query_string; > + thd->m_reprepare_observer= NULL; > + thd->lex->reset_n_backup_query_tables_list(&save_query_tables); > + thd->in_sub_stmt|= SUB_STMT_AUTO_HIST; > + thd->lex->no_write_to_binlog= !thd->is_current_stmt_binlog_format_row(); > + TABLE_LIST *tl; > + > + DBUG_ASSERT(!thd->vers_auto_part_tables.is_empty()); > + > + for (TABLE_SHARE &share: thd->vers_auto_part_tables) > + { > + tl= (TABLE_LIST *) thd->alloc(sizeof(TABLE_LIST)); > + tl->init_one_table(&share.db, &share.table_name, NULL, TL_READ_NO_INSERT); > + tl->open_type= OT_BASE_ONLY; > + tl->i_s_requested_object= OPEN_TABLE_ONLY; > + tl->next_global= table_list; > + table_list= tl; > + } > + > + /* NB: mysql_execute_command() can be recursive because of PS/SP. > + Don't duplicate any processing including error messages. */ > + thd->vers_auto_part_tables.empty(); > + > + DBUG_ASSERT(!thd->is_error()); > + /* NB: we have to preserve m_affected_rows, m_row_count_func, m_last_insert_id, etc */ > + thd->set_stmt_da(&new_stmt_da); > + new_stmt_da.set_overwrite_status(true); > + > + DDL_options_st ddl_opts_none; > + ddl_opts_none.init(); > + if (open_and_lock_tables(thd, ddl_opts_none, table_list, false, 0)) > + goto open_err; > + > + for (tl= table_list; tl; tl= tl->next_global) > + { > + TABLE *table= tl->table; > + DBUG_ASSERT(table); > + DBUG_ASSERT(table->s->get_table_ref_type() == TABLE_REF_BASE_TABLE); > + DBUG_ASSERT(table->versioned()); > + DBUG_ASSERT(table->part_info); > + DBUG_ASSERT(table->part_info->vers_info); > + alter_info.reset(); > + alter_info.partition_flags= ALTER_PARTITION_ADD|ALTER_PARTITION_AUTO_HIST; > + create_info.init(); > + create_info.alter_info= &alter_info; > + Alter_table_ctx alter_ctx(thd, tl, 1, &table->s->db, &table->s->table_name); > + > + if (thd->mdl_context.upgrade_shared_lock(table->mdl_ticket, > + MDL_SHARED_NO_WRITE, > + thd->variables.lock_wait_timeout)) > + goto exit; > + > + create_info.db_type= table->s->db_type(); > + create_info.options|= HA_VERSIONED_TABLE; > + DBUG_ASSERT(create_info.db_type); > + > + create_info.vers_info.set_start(table->s->vers_start_field()->field_name); > + create_info.vers_info.set_end(table->s->vers_end_field()->field_name); > + > + partition_info *part_info= new partition_info(); > + if (unlikely(!part_info)) > + { > + my_error(ER_OUT_OF_RESOURCES, MYF(0)); > + goto exit; > + } > + part_info->use_default_num_partitions= false; > + part_info->use_default_num_subpartitions= false; > + part_info->num_parts= 1; > + part_info->num_subparts= table->part_info->num_subparts; > + part_info->subpart_type= table->part_info->subpart_type; > + if (unlikely(part_info->vers_init_info(thd))) > + { > + my_error(ER_OUT_OF_RESOURCES, MYF(0)); > + goto exit; > + } > + /* Choose first non-occupied name suffix */ > + uint32 suffix= table->part_info->num_parts - 1; > + DBUG_ASSERT(suffix > 0); > + char part_name[MAX_PART_NAME_SIZE + 1]; > + if (make_partition_name(part_name, suffix)) > + { > +vers_make_name_err: > + push_warning_printf(thd, Sql_condition::WARN_LEVEL_WARN, > + WARN_VERS_HIST_PART_ERROR, > + "Auto-increment history partition: " > + "name generation failed for suffix %d", suffix); > + my_error(WARN_VERS_HIST_PART_ERROR, MYF(ME_WARNING), > + table->s->db.str, table->s->table_name.str, 0); > + goto exit; > + } > + List_iterator_fast<partition_element> it(table->part_info->partitions); > + partition_element *el; > + while ((el= it++)) > + { > + if (0 == my_strcasecmp(&my_charset_latin1, el->partition_name, part_name)) > + { > + if (make_partition_name(part_name, ++suffix)) > + goto vers_make_name_err; > + it.rewind(); > + } > + } > > - while ((next= it++) != vers_info->now_part) > + // NB: set_ok_status() requires DA_EMPTY > + thd->get_stmt_da()->reset_diagnostics_area(); > + > + thd->work_part_info= part_info; > + if (part_info->set_up_defaults_for_partitioning(thd, table->file, > + NULL, suffix + 1)) > { > - vers_info->hist_part= next; > - if (next->range_value > thd->query_start()) > - return; > + push_warning(thd, Sql_condition::WARN_LEVEL_WARN, > + WARN_VERS_HIST_PART_ERROR, > + "Auto-increment history partition: " > + "setting up defaults failed"); > + goto exit; > + } > + bool partition_changed= false; > + bool fast_alter_partition= false; > + if (prep_alter_part_table(thd, table, &alter_info, &create_info, > + &partition_changed, &fast_alter_partition)) > + { > + push_warning(thd, Sql_condition::WARN_LEVEL_WARN, WARN_VERS_HIST_PART_ERROR, > + "Auto-increment history partition: " > + "alter partitition prepare failed"); > + goto exit; > + } > + if (!fast_alter_partition) > + { > + push_warning(thd, Sql_condition::WARN_LEVEL_WARN, WARN_VERS_HIST_PART_ERROR, > + "Auto-increment history partition: " > + "fast alter partitition is not possible"); > + my_error(WARN_VERS_HIST_PART_ERROR, MYF(ME_WARNING), > + table->s->db.str, table->s->table_name.str, 0); > + goto exit; > + } > + DBUG_ASSERT(partition_changed); > + if (mysql_prepare_alter_table(thd, table, &create_info, &alter_info, > + &alter_ctx)) > + { > + push_warning(thd, Sql_condition::WARN_LEVEL_WARN, WARN_VERS_HIST_PART_ERROR, > + "Auto-increment history partition: " > + "alter prepare failed"); > + goto exit; > + } > + > + // Forge query string for rpl logging > + if (!thd->lex->no_write_to_binlog) > + { > + query.set(STRING_WITH_LEN("ALTER TABLE `"), &my_charset_latin1); > + > + if (query.append(table->s->db) || > + query.append(STRING_WITH_LEN("`.`")) || > + query.append(table->s->table_name) || > + query.append("` ADD PARTITION (PARTITION `") || > + query.append(part_name) || > + query.append("` HISTORY) AUTO_INCREMENT")) > + { > + my_error(ER_OUT_OF_RESOURCES, MYF(ME_ERROR_LOG)); > + goto exit; > + } > + CSET_STRING qs(query.c_ptr(), query.length(), &my_charset_latin1); > + thd->set_query(qs); > + } > + > + if (fast_alter_partition_table(thd, table, &alter_info, &create_info, > + tl, &table->s->db, &table->s->table_name)) > + { > + push_warning(thd, Sql_condition::WARN_LEVEL_WARN, WARN_VERS_HIST_PART_ERROR, > + "Auto-increment history partition: " > + "alter partition table failed"); > + my_error(WARN_VERS_HIST_PART_ERROR, MYF(ME_WARNING), > + tl->db.str, tl->table_name.str, 0); > } > - my_error(WARN_VERS_PART_FULL, MYF(ME_WARNING|ME_ERROR_LOG), > - table->s->db.str, table->s->table_name.str, > - vers_info->hist_part->partition_name, "INTERVAL"); > } > + > + if (!thd->transaction.stmt.is_empty()) > + trans_commit_stmt(thd); > + > +exit: > + // If we failed with error allow non-processed tables to be processed next time > + if (tl) > + while ((tl= tl->next_global)) > + tl->table->s->vers_auto_part= false; > + close_thread_tables(thd); > +open_err: > + thd->work_part_info= save_part_info; > + thd->m_reprepare_observer= save_reprepare_observer; > + thd->lex->restore_backup_query_tables_list(&save_query_tables); > + thd->in_sub_stmt&= ~SUB_STMT_AUTO_HIST; > + if (!new_stmt_da.is_warning_info_empty()) > + save_stmt_da->copy_sql_conditions_from_wi(thd, new_stmt_da.get_warning_info()); > + thd->set_stmt_da(save_stmt_da); > + thd->lex->no_write_to_binlog= save_no_write_to_binlog; > + thd->set_query(save_query); > } > > > diff --git a/sql/partition_info.h b/sql/partition_info.h > index eb8e53a381a..d02eecea073 100644 > --- a/sql/partition_info.h > +++ b/sql/partition_info.h > @@ -34,10 +34,19 @@ typedef bool (*check_constants_func)(THD *thd, partition_info *part_info); > > struct st_ddl_log_memory_entry; > > + > +/* Auto-create history partition configuration */ > +static const uint VERS_MIN_EMPTY= 1; No, this doesn't work. One can easily set @@timestamp to do a multi-hour jump and no fixed value of VERS_MIN_EMPTY will help. You need to add partitions before the statement, as I wrote in my previous review. > +static const uint VERS_MIN_INTERVAL= 3600; // seconds > +static const uint VERS_MIN_LIMIT= 1000; > +static const uint VERS_ERROR_TIMEOUT= 300; // seconds > + > + > struct Vers_part_info : public Sql_alloc > { > Vers_part_info() : > limit(0), > + auto_inc(false), > now_part(NULL), > hist_part(NULL) > { > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index 7cc1faea79b..4be3342e78e 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -7521,14 +7533,19 @@ add_partition_rule: > > add_part_extra: > /* empty */ > - | '(' part_def_list ')' > + | '(' part_def_list ')' opt_vers_auto_inc > { > LEX *lex= Lex; > lex->part_info->num_parts= lex->part_info->partitions.elements; > + if ($4) > + lex->alter_info.partition_flags|= ALTER_PARTITION_AUTO_HIST; > } > - | PARTITIONS_SYM real_ulong_num > + | PARTITIONS_SYM real_ulong_num opt_vers_auto_inc > { > - Lex->part_info->num_parts= $2; > + LEX *lex= Lex; ^^^ this pattern is obsolete for, like, 10 years. It does no harm, but adds no value either. > + lex->part_info->num_parts= $2; > + if ($3) > + lex->alter_info.partition_flags|= ALTER_PARTITION_AUTO_HIST; > } > ; > > diff --git a/storage/mroonga/mrn_table.cpp b/storage/mroonga/mrn_table.cpp > index b10668cfcce..6458402f572 100644 > --- a/storage/mroonga/mrn_table.cpp > +++ b/storage/mroonga/mrn_table.cpp > @@ -932,7 +932,7 @@ MRN_SHARE *mrn_get_share(const char *table_name, TABLE *table, int *error) > share->wrap_key_info = NULL; > share->wrap_primary_key = MAX_KEY; > } > - memcpy(wrap_table_share, table->s, sizeof(*wrap_table_share)); > + memcpy((void *)wrap_table_share, (void *)table->s, sizeof(*wrap_table_share)); why is that? > mrn_init_sql_alloc(current_thd, &(wrap_table_share->mem_root)); > wrap_table_share->keys = share->wrap_keys; > wrap_table_share->key_info = share->wrap_key_info; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 3

Re: [Maria-developers] d4d332d196d: Added error message to --die in mtr
by Sergei Golubchik 13 Apr '20

13 Apr '20

Hi, Michael! On Apr 13, Michael Widenius wrote: > revision-id: d4d332d196d (mariadb-10.5.2-119-gd4d332d196d) > parent(s): c7ab676192d > author: Michael Widenius <monty(a)mariadb.com> > committer: Michael Widenius <monty(a)mariadb.com> > timestamp: 2020-04-09 01:37:01 +0300 > message: > > Added error message to --die in mtr > > --- > client/mysqltest.cc | 3 ++- > 1 file changed, 2 insertions(+), 1 deletion(-) > > diff --git a/client/mysqltest.cc b/client/mysqltest.cc > index 3e08d3f62d7..f593d8e949e 100644 > --- a/client/mysqltest.cc > +++ b/client/mysqltest.cc > @@ -9748,7 +9748,8 @@ int main(int argc, char **argv) > break; > case Q_DIE: > /* Abort test with error code and error message */ > - die("%s", command->first_argument); > + die("%s", command->first_argument[0] ? command->first_argument : > + "Explicite --die command executed"); typo. "Explicit" > break; > case Q_EXIT: > /* Stop processing any more commands */ Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] MDEV-17399 Add support for JSON_TABLE.
by Sergey Petrunia 12 Apr '20

12 Apr '20

Hi Alexey, Please find my review input below. There is one big issue and a number of smaller ones. > commit 654fdfee33e3eafe3b7f25d7e213717c22ea1e18 > Author: Alexey Botchkov <holyfoot(a)askmonty.org> > Date: Mon Mar 30 01:00:28 2020 +0400 > > MDEV-17399 Add support for JSON_TABLE. > > Syntax for JSON_TABLE added. > The ha_json_table handler added. Executing the JSON_TABLE we > create the temporary table of the ha_json_table, add dependencies > of other tables and sometimes conditions to WHERE. > == The big issue == I think some design choices of this patch are questionable: The temporary table has a unique key. What is it for? The key is defined over the field holding the JSON text. What if the JSON text gets bigger than MAX_KEY_LEN? Then, there is an Item_func_eq(json_text, temp_table_field), which is set to be "always equal" with set_always_equal(). This looks like a hack. EXPLAIN shows that JSON_TABLE uses a kind of ref access which I think is counter-intuitive. What is the goal of all this? The JSON_TABLE table is "laterally dependent" on the tables that are referred from its arguments. For the optimizer, this means: 1. The JSON_TABLE table must be put after its dependencies (this is similar to outer joins) 2. The JSON_TABLE can only be read when there are "current records" available for each of the dependencies (an example when they are not available: join buffering. There are many records in the buffer, which one we should produce matches for? For outer joins, join buffering code has a complex system to track this). The above "kludges" make an attempt to trick the optimizer into meeting requrirements #1 and #2 but I think this is approach is hack-ish and has holes. It would be much better if the optimizer was explicitly aware that a table is "laterally dependent" on other tables. "table dependencies" are already there, so the property #1 is already taken care of. We still need to take care of #2 which means disable join buffering in such cases. I think this should be easy to do. == Small bits from trying out the patch == === FOR ORDINALITY doesn't seem to work === select * from json_table('[{"a": 1, "b": [11,111]}, {"a": 2, "b": [22,222]}]', '$[*]' COLUMNS(a for ordinality) ) as tt; +------+ | a | +------+ | | | | +------+ 2 rows in set (0.001 sec) === NULL ON ERROR doesn't seem to work === select * from json_table('[{"a": 1, "b": [11,111]}, {"a": "bbbb", "b": [22,222]}]', '$[*]' COLUMNS( a DECIMAL(6,3) PATH '$.a' NULL ON ERROR)) as tt; +-------+ | a | +-------+ | 1.000 | | 0.000 | +-------+ I would expect the second row to have NULL, not 0. === Lack of test coverage === Other parts, like NESTED PATH, also have no test coverage at all. I think this must be addressed before this is done. === No error on Invalid Json input === If I pass invalid JSON, I get no warning or error or anything: MariaDB [test]> select * from json_table('[{"a": 1, invalid-json', '$[*]' COLUMNS( a INT PATH '$.a')) as tt; +------+ | a | +------+ | 1 | +------+ 1 row in set (0.001 sec) === Recursive ON-rule in the grammar === The json_opt_on_empty_or_error rule in sql_yacc.yy is recursive and causes the following to be accepted: select * from json_table( '[{"a": 1, "b": [11,111]}, {"a": 2, "b": [22,222]}]', '$[*]' COLUMNS(a INT PATH '$.a' NULL ON EMPTY NULL ON EMPTY NULL ON EMPTY) ) as tt; Is this intentional? === No error on invalid LATERAL dependency === create table t1 (item_name varchar(32), item_props varchar(1024)); insert into t1 values ('Laptop', '{"color": "black", "price": 1000}'); insert into t1 values ('Jeans', '{"color": "blue", "price": 50}'); MariaDB [test]> select * from t1 right join json_table(t1.item_props,'$' columns( color varchar(100) path '$.color')) as T on 1; Empty set (0.000 sec) The above query cannot be executed as left join execution requires that T is computed before 1, but t is dependent on t1. We dont get an error for this though. MySQL produces this: ERROR 3668 (HY000): INNER or LEFT JOIN must be used for LATERAL references made by 'T' with left join, there's a demo of how the trickery with the optimizer was successful (and I think one can construct other examples of this): MariaDB [test]> select * from t1 left join json_table(t1.item_props,'$' columns( color varchar(100) path '$.color')) as T on 1; +-----------+-----------------------------------+-------+ | item_name | item_props | color | +-----------+-----------------------------------+-------+ | Laptop | {"color": "black", "price": 1000} | blue | | Jeans | {"color": "blue", "price": 50} | blue | +-----------+-----------------------------------+-------+ 2 rows in set (0.002 sec) MariaDB [test]> explain select * from t1 left join json_table(t1.item_props,'$' columns( color varchar(100) path '$.color')) as T on 1; +------+-------------+-------+------+---------------+------+---------+------+------+-------------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +------+-------------+-------+------+---------------+------+---------+------+------+-------------------------------------------------+ | 1 | SIMPLE | t1 | ALL | NULL | NULL | NULL | NULL | 2 | | | 1 | SIMPLE | T | ALL | NULL | NULL | NULL | NULL | 40 | Using where; Using join buffer (flat, BNL join) | +------+-------------+-------+------+---------------+------+---------+------+------+-------------------------------------------------+ 2 rows in set (0.001 sec) === AS is required with table alias === sql_yacc.yy has this: > +table_function: > + JSON_TABLE_SYM '(' expr ',' TEXT_STRING_sys > + { ... > + } > + json_table_columns_clause ')' AS ident_table_alias Is the 'AS' really required? MySQL-8 doesn't seem to require it. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 1

Re: [Maria-developers] [Commits] c1394ab6b5c: MDEV-22191: Range access is not picked when index_merge_sort_union is turned off
by Sergey Petrunia 09 Apr '20

09 Apr '20

Hi Varun, On Wed, Apr 08, 2020 at 11:48:28PM +0530, Varun wrote: > revision-id: c1394ab6b5c0830ec09f6afdae11fa82bae1a123 (mariadb-5.5.67-9-gc1394ab6b5c) > parent(s): 64b70b09e6ac253b7915f6120ade5e69fa750b18 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2020-04-08 23:47:03 +0530 > message: > > MDEV-22191: Range access is not picked when index_merge_sort_union is turned off > > When index_merge_sort_union is turned off only ror scans were considered for range > scans, which is wrong. > To fix the problem ensure both ror scans and non ror scans are considered for range > access > > --- > mysql-test/r/range.result | 19 +++++++++++++++++++ > mysql-test/r/range_mrr_icp.result | 19 +++++++++++++++++++ > mysql-test/t/range.test | 14 ++++++++++++++ > sql/opt_range.cc | 16 ++++++++++------ > 4 files changed, 62 insertions(+), 6 deletions(-) ... > --- a/sql/opt_range.cc > +++ b/sql/opt_range.cc > @@ -949,7 +949,8 @@ QUICK_RANGE_SELECT *get_quick_select(PARAM *param,uint index, > static TRP_RANGE *get_key_scans_params(PARAM *param, SEL_TREE *tree, > bool index_read_must_be_used, > bool update_tbl_stats, > - double read_time); > + double read_time, > + bool ror_scans_required); > static > TRP_INDEX_INTERSECT *get_best_index_intersect(PARAM *param, SEL_TREE *tree, > double read_time); > @@ -3146,7 +3147,7 @@ int SQL_SELECT::test_quick_select(THD *thd, key_map keys_to_use, > > /* Get best 'range' plan and prepare data for making other plans */ > if ((range_trp= get_key_scans_params(&param, tree, FALSE, TRUE, > - best_read_time))) > + best_read_time, FALSE))) > { > best_trp= range_trp; > best_read_time= best_trp->read_cost; > @@ -4708,7 +4709,8 @@ TABLE_READ_PLAN *get_best_disjunct_quick(PARAM *param, SEL_IMERGE *imerge, > { > DBUG_EXECUTE("info", print_sel_tree(param, *ptree, &(*ptree)->keys_map, > "tree in SEL_IMERGE");); > - if (!(*cur_child= get_key_scans_params(param, *ptree, TRUE, FALSE, read_time))) > + if (!(*cur_child= get_key_scans_params(param, *ptree, TRUE, FALSE, > + read_time, TRUE))) > { > /* > One of index scans in this index_merge is more expensive than entire > @@ -5030,7 +5032,7 @@ TABLE_READ_PLAN *merge_same_index_scans(PARAM *param, SEL_IMERGE *imerge, > index merge retrievals are not well calibrated > */ > trp= get_key_scans_params(param, *imerge->trees, FALSE, TRUE, > - read_time); > + read_time, TRUE); > } As far as I understand, this call constructs range scan, not a portion of index_merge scan. So, it is not correct to require a ROR scan here. > > DBUG_RETURN(trp); > @@ -6747,6 +6749,7 @@ TRP_ROR_INTERSECT *get_best_covering_ror_intersect(PARAM *param, > index_read_must_be_used if TRUE, assume 'index only' option will be set > (except for clustered PK indexes) > read_time don't create read plans with cost > read_time. > + ror_scans_required set to TRUE for index merge > RETURN > Best range read plan > NULL if no plan found or error occurred > @@ -6755,7 +6758,8 @@ TRP_ROR_INTERSECT *get_best_covering_ror_intersect(PARAM *param, > static TRP_RANGE *get_key_scans_params(PARAM *param, SEL_TREE *tree, > bool index_read_must_be_used, > bool update_tbl_stats, > - double read_time) > + double read_time, > + bool ror_scans_required) > { > uint idx; > SEL_ARG **key,**end, **key_to_read= NULL; > @@ -6802,7 +6806,7 @@ static TRP_RANGE *get_key_scans_params(PARAM *param, SEL_TREE *tree, > update_tbl_stats, &mrr_flags, > &buf_size, &cost); > > - if (!param->is_ror_scan && > + if (ror_scans_required && !param->is_ror_scan && > !optimizer_flag(param->thd, OPTIMIZER_SWITCH_INDEX_MERGE_SORT_UNION)) So, the meaning if "ror_scans_required" parameter is actually: " require ror scans if index_merge optimizer settings are such that it cannot use ROR-scans" This is very complicated and non-orthogonal. How about moving the "!optimizer_flag(param->thd, OPTIMIZER_SWITCH_INDEX_MERGE_SORT_UNION)" check up into get_best_disjunct_quick()? That is, get_best_disjunct_quick() will pass ror_scans_required=true iff OPTIMIZER_SWITCH_INDEX_MERGE_SORT_UNION is not set. This way, index_merge switches will be only checked in index_merge code and ror_scans_required will mean what its name says. > { > /* The scan is not a ROR-scan, just skip it */ BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] 383e1a5b71e: MENT-703: Disable rpl.rpl_skip_replication, rpl.rpl_set_statement_default_master, rpl.rpl_extra_col_master_myisam, rpl.rpl_slave_load_tmpdir_not_exist tests in ES 10.2 and above
by sujatha 08 Apr '20

08 Apr '20

revision-id: 383e1a5b71e164f1c1b17a085ba90a8c30cbdf46 (mariadb-10.2.24-716-g383e1a5b71e) parent(s): 2969d0702d56405d1aec8c16a272ac85fef7bd61 author: Sujatha committer: Sujatha timestamp: 2020-04-08 13:31:33 +0530 message: MENT-703: Disable rpl.rpl_skip_replication, rpl.rpl_set_statement_default_master, rpl.rpl_extra_col_master_myisam, rpl.rpl_slave_load_tmpdir_not_exist tests in ES 10.2 and above Problem: ======= Disable following tests in ES 10.2 and above, till MDEV-13258 and MDEV-14203 are fixed. rpl.rpl_skip_replication, rpl.rpl_set_statement_default_master, rpl.rpl_extra_col_master_myisam, rpl.rpl_slave_load_tmpdir_not_exist --- mysql-test/suite/rpl/disabled.def | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/mysql-test/suite/rpl/disabled.def b/mysql-test/suite/rpl/disabled.def index bdefb1660bd..951f8a6aa0f 100644 --- a/mysql-test/suite/rpl/disabled.def +++ b/mysql-test/suite/rpl/disabled.def @@ -16,3 +16,7 @@ rpl_partition_archive : MDEV-5077 2013-09-27 svoj Cannot exchange partition rpl_row_binlog_max_cache_size : MDEV-11092 rpl_row_index_choice : MDEV-11666 rpl_slave_grp_exec: MDEV-10514 +rpl_skip_replication : MDEV-13258 +rpl_set_statement_default_master : MDEV-13258 +rpl_extra_col_master_myisam : MDEV-14203 +rpl_slave_load_tmpdir_not_exist : MDEV-14203

1 0

Re: [Maria-developers] 7665323d02a: MENT-202 Implement system variable that makes slow master shutdown the default behavior
by Sergei Golubchik 07 Apr '20

07 Apr '20

Hi, Andrei! ok to push. one comment below. On Apr 07, Andrei Elkin wrote: > revision-id: 7665323d02a (mariadb-10.3.22-194-g7665323d02a) > parent(s): 04e38549af8 > author: Andrei Elkin <andrei.elkin(a)mariadb.com> > committer: Andrei Elkin <andrei.elkin(a)mariadb.com> > timestamp: 2020-04-07 14:25:25 +0300 > message: > > MENT-202 Implement system variable that makes slow master shutdown the default behavior > > @@global.shutdown_wait_slaves is introduced to hold a desirable > shutdown mode that the SHUTDOWN sql command runs with implicitly > when invoked without any option. > > diff --git a/sql/sys_vars.cc b/sql/sys_vars.cc > index 8bb9f86c8b6..2b23edc7000 100644 > --- a/sql/sys_vars.cc > +++ b/sql/sys_vars.cc > @@ -4270,6 +4270,12 @@ static Sys_var_ulong Sys_profiling_history_size( > VALID_RANGE(0, 100), DEFAULT(15), BLOCK_SIZE(1)); > #endif > > +static Sys_var_mybool Sys_shutdown_wait_for_slaves( > + "shutdown_wait_for_slaves", > + "when ON, SHUTDOWN command runs with implicit WAIT FOR ALL SLAVES option.", > + GLOBAL_VAR(opt_shutdown_wait_for_slaves), CMD_LINE(OPT_ARG), > + DEFAULT(FALSE), NO_MUTEX_GUARD, NOT_IN_BINLOG, ON_CHECK(0), ON_UPDATE(0)); You don't need to write all this tail of the default behavior. The last line could be just DEFAULT(FALSE)); Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] 2888a225214: MDEV-21117: --tc-heuristic-recover=rollback is not replication safe
by sujatha 07 Apr '20

07 Apr '20

revision-id: 2888a22521475f734b28ed65d199cfe2a754aed7 (mariadb-10.1.43-100-g2888a225214) parent(s): 5720db2b43491e5aec9265bce9637e00c72fa0aa author: Sujatha committer: Sujatha timestamp: 2020-04-07 19:21:45 +0530 message: MDEV-21117: --tc-heuristic-recover=rollback is not replication safe Problem: ======= When run after master server crash --tc-heuristic-recover=rollback produces inconsistent server state with binlog still containing transactions that were rolled back by the option. Such way recovered server may not be used for replication. Fix: === During "--tc-heuristic-recover=ROLLBACK", query the storage engine to get binlog file name and position corresponding to the last committed transaction. This marks the consistent binlog state. If last_commit_file is not set then checkpoint binary log file is considered as starting point. Look for first transactional event beyond the consistent binlog state. This will be the starting point for heuristic rollback. Consider this event specific starting point as binlog truncation position. Now traverse the rest of binlogs beyond this point. During this traversal check for the presence of DDL or non transactional operations, as they cannot be safely rolled back. If such events are found the truncation will not happen it will return an error. If only transactional events are found beyond the binlog truncation position it is safe to truncate binlog. The log gets truncated to the identified position and the GTID state is adjusted accordingly. If there are more binary logs beyond the being truncated file they are all removed. --- .../r/binlog_heuristic_rollback_active_log.result | 18 + .../binlog/r/binlog_truncate_multi_log.result | 21 + .../suite/binlog/r/binlog_truncate_none.result | 42 ++ .../t/binlog_heuristic_rollback_active_log.test | 75 ++++ .../suite/binlog/t/binlog_truncate_multi_log.test | 81 ++++ .../suite/binlog/t/binlog_truncate_none.test | 130 ++++++ .../suite/rpl/r/rpl_heuristic_fail_over.result | 53 +++ .../suite/rpl/t/rpl_heuristic_fail_over.test | 160 +++++++ sql/log.cc | 479 ++++++++++++++++++++- sql/log.h | 7 +- storage/innobase/handler/ha_innodb.cc | 3 + storage/xtradb/handler/ha_innodb.cc | 3 + 12 files changed, 1068 insertions(+), 4 deletions(-) diff --git a/mysql-test/suite/binlog/r/binlog_heuristic_rollback_active_log.result b/mysql-test/suite/binlog/r/binlog_heuristic_rollback_active_log.result new file mode 100644 index 00000000000..eff95d05aac --- /dev/null +++ b/mysql-test/suite/binlog/r/binlog_heuristic_rollback_active_log.result @@ -0,0 +1,18 @@ +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); +RESET MASTER; +CREATE TABLE t ( f INT ) ENGINE=INNODB; +INSERT INTO t VALUES (10); +SET DEBUG_SYNC= "commit_before_update_end_pos SIGNAL con1_ready WAIT_FOR con1_go"; +INSERT INTO t VALUES (20); +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; +# Kill the server +"One row should be present in table 't'" +SELECT COUNT(*) FROM t; +COUNT(*) +1 +"gtid_binlog_state should be 0-1-2 +SELECT @@GLOBAL.gtid_binlog_state; +@@GLOBAL.gtid_binlog_state +0-1-2 +DROP TABLE t; diff --git a/mysql-test/suite/binlog/r/binlog_truncate_multi_log.result b/mysql-test/suite/binlog/r/binlog_truncate_multi_log.result new file mode 100644 index 00000000000..9796a480cca --- /dev/null +++ b/mysql-test/suite/binlog/r/binlog_truncate_multi_log.result @@ -0,0 +1,21 @@ +SET @old_max_binlog_size= @@global.max_binlog_size; +SET GLOBAL max_binlog_size= 4096; +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); +RESET MASTER; +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +SET DEBUG_SYNC= "commit_before_update_end_pos SIGNAL con1_ready WAIT_FOR con1_go"; +INSERT INTO t1 VALUES (2, REPEAT("x", 4100)); +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; +# Kill the server +"Zero rows shoule be present in table" +SELECT COUNT(*) FROM t1; +COUNT(*) +0 +SELECT @@GLOBAL.gtid_current_pos; +@@GLOBAL.gtid_current_pos +0-1-1 +DROP TABLE t1; +SELECT @@GLOBAL.gtid_binlog_state; +@@GLOBAL.gtid_binlog_state +0-1-2 diff --git a/mysql-test/suite/binlog/r/binlog_truncate_none.result b/mysql-test/suite/binlog/r/binlog_truncate_none.result new file mode 100644 index 00000000000..bae87985208 --- /dev/null +++ b/mysql-test/suite/binlog/r/binlog_truncate_none.result @@ -0,0 +1,42 @@ +SET @old_max_binlog_size= @@global.max_binlog_size; +SET GLOBAL max_binlog_size= 4096; +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); +RESET MASTER; +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +INSERT INTO t1 VALUES (1, REPEAT("x", 4100)); +CREATE TABLE t2 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +show binary logs; +Log_name File_size +master-bin.000001 # +master-bin.000002 # +# Kill the server +"Zero records should be there." +SELECT COUNT(*) FROM t1; +COUNT(*) +1 +show binary logs; +Log_name File_size +master-bin.000001 # +master-bin.000002 # +master-bin.000003 # +master-bin.000004 # +DROP TABLE t1,t2; +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); +RESET MASTER; +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +CREATE TABLE t2 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +show binary logs; +Log_name File_size +master-bin.000001 # +# Kill the server +"Zero records should be there." +SELECT COUNT(*) FROM t1; +COUNT(*) +0 +SHOW TABLES; +Tables_in_test +t1 +t2 +DROP TABLE t1,t2; diff --git a/mysql-test/suite/binlog/t/binlog_heuristic_rollback_active_log.test b/mysql-test/suite/binlog/t/binlog_heuristic_rollback_active_log.test new file mode 100644 index 00000000000..87af83a7a92 --- /dev/null +++ b/mysql-test/suite/binlog/t/binlog_heuristic_rollback_active_log.test @@ -0,0 +1,75 @@ +# ==== Purpose ==== +# +# Test verifies the truncation of single binary log file. +# +# ==== Implementation ==== +# +# Steps: +# 0 - Create table t1 and insert a row. +# 1 - Insert an another row such that it gets written to binlog but commit +# in engine fails as server crashed at this point. +# 2 - Restart server with --tc-heuristic-recover=ROLLBACK +# 3 - Upon server start 'master-bin.000001' will be truncated to contain +# only the first insert +# +# ==== References ==== +# +# MDEV-21117: --tc-heuristic-recover=rollback is not replication safe + + +--source include/have_innodb.inc +--source include/have_log_bin.inc +--source include/have_debug.inc +--source include/have_binlog_format_statement.inc + +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); + +connect(master,localhost,root,,); +connect(master1,localhost,root,,); + +--connection master +RESET MASTER; +CREATE TABLE t ( f INT ) ENGINE=INNODB; +INSERT INTO t VALUES (10); + +--connection master1 +# Hold insert after write to binlog and before "run_commit_ordered" in engine +SET DEBUG_SYNC= "commit_before_update_end_pos SIGNAL con1_ready WAIT_FOR con1_go"; +send INSERT INTO t VALUES (20); + +--connection master +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; + +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +wait +EOF + +--source include/kill_mysqld.inc +--source include/wait_until_disconnected.inc +# +# Server restart +# +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: --tc-heuristic-recover=ROLLBACK +EOF +--source include/wait_until_disconnected.inc + +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: +EOF + +connection default; +--enable_reconnect +--source include/wait_until_connected_again.inc + +--connection master +--enable_reconnect +--source include/wait_until_connected_again.inc + +--echo "One row should be present in table 't'" +SELECT COUNT(*) FROM t; + +--echo "gtid_binlog_state should be 0-1-2 +SELECT @@GLOBAL.gtid_binlog_state; +DROP TABLE t; diff --git a/mysql-test/suite/binlog/t/binlog_truncate_multi_log.test b/mysql-test/suite/binlog/t/binlog_truncate_multi_log.test new file mode 100644 index 00000000000..976b987d3bc --- /dev/null +++ b/mysql-test/suite/binlog/t/binlog_truncate_multi_log.test @@ -0,0 +1,81 @@ +# ==== Purpose ==== +# +# Test verifies truncation of multiple binary logs. +# +# ==== Implementation ==== +# +# Steps: +# 0 - Set max_binlog_size= 4096. Create a table do an insert such that the +# max_binlog_size is reached and binary log gets rotated. +# 1 - Using debug simulation make the server crash at a point where the DML +# transaction is written to binary log but not committed in engine. +# 2 - At the time of crash two binary logs will be there master-bin.0000001 +# and master-bin.000002. +# 3 - Restart server with --tc-heuristic-recover=ROLLBACK +# 4 - Since the prepared DML in master-bin.000001 is rolled back the first +# binlog will be truncated prior to the DML and master-bin.000002 will be +# removed. +# +# ==== References ==== +# +# MDEV-21117: --tc-heuristic-recover=rollback is not replication safe + + +--source include/have_innodb.inc +--source include/have_log_bin.inc +--source include/have_debug.inc +--source include/have_debug_sync.inc +--source include/have_binlog_format_row.inc + +SET @old_max_binlog_size= @@global.max_binlog_size; +SET GLOBAL max_binlog_size= 4096; + +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); + +RESET MASTER; + +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; + +connect(master1,localhost,root,,); +connect(master2,localhost,root,,); + +--connection master1 +# Hold insert after write to binlog and before "run_commit_ordered" in engine +SET DEBUG_SYNC= "commit_before_update_end_pos SIGNAL con1_ready WAIT_FOR con1_go"; +send INSERT INTO t1 VALUES (2, REPEAT("x", 4100)); + +--connection master2 +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; + +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +wait +EOF + +--source include/kill_mysqld.inc +--source include/wait_until_disconnected.inc + +# +# Server restart +# +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: --tc-heuristic-recover=ROLLBACK --debug-dbug=d,simulate_innodb_forget_commit_pos +EOF +--source include/wait_until_disconnected.inc + +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: +EOF + +connection default; +--enable_reconnect +--source include/wait_until_connected_again.inc + +--echo "Zero rows shoule be present in table" +SELECT COUNT(*) FROM t1; + +SELECT @@GLOBAL.gtid_current_pos; + +DROP TABLE t1; +SELECT @@GLOBAL.gtid_binlog_state; + diff --git a/mysql-test/suite/binlog/t/binlog_truncate_none.test b/mysql-test/suite/binlog/t/binlog_truncate_none.test new file mode 100644 index 00000000000..cc3fd4f0b37 --- /dev/null +++ b/mysql-test/suite/binlog/t/binlog_truncate_none.test @@ -0,0 +1,130 @@ +# ==== Purpose ==== +# +# Test case verifies no binlog truncation happens when non transactional +# events are found in binlog after the last committed transaction. +# +# ==== Implementation ==== +# +# Steps: +# 0 - Set max_binlog_size= 4096. Create a table and do an insert such that +# the max_binlog_size is reached and binary log gets rotated. +# 1 - Create a table in newly created binary log and crash the server +# 2 - Restart server with --tc-heuristic-recover=ROLLBACK +# 3 - Recovery code will get the last committed DML specific postion and +# will try to check if binlog can be truncated upto this position. +# Since a DDL is present beyond this no truncation will happen. +# ==== References ==== +# +# MDEV-21117: --tc-heuristic-recover=rollback is not replication safe + + +--source include/have_innodb.inc +--source include/have_log_bin.inc +--source include/have_debug.inc +--source include/have_binlog_format_row.inc + +SET @old_max_binlog_size= @@global.max_binlog_size; +SET GLOBAL max_binlog_size= 4096; + +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); + +RESET MASTER; + +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +INSERT INTO t1 VALUES (1, REPEAT("x", 4100)); +CREATE TABLE t2 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +--source include/show_binary_logs.inc + +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +wait +EOF + +--source include/kill_mysqld.inc +--source include/wait_until_disconnected.inc + +# +# Server restart +# +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: --tc-heuristic-recover=ROLLBACK +EOF +--source include/wait_until_disconnected.inc + +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: +EOF + +connection default; +--enable_reconnect +--source include/wait_until_connected_again.inc + +--echo "Zero records should be there." +SELECT COUNT(*) FROM t1; +--source include/show_binary_logs.inc +DROP TABLE t1,t2; + +# ==== Purpose ==== +# +# Test case verifies no binlog truncation happens when only DDLs are present in +# the binary log. Since none of the DMLs are performed in storage engine, +# Engine will not have last committed transaction file name or position. +# Truncation code should return success. +# +# ==== Implementation ==== +# +# Steps: +# 0 - Create two table t1, t2 +# 1 - Kill and restart server with --tc-heuristic-recover=ROLLBACK +# 2 - Only DDL statements are present in the binary log. Since +# no DML was performed engine will not have last commited transaction +# specific binary log name and position. Since no transactional events +# are found, truncation code should simply return. +# +# ==== References ==== +# +# MDEV-21117: --tc-heuristic-recover=rollback is not replication safe + + +--source include/have_innodb.inc +--source include/have_log_bin.inc +--source include/have_debug.inc +--source include/have_binlog_format_row.inc + +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); + +RESET MASTER; + +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +CREATE TABLE t2 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +--source include/show_binary_logs.inc + +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +wait +EOF + +--source include/kill_mysqld.inc +--source include/wait_until_disconnected.inc + +# +# Server restart +# +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: --tc-heuristic-recover=ROLLBACK --debug-dbug=d,simulate_innodb_forget_commit_pos +EOF +--source include/wait_until_disconnected.inc + +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: +EOF + +connection default; +--enable_reconnect +--source include/wait_until_connected_again.inc + +--echo "Zero records should be there." +SELECT COUNT(*) FROM t1; +SHOW TABLES; +DROP TABLE t1,t2; + diff --git a/mysql-test/suite/rpl/r/rpl_heuristic_fail_over.result b/mysql-test/suite/rpl/r/rpl_heuristic_fail_over.result new file mode 100644 index 00000000000..8391245ba27 --- /dev/null +++ b/mysql-test/suite/rpl/r/rpl_heuristic_fail_over.result @@ -0,0 +1,53 @@ +include/rpl_init.inc [topology=1->2] +include/stop_slave.inc +set global rpl_semi_sync_slave_enabled = 1; +CHANGE MASTER TO master_use_gtid= current_pos; +include/start_slave.inc +ALTER TABLE mysql.gtid_slave_pos ENGINE=InnoDB; +set global rpl_semi_sync_master_enabled = 1; +set global rpl_semi_sync_master_wait_point=AFTER_SYNC; +SET @old_max_binlog_size= @@global.max_binlog_size; +SET GLOBAL max_binlog_size= 4096; +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +INSERT INTO t1 VALUES (1, 'dummy1'); +SET DEBUG_SYNC= "commit_before_update_end_pos SIGNAL con1_ready WAIT_FOR con1_go"; +INSERT INTO t1 VALUES (2, REPEAT("x", 4100)); +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; +# Kill the server +include/stop_slave.inc +SELECT @@GLOBAL.gtid_current_pos; +@@GLOBAL.gtid_current_pos +0-1-5 +FOUND /Crashed binlog file \.\/master\-bin\.000001 size is [0-9]*, but truncated to */ in mysqld.1.err +set global rpl_semi_sync_master_enabled = 1; +set global rpl_semi_sync_master_wait_point=AFTER_SYNC; +CHANGE MASTER TO master_host='127.0.0.1', master_port=SERVER_MYPORT_2, master_user='root', master_use_gtid=CURRENT_POS; +set global rpl_semi_sync_slave_enabled = 1; +include/start_slave.inc +INSERT INTO t1 VALUES (3, 'dummy3'); +SELECT COUNT(*) FROM t1; +COUNT(*) +2 +SHOW VARIABLES LIKE 'gtid_current_pos'; +Variable_name Value +gtid_current_pos 0-2-6 +SHOW VARIABLES LIKE 'gtid_current_pos'; +Variable_name Value +gtid_current_pos 0-2-6 +DROP TABLE t1; +set global rpl_semi_sync_master_enabled = 0; +set global rpl_semi_sync_slave_enabled = 0; +set global rpl_semi_sync_master_wait_point=default; +set global rpl_semi_sync_master_enabled = 0; +set global rpl_semi_sync_slave_enabled = 0; +set global rpl_semi_sync_master_wait_point=default; +include/stop_slave.inc +RESET MASTER; +RESET SLAVE; +RESET MASTER; +RESET SLAVE; +CHANGE MASTER TO master_host='127.0.0.1', master_port=SERVER_MYPORT_1, master_user='root', master_use_gtid=no; +include/start_slave.inc +include/rpl_end.inc diff --git a/mysql-test/suite/rpl/t/rpl_heuristic_fail_over.test b/mysql-test/suite/rpl/t/rpl_heuristic_fail_over.test new file mode 100644 index 00000000000..5e9273de62d --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_heuristic_fail_over.test @@ -0,0 +1,160 @@ +# ==== Purpose ==== +# +# Test verifies replication failover scenario. +# +# ==== Implementation ==== +# +# Steps: +# 0 - Have two servers with id's 1 and 2. Enable semi-sync based +# replication. Have semi sync master wait point as 'after_sync'. +# 1 - Create a table and insert a row. While inserting second row simulate +# a server crash at once the transaction is written to binlog, flushed +# and synced but the binlog position is not updated. +# 2 - Restart the server using tc-heuristic-recover=ROLLBACK +# 3 - Post restart switch the crashed master to slave. Execute CHANGE MASTER +# TO command to connect to server id 2. +# 4 - Slave should be able to connect to master. +# 5 - Execute some DML on new master with server id 2. Ensure that it gets +# replicated to server id 1. +# 6 - Verify the "gtid_current_pos" for correctness. +# 7 - Clean up +# +# ==== References ==== +# +# MDEV-21117: --tc-heuristic-recover=rollback is not replication safe + + +--source include/have_semisync.inc +--source include/have_innodb.inc +--source include/have_log_bin.inc +--source include/have_debug.inc +--source include/have_debug_sync.inc +--source include/have_binlog_format_row.inc +--let $rpl_topology=1->2 +--source include/rpl_init.inc + +--connection server_2 +--source include/stop_slave.inc +set global rpl_semi_sync_slave_enabled = 1; +CHANGE MASTER TO master_use_gtid= current_pos; +--source include/start_slave.inc + + +--connection server_1 +ALTER TABLE mysql.gtid_slave_pos ENGINE=InnoDB; +set global rpl_semi_sync_master_enabled = 1; +set global rpl_semi_sync_master_wait_point=AFTER_SYNC; +SET @old_max_binlog_size= @@global.max_binlog_size; +SET GLOBAL max_binlog_size= 4096; + +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); + +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +INSERT INTO t1 VALUES (1, 'dummy1'); +--save_master_pos + +--connection server_2 +--sync_with_master + +--connection server_1 +connect(master1,localhost,root,,); +connect(master2,localhost,root,,); + +--connection master1 +# Hold insert after write to binlog and before "run_commit_ordered" in engine +SET DEBUG_SYNC= "commit_before_update_end_pos SIGNAL con1_ready WAIT_FOR con1_go"; +send INSERT INTO t1 VALUES (2, REPEAT("x", 4100)); + +--connection master2 +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +wait +EOF + +--source include/kill_mysqld.inc +--source include/wait_until_disconnected.inc + +--connection server_2 +--error 2003 +--source include/stop_slave.inc +SELECT @@GLOBAL.gtid_current_pos; + +--connection server_1 +# +# Server restart +# +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: --tc-heuristic-recover=ROLLBACK +EOF +--source include/wait_until_disconnected.inc + +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: +EOF + +connection default; +--enable_reconnect +--source include/wait_until_connected_again.inc + +--connection server_1 +--enable_reconnect +--source include/wait_until_connected_again.inc + +# Check error log for correct messages. +let $log_error_= `SELECT @@GLOBAL.log_error`; +if(!$log_error_) +{ + # MySQL Server on windows is started with --console and thus + # does not know the location of its .err log, use default location + let $log_error_ = $MYSQLTEST_VARDIR/log/mysqld.1.err; +} +--let SEARCH_FILE=$log_error_ +--let SEARCH_RANGE=-50000 +--let SEARCH_PATTERN=Crashed binlog file \.\/master\-bin\.000001 size is [0-9]*, but truncated to * +--source include/search_pattern_in_file.inc + +--connection server_2 +set global rpl_semi_sync_master_enabled = 1; +set global rpl_semi_sync_master_wait_point=AFTER_SYNC; + +--connection server_1 +--replace_result $SERVER_MYPORT_2 SERVER_MYPORT_2 +eval CHANGE MASTER TO master_host='127.0.0.1', master_port=$SERVER_MYPORT_2, master_user='root', master_use_gtid=CURRENT_POS; +set global rpl_semi_sync_slave_enabled = 1; +--source include/start_slave.inc + +--connection server_2 +INSERT INTO t1 VALUES (3, 'dummy3'); +--save_master_pos + +--connection server_1 +--sync_with_master +SELECT COUNT(*) FROM t1; +SHOW VARIABLES LIKE 'gtid_current_pos'; + +--connection server_2 +SHOW VARIABLES LIKE 'gtid_current_pos'; +DROP TABLE t1; +set global rpl_semi_sync_master_enabled = 0; +set global rpl_semi_sync_slave_enabled = 0; +set global rpl_semi_sync_master_wait_point=default; +--save_master_pos + +--connection server_1 +--sync_with_master +set global rpl_semi_sync_master_enabled = 0; +set global rpl_semi_sync_slave_enabled = 0; +set global rpl_semi_sync_master_wait_point=default; +--source include/stop_slave.inc +RESET MASTER; +RESET SLAVE; + +--connection server_2 +RESET MASTER; +RESET SLAVE; +--replace_result $SERVER_MYPORT_1 SERVER_MYPORT_1 +eval CHANGE MASTER TO master_host='127.0.0.1', master_port=$SERVER_MYPORT_1, master_user='root', master_use_gtid=no; +--source include/start_slave.inc + +--source include/rpl_end.inc diff --git a/sql/log.cc b/sql/log.cc index 0efef6d1e29..f383033e975 100644 --- a/sql/log.cc +++ b/sql/log.cc @@ -3164,6 +3164,7 @@ MYSQL_BIN_LOG::MYSQL_BIN_LOG(uint *sync_period) checksum_alg_reset(BINLOG_CHECKSUM_ALG_UNDEF), relay_log_checksum_alg(BINLOG_CHECKSUM_ALG_UNDEF), description_event_for_exec(0), description_event_for_queue(0), + last_commit_pos_offset(0), current_binlog_id(0) { /* @@ -3173,6 +3174,7 @@ MYSQL_BIN_LOG::MYSQL_BIN_LOG(uint *sync_period) before main(). */ index_file_name[0] = 0; + last_commit_pos_file[0]= 0; bzero((char*) &index_file, sizeof(index_file)); bzero((char*) &purge_index_file, sizeof(purge_index_file)); } @@ -7877,6 +7879,7 @@ MYSQL_BIN_LOG::trx_group_commit_leader(group_commit_entry *leader) first= false; } + DEBUG_SYNC(leader->thd, "commit_before_update_end_pos"); /* update binlog_end_pos so it can be read by dump thread * * note: must be _after_ the RUN_HOOK(after_flush) or else @@ -8964,7 +8967,7 @@ int TC_LOG_MMAP::open(const char *opt_name) { if (my_errno != ENOENT) goto err; - if (using_heuristic_recover()) + if (using_heuristic_recover(opt_name)) return 1; if ((fd= mysql_file_create(key_file_tclog, logname, CREATE_MODE, O_RDWR | O_CLOEXEC, MYF(MY_WME))) < 0) @@ -9497,14 +9500,42 @@ TC_LOG_MMAP tc_log_mmap; 1 heuristic recovery was performed */ -int TC_LOG::using_heuristic_recover() +int TC_LOG::using_heuristic_recover(const char* opt_name) { + LOG_INFO log_info; + int error; + if (!tc_heuristic_recover) return 0; sql_print_information("Heuristic crash recovery mode"); + if (ha_recover(0)) + { sql_print_error("Heuristic crash recovery failed"); + } + + /* + Check if TC log is referring to binary log not memory map. Ensure that + tc_heuristic_recover being ROLLBACK". If both match initiate binlog + truncation mechanism. + */ + if (!strcmp(opt_name,opt_bin_logname) && + tc_heuristic_recover == TC_HEURISTIC_RECOVER_ROLLBACK) + { + if ((error= mysql_bin_log.find_log_pos(&log_info, NullS, 1))) + { + if (error != LOG_INFO_EOF) + sql_print_error("find_log_pos() failed (error: %d)", error); + } + else + { + if ((error= heuristic_binlog_rollback())) + sql_print_error("Heuristic crash recovery failed to remove " + "rolled back trancations from binary log"); + } + } + sql_print_information("Please restart mysqld without --tc-heuristic-recover"); return 1; } @@ -9512,6 +9543,448 @@ int TC_LOG::using_heuristic_recover() /****** transaction coordinator log for 2pc - binlog() based solution ******/ #define TC_LOG_BINLOG MYSQL_BIN_LOG +/** + Truncates the current binlog to specified position. Removes the rest of binlogs + which are present after this binlog file. + + @param truncate_file Holds the binlog name to be truncated + @param truncate_pos Position within binlog from where it needs to + truncated. + + @retval true ok + @retval false error + +*/ +bool MYSQL_BIN_LOG::truncate_and_remove_binlogs(const char *truncate_file, + my_off_t truncate_pos) +{ + int error= 0; +#ifdef HAVE_REPLICATION + LOG_INFO log_info; + THD *thd= current_thd; + my_off_t index_file_offset=0; + File file= -1; + MY_STAT s; + my_off_t binlog_size; + + if ((error= open_purge_index_file(TRUE))) + { + sql_print_error("MYSQL_BIN_LOG::purge_logs failed to sync the index file."); + goto err; + } + + if ((error=find_log_pos(&log_info, truncate_file, 1))) + goto err; + + while (!(error= find_next_log(&log_info, 1))) + { + if (!index_file_offset) + index_file_offset= log_info.index_file_start_offset; + if((error= register_purge_index_entry(log_info.log_file_name))) + { + sql_print_error("MYSQL_BIN_LOG::purge_logs failed to copy %s to register file.", + log_info.log_file_name); + goto err; + } + } + + if (error != LOG_INFO_EOF) + { + sql_print_error("Error while looking for next binlog file to be truncated. " + "Error:%d", error); + goto err; + } + + if (!index_file_offset) + index_file_offset= log_info.index_file_start_offset; + + if ((error= sync_purge_index_file())) + { + sql_print_error("MYSQL_BIN_LOG::purge_logs failed to flush register file."); + goto err; + } + + DBUG_ASSERT(index_file_offset != 0); + // Trim index file + if (mysql_file_chsize(index_file.file, index_file_offset, '\n', MYF(MY_WME)) || + mysql_file_sync(index_file.file, MYF(MY_WME|MY_SYNC_FILESIZE))) + { + sql_print_error("Failed to trim binlog index file " + "when master server is recovering it."); + mysql_file_close(index_file.file, MYF(MY_WME)); + goto err; + } + + /* Reset data in old index cache */ + reinit_io_cache(&index_file, READ_CACHE, (my_off_t) 0, 0, 1); + + /* Read each entry from purge_index_file and delete the file. */ + if (is_inited_purge_index_file() && + (error= purge_index_entry(thd, NULL, TRUE))) + { + sql_print_error("MYSQL_BIN_LOG::purge_logs failed to process registered files" + " that would be purged."); + goto err; + } + + if (truncate_pos) + { + if ((file= mysql_file_open(key_file_binlog, truncate_file, + O_RDWR | O_BINARY, MYF(MY_WME))) < 0) + { + sql_print_error("Failed to open binlog file:%s for truncation.", + truncate_file); + error= 1; + goto err; + } + my_stat(truncate_file, &s, MYF(0)); + binlog_size= s.st_size; + + /* Change binlog file size to truncate_pos */ + if (mysql_file_chsize(file, truncate_pos, 0, MYF(MY_WME)) || + mysql_file_sync(file, MYF(MY_WME|MY_SYNC_FILESIZE))) + { + sql_print_error("Failed to trim the crashed binlog file:%s to size:%llu", + truncate_file, truncate_pos); + mysql_file_close(file, MYF(MY_WME)); + error= 1; + goto err; + } + else + { + sql_print_information("Crashed binlog file %s size is %llu, " + "but truncated to %llu.", + truncate_file, binlog_size, truncate_pos); + } + mysql_file_close(file, MYF(MY_WME)); + file= -1; + } + +err: + if (file >= 0) + mysql_file_close(file, MYF(MY_WME)); + close_purge_index_file(); +#endif + return error; +} + +/** + Returns the checkpoint binlog file name found in the lastest binlog file. + + @param checkpoint_file Holds the binlog checkpoint file name. + + @retval 0 ok + @retval 1 error + +*/ +int TC_LOG_BINLOG::get_binlog_checkpoint_file(char* checkpoint_file) +{ + Log_event *ev= NULL; + bool binlog_checkpoint_found= false; + LOG_INFO log_info; + const char *errmsg; + IO_CACHE log; + File file; + Format_description_log_event fdle(BINLOG_VERSION); + char log_name[FN_REFLEN]; + int error=1; + + if (!fdle.is_valid()) + return 1; + + if ((error= find_log_pos(&log_info, NullS, 1))) + { + sql_print_error("find_log_pos() failed (error: %d)", error); + return error; + } + + // Move to the last binary log. + do + { + strmake_buf(log_name, log_info.log_file_name); + } while (!(error= find_next_log(&log_info, 1))); + + if (error != LOG_INFO_EOF) + { + sql_print_error("find_log_pos() failed (error: %d)", error); + return error; + } + if ((file= open_binlog(&log, log_name, &errmsg)) < 0) + { + sql_print_error("%s", errmsg); + return error; + } + while (!binlog_checkpoint_found && + (ev= Log_event::read_log_event(&log, 0, &fdle, + opt_master_verify_checksum)) && + ev->is_valid()) + { + enum Log_event_type typ= ev->get_type_code(); + if (typ == BINLOG_CHECKPOINT_EVENT) + { + uint dir_len; + Binlog_checkpoint_log_event *cev= (Binlog_checkpoint_log_event *)ev; + if (cev->binlog_file_len >= FN_REFLEN) + { + sql_print_error("Incorrect binlog checkpoint event with too " + "long file name found."); + delete ev; + ev= NULL; + end_io_cache(&log); + mysql_file_close(file, MYF(MY_WME)); + return 1; + } + else + { + dir_len= dirname_length(log_name); + strmake(strnmov(checkpoint_file, log_name, dir_len), + cev->binlog_file_name, FN_REFLEN - 1 - dir_len); + binlog_checkpoint_found= true; + } + } + delete ev; + ev= NULL; + } // End of while + end_io_cache(&log); + mysql_file_close(file, MYF(MY_WME)); + file= -1; + /* + Old binary log without checkpoint found, binlog truncation is not + possible. Hence return error. + */ + if (!binlog_checkpoint_found) + return 1; + else + { + // Look for binlog checkpoint file in binlog index file. + if (find_log_pos(&log_info, checkpoint_file, 1)) + { + sql_print_error("Binlog file '%s' not found in binlog index, needed " + "for recovery. Aborting.", checkpoint_file); + return 1; + } + } + return 0; +} + + +/** + Truncates the binary log, according to the transactions that got rolled + back from engine, during heuristic-recover=ROLLBACK. Global GTID state is + adjusted as per the truncated binlog. + + Called from @c TC_LOG::using_heuristic_recover(const char* opt_name) + + @param opt_name The base name of binary log. + + @return indicates success or failure of binlog rollback + @retval 0 success + @retval 1 failure + +*/ +int TC_LOG_BINLOG::heuristic_binlog_rollback() +{ + int error=0; +#ifdef HAVE_REPLICATION + Log_event *ev= NULL; + char engine_commit_file[FN_REFLEN]; + char binlog_truncate_file_name[FN_REFLEN]; + char checkpoint_file[FN_REFLEN]; + my_off_t binlog_truncate_pos= 0; + my_off_t engine_commit_pos= 0; + LOG_INFO log_info; + const char *errmsg; + IO_CACHE log; + File file; + Format_description_log_event fdle(BINLOG_VERSION); + bool found_engine_commit_pos= false; + bool found_truncate_pos= false; + bool is_safe= true; + my_off_t tmp_truncate_pos=0; + rpl_gtid last_gtid; + bool last_gtid_standalone= false; + bool last_gtid_valid= false; + + if (!fdle.is_valid()) + return 1; + + DBUG_EXECUTE_IF("simulate_innodb_forget_commit_pos", + { + last_commit_pos_file[0]= 0; + };); + + // Initialize engine_commit_file/pos + if (last_commit_pos_file[0] != 0) + { + strmake_buf(engine_commit_file, last_commit_pos_file); + engine_commit_pos= last_commit_pos_offset; + } + else + { + if ((error= get_binlog_checkpoint_file(checkpoint_file))) + return error; + strmake_buf(engine_commit_file, checkpoint_file); + } + + // Look for last last_commit_pos_file in binlog index file + if ((error= find_log_pos(&log_info, engine_commit_file, 1))) + { + sql_print_error("find_log_pos() failed (error: %d)", error); + return error; + } + if ((file= open_binlog(&log, log_info.log_file_name, &errmsg)) < 0) + { + sql_print_error("Failed to open the binlog:%s for recovery. Error: %s", + binlog_truncate_file_name, errmsg); + goto err; + } + + /* + If there is no engine specific commit file we are doing checkpoint file + based recovery. Hence we mark "found_engine_commit_pos" true. Next start + looking for the first transactional event group with is the candidate for + rollback. + */ + if (engine_commit_pos == 0) + found_engine_commit_pos= true; + + error= read_state_from_file(); + if (error && error != 2) + { + sql_print_error("Failed to load global gtid binlog state from file"); + goto err; + } + + for(;;) + { + while (is_safe && (ev= Log_event::read_log_event(&log, 0, &fdle, + opt_master_verify_checksum)) && + ev->is_valid()) + { + enum Log_event_type typ= ev->get_type_code(); + switch (typ) + { + case XID_EVENT: + if (ev->log_pos == engine_commit_pos) + { + found_engine_commit_pos= true; + } + break; + case GTID_LIST_EVENT: + { + Gtid_list_log_event *glev= (Gtid_list_log_event *)ev; + /* Initialise the binlog state from the Gtid_list event. */ + if (!found_truncate_pos && glev->count > 0 && + rpl_global_gtid_binlog_state.load(glev->list, glev->count)) + goto err; + } + break; + case GTID_EVENT: + { + Gtid_log_event *gev= (Gtid_log_event *)ev; + + /* Update the binlog state with any GTID logged after Gtid_list. */ + last_gtid.domain_id= gev->domain_id; + last_gtid.server_id= gev->server_id; + last_gtid.seq_no= gev->seq_no; + last_gtid_standalone= + ((gev->flags2 & Gtid_log_event::FL_STANDALONE) ? true : false); + last_gtid_valid= true; + if (gev->flags2 & Gtid_log_event::FL_TRANSACTIONAL && + !found_truncate_pos) + { + if ((engine_commit_pos == 0 || found_engine_commit_pos)) + { + found_truncate_pos=true; + strmake_buf(binlog_truncate_file_name, log_info.log_file_name); + binlog_truncate_pos= tmp_truncate_pos; + } + } + else + { + if (found_truncate_pos) + is_safe=false; + } + } + break; + default: + /* Nothing. */ + break; + }// End switch + if (!found_truncate_pos && last_gtid_valid && + ((last_gtid_standalone && !ev->is_part_of_group(typ)) || + (!last_gtid_standalone && + (typ == XID_EVENT || + (typ == QUERY_EVENT && + (((Query_log_event *)ev)->is_commit() || + ((Query_log_event *)ev)->is_rollback())))))) + { + if (rpl_global_gtid_binlog_state.update_nolock(&last_gtid, false)) + goto err; + last_gtid_valid= false; + } + // Used to identify the last group specific end position. + tmp_truncate_pos= ev->log_pos; + delete ev; + ev= NULL; + }// End While + end_io_cache(&log); + mysql_file_close(file, MYF(MY_WME)); + file= -1; + + if (is_safe) + { + if ((error=find_next_log(&log_info, 1))) + { + if (error != LOG_INFO_EOF) + { + sql_print_error("Error reading binlog files during recovery. Aborting."); + goto err; + } + else + break; + } + + if ((file= open_binlog(&log, log_info.log_file_name, &errmsg)) < 0) + { + sql_print_error("%s", errmsg); + goto err; + } + } + } //end of for(;;) + if (!found_truncate_pos) + { + return 0; // Nothing to truncate + } + else + DBUG_ASSERT(binlog_truncate_pos > 0); + + if (is_safe) + { + if (truncate_and_remove_binlogs(binlog_truncate_file_name, + binlog_truncate_pos)) + goto err; + } + if (write_state_to_file()) + { + sql_print_error("Failed to save binlog GTID state during heuristic " + "binlog rollback. "); + goto err; + } + return 0; + +err: + error= 1; + if (file >= 0) + { + end_io_cache(&log); + mysql_file_close(file, MYF(MY_WME)); + } + sql_print_error("Binlog truncation failed"); +#endif + return error; +} + int TC_LOG_BINLOG::open(const char *opt_name) { int error= 1; @@ -9526,7 +9999,7 @@ int TC_LOG_BINLOG::open(const char *opt_name) return 1; } - if (using_heuristic_recover()) + if (using_heuristic_recover(opt_name)) { mysql_mutex_lock(&LOCK_log); /* generate a new binlog to mask a corrupted one */ diff --git a/sql/log.h b/sql/log.h index 277e5c6f69c..e91bdd94083 100644 --- a/sql/log.h +++ b/sql/log.h @@ -41,7 +41,8 @@ bool stmt_has_updated_non_trans_table(const THD* thd); class TC_LOG { public: - int using_heuristic_recover(); + int using_heuristic_recover(const char* opt_name); + virtual int heuristic_binlog_rollback() { return 0; }; TC_LOG() {} virtual ~TC_LOG() {} @@ -694,6 +695,7 @@ class MYSQL_BIN_LOG: public TC_LOG, private MYSQL_LOG void commit_checkpoint_notify(void *cookie); int recover(LOG_INFO *linfo, const char *last_log_name, IO_CACHE *first_log, Format_description_log_event *fdle, bool do_xa); + int heuristic_binlog_rollback(); int do_binlog_recovery(const char *opt_name, bool do_xa_recovery); #if !defined(MYSQL_CLIENT) @@ -794,6 +796,9 @@ class MYSQL_BIN_LOG: public TC_LOG, private MYSQL_LOG int purge_first_log(Relay_log_info* rli, bool included); int set_purge_index_file_name(const char *base_file_name); int open_purge_index_file(bool destroy); + bool truncate_and_remove_binlogs(const char *truncate_file, + my_off_t truncate_pos); + int get_binlog_checkpoint_file(char* checkpoint_file); bool is_inited_purge_index_file(); int close_purge_index_file(); int clean_purge_index_file(); diff --git a/storage/innobase/handler/ha_innodb.cc b/storage/innobase/handler/ha_innodb.cc index 4de2cdbeaec..42d655ee0e8 100644 --- a/storage/innobase/handler/ha_innodb.cc +++ b/storage/innobase/handler/ha_innodb.cc @@ -16493,6 +16493,9 @@ innobase_xa_recover( { DBUG_ASSERT(hton == innodb_hton_ptr); + mysql_bin_log.last_commit_pos_offset= trx_sys_mysql_bin_log_pos; + strmake_buf(mysql_bin_log.last_commit_pos_file, trx_sys_mysql_bin_log_name); + if (len == 0 || xid_list == NULL) { return(0); diff --git a/storage/xtradb/handler/ha_innodb.cc b/storage/xtradb/handler/ha_innodb.cc index 2aafb1a44ee..dec106cac54 100644 --- a/storage/xtradb/handler/ha_innodb.cc +++ b/storage/xtradb/handler/ha_innodb.cc @@ -17165,6 +17165,9 @@ innobase_xa_recover( { DBUG_ASSERT(hton == innodb_hton_ptr); + mysql_bin_log.last_commit_pos_offset= trx_sys_mysql_bin_log_pos; + strmake_buf(mysql_bin_log.last_commit_pos_file, trx_sys_mysql_bin_log_name); + if (len == 0 || xid_list == NULL) { return(0);

1 0

Re: [Maria-developers] 7b03ce7af3d: MDEV-17395 REPLACE/INSERT ODKU: support WITHOUT OVERLAPS
by Sergei Golubchik 06 Apr '20

06 Apr '20

Hi, Nikita! First, about tests: On Mar 06, Nikita Malyavin wrote: > revision-id: 7b03ce7af3d (mariadb-10.4.4-504-g7b03ce7af3d) > parent(s): 5c94cf3bf44 > author: Nikita Malyavin <nikitamalyavin(a)gmail.com> > committer: Nikita Malyavin <nikitamalyavin(a)gmail.com> > timestamp: 2019-11-28 01:38:53 +1000 > message: > > MDEV-17395 REPLACE/INSERT ODKU: support WITHOUT OVERLAPS > > --- > mysql-test/suite/period/r/insert_replace.result | 56 +++ > mysql-test/suite/period/t/insert_replace.test | 37 ++ > sql/handler.cc | 14 + > sql/handler.h | 3 +- > sql/sql_insert.cc | 492 ++++++++++++++---------- > sql/sql_load.cc | 6 + > sql/table.cc | 4 +- > sql/table.h | 5 +- > 8 files changed, 409 insertions(+), 208 deletions(-) > > diff --git a/mysql-test/suite/period/r/insert_replace.result b/mysql-test/suite/period/r/insert_replace.result > --- /dev/null > +++ b/mysql-test/suite/period/r/insert_replace.result > @@ -0,0 +1,56 @@ > +create or replace table t(id int, val int, s date, e date, > +period for p(s,e), > +primary key(id, p without overlaps)) engine=myisam; > +insert into t values (1, 1, '2003-01-01', '2003-03-01'), > +(1, 2, '2003-05-01', '2003-07-01'); > +# This just inserts a row; no rows matched > +insert into t values (2, 3, '2003-01-01', '2003-04-01') > +on duplicate key update val=3; > +# The following command is equivalent to > +# MERGE INTO t USING t > +# ON id = 1 AND s <= 2003-04-01 AND e > 2003-01-01 > +# WHEN MATCHED UPDATE SET val=3 > +# WHEN NOT MATCHED INSERT VALUES (1, 3, '2003-01-01', '2003-04-01'); > +insert into t values (1, 3, '2003-01-01', '2003-04-01') > +on duplicate key update val=3; > +select row_count(); > +row_count() > +2 > +select * from t; > +id val s e > +1 3 2003-01-01 2003-03-01 > +1 2 2003-05-01 2003-07-01 > +2 3 2003-01-01 2003-04-01 It's somewhat misleading, because you've used '3' everywhere. Please rewrite your tests (all of them, also tests below) to identify every operation uniquely. For example: insert into t values (1, 1, '2003-01-01', '2003-03-01'), (1, 2, '2003-05-01', '2003-07-01'); # This just inserts a row; no rows matched insert into t values (2, 3, '2003-01-01', '2003-04-01') on duplicate key update val=4; # The following command is equivalent to # MERGE INTO t USING t # ON id = 1 AND s <= 2003-04-01 AND e > 2003-01-01 # WHEN MATCHED UPDATE SET val=6 # WHEN NOT MATCHED INSERT VALUES (1, 5, '2003-01-01', '2003-04-01'); insert into t values (1, 5, '2003-01-01', '2003-04-01') on duplicate key update val=6; > +insert into t values (1, 3, '2003-01-01', '2003-06-01') > +on duplicate key update val=4; > +select row_count(); > +row_count() > +4 > +select * from t; > +id val s e > +1 4 2003-01-01 2003-03-01 > +1 4 2003-05-01 2003-07-01 > +2 3 2003-01-01 2003-04-01 I don't think IODKU is defined via MERGE. Unfortunately. See, how it works: create table t1 (a int not null, b int not null, c int, unique(a), unique(b)); insert t1 values (1,1,1), (2,2,2); insert t1 values (1,2,3) on duplicate key update c=4; select * from t1; a b c 1 1 4 2 2 2 here only one row was updated. If it would've been defined as MERGE INTO t1 USING t1 ON a=1 OR b=2 WHEN MATCHED UPDATE c=4 WHEN NOT MATCHED INSERT VALUES (1,2,3) then it would've updated both rows. As you can see it literally is defined as "insert, and if there's a duplicate key error, then update the conflicting row instead" That is, in your case it should've updated only one row too. Also, please, add this statement to your test: insert into t values (1, 3, '2003-01-01', '2003-02-01') on duplicate key update val=4; > +# No rows matched > +insert into t values (1, 3, '2003-07-01', '2003-08-01') > +on duplicate key update val=5; > +select row_count(); > +row_count() > +1 > +select * from t; > +id val s e > +1 4 2003-01-01 2003-03-01 > +1 4 2003-05-01 2003-07-01 > +2 3 2003-01-01 2003-04-01 > +1 3 2003-07-01 2003-08-01 > +replace into t values(1, 6, '2003-01-01', '2003-06-01'); > +select row_count(); > +row_count() > +4 > +select * from t; > +id val s e > +1 6 2003-01-01 2003-06-01 > +1 4 2003-06-01 2003-07-01 > +2 3 2003-01-01 2003-04-01 > +1 3 2003-07-01 2003-08-01 Here you do DELETE FOR PERIOD. But above you didn't do UPDATE FOR PERIOD. Add also this: replace into t values(1, 6, '2003-01-01', '2003-02-01'); And tests for INSERT SELECT (also with IGNORE, REPLACE, ODKU) and for LOAD DATA (also with IGNORE and REPLACE). > +drop table t; Now, about semantics. It is very arguable here. One options is to do, literally, "insert, if fails delete/update the conflicting row". No periods involved here. The other option is to use FOR PERIOD implicitly for updates and deletes. Example, in all test cases below I'll assume: insert t1 values (1,1,'2003-01-01','2003-03-01'); insert t1 values (1,2,'2003-05-01','2003-06-01'); -> 1 1 2003-01-01 2003-03-01 1 2 2003-05-01 2003-06-01 So, option one: insert t1 values (1,3,'2003-01-01','2003-02-01') on duplicate key update val=4; -> 1 4 2003-01-01 2003-02-01 1 2 2003-05-01 2003-06-01 Option two: insert t1 values (1,3,'2003-01-01','2003-02-01') on duplicate key update val=4; -> 1 4 2003-01-01 2003-02-01 1 1 2003-02-01 2003-03-01 1 2 2003-05-01 2003-06-01 Overlapping range, option two: insert t1 values (1,3,'2003-01-01','2003-04-01') on duplicate key update val=4; -> 1 4 2003-01-01 2003-03-01 1 3 2003-03-01 2003-04-01 1 2 2003-05-01 2003-06-01 And the same for replace, basically: replace t1 values (1,3,'2003-01-01','2003-02-01'); -> 1 3 2003-01-01 2003-02-01 1 2 2003-05-01 2003-06-01 or -> 1 3 2003-01-01 2003-02-01 1 1 2003-02-01 2003-03-01 1 2 2003-05-01 2003-06-01 Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 3

Re: [Maria-developers] 6daf451415f: Let "FTWRL <table_list>" use extra(HA_EXTRA_FLUSH)
by Sergei Golubchik 02 Apr '20

02 Apr '20

Hi, Sergey! On Apr 01, Sergey Vojtovich wrote: > revision-id: 6daf451415f (mariadb-10.5.0-69-g6daf451415f) > parent(s): c24253d0fa3 > author: Sergey Vojtovich <svoj(a)mariadb.org> > committer: Sergey Vojtovich <svoj(a)mariadb.org> > timestamp: 2019-12-25 20:24:24 +0400 > message: > > Let "FTWRL <table_list>" use extra(HA_EXTRA_FLUSH) > > Rather than flushing caches with tdc_remove_table(TDC_RT_REMOVE_UNUSED) > flush them with extra(HA_EXTRA_FLUSH) instead. This goes inline with > regular FTWRL. Not quite. FTWRL calls flush_tables(thd, FLUSH_ALL), and flush_tables() does extra(HA_EXTRA_FLUSH) + tc_release_table() + tdc_release_share(share) So FTWRL still closes all tables properly. > diff --git a/sql/sql_reload.cc b/sql/sql_reload.cc > --- a/sql/sql_reload.cc > +++ b/sql/sql_reload.cc > @@ -539,16 +538,10 @@ bool flush_tables_with_read_lock(THD *thd, TABLE_LIST *all_tables) > > DEBUG_SYNC(thd,"flush_tables_with_read_lock_after_acquire_locks"); > > - for (table_list= all_tables; table_list; > + /* Reset ticket to satisfy asserts in open_tables(). */ > + for (auto table_list= all_tables; table_list; > table_list= table_list->next_global) > - { > - /* Request removal of table from cache. */ > - tdc_remove_table(thd, TDC_RT_REMOVE_UNUSED, > - table_list->db.str, > - table_list->table_name.str); > - /* Reset ticket to satisfy asserts in open_tables(). */ > table_list->mdl_request.ticket= NULL; > - } > } > > thd->variables.option_bits|= OPTION_TABLE_LOCK; > @@ -583,6 +576,16 @@ bool flush_tables_with_read_lock(THD *thd, TABLE_LIST *all_tables) > } > } > > + if (thd->lex->type & REFRESH_READ_LOCK) > + { > + for (auto table_list= all_tables; table_list; > + table_list= table_list->next_global) > + { > + if (table_list->table->file->extra(HA_EXTRA_FLUSH)) > + goto error_reset_bits; > + } > + } > + > if (thd->locked_tables_list.init_locked_tables(thd)) > goto error_reset_bits; > > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 3

Re: [Maria-developers] a737b71295e: Cleanup mysql_inplace_alter_table()
by Sergei Golubchik 02 Apr '20

02 Apr '20

Hi, Sergey! ok to push On Apr 02, Sergey Vojtovich wrote: > revision-id: a737b71295e (mariadb-10.5.0-71-ga737b71295e) > parent(s): 5c26e91f755 > author: Sergey Vojtovich <svoj(a)mariadb.org> > committer: Sergey Vojtovich <svoj(a)mariadb.org> > timestamp: 2019-12-25 20:24:25 +0400 > message: > > Cleanup mysql_inplace_alter_table() > > Removed redundant tdc_remove_table(TDC_RT_REMOVE_ALL). Share was marked > flushed by preceding wait_while_table_is_used() and eventually flushed by > close_all_tables_for_name(). > > Part of MDEV-17882 - Cleanup refresh version > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 5c26e91f755: Cleanup close_all_tables_for_name()
by Sergei Golubchik 02 Apr '20

02 Apr '20

Hi, Sergey! On Apr 01, Sergey Vojtovich wrote: > revision-id: 5c26e91f755 (mariadb-10.5.0-70-g5c26e91f755) > parent(s): 6daf451415f > author: Sergey Vojtovich <svoj(a)mariadb.org> > committer: Sergey Vojtovich <svoj(a)mariadb.org> > timestamp: 2019-12-25 20:24:25 +0400 > message: > > Cleanup close_all_tables_for_name() > > close_all_tables_for_name() is always preceded by > wait_while_table_is_used(), which makes tdc_remove_table() redundant. > The only (now fixed) exception was close_cached_tables(). > > Part of MDEV-17882 - Cleanup refresh version > > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > index 60e2b9957a4..9494c0b7bd2 100644 > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -391,13 +391,12 @@ bool close_cached_tables(THD *thd, TABLE_LIST *tables, > if (! table) > continue; > > - if (thd->mdl_context.upgrade_shared_lock(table->mdl_ticket, MDL_EXCLUSIVE, > - timeout)) > + if (wait_while_table_is_used(thd, table, > + HA_EXTRA_PREPARE_FOR_FORCED_CLOSE)) the comment just above this block says /* If we are under LOCK TABLES, we need to reopen the tables without opening a door for any concurrent threads to sneak in and get lock on our tables. To achieve this we use exclusive metadata locks. */ how do you achieve it now without the MDL_EXCLUSIVE ? > { > result= true; > break; > } > - table->file->extra(HA_EXTRA_PREPARE_FOR_FORCED_CLOSE); > close_all_tables_for_name(thd, table->s, HA_EXTRA_NOT_USED, NULL); > } the rest is fine Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 2

Re: [Maria-developers] eeba5b2b158: Fixed close_cached_connection_tables() flushing
by Sergei Golubchik 02 Apr '20

02 Apr '20

Hi, Sergey! On Apr 02, Sergey Vojtovich wrote: > revision-id: eeba5b2b158 (mariadb-10.5.0-72-geeba5b2b158) > parent(s): a737b71295e > author: Sergey Vojtovich <svoj(a)mariadb.org> > committer: Sergey Vojtovich <svoj(a)mariadb.org> > timestamp: 2019-12-25 20:24:26 +0400 > message: > > Fixed close_cached_connection_tables() flushing > > Let DROP SERVER and ALTER SERVER perform fair affected tables flushing. > That is acquire MDL_EXCLUSIVE and do tdc_remove_table(TDC_RT_REMOVE_ALL). > > Aim of this patch is elimination of another inconsistent use of > TDC_RT_REMOVE_UNUSED. It fixes (to some extent) a problem described in the > beginning of sql_server.cc, when close_cached_connection_tables() > interferes with concurrent transaction. > > A better fix should probably introduce proper MDL locks for server > objects? > > Part of MDEV-17882 - Cleanup refresh version > > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > index 9494c0b7bd2..5c6d366ce6f 100644 > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -689,33 +689,21 @@ static my_bool close_cached_connection_tables_callback( > Close cached connections Could you, perhaps, move this function to sql_server.cc ? Also it could be made static there. And please clarify the comment, "Close cached connections' was not helpful at all. May be "Close all tables with a given CONNECTION= value" > @return false ok > - @return true If there was an error from closed_cached_connection_tables or > - if there was any open connections that we had to force closed > + @return true error, some tables may keep using old server info > */ > > bool close_cached_connection_tables(THD *thd, LEX_CSTRING *connection) > { > - bool res= false; > - close_cached_connection_tables_arg argument; > + close_cached_connection_tables_arg argument= { thd, connection, 0 }; > DBUG_ENTER("close_cached_connections"); > - DBUG_ASSERT(thd); > - > - argument.thd= thd; > - argument.connection= connection; > - argument.tables= NULL; > > if (tdc_iterate(thd, > (my_hash_walk_action) close_cached_connection_tables_callback, > &argument)) > DBUG_RETURN(true); > > - for (TABLE_LIST *table= argument.tables; table; table= table->next_local) > - res|= tdc_remove_table(thd, TDC_RT_REMOVE_UNUSED, > - table->db.str, > - table->table_name.str); > - > - /* Return true if we found any open connections */ > - DBUG_RETURN(res); > + DBUG_RETURN(close_cached_tables(thd, argument.tables, true, > + thd->variables.lock_wait_timeout)); are you sure you want to do it when argument.tables is empty? It'll do purge_tables() which seems to be a bit extreme for DROP SERVER > } > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] df4db9a1ea1: Yet less TDC hash lookups
by Sergei Golubchik 02 Apr '20

02 Apr '20

Hi, Sergey! ok to push On Apr 02, Sergey Vojtovich wrote: > revision-id: df4db9a1ea1 (mariadb-10.5.0-76-gdf4db9a1ea1) > parent(s): 269793896c4 > author: Sergey Vojtovich <svoj(a)mariadb.org> > committer: Sergey Vojtovich <svoj(a)mariadb.org> > timestamp: 2019-12-27 20:37:11 +0400 > message: > > Yet less TDC hash lookups > > Let auto repair table and truncate table routines flush TABLE_SHARE > directly. > > Part of MDEV-17882 - Cleanup refresh version > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] f92e6520a62: tc_remove_all_unused_tables() cleanup
by Sergei Golubchik 02 Apr '20

02 Apr '20

Hi, Sergey! On Apr 02, Sergey Vojtovich wrote: > revision-id: f92e6520a62 (mariadb-10.5.0-74-gf92e6520a62) > parent(s): df0de9c652d > author: Sergey Vojtovich <svoj(a)mariadb.org> > committer: Sergey Vojtovich <svoj(a)mariadb.org> > timestamp: 2019-12-26 16:54:19 +0400 > message: > > tc_remove_all_unused_tables() cleanup > > As tc_purge() never marks share flushed, let tdc_remove_table() do it > directly. sure Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] df0de9c652d: Proper locking for mysql.gtid_slave_pos truncation
by Sergei Golubchik 02 Apr '20

02 Apr '20

Hi, Sergey! look ok On Apr 02, Sergey Vojtovich wrote: > revision-id: df0de9c652d (mariadb-10.5.0-73-gdf0de9c652d) > parent(s): eeba5b2b158 > author: Sergey Vojtovich <svoj(a)mariadb.org> > committer: Sergey Vojtovich <svoj(a)mariadb.org> > timestamp: 2019-12-26 16:50:51 +0400 > message: > > Proper locking for mysql.gtid_slave_pos truncation > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 269793896c4: Split tdc_remove_table()
by Sergei Golubchik 02 Apr '20

02 Apr '20

Hi, Sergey! ok to push On Apr 02, Sergey Vojtovich wrote: > revision-id: 269793896c4 (mariadb-10.5.0-75-g269793896c4) > parent(s): f92e6520a62 > author: Sergey Vojtovich <svoj(a)mariadb.org> > committer: Sergey Vojtovich <svoj(a)mariadb.org> > timestamp: 2019-12-27 17:14:57 +0400 > message: > > Split tdc_remove_table() > > TDC_RT_REMOVE_ALL -> tdc_remove_table(). Some occurrences replaced with > TDC_element::flush() (whenver TABLE_SHARE is available). > > TDC_RT_REMOVE_NOT_OWN[_KEEP_SHARE] -> TDC_element::flush(). These modes > assume that current thread owns TABLE_SHARE reference, which means we can > avoid hash lookup and flush unused TABLE instances directly. > > TDC_RT_REMOVE_UNUSED -> TDC_element::flush_unused(). Only [ab]used by > mysql_admin_table() currently. Should be removed eventually. > > Part of MDEV-17882 - Cleanup refresh version > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] b62432fc245: MDEV-21117: --tc-heuristic-recover=rollback is not replication safe
by sujatha 01 Apr '20

01 Apr '20

revision-id: b62432fc2459a312aaa0e27c0950de3420a34d53 (mariadb-10.1.43-96-gb62432fc245) parent(s): f9639c2d1a5e24f1a1533b6277fe7eca3aa3c3c0 author: Sujatha committer: Sujatha timestamp: 2020-04-01 23:17:10 +0530 message: MDEV-21117: --tc-heuristic-recover=rollback is not replication safe Problem: ======= When run after master server crash --tc-heuristic-recover=rollback produces inconsistent server state with binlog still containing transactions that were rolled back by the option. Such way recovered server may not be used for replication. Fix: === During --tc-heuristic-recover query the storage engine to get binlog file name and position corrensponding to the last committed transaction. If last_commit_pos_file < checkpoint_file then we need to start looking from checkpoint file. Look for first transactional event as this is the candidate for heuristic rollback. If a transactional event is found then this is used as reference for binlog truncation. Otherwise no truncation is needed. If last_commit_pos_file is equal to checkpoint_file then truncate binlog as per the last_commit_pos. If last_commit_file is not set then checkpoint_file is what we have. Look for first transactional event and consider this as truncate position. Once the binlog file name and position for truncation are identified start reading all the events beyond this position. Check if they are all transactional. If all are transactional truncate the binary log. If any non transactional events are found then truncation will fail. Adjust the global gtid state as per the truncated binlog. --- .../r/binlog_heuristic_rollback_active_log.result | 22 + .../binlog/r/binlog_truncate_multi_log.result | 26 + .../suite/binlog/r/binlog_truncate_none.result | 24 + .../t/binlog_heuristic_rollback_active_log.test | 76 +++ .../suite/binlog/t/binlog_truncate_multi_log.test | 88 ++++ .../suite/binlog/t/binlog_truncate_none.test | 65 +++ sql/log.cc | 565 ++++++++++++++++++++- sql/log.h | 7 +- storage/innobase/handler/ha_innodb.cc | 3 + storage/xtradb/handler/ha_innodb.cc | 3 + 10 files changed, 875 insertions(+), 4 deletions(-) diff --git a/mysql-test/suite/binlog/r/binlog_heuristic_rollback_active_log.result b/mysql-test/suite/binlog/r/binlog_heuristic_rollback_active_log.result new file mode 100644 index 00000000000..512644459f0 --- /dev/null +++ b/mysql-test/suite/binlog/r/binlog_heuristic_rollback_active_log.result @@ -0,0 +1,22 @@ +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); +RESET MASTER; +CREATE TABLE t ( f INT ) ENGINE=INNODB; +INSERT INTO t VALUES (10); +SET GLOBAL DEBUG_DBUG="d,simulate_crash_after_binlog_write"; +INSERT INTO t VALUES (20); +ERROR HY000: Lost connection to MySQL server during query +SELECT * FROM t; +f +10 +include/show_binlog_events.inc +Log_name Pos Event_type Server_id End_log_pos Info +master-bin.000001 # Gtid # # GTID #-#-# +master-bin.000001 # Query # # use `test`; CREATE TABLE t ( f INT ) ENGINE=INNODB +master-bin.000001 # Gtid # # BEGIN GTID #-#-# +master-bin.000001 # Query # # use `test`; INSERT INTO t VALUES (10) +master-bin.000001 # Xid # # COMMIT /* XID */ +DROP TABLE t; +show binlog events in 'master-bin.000003' limit 1,1; +Log_name Pos Event_type Server_id End_log_pos Info +master-bin.000003 # Gtid_list 1 # [0-1-2] diff --git a/mysql-test/suite/binlog/r/binlog_truncate_multi_log.result b/mysql-test/suite/binlog/r/binlog_truncate_multi_log.result new file mode 100644 index 00000000000..b6a3580f916 --- /dev/null +++ b/mysql-test/suite/binlog/r/binlog_truncate_multi_log.result @@ -0,0 +1,26 @@ +SET @old_max_binlog_size= @@global.max_binlog_size; +SET GLOBAL max_binlog_size= 4096; +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); +RESET MASTER; +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +SET DEBUG_SYNC= "commit_after_release_LOCK_log SIGNAL con1_ready WAIT_FOR con1_go"; +INSERT INTO t1 VALUES (1, REPEAT("x", 4100)); +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; +show binary logs; +Log_name File_size +master-bin.000001 # +master-bin.000002 # +SET GLOBAL DEBUG_DBUG="d,simulate_crash_after_binlog_write"; +SET DEBUG_SYNC= "now SIGNAL con1_go"; +ERROR HY000: Lost connection to MySQL server during query +"Only CREATE TABLE t1 should be present in the binary log" +include/show_binlog_events.inc +Log_name Pos Event_type Server_id End_log_pos Info +master-bin.000001 # Gtid # # GTID #-#-# +master-bin.000001 # Query # # use `test`; CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb +"Zero records should be there." +SELECT COUNT(*) FROM t1; +COUNT(*) +0 +DROP TABLE t1; diff --git a/mysql-test/suite/binlog/r/binlog_truncate_none.result b/mysql-test/suite/binlog/r/binlog_truncate_none.result new file mode 100644 index 00000000000..1eee3fb1cd1 --- /dev/null +++ b/mysql-test/suite/binlog/r/binlog_truncate_none.result @@ -0,0 +1,24 @@ +SET @old_max_binlog_size= @@global.max_binlog_size; +SET GLOBAL max_binlog_size= 4096; +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); +RESET MASTER; +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +INSERT INTO t1 VALUES (1, REPEAT("x", 4100)); +CREATE TABLE t2 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +show binary logs; +Log_name File_size +master-bin.000001 # +master-bin.000002 # +# Kill the server +"Zero records should be there." +SELECT COUNT(*) FROM t1; +COUNT(*) +1 +show binary logs; +Log_name File_size +master-bin.000001 # +master-bin.000002 # +master-bin.000003 # +master-bin.000004 # +DROP TABLE t1,t2; diff --git a/mysql-test/suite/binlog/t/binlog_heuristic_rollback_active_log.test b/mysql-test/suite/binlog/t/binlog_heuristic_rollback_active_log.test new file mode 100644 index 00000000000..1abb3b6495a --- /dev/null +++ b/mysql-test/suite/binlog/t/binlog_heuristic_rollback_active_log.test @@ -0,0 +1,76 @@ +# ==== Purpose ==== +# +# Test verifies the truncation of single binary log file. +# +# ==== Implementation ==== +# +# Steps: +# 0 - Create table t1 and insert a row. +# 1 - Insert an another row such that it gets written to binlog but commit +# in engine fails as server crashed at this point. +# 2 - Restart server with --tc-heuristic-recover=ROLLBACK +# 3 - Upon server start 'master-bin.000001' will be truncated to contain +# only the first insert +# +# ==== References ==== +# +# MDEV-21117: --tc-heuristic-recover=rollback is not replication safe + + +--source include/have_innodb.inc +--source include/have_log_bin.inc +--source include/have_debug.inc +--source include/have_binlog_format_statement.inc + +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); + +connect(master,localhost,root,,); +connect(master1,localhost,root,,); + +--connection master +RESET MASTER; +CREATE TABLE t ( f INT ) ENGINE=INNODB; +INSERT INTO t VALUES (10); + +--connection master1 +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +wait +EOF + +SET GLOBAL DEBUG_DBUG="d,simulate_crash_after_binlog_write"; +--error 2013 # CR_SERVER_LOST +INSERT INTO t VALUES (20); +--source include/wait_until_disconnected.inc + +--connection master +--source include/wait_until_disconnected.inc + +# +# Server restart +# +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: --tc-heuristic-recover=ROLLBACK +EOF +--source include/wait_until_disconnected.inc + +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: +EOF + +connection default; +--enable_reconnect +--source include/wait_until_connected_again.inc + +--connection master +--enable_reconnect +--source include/wait_until_connected_again.inc + +--connection master +SELECT * FROM t; +--source include/show_binlog_events.inc + +--connection master +DROP TABLE t; +let $binlog_file= query_get_value(SHOW MASTER STATUS, File, 1); +--source include/show_gtid_list.inc diff --git a/mysql-test/suite/binlog/t/binlog_truncate_multi_log.test b/mysql-test/suite/binlog/t/binlog_truncate_multi_log.test new file mode 100644 index 00000000000..5afa3b782cc --- /dev/null +++ b/mysql-test/suite/binlog/t/binlog_truncate_multi_log.test @@ -0,0 +1,88 @@ +# ==== Purpose ==== +# +# Test verifies truncation of multiple binary logs. +# +# ==== Implementation ==== +# +# Steps: +# 0 - Set max_binlog_size= 4096. Create a table and do an insert such that +# the max_binlog_size is reached and binary log gets rotated. +# 1 - Using debug simulation make the server crash at a point where the DML +# transaction is written to binary log but not committed in engine. +# 2 - At the time of crash two binary logs will be there master-bin.0000001 +# and master-bin.000002. +# 3 - Restart server with --tc-heuristic-recover=ROLLBACK +# 4 - Since the prepared DML in master-bin.000001 is rolled back the first +# binlog will be truncated prior to the DML and master-bin.000002 will be +# removed. +# +# ==== References ==== +# +# MDEV-21117: --tc-heuristic-recover=rollback is not replication safe + + +--source include/have_innodb.inc +--source include/have_log_bin.inc +--source include/have_debug.inc +--source include/have_binlog_format_row.inc + +SET @old_max_binlog_size= @@global.max_binlog_size; +SET GLOBAL max_binlog_size= 4096; + +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); + +RESET MASTER; + +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; + +# Hold insert after write to binlog and before "run_commit_ordered" in engine +connect(master1,localhost,root,,); +connect(master2,localhost,root,,); + +--connection master1 +SET DEBUG_SYNC= "commit_after_release_LOCK_log SIGNAL con1_ready WAIT_FOR con1_go"; +send INSERT INTO t1 VALUES (1, REPEAT("x", 4100)); + +--connection default +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; +--source include/show_binary_logs.inc +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +wait +EOF + +--connection master2 +SET GLOBAL DEBUG_DBUG="d,simulate_crash_after_binlog_write"; +SET DEBUG_SYNC= "now SIGNAL con1_go"; + +--connection master1 +--error 2013 # CR_SERVER_LOST +--reap +--source include/wait_until_disconnected.inc + +--connection master2 +--source include/wait_until_disconnected.inc + +# +# Server restart +# +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: --tc-heuristic-recover=ROLLBACK +EOF +--source include/wait_until_disconnected.inc + +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: +EOF + +connection default; +--enable_reconnect +--source include/wait_until_connected_again.inc +--echo "Only CREATE TABLE t1 should be present in the binary log" +--source include/show_binlog_events.inc + +--echo "Zero records should be there." +SELECT COUNT(*) FROM t1; + +DROP TABLE t1; + diff --git a/mysql-test/suite/binlog/t/binlog_truncate_none.test b/mysql-test/suite/binlog/t/binlog_truncate_none.test new file mode 100644 index 00000000000..9254cfd4d72 --- /dev/null +++ b/mysql-test/suite/binlog/t/binlog_truncate_none.test @@ -0,0 +1,65 @@ +# ==== Purpose ==== +# +# Test case verifies no binlog truncation happens when non transactional +# events are found in binlog after the last committed transaction. +# +# ==== Implementation ==== +# +# Steps: +# 0 - Set max_binlog_size= 4096. Create a table and do an insert such that +# the max_binlog_size is reached and binary log gets rotated. +# 1 - Create a table in newly created binary log and crash the server +# 2 - Restart server with --tc-heuristic-recover=ROLLBACK +# 3 - Recovery code will get the last committed DML specific postion and +# will try to check if binlog can be truncated upto this position. +# Since a DDL is present beyond this no truncation will happen. +# ==== References ==== +# +# MDEV-21117: --tc-heuristic-recover=rollback is not replication safe + + +--source include/have_innodb.inc +--source include/have_log_bin.inc +--source include/have_debug.inc +--source include/have_binlog_format_row.inc + +SET @old_max_binlog_size= @@global.max_binlog_size; +SET GLOBAL max_binlog_size= 4096; + +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); + +RESET MASTER; + +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +INSERT INTO t1 VALUES (1, REPEAT("x", 4100)); +CREATE TABLE t2 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +--source include/show_binary_logs.inc + +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +wait +EOF + +--source include/kill_mysqld.inc +--source include/wait_until_disconnected.inc + +# +# Server restart +# +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: --tc-heuristic-recover=ROLLBACK +EOF +--source include/wait_until_disconnected.inc + +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: +EOF + +connection default; +--enable_reconnect +--source include/wait_until_connected_again.inc + +--echo "Zero records should be there." +SELECT COUNT(*) FROM t1; +--source include/show_binary_logs.inc +DROP TABLE t1,t2; diff --git a/sql/log.cc b/sql/log.cc index 0efef6d1e29..8c997d547da 100644 --- a/sql/log.cc +++ b/sql/log.cc @@ -3164,6 +3164,7 @@ MYSQL_BIN_LOG::MYSQL_BIN_LOG(uint *sync_period) checksum_alg_reset(BINLOG_CHECKSUM_ALG_UNDEF), relay_log_checksum_alg(BINLOG_CHECKSUM_ALG_UNDEF), description_event_for_exec(0), description_event_for_queue(0), + last_commit_pos_offset(0), current_binlog_id(0) { /* @@ -3173,6 +3174,7 @@ MYSQL_BIN_LOG::MYSQL_BIN_LOG(uint *sync_period) before main(). */ index_file_name[0] = 0; + last_commit_pos_file[0]= 0; bzero((char*) &index_file, sizeof(index_file)); bzero((char*) &purge_index_file, sizeof(purge_index_file)); } @@ -7936,6 +7938,8 @@ MYSQL_BIN_LOG::trx_group_commit_leader(group_commit_entry *leader) mysql_mutex_unlock(&LOCK_log); DEBUG_SYNC(leader->thd, "commit_after_release_LOCK_log"); + DBUG_EXECUTE_IF("simulate_crash_after_binlog_write", + DBUG_SUICIDE();); /* Loop through threads and run the binlog_sync hook @@ -8964,7 +8968,7 @@ int TC_LOG_MMAP::open(const char *opt_name) { if (my_errno != ENOENT) goto err; - if (using_heuristic_recover()) + if (using_heuristic_recover(opt_name)) return 1; if ((fd= mysql_file_create(key_file_tclog, logname, CREATE_MODE, O_RDWR | O_CLOEXEC, MYF(MY_WME))) < 0) @@ -9497,14 +9501,40 @@ TC_LOG_MMAP tc_log_mmap; 1 heuristic recovery was performed */ -int TC_LOG::using_heuristic_recover() +int TC_LOG::using_heuristic_recover(const char* opt_name) { + LOG_INFO log_info; + int error; + if (!tc_heuristic_recover) return 0; sql_print_information("Heuristic crash recovery mode"); + if (ha_recover(0)) + { sql_print_error("Heuristic crash recovery failed"); + } + + /* + Check if TC log is referring to binary log not memory map. Ensure that + tc_heuristic_recover being ROLLBACK". If both match initiate binlog + truncation mechanism. + */ + if (!strcmp(opt_name,opt_bin_logname) && + tc_heuristic_recover == TC_HEURISTIC_RECOVER_ROLLBACK) + { + if ((error= mysql_bin_log.find_log_pos(&log_info, NullS, 1))) + { + if (error != LOG_INFO_EOF) + sql_print_error("find_log_pos() failed (error: %d)", error); + else + return 0; + } + if ((error= mysql_bin_log.heuristic_binlog_rollback())) + return error; + } + sql_print_information("Please restart mysqld without --tc-heuristic-recover"); return 1; } @@ -9512,6 +9542,535 @@ int TC_LOG::using_heuristic_recover() /****** transaction coordinator log for 2pc - binlog() based solution ******/ #define TC_LOG_BINLOG MYSQL_BIN_LOG +/** + Truncates the current binlog to specified position. Removes the rest of binlogs + which are present after this binlog file. + + @param truncate_file Holds the binlog name to be truncated + @param truncate_pos Position within binlog from where it needs to + truncated. + + @retval true ok + @retval false error + +*/ +bool MYSQL_BIN_LOG::truncate_and_remove_binlogs(const char *truncate_file, + my_off_t truncate_pos) +{ + int error= 0; +#ifdef HAVE_REPLICATION + LOG_INFO log_info; + THD *thd= current_thd; + my_off_t index_file_offset=0; + File file= -1; + bool included= (!(truncate_pos) ? 1 : 0); + MY_STAT s; + my_off_t binlog_size; + + if ((error= open_purge_index_file(TRUE))) + { + sql_print_error("MYSQL_BIN_LOG::purge_logs failed to sync the index file."); + goto err; + } + + if ((error=find_log_pos(&log_info, truncate_file, 1))) + goto err; + + if (included) + { + index_file_offset= log_info.index_file_start_offset; + if((error= register_purge_index_entry(log_info.log_file_name))) + { + sql_print_error("MYSQL_BIN_LOG::purge_logs failed to copy %s to register file.", + log_info.log_file_name); + goto err; + } + } + while (!(error= find_next_log(&log_info, 1))) + { + if (!index_file_offset) + index_file_offset= log_info.index_file_start_offset; + if((error= register_purge_index_entry(log_info.log_file_name))) + { + sql_print_error("MYSQL_BIN_LOG::purge_logs failed to copy %s to register file.", + log_info.log_file_name); + goto err; + } + } + + if (error != LOG_INFO_EOF) + { + sql_print_error("Error while looking for next binlog file to be truncated. " + "Error:%d", error); + goto err; + } + + if (!index_file_offset) + index_file_offset= log_info.index_file_start_offset; + + if ((error= sync_purge_index_file())) + { + sql_print_error("MYSQL_BIN_LOG::purge_logs failed to flush register file."); + goto err; + } + + DBUG_ASSERT(index_file_offset != 0); + // Trim index file + if (mysql_file_chsize(index_file.file, index_file_offset, '\n', MYF(MY_WME)) || + mysql_file_sync(index_file.file, MYF(MY_WME|MY_SYNC_FILESIZE))) + { + sql_print_error("Failed to trim binlog index file " + "when master server is recovering it."); + mysql_file_close(index_file.file, MYF(MY_WME)); + goto err; + } + + /* Reset data in old index cache */ + reinit_io_cache(&index_file, READ_CACHE, (my_off_t) 0, 0, 1); + + /* Read each entry from purge_index_file and delete the file. */ + if (is_inited_purge_index_file() && + (error= purge_index_entry(thd, NULL, TRUE))) + { + sql_print_error("MYSQL_BIN_LOG::purge_logs failed to process registered files" + " that would be purged."); + goto err; + } + + if (truncate_pos) + { + if ((file= mysql_file_open(key_file_binlog, truncate_file, + O_RDWR | O_BINARY, MYF(MY_WME))) < 0) + { + sql_print_error("Failed to open binlog file:%s for truncation.", + truncate_file); + error= 1; + goto err; + } + my_stat(truncate_file, &s, MYF(0)); + binlog_size= s.st_size; + + /* Change binlog file size to truncate_pos */ + if (mysql_file_chsize(file, truncate_pos, 0, MYF(MY_WME)) || + mysql_file_sync(file, MYF(MY_WME|MY_SYNC_FILESIZE))) + { + sql_print_error("Failed to trim the crashed binlog file:%s to size:%llu", + truncate_file, truncate_pos); + mysql_file_close(file, MYF(MY_WME)); + error= 1; + goto err; + } + else + { + sql_print_information("Crashed binlog file %s size is %llu, " + "but truncated to %llu.", + truncate_file, binlog_size, truncate_pos); + } + clear_inuse_flag_when_closing(file); + mysql_file_close(file, MYF(MY_WME)); + } + +err: + close_purge_index_file(); +#endif + return error; +} + +/** + Returns the checkpoint binlog file name. Iterate though the list of binary + logs. Move to the last one. Read the binlog checkpoint event within the last + log and return it. + + @param checkpoint_file Holds the binlog checkpoint file name. + + @retval 0 ok + @retval 1 error + +*/ +int TC_LOG_BINLOG::get_binlog_checkpoint_file(char* checkpoint_file) +{ + Log_event *ev= NULL; + bool binlog_checkpoint_found= false; + LOG_INFO log_info; + const char *errmsg; + IO_CACHE log; + File file; + Format_description_log_event fdle(BINLOG_VERSION); + char log_name[FN_REFLEN]; + int error=1; + + if (!fdle.is_valid()) + return 1; + + if ((error= find_log_pos(&log_info, NullS, 1))) + { + sql_print_error("find_log_pos() failed (error: %d)", error); + return error; + } + + // Move to the last binary log. + do + { + strmake_buf(log_name, log_info.log_file_name); + } while (!(error= find_next_log(&log_info, 1))); + + if (error != LOG_INFO_EOF) + { + sql_print_error("find_log_pos() failed (error: %d)", error); + return error; + } + if ((file= open_binlog(&log, log_name, &errmsg)) < 0) + { + sql_print_error("%s", errmsg); + return error; + } + while (!binlog_checkpoint_found && + (ev= Log_event::read_log_event(&log, 0, &fdle, + opt_master_verify_checksum)) && + ev->is_valid()) + { + enum Log_event_type typ= ev->get_type_code(); + if (typ == BINLOG_CHECKPOINT_EVENT) + { + uint dir_len; + Binlog_checkpoint_log_event *cev= (Binlog_checkpoint_log_event *)ev; + if (cev->binlog_file_len >= FN_REFLEN) + { + sql_print_error("Incorrect binlog checkpoint event with too " + "long file name found."); + delete ev; + ev= NULL; + end_io_cache(&log); + mysql_file_close(file, MYF(MY_WME)); + return 1; + } + else + { + dir_len= dirname_length(log_name); + strmake(strnmov(checkpoint_file, log_name, dir_len), + cev->binlog_file_name, FN_REFLEN - 1 - dir_len); + binlog_checkpoint_found= true; + } + } + delete ev; + ev= NULL; + } // End of while + end_io_cache(&log); + mysql_file_close(file, MYF(MY_WME)); + file= -1; + /* + Old binary log without checkpoint found, binlog truncation is not + possible. Hence return error. + */ + if (!binlog_checkpoint_found) + return 1; + else + { + // Look for binlog checkpoint file in binlog index file. + if (find_log_pos(&log_info, checkpoint_file, 1)) + { + sql_print_error("Binlog file '%s' not found in binlog index, needed " + "for recovery. Aborting.", checkpoint_file); + return 1; + } + } + return 0; +} + +/** + Given a binlog file name and position the following function will parse the + binlog for the presense of only transactional event groups beyond the + position to be truncated. If all are transactional then truncation is + considered to be safe. If any non transactional or DDL groups are found then + truncation is not safe. + + @param truncate_file Holds the name of binlog to be truncated + + @retval true Safe to truncate + @retval false Truncation is not safe + +*/ +bool MYSQL_BIN_LOG::is_binlog_truncate_safe(const char* truncate_file, my_off_t pos) +{ + Log_event *ev= NULL; + LOG_INFO log_info; + const char *errmsg; + IO_CACHE log; + File file; + Format_description_log_event fdle(BINLOG_VERSION); + int error; + + if (!fdle.is_valid()) + goto err; + + if ((error= find_log_pos(&log_info, truncate_file, 1))) + { + sql_print_error("find_log_pos() failed (error: %d)", error); + goto err; + } + + if ((file= open_binlog(&log, log_info.log_file_name, &errmsg)) < 0) + { + sql_print_error("Failed to open the binlog checkpoint file:%s for recovery." + "Error: %s", truncate_file, errmsg); + goto err; + } + if (pos) + my_b_seek(&log, pos); + + for (;;) + { + while ((ev= Log_event::read_log_event(&log, 0, &fdle, + opt_master_verify_checksum)) && + ev->is_valid()) + { +#ifdef HAVE_REPLICATION + enum Log_event_type typ= ev->get_type_code(); + if (typ == GTID_EVENT) + { + Gtid_log_event *gev= (Gtid_log_event *)ev; + if (!(gev->flags2 & Gtid_log_event::FL_TRANSACTIONAL)) + { + delete ev; + ev= NULL; + goto err; + } + } +#endif + delete ev; + ev= NULL; + }// End While + end_io_cache(&log); + mysql_file_close(file, MYF(MY_WME)); + file= -1; + + if ((error=find_next_log(&log_info, 1))) + { + if (error != LOG_INFO_EOF) + { + sql_print_error("Error reading binlog files during recovery. Aborting."); + goto err; + } + else + break; + } + + if ((file= open_binlog(&log, log_info.log_file_name, &errmsg)) < 0) + { + sql_print_error("%s", errmsg); + goto err; + } + } // End of for(;;) + return true; + +err: + if (file >= 0) + { + end_io_cache(&log); + mysql_file_close(file, MYF(MY_WME)); + file= -1; + } + return false; +} + + +/** + Truncates the binary log, according to the transactions that got rolled + back from engine, during heuristic-recover=ROLLBACK. Global GTID state is + adjusted as per the truncated binlog. + + Called from @c TC_LOG::using_heuristic_recover(const char* opt_name) + + @param opt_name The base name of binary log. + + @return indicates success or failure of binlog rollback + @retval 0 success + @retval 1 failure + +*/ +int TC_LOG_BINLOG::heuristic_binlog_rollback() +{ + int error=0; +#ifdef HAVE_REPLICATION + Log_event *ev= NULL; + char binlog_truncate_file_name[FN_REFLEN]; + char checkpoint_file[FN_REFLEN]; + my_off_t binlog_truncate_pos=0; + LOG_INFO log_info; + const char *errmsg; + IO_CACHE log; + File file; + Format_description_log_event fdle(BINLOG_VERSION); + bool found_truncate_pos= false; + bool is_safe= false; + my_off_t tmp_truncate_pos=0; + rpl_gtid last_gtid; + bool last_gtid_standalone= false; + bool last_gtid_valid= false; + + if (!fdle.is_valid()) + return 1; + + if ((error= get_binlog_checkpoint_file(checkpoint_file))) + return error; + + sql_print_information("Binlog checkpoint_file name:%s", checkpoint_file); + + if (last_commit_pos_file[0] != 0) + { + if (strcmp(last_commit_pos_file, checkpoint_file) > 0) + { + sql_print_error("Engine specific binlog_file:%s cannot be advanced than " + "checkpoint_file:%s", last_commit_pos_file, checkpoint_file); + return 1; + } + else + binlog_truncate_pos= last_commit_pos_offset; + } + /* + If last_commit_pos_file < checkpoint_file then we need to start looking + from checkpoint file. Look for first transactional event as this is the + candidate for heuristic rollback. If a transactional event is found then + this is used as reference for binlog truncation. Otherwise no truncation + is needed. + + If last_commit_pos_file is equal to checkpoint_file then truncate binlog as + per the last_commit_pos. + + If last_commit_file is not set then checkpoint_file is what we have. + Look for first trans event and consider this as truncate position. + */ + strmake_buf(binlog_truncate_file_name, checkpoint_file); + + error= read_state_from_file(); + if (error && error != 2) + { + sql_print_error("Failed to load global gtid binlog state from file"); + return error; + } + + if ((error= find_log_pos(&log_info, checkpoint_file, 1))) + { + sql_print_error("find_log_pos() failed (error: %d)", error); + return error; + } + + if ((file= open_binlog(&log, log_info.log_file_name, &errmsg)) < 0) + { + sql_print_error("Failed to open the binlog checkpoint file:%s for recovery." + "Error: %s", checkpoint_file, errmsg); + return 1; + } + + while (!found_truncate_pos && + ((ev= Log_event::read_log_event(&log, 0, &fdle, + opt_master_verify_checksum)) && + ev->is_valid())) + { + enum Log_event_type typ= ev->get_type_code(); + switch (typ) + { + case XID_EVENT: + if (ev->log_pos == binlog_truncate_pos) + { + found_truncate_pos= true; + } + break; + case GTID_LIST_EVENT: + { + Gtid_list_log_event *glev= (Gtid_list_log_event *)ev; + /* Initialise the binlog state from the Gtid_list event. */ + if (glev->count > 0 && + rpl_global_gtid_binlog_state.load(glev->list, glev->count)) + goto err; + } + break; + case GTID_EVENT: + if (!found_truncate_pos) + { + Gtid_log_event *gev= (Gtid_log_event *)ev; + + /* Update the binlog state with any GTID logged after Gtid_list. */ + last_gtid.domain_id= gev->domain_id; + last_gtid.server_id= gev->server_id; + last_gtid.seq_no= gev->seq_no; + last_gtid_standalone= + ((gev->flags2 & Gtid_log_event::FL_STANDALONE) ? true : false); + last_gtid_valid= true; + if ((binlog_truncate_pos == 0) && gev->flags2 & Gtid_log_event::FL_TRANSACTIONAL) + { + found_truncate_pos=true; + binlog_truncate_pos= tmp_truncate_pos; + } + } + break; + default: + /* Nothing. */ + break; + }// End switch + if (last_gtid_valid && + ((last_gtid_standalone && !ev->is_part_of_group(typ)) || + (!last_gtid_standalone && + (typ == XID_EVENT || + (typ == QUERY_EVENT && + (((Query_log_event *)ev)->is_commit() || + ((Query_log_event *)ev)->is_rollback())))))) + { + if (rpl_global_gtid_binlog_state.update_nolock(&last_gtid, false)) + goto err; + last_gtid_valid= false; + } + // Used to identify the last group specific end position. + tmp_truncate_pos= ev->log_pos; + delete ev; + ev= NULL; + }// End While + end_io_cache(&log); + mysql_file_close(file, MYF(MY_WME)); + file= -1; + + if (!found_truncate_pos) + { + if (binlog_truncate_pos != 0) + { + sql_print_error("Failed to locate event with offset:%llu in binlog:%s" + "Unable to truncate the binary log", binlog_truncate_pos, + binlog_truncate_file_name); + goto err; + } + else + return 0; // Nothing to truncate + } + + is_safe= is_binlog_truncate_safe(binlog_truncate_file_name, + binlog_truncate_pos); + + if (is_safe) + { + if (truncate_and_remove_binlogs(binlog_truncate_file_name, + binlog_truncate_pos)) + goto err; + } + if (write_state_to_file()) + { + sql_print_error("Failed to save binlog GTID state during heuristic " + "binlog rollback. "); + goto err; + } + return 0; + +err: + error= 1; + if (file >= 0) + { + end_io_cache(&log); + mysql_file_close(file, MYF(MY_WME)); + } + sql_print_error("Binlog truncation failed"); +#endif + return error; +} + int TC_LOG_BINLOG::open(const char *opt_name) { int error= 1; @@ -9526,7 +10085,7 @@ int TC_LOG_BINLOG::open(const char *opt_name) return 1; } - if (using_heuristic_recover()) + if (using_heuristic_recover(opt_name)) { mysql_mutex_lock(&LOCK_log); /* generate a new binlog to mask a corrupted one */ diff --git a/sql/log.h b/sql/log.h index 277e5c6f69c..c6179c27d8e 100644 --- a/sql/log.h +++ b/sql/log.h @@ -41,7 +41,7 @@ bool stmt_has_updated_non_trans_table(const THD* thd); class TC_LOG { public: - int using_heuristic_recover(); + int using_heuristic_recover(const char* opt_name); TC_LOG() {} virtual ~TC_LOG() {} @@ -692,6 +692,7 @@ class MYSQL_BIN_LOG: public TC_LOG, private MYSQL_LOG bool need_prepare_ordered, bool need_commit_ordered); int unlog(ulong cookie, my_xid xid); void commit_checkpoint_notify(void *cookie); + int heuristic_binlog_rollback(); int recover(LOG_INFO *linfo, const char *last_log_name, IO_CACHE *first_log, Format_description_log_event *fdle, bool do_xa); int do_binlog_recovery(const char *opt_name, bool do_xa_recovery); @@ -794,6 +795,10 @@ class MYSQL_BIN_LOG: public TC_LOG, private MYSQL_LOG int purge_first_log(Relay_log_info* rli, bool included); int set_purge_index_file_name(const char *base_file_name); int open_purge_index_file(bool destroy); + bool truncate_and_remove_binlogs(const char *truncate_file, + my_off_t truncate_pos); + int get_binlog_checkpoint_file(char* checkpoint_file); + bool is_binlog_truncate_safe(const char* checkpoint_file, my_off_t pos); bool is_inited_purge_index_file(); int close_purge_index_file(); int clean_purge_index_file(); diff --git a/storage/innobase/handler/ha_innodb.cc b/storage/innobase/handler/ha_innodb.cc index 4de2cdbeaec..42d655ee0e8 100644 --- a/storage/innobase/handler/ha_innodb.cc +++ b/storage/innobase/handler/ha_innodb.cc @@ -16493,6 +16493,9 @@ innobase_xa_recover( { DBUG_ASSERT(hton == innodb_hton_ptr); + mysql_bin_log.last_commit_pos_offset= trx_sys_mysql_bin_log_pos; + strmake_buf(mysql_bin_log.last_commit_pos_file, trx_sys_mysql_bin_log_name); + if (len == 0 || xid_list == NULL) { return(0); diff --git a/storage/xtradb/handler/ha_innodb.cc b/storage/xtradb/handler/ha_innodb.cc index 2aafb1a44ee..dec106cac54 100644 --- a/storage/xtradb/handler/ha_innodb.cc +++ b/storage/xtradb/handler/ha_innodb.cc @@ -17165,6 +17165,9 @@ innobase_xa_recover( { DBUG_ASSERT(hton == innodb_hton_ptr); + mysql_bin_log.last_commit_pos_offset= trx_sys_mysql_bin_log_pos; + strmake_buf(mysql_bin_log.last_commit_pos_file, trx_sys_mysql_bin_log_name); + if (len == 0 || xid_list == NULL) { return(0);

1 0

Re: [Maria-developers] 46c66698d63: MENT-566 jUnit xml fixes
by Sergei Golubchik 01 Apr '20

01 Apr '20

Hi, Rasmus! Please, see below On Mar 30, Rasmus Johansson wrote: > revision-id: 46c66698d63 (mariadb-10.2.31-189-g46c66698d63) > parent(s): 065a3d3eed7 > author: Rasmus Johansson <razze(a)iki.fi> > committer: Rasmus Johansson <razze(a)iki.fi> > timestamp: 2020-03-26 10:50:06 +0000 > message: > > MENT-566 jUnit xml fixes > > diff --git a/mysql-test/lib/mtr_report.pm b/mysql-test/lib/mtr_report.pm > index 3701ad79b15..1c6825d8fb7 100644 > --- a/mysql-test/lib/mtr_report.pm > +++ b/mysql-test/lib/mtr_report.pm > @@ -35,6 +37,7 @@ use My::Platform; > use POSIX qw[ _exit ]; > use IO::Handle qw[ flush ]; > use mtr_results; > +use Cwd 'abs_path'; this seems to be unused now > > use Term::ANSIColor; > > @@ -60,6 +63,8 @@ our $verbose; > our $verbose_restart= 0; > our $timer= 1; > > +my $xml_report = ""; this could be moved down inside if ($::opt_xml_report) > + > sub report_option { > my ($opt, $value)= @_; > > @@ -402,6 +409,92 @@ sub mtr_report_stats ($$$$) { > print "All $tot_tests tests were successful.\n\n"; > } > > + if ($::opt_xml_report) { > + my @sorted_tests = sort {$a->{'name'} cmp $b->{'name'}} @$tests; > + my $last_suite = ""; > + my $current_suite = ""; > + my $timest = isotime(time); > + my %suite_totals; > + my %suite_time; > + my %suite_tests; > + my %suite_failed; > + my %suite_disabled; > + my %suite_skipped; > + my $host = hostname; > + my $suiteNo = 0; > + > + # loop through test results to count totals > + foreach my $test ( @sorted_tests ) { > + $current_suite = $test->{'suite'}->{'name'}; > + > + if ($test->{'timer'} eq "") { > + $test->{'timer'} = 0; > + } > + > + # $suite_totals{$current_suite . "_time"} = $suite_totals{$current_suite . "_time"} + $test->{'timer'}; > + # $suite_totals{$current_suite . "_tests"} = $suite_totals{$current_suite . "_tests"} + 1; don't forget to remove all these commented-out lines before pushing > + $suite_time{$current_suite} = $suite_time{$current_suite} + $test->{'timer'}; > + $suite_tests{$current_suite} = $suite_tests{$current_suite} + 1; > + > + if ($test->{'result'} eq "MTR_RES_FAILED") { > + # $suite_totals{$current_suite . "_failed"} = $suite_totals{$current_suite . "_failed"} + 1; > + $suite_failed{$current_suite} = $suite_failed{$current_suite} + 1; > + } elsif ($test->{'result'} eq "MTR_RES_SKIPPED" && $test->{'disable'}) { > + # $suite_totals{$current_suite . "_disabled"} = $suite_totals{$current_suite . "_disabled"} + 1; > + $suite_disabled{$current_suite} = $suite_disabled{$current_suite} + 1; > + } elsif ($test->{'result'} eq "MTR_RES_SKIPPED") { > + $suite_skipped{$current_suite} = $suite_skipped{$current_suite} + 1; > + } > + > + $suite_totals{"all_time"} = $suite_totals{"all_time"} + $test->{'timer'}; > + } > + > + # generate xml > + $xml_report = "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n"; > + $xml_report = $xml_report . "<testsuites disabled=\"" . $tot_disabled . "\" errors=\"\" failures=\"" . $tot_failed . "\" name=\"\" tests=\"" . $tot_tests . "\" time=\"" . $suite_totals{"all_time"} . "\">\n"; I find it more concise to write $xml_report .= "<testsuites ..." instead of $xml_report = $xml_report . "<testsuites ..." and also it's easier to read, because the first form directly means "append to the variable", while the second means "concatenate two strings and store the result" and by using the target variable also as first string one gets the append effect. > + > + foreach my $test ( @sorted_tests ) { > + $current_suite = $test->{'suite'}->{'name'}; > + > + if ($current_suite ne $last_suite) { > + if ($last_suite ne "") { > + $xml_report = $xml_report . "\t</testsuite>\n"; > + $suiteNo++; > + } > + > + $xml_report = $xml_report . "\t<testsuite disabled=\"" . $suite_disabled{$current_suite} . "\" errors=\"\" failures=\"" . $suite_failed{$current_suite} . "\" hostname=\"" . $host . "\" id=\"" . $suiteNo . "\" name=\"" . $current_suite . "\" package=\"\" skipped=\"" . $suite_skipped{$current_suite} . "\" tests=\"" . $suite_tests{$current_suite} . "\" time=\"" . $suite_time{$current_suite} . "\" timestamp=\"" . $timest . "\">\n"; why do you always avoid variable interapolation? The above line could've been written as $xml_report .= qq(\t<testsuite disabled="$suite_disabled{$current_suite}" errors="" failures="$suite_failed{$current_suite}" hostname="$host" id="$suiteNo" name="$current_suite" package="" skipped="$suite_skipped{$current_suite}" tests="$suite_tests{$current_suite}" time="$suite_time{$current_suite}" timestamp="$timest">\n); which is notably shorter and more readable > + $last_suite = $current_suite; > + } > + > + $xml_report = $xml_report . "\t\t<testcase assertions=\"\" classname=\"" . $current_suite . "\" name=\"$test->{'name'}\" status=\"$test->{'result'}\" time=\"" . $test->{timer} . "\""; > + > + if ($test->{'result'} eq "MTR_RES_FAILED") { > + $xml_report = $xml_report . ">\n\t\t\t<failure message=\"\" type=\"" . $test->{'result'} . "\">\n<![CDATA[" . $test->{'logfile'} . "]]>\n\t\t\t</failure>\n\t\t</testcase>\n"; > + } elsif ($test->{'result'} eq "MTR_RES_SKIPPED" && $test->{'disable'}) { > + $xml_report = $xml_report . ">\n\t\t\t<failure message=\"disabled\" type=\"" . $test->{'result'} . "\"/>\n\t\t</testcase>\n"; > + } elsif ($test->{'result'} eq "MTR_RES_SKIPPED") { > + $xml_report = $xml_report . ">\n\t\t\t<failure message=\"skipped\" type=\"" . $test->{'result'} . "\"/>\n\t\t</testcase>\n"; > + } else { > + $xml_report = $xml_report . " />\n"; > + } > + > + # check if last test > + if ($test eq @sorted_tests[$#sorted_tests]) { > + $xml_report = $xml_report . "\t</testsuite>\n"; > + } better to print this after the loop, then to do a check for every test. it will not be wrong, because your condition $test eq @sorted_tests[$#sorted_tests] can only be true once, for the very last test. So you can as well remove the whole if() and put the unconditional $xml_report .= "\t</testsuite>\n"; after the loop. > + } > + > + $xml_report = $xml_report . "</testsuites>\n"; or just combine it with this one: $xml_report .= "\t</testsuite>\n</testsuites>\n"; > + > + # save to file > + # my $xml_file = $::opt_vardir . "/log/" . $::opt_xml_report; > + my $xml_file = $::opt_xml_report; > + > + open XML_FILE, ">" . $xml_file; error checking? Even if it's just open XML_FILE, ">" , $xml_file or die "$!"; and always use the 3-form open(). Where ">" is a separate argument, not concatenated to the file name (see how I wrote it above) > + print XML_FILE $xml_report; > + close XML_FILE; > + } > + > if (@$extra_warnings) > { > print <<MSG; > diff --git a/mysql-test/mysql-test-run.pl b/mysql-test/mysql-test-run.pl > index ef37cb4144d..298ba0015a3 100755 > --- a/mysql-test/mysql-test-run.pl > +++ b/mysql-test/mysql-test-run.pl > @@ -80,6 +80,7 @@ use lib "lib"; > > use Cwd ; > use Cwd 'realpath'; > +use Cwd 'abs_path'; this seems to be unused now > use Getopt::Long; > use My::File::Path; # Patched version of File::Path > use File::Basename; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 2

Re: [Maria-developers] 7740cb24572: Don't use plugin->data for storage engine plugins
by Sergei Golubchik 31 Mar '20

31 Mar '20

Hi, Sergey! On Mar 31, Sergey Vojtovich wrote: > revision-id: 7740cb24572 (mariadb-10.4.4-472-g7740cb24572) > parent(s): da6d7f72b0a > author: Sergey Vojtovich <svoj(a)mariadb.org> > committer: Sergey Vojtovich <svoj(a)mariadb.org> > timestamp: 2019-11-15 15:23:42 +0400 > message: > > Don't use plugin->data for storage engine plugins > > Use plugin->plugin->info->hton instead. > plugin_data() replaced with plugin_hton(). > plugin_hton() must never return NULL anymore and is only good to be called > against plugins in PLUGIN_IS_READY state. Why is that? Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

[Maria-developers] GSOC 2020 - MDEV-11263 Aggregate Window Functions
by Tavneet Sarna 30 Mar '20

30 Mar '20

Hi everyone, As part of GSOC 2020, this is one of the two projects I am interested in pursuing. As guided by last years' comments by Varun and Vicențiu, I have set up a debugger and corresponding breakpoints in the Item_sum_sum::add and Item_sum_sp::add for a custom sum aggregate function to understand the code flow. I had a couple of queries regarding the same: 1. In *do_add *from decimal.c, there are three parts with comments - /* part 1 - MY_MAX(frac) ... min (frac) */, /* part 2 - MY_MIN(frac) ... MY_MIN(intg) */. Can someone please elaborate on what do the comments mean ? 2. In *Item_sum_sum::add_helper*, there is an unlikely branch for variable direct_added. Can someone please give an idea about when will direct added be true ? In fact in all the uses for direct_added, it is always in an unlikely branch in Item_sum.cc. Regards Tavneet

2 1

[Maria-developers] Guidance regarding GSoC Projects.
by Arhant Jain 27 Mar '20

27 Mar '20

Hello Everyone, I'm interested in the work associated with these projects. I tried to explore them on my level. It will be very helpful if the concerned developers can direct some more information regarding these projects. 1. Switching off/on separate tests in mysql-test-run suite 2. Add a bitmap for NULL/empty data 3. Speed up RowGroup data access methods reducing level of indirection Thanks Arhant

1 0

[Maria-developers] Regarding GSoc :ADD FULL OUTER JOIN
by Jeeukrishnan Kayshyap. 26 Mar '20

26 Mar '20

Hi ,I am a GSoC aspirant . I am trying to solve https://jira.mariadb.org/browse/MDEV-13648 . Firstly , I discussed a new algorithm with Igor Babev sir , but it was wrong as it will create space problem. Now , I am planning to add some code in Sql_yacc.yy : /*full join*/ after the right join where /*full join */ will contain : 1.left join implementation (similar to the implementation of left join in sql_yacc.yy (line no. 11956-11950)) The result table will be in a temporary table ->tab1. 2.Use of right join doing the swapping of table (similar to right join in this part ;(line no. 11992-12030)) the result of this will be stored in another temporary table ->tab2 3.Union all clause: implementation of union all clause to combine tab1 and tab2. So , Oleksandr Byelkin sir replied to me that , addition of UNION will make it more complicated.it could be done with less effort than adding full UNION. But at present I am not able to get any idea how to reduce the complexity or any kind of new idea to implement it . Please guide me in the right direction and help me to solve this issue.

2 4

Re: [Maria-developers] fc90a39426c: MDEV-20604: Duplicate key value is silently truncated to 64 characters in print_keydup_error
by Sergei Golubchik 25 Mar '20

25 Mar '20

Hi, Oleksandr! Looks good. Thanks for tests. Ok to push. One comment, see below. On Mar 25, Oleksandr Byelkin wrote: > revision-id: fc90a39426c (mariadb-10.2.31-87-gfc90a39426c) > parent(s): fb74de97287 > author: Oleksandr Byelkin <sanja(a)mariadb.com> > committer: Oleksandr Byelkin <sanja(a)mariadb.com> > timestamp: 2020-03-24 10:34:14 +0100 > message: > > MDEV-20604: Duplicate key value is silently truncated to 64 characters in print_keydup_error > > Added indication of truncated string for "s" and "M" formats > > diff --git a/unittest/mysys/my_vsnprintf-t.c b/unittest/mysys/my_vsnprintf-t.c > index 6ba0a42cf7e..872e88ddd7e 100644 > --- a/unittest/mysys/my_vsnprintf-t.c > +++ b/unittest/mysys/my_vsnprintf-t.c > @@ -168,14 +184,14 @@ int main(void) > test1("M with positional: 0 \"Internal error/check (Not system error)\"", > "M with positional: %1$M", 0); > > - test1("M with width: 0 \"Internal error/ch", > + test1("M with width: 0 \"Internal error...", > "M with width: %.20M", 0); > - test1("M with width positional: 0 \"Internal error/ch", > + test1("M with width positional: 0 \"Internal error...", > "M with width positional: %2$.*1$M", 20, 0); > > - test_w_len("M small buf: 0 \"In", > + test_w_len("M small buf: 0 \"..", > 19, "M small buf: %M", 0); > - test_w_len("M small buf positional: 0 \"In", > + test_w_len("M small buf positional: 0 \"..", > 30, "M small buf positional: %1$M", 0); > > return exit_status(); We didn't discuss truncation when the buffer is too small, only truncation by width. It's not at all clear why truncation by small buffer should also use ellipsis. But it's not clear that it shouldn't either. I don't have a strong opinion in favor of either, so feel free to keep it your way. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] GSOC 2020 - MDEV-383
by Tavneet Sarna 25 Mar '20

25 Mar '20

Hi, I was looking into MDEV-383 <https://jira.mariadb.org/browse/MDEV-383> - "*Evaluate subquery predicates earlier or later depending on their SELECTIVITY*" as a possible GSOC 2020 project and I have a few queries about the same. 1. "There are a lot of subquery conditions out there that are inexpensive to evaluate and have good selectivity." - How is the cost of a subquery predicate calculated ? 2. One of the assumptions in MDEV-83 <https://jira.mariadb.org/browse/MDEV-83> was that "selectivity of a subquery predicate cannot be estimated during optimization." - what has changed since that we are looking into estimating selectivity ( or maybe my understanding of the task is incorrect so I would be grateful if you could just point me to any resources regarding this). 3. Igor had responded on the jira - "for starters, you could introduce an option that would allow not to push correlated subquery predicates to joined tables." - did you mean a command line argument/option and I don't get how this would help with the main task. 4. Also, can anyone please point me to the classes where predicate cost and join plans are evaluated ? Apologies if the queries are a bit rudimentary. Regards Tavneet

1 0

[Maria-developers] Regarding GSoc :ADD FULL OUTER JOIN
by Jeeukrishnan Kayshyap. 24 Mar '20

24 Mar '20

HI , to implement FULL OUTER JOIN between T1 and T2 can we do "T1 Left Outer Join T2 union all T2 Anti Join T1" ?? Is this method already tried ? Please guide me to solve this issue and if possible someone please reply to my last email.

1 0

[Maria-developers] MDEV-14296: Moving wsrep_sst_common to /usr/libexec
by Vicențiu Ciorbaru 23 Mar '20

23 Mar '20

Hi Otto! I am doing a push towards cleaning up as much as possible from our packaging issues. Starting with the older ones: *Background:* wsrep_sst_common.sh is a script that should never be run standalone, it is rather a helper script used for other wsrep_* scripts. As such, it *does not really belong in /usr/bin*. See MDEV-14296 [1] for more details. <https://jira.mariadb.org/browse/MDEV-14296> It looks like rpm packagers - read Fedora - [1a] require us to ship wsrep_sst_common in */usr/libexec* (or alternatively /usr/share, but /usr/libexec makes much more sense semantically to me, after reading the FHS [2]). There is PR #603 [3] from Daniel Black, which has some missing changes, which I've done myself, namely updating the debian/*.install files to put the file in the correct location and a change in path for our apparmor profile [4] [5]. *My question for you is*: Is this acceptable to do in 10.1 (merge PR 603 + commits [4] [5] into 10.1) from a packaging point of view? The debian packaging policy here [6], allows one to follow FSH version 3.0 and does not mention any exception for libexec. <https://www.debian.org/doc/debian-policy/ch-opersys.html> I have tested this and I have not encountered any regressions while upgrading and the script seems to function as intended (although I am not an expert DBA to make use of wsrep_* scripts). Thanks, Vicențiu [1] https://jira.mariadb.org/browse/MDEV-14296 [1a] https://src.fedoraproject.org/rpms/mariadb/blob/ecb40d449c382ea7e4052c75d5d… [2] https://refspecs.linuxfoundation.org/FHS_3.0/fhs/ch04s07.html [3] https://github.com/MariaDB/server/pull/603 [4] https://github.com/MariaDB/server/commit/8042265659656c312f39304a5dc60de708… [5] https://github.com/MariaDB/server/commit/dbed51d86e4aae9b8a54d6b5aa31c974fe… [6] https://www.debian.org/doc/debian-policy/ch-opersys.html

1 0

[Maria-developers] Regarding GSoc :ADD FULL OUTER JOIN
by Jeeukrishnan Kayshyap. 23 Mar '20

23 Mar '20

Hi , I studied the join_table part of sql_yacc.yy and the the convert_right_join() part from sql_parse.cc . Sir to add the full outer join : if we add a /*Full outer join variant */ , where first we will do left outer join and then right outer join in that , after that we can use the "UNION" part in /*Full outer join variant */ to join the left outer join and right outer join table to get the full outer join table. Will this approach gonna work Sir? please guide me for this issue.Is it completely wrong or need some modification?

2 1

[Maria-developers] Regarding GSoC : ADD FULL OUTER JOIN
by Jeeukrishnan Kayshyap. 23 Mar '20

23 Mar '20

What are the other thing we need to consider if we include *st_select_lex::convert_full_join() similar to *st_select_lex::convert_right_join() ?

1 0

Re: [Maria-developers] 4b142536a24: MDEV-20604: Duplicate key value is silently truncated to 64 characters in print_keydup_error
by Sergei Golubchik 22 Mar '20

22 Mar '20

Hi, Oleksandr! On Mar 22, Oleksandr Byelkin wrote: > revision-id: 4b142536a24 (mariadb-10.2.31-56-g4b142536a24) > parent(s): ed21202a14e > author: Oleksandr Byelkin <sanja(a)mariadb.com> > committer: Oleksandr Byelkin <sanja(a)mariadb.com> > timestamp: 2020-03-18 08:25:13 +0100 > message: > > MDEV-20604: Duplicate key value is silently truncated to 64 characters in print_keydup_error > > Added indication of truncated string with special width/precission suffix ':'. The comment looks wrong :) > diff --git a/client/mysql_upgrade.c b/client/mysql_upgrade.c > index 0bbb8a149ff..003701514cf 100644 > --- a/client/mysql_upgrade.c > +++ b/client/mysql_upgrade.c > @@ -501,10 +499,10 @@ static void find_tool(char *tool_executable_name, const char *tool_name, > last_fn_libchar -= 6; > } > > - len= (int)(last_fn_libchar - self_name); > - > - my_snprintf(tool_executable_name, FN_REFLEN, "%.*s%c%s", > - len, self_name, FN_LIBCHAR, tool_name); > + last_fn_libchar[0]= 0; > + my_snprintf(tool_executable_name, FN_REFLEN, "%s%c%s", > + self_name, FN_LIBCHAR, tool_name); > + last_fn_libchar[0]= FN_LIBCHAR; Okay, but why not to use %b here? > } > > if (opt_verbose) > diff --git a/client/mysqltest.cc b/client/mysqltest.cc > index 60a203ccedd..045ab566fdb 100644 > --- a/client/mysqltest.cc > +++ b/client/mysqltest.cc > @@ -9534,6 +9533,7 @@ int main(int argc, char **argv) > case Q_LET: do_let(command); break; > case Q_EVAL_RESULT: > die("'eval_result' command is deprecated"); > + break; // never called but keep compiler calm wouldn't it be better to specify __attribute__ ((noreturn)) for die()? > case Q_EVAL: > case Q_EVALP: > case Q_QUERY_VERTICAL: > diff --git a/strings/my_vsnprintf.c b/strings/my_vsnprintf.c > index ad3517e4252..9ae10f337cc 100644 > --- a/strings/my_vsnprintf.c > +++ b/strings/my_vsnprintf.c > @@ -28,7 +28,9 @@ > #define LENGTH_ARG 1 > #define WIDTH_ARG 2 > #define PREZERO_ARG 4 > -#define ESCAPED_ARG 8 > +#define ESCAPED_ARG 8 > +// Show truncation of string argument > +#define SH_TRUNC_ARG 16 you don't really need a flag, because process_str_arg should always do it. By the way, in your code it doesn't. It doesn't print dots for %M and for %1$s. > > typedef struct pos_arg_info ARGS_INFO; > typedef struct print_info PRINT_INFO; > @@ -198,18 +199,42 @@ static char *process_str_arg(CHARSET_INFO *cs, char *to, const char *end, > size_t width, char *par, uint print_type) > { > int well_formed_error; > - size_t plen, left_len= (size_t) (end - to) + 1; > + uint dots= 0; > + size_t plen, left_len= (size_t) (end - to) + 1, slen=0; > if (!par) > par = (char*) "(null)"; > > - plen= strnlen(par, width); > + plen= slen= strnlen(par, width + 1); > + if (plen > width) > + plen= width; > if (left_len <= plen) > plen = left_len - 1; > + if ((slen > plen) && (print_type & SH_TRUNC_ARG)) > + { > + if (plen < 3) > + { > + dots= plen; > + plen= 0; > + } > + else > + { > + plen-= 3; > + dots= 3; > + } > + } > + > plen= my_well_formed_length(cs, par, par + plen, width, &well_formed_error); > if (print_type & ESCAPED_ARG) > to= backtick_string(cs, to, end, par, plen, '`'); > else > to= strnmov(to,par,plen); > + > + if (dots) > + { > + for (; dots; dots--) > + *(to++)= '.'; > + *(to)= 0; > + } This should look strange when truncating a quoted string. See the test case below > return to; > } > > diff --git a/unittest/mysys/my_vsnprintf-t.c b/unittest/mysys/my_vsnprintf-t.c > index 6ba0a42cf7e..e8831f7c1f4 100644 > --- a/unittest/mysys/my_vsnprintf-t.c > +++ b/unittest/mysys/my_vsnprintf-t.c > @@ -99,12 +99,12 @@ int main(void) > test1("Width is ignored for strings <x> <y>", > "Width is ignored for strings <%04s> <%5s>", "x", "y"); > > - test1("Precision works for strings <abcde>", > + test1("Precision works for strings <ab...>", > "Precision works for strings <%.5s>", "abcdef!"); > > - test1("Flag '`' (backtick) works: `abcd` `op``q` (mysql extension)", This > - "Flag '`' (backtick) works: %`s %`.4s (mysql extension)", This > - "abcd", "op`qrst"); > + test1("Flag '`' (backtick) works: `abcd` `op``q`... (mysql extension)", > + "Flag '`' (backtick) works: %`s %`.7s (mysql extension)", > + "abcd", "op`qrstuuuuuuuuu"); Here, the test case with a quoted string. First, you print 10 characters instead of 7. Second, you put dots after the closing backtick, which looks confusing, the truncation happens before the backtick. Third, as you can see, you cannot use strlen() for a string length because there can be backticks inside the string. perhaps it would be simpler to move ... inside backtick_string() > > test1("Length modifiers work: 1 * -1 * 2 * 3", > "Length modifiers work: %d * %ld * %lld * %zd", 1, -1L, 2LL, (size_t)3); > @@ -125,7 +125,7 @@ int main(void) > test1("Asterisk '*' as a width works: < 4>", > "Asterisk '*' as a width works: <%*d>", 5, 4); > > - test1("Asterisk '*' as a precision works: <qwerty>", > + test1("Asterisk '*' as a precision works: <qwe...>", > "Asterisk '*' as a precision works: <%.*s>", 6, "qwertyuiop"); please add tests for ...truncation in %M and %1$s > > test1("Positional arguments for a width: < 4>", > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] cf70893ac9d: MDEV-21303 Make executables MariaDB named
by Sergei Golubchik 18 Mar '20

18 Mar '20

Hi, Rasmus! See my comments/questions below: On Mar 16, Rasmus Johansson wrote: > revision-id: cf70893ac9d (mariadb-10.5.0-391-gcf70893ac9d) > parent(s): 56402e84b5b > author: Rasmus Johansson <razze(a)iki.fi> > committer: Rasmus Johansson <razze(a)iki.fi> > timestamp: 2020-03-16 11:10:25 +0000 > message: > > MDEV-21303 Make executables MariaDB named > > To change all executables to have a mariadb name I had to: > - Do name changes in every CMakeLists.txt that produces executables > - CREATE_MARIADB_SYMLINK was removed and GET_SYMLINK added by Wlad to reuse the function in other places also > - The scripts/CMakeLists.txt could make use of GET_SYMLINK instead of introducing redundant code, but I thought I'll leave that for next release > - A lot of changes to debian/.install and debian/.links files due to swapping of real executable and symlink. I did not however change the name of the manpages, so the real name is still mysql there and mariadb are symlinks. > - The Windows part needed a change now when we made the executables mariadb -named. MSI (and ZIP) do not support symlinks and to not break backward compatibility we had to include mysql named binaries also. Done by Wlad > > diff --git a/cmake/mysql_add_executable.cmake b/cmake/mysql_add_executable.cmake > index eec370d51af..f4d71ae9cef 100644 > --- a/cmake/mysql_add_executable.cmake > +++ b/cmake/mysql_add_executable.cmake > @@ -48,7 +48,7 @@ FUNCTION (MYSQL_ADD_EXECUTABLE) > ENDIF() > > IF (ARG_WIN32) > - SET(WIN32 WIN32) > + SET(WIN32 ARG_WIN32) This looks wrong, see below > ELSE() > UNSET(WIN32) > ENDIF() > @@ -62,6 +62,7 @@ FUNCTION (MYSQL_ADD_EXECUTABLE) > ELSE() > UNSET(EXCLUDE_FROM_ALL) > ENDIF() > + > ADD_EXECUTABLE(${target} ${WIN32} ${MACOSX_BUNDLE} ${EXCLUDE_FROM_ALL} ${sources}) here it'll pass ARG_WIN32 to ADD_EXECUTABLE. But ADD_EXECUTABLE takes an optional WIN32 keyword, not ARG_WIN32. > > # tell CPack where to install > @@ -79,16 +80,49 @@ FUNCTION (MYSQL_ADD_EXECUTABLE) > IF (COMP MATCHES ${SKIP_COMPONENTS}) > RETURN() > ENDIF() > + > IF (WITH_STRIPPED_CLIENT AND NOT target STREQUAL mysqld) this should be `STREQUAL mariadbd` now, I believe. > INSTALL(CODE "SET(CMAKE_INSTALL_DO_STRIP 1)" COMPONENT ${COMP}) > SET(reset_strip ON) > ENDIF() > - MYSQL_INSTALL_TARGETS(${target} DESTINATION ${ARG_DESTINATION} COMPONENT ${COMP}) > + > + IF(NOT ${mariadbname} STREQUAL "") where is mariadbname set? > + MYSQL_INSTALL_TARGETS(${mariadbname} DESTINATION ${ARG_DESTINATION} COMPONENT ${COMP}) > + ELSE() > + MYSQL_INSTALL_TARGETS(${target} DESTINATION ${ARG_DESTINATION} COMPONENT ${COMP}) > + ENDIF() > + > IF (reset_strip) > INSTALL(CODE "SET(CMAKE_INSTALL_DO_STRIP 0)" COMPONENT ${COMP}) > ENDIF() > ENDIF() > > - # create mariadb named symlink > - CREATE_MARIADB_SYMLINK(${target} ${ARG_DESTINATION} ${COMP}) > + # create MySQL named "legacy links" > + > + # Windows note: > + # Here, hardlinks are used, because cmake can't install symlinks. > + # In packages, there are won't be links, just copies. better to put the comment below in the ELSE branch, not here, far from the code that it is supposed to explain. > + GET_SYMLINK(${target} link) > + IF(link) > + IF(UNIX) > + ADD_CUSTOM_COMMAND(TARGET ${target} POST_BUILD > + COMMAND ${CMAKE_COMMAND} -E create_symlink > + ${target} ${link} > + COMMENT "Creating ${link} link" > + WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR}/${CMAKE_CFG_INTDIR}) > + INSTALL(PROGRAMS > + ${CMAKE_CURRENT_BINARY_DIR}/${CMAKE_CFG_INTDIR}/${link} > + DESTINATION > + ${ARG_DESTINATION} > + COMPONENT ${COMP}) > + ELSE() > + SET(link ${link}.exe) > + ADD_CUSTOM_COMMAND(TARGET ${target} POST_BUILD > + COMMAND cmake -E remove -f ${link} > + COMMAND mklink /H ${link} $<TARGET_FILE_NAME:${target}> > + COMMENT "Creating ${link} link" > + WORKING_DIRECTORY $<TARGET_FILE_DIR:${target}>) > + INSTALL(PROGRAMS $<TARGET_FILE_DIR:${target}>/${link} DESTINATION ${ARG_DESTINATION} COMPONENT ${COMP}) > + ENDIF() > + ENDIF() > ENDFUNCTION() > diff --git a/cmake/symlinks.cmake b/cmake/symlinks.cmake > index ec638bc82de..e040ff19f77 100644 > --- a/cmake/symlinks.cmake > +++ b/cmake/symlinks.cmake > @@ -9,68 +9,46 @@ macro(REGISTER_SYMLINK from to) > endmacro() > > # MariaDB names for executables > -REGISTER_SYMLINK("mysql_client_test_embedded" "mariadb-client-test-embedded") > -REGISTER_SYMLINK("mysql_client_test" "mariadb-client-test") > +REGISTER_SYMLINK("mariadb-client-test-embedded" "mysql_client_test_embedded") > +REGISTER_SYMLINK("mariadb-client-test" "mysql_client_test") ... I suppose you need to make sure that everything works with new names, without symlinks. That is all internal tools and scripts invoke other tools and scripts by their mariadb* names. A way to do it could be to comment out all REGISTER_SYMLINK lines and fix whatever that will break. grep is also a useful tool here, but with rather low signal-to-noise ratio. > + > +MACRO(GET_SYMLINK name out) > + set(${out}) > + list(FIND MARIADB_SYMLINK_FROMS ${name} _index) > if (${_index} GREATER -1) > - list(GET MARIADB_SYMLINK_TOS ${_index} mariadbname) > - endif() > - > - if (mariadbname) > - CREATE_MARIADB_SYMLINK_IN_DIR(${src} ${mariadbname} ${dir} ${comp}) > - endif() > -endmacro(CREATE_MARIADB_SYMLINK) > - > -# Add MariaDB symlinks in directory > -macro(CREATE_MARIADB_SYMLINK_IN_DIR src dest dir comp) > - if(UNIX) > - add_custom_target( > - SYM_${dest} ALL > - DEPENDS ${CMAKE_CURRENT_BINARY_DIR}/${dest} > - ) > - > - add_custom_command(OUTPUT ${CMAKE_CURRENT_BINARY_DIR}/${dest} POST_BUILD > - COMMAND ${CMAKE_COMMAND} -E create_symlink ${src} ${dest} > - COMMENT "mklink ${src} -> ${dest}") > - > - install(FILES ${CMAKE_CURRENT_BINARY_DIR}/${dest} DESTINATION ${dir} COMPONENT ${comp}) > + list(GET MARIADB_SYMLINK_TOS ${_index} ${out}) > endif() > -endmacro(CREATE_MARIADB_SYMLINK_IN_DIR) > +ENDMACRO() > diff --git a/debian/mariadb-backup.install b/debian/mariadb-backup.install > index 2bc61c12079..e1967495b00 100644 > --- a/debian/mariadb-backup.install > +++ b/debian/mariadb-backup.install > @@ -1,5 +1,4 @@ > -usr/bin/mariabackup > +usr/bin/mariadb-backup > usr/bin/mbstream > usr/share/man/man1/mariabackup.1 > -usr/share/man/man1/mariadb-backup.1 this is strange. Why did you remove mariadb-backup.1 and not mariabackup.1 ? > usr/share/man/man1/mbstream.1 > diff --git a/debian/mariadb-client-10.5.install b/debian/mariadb-client-10.5.install > index 67c0c2619c3..216e4015851 100644 > --- a/debian/mariadb-client-10.5.install > +++ b/debian/mariadb-client-10.5.install > @@ -1,29 +1,17 @@ > debian/additions/innotop/innotop usr/bin/ > debian/additions/mysqlreport usr/bin/ > +usr/bin/mariadb-access > +usr/bin/mariadb-admin > usr/bin/mariadb-conv > -usr/bin/mysql_find_rows > -usr/bin/mysql_fix_extensions > -usr/bin/mysql_waitpid > -usr/bin/mysqlaccess > -usr/bin/mysqladmin > -usr/bin/mysqldump > -usr/bin/mysqldumpslow > -usr/bin/mysqlimport > -usr/bin/mysqlshow you removed mysqlshow here, and added mariadb-show to the server package. Intentional? UPD: I see that before your patch mysqlshow was in the client package, while mariadb-show symlink was in the server package. It's a bug, both should be in the client package. > +usr/bin/mariadb-dump > +usr/bin/mariadb-dumpslow > +usr/bin/mariadb-find-rows > +usr/bin/mariadb-fix-extensions > +usr/bin/mariadb-import > +usr/bin/mariadb-slap > +usr/bin/mariadb-waitpid > usr/bin/mysqlslap you did not remove mysqlslap, but added mariadb-slap > usr/bin/mytop > -usr/share/man/man1/mariadb-access.1 > -usr/share/man/man1/mariadb-admin.1 > -usr/share/man/man1/mariadb-binlog.1 > -usr/share/man/man1/mariadb-conv.1 > -usr/share/man/man1/mariadb-dump.1 > -usr/share/man/man1/mariadb-dumpslow.1 > -usr/share/man/man1/mariadb-find-rows.1 > -usr/share/man/man1/mariadb-fix-extensions.1 > -usr/share/man/man1/mariadb-import.1 > -usr/share/man/man1/mariadb-plugin.1 > -usr/share/man/man1/mariadb-slap.1 > -usr/share/man/man1/mariadb-waitpid.1 same weirdness as elsewhere, mariadb*.1 man pages are symlinks to mysql*.1 manpages. Should be vice versa. > usr/share/man/man1/mysql_find_rows.1 > usr/share/man/man1/mysql_fix_extensions.1 > usr/share/man/man1/mysql_waitpid.1 > diff --git a/debian/mariadb-client-10.5.links b/debian/mariadb-client-10.5.links > index 5d966575b76..d1bdfcfa3c8 100644 > --- a/debian/mariadb-client-10.5.links > +++ b/debian/mariadb-client-10.5.links > @@ -1,21 +1,19 @@ > -usr/bin/mysql_find_rows usr/bin/mariadb-find-rows > -usr/bin/mysql_fix_extensions usr/bin/mariadb-fix-extensions > -usr/bin/mysql_plugin usr/bin/mariadb-plugin > -usr/bin/mysql_waitpid usr/bin/mariadb-waitpid > -usr/bin/mysqlaccess usr/bin/mariadb-access > -usr/bin/mysqladmin usr/bin/mariadb-admin > -usr/bin/mysqlbinlog usr/bin/mariadb-binlog you removed mysqlbinlog, but added mariadb-binlog to the server package. Intentional? same for mysql_plugin/mariadb-plugin. UPD: mysqlbinlog was in the server package, while mariadb-binlog symlink was in the client package. So your change is correct, both will be in the server package now. Good. same for mysql_plugin/mariadb-plugin. > -usr/bin/mysqlcheck usr/bin/mariadb-analyze > -usr/bin/mysqlcheck usr/bin/mariadb-optimize > diff --git a/debian/mariadb-plugin-rocksdb.install b/debian/mariadb-plugin-rocksdb.install > index 80987612c30..0fc868a0721 100644 > --- a/debian/mariadb-plugin-rocksdb.install > +++ b/debian/mariadb-plugin-rocksdb.install > @@ -2,6 +2,5 @@ etc/mysql/conf.d/rocksdb.cnf etc/mysql/mariadb.conf.d > usr/bin/myrocks_hotbackup > usr/bin/mysql_ldb you didn't replace mysql_ldb with mariadb-ldb? > usr/lib/mysql/plugin/ha_rocksdb.so > -usr/share/man/man1/mariadb-ldb.1 > usr/share/man/man1/myrocks_hotbackup.1 > usr/share/man/man1/mysql_ldb.1 > diff --git a/debian/mariadb-server-10.5.install b/debian/mariadb-server-10.5.install > index 4a860ff57df..38d9f984276 100644 > --- a/debian/mariadb-server-10.5.install > +++ b/debian/mariadb-server-10.5.install > @@ -14,22 +14,23 @@ usr/bin/aria_pack > usr/bin/aria_read_log > usr/bin/galera_new_cluster > usr/bin/galera_recovery > +usr/bin/mariadb-binlog > +usr/bin/mariadb-convert-table-format > +usr/bin/mariadb-hotcopy > +usr/bin/mariadb-plugin > +usr/bin/mariadb-secure-installation > usr/bin/mariadb-service-convert > +usr/bin/mariadb-setpermission > +usr/bin/mariadb-show > +usr/bin/mariadb-tzinfo-to-sql > +usr/bin/mariadbd-multi > +usr/bin/mariadbd-safe > +usr/bin/mariadbd-safe duplicate line ^^^ > usr/bin/msql2mysql > usr/bin/myisam_ftdump > usr/bin/myisamchk > usr/bin/myisamlog > usr/bin/myisampack > -usr/bin/mysql_convert_table_format > -usr/bin/mysql_plugin > -usr/bin/mysql_secure_installation > -usr/bin/mysql_setpermission > -usr/bin/mysql_tzinfo_to_sql > -usr/bin/mysqlbinlog > -usr/bin/mysqld_multi > -usr/bin/mysqld_safe > -usr/bin/mysqld_safe_helper you don't install mariadbd-safe-helper. Perhaps that's where your duplicate line comes from, it's not a duplicate, it's truncated :) > -usr/bin/mysqlhotcopy > usr/bin/perror > usr/bin/replace > usr/bin/resolve_stack_dump > diff --git a/debian/mariadb-server-10.5.links b/debian/mariadb-server-10.5.links > index f2d97460371..e3a2d68541a 100644 > --- a/debian/mariadb-server-10.5.links > +++ b/debian/mariadb-server-10.5.links > @@ -1,9 +1,20 @@ > -usr/bin/mysql_convert_table_format usr/bin/mariadb-convert-table-format > -usr/bin/mysql_secure_installation usr/bin/mariadb-secure-installation > -usr/bin/mysql_setpermission usr/bin/mariadb-setpermission > -usr/bin/mysql_tzinfo_to_sql usr/bin/mariadb-tzinfo-to-sql > -usr/bin/mysqld_multi usr/bin/mariadbd-multi > -usr/bin/mysqld_safe usr/bin/mariadbd-safe > -usr/bin/mysqld_safe_helper usr/bin/mariadbd-safe-helper > -usr/bin/mysqlhotcopy usr/bin/mariadb-hotcopy > -usr/bin/mysqlshow usr/bin/mariadb-show > +usr/bin/mariadb-binlog usr/bin/mysqlbinlog > +usr/bin/mariadb-convert-table-format usr/bin/mysql_convert_table_format > +usr/bin/mariadb-hotcopy usr/bin/mysqlhotcopy > +usr/bin/mariadb-plugin usr/bin/mysql_plugin > +usr/bin/mariadb-secure-installation usr/bin/mysql_secure_installation > +usr/bin/mariadb-setpermission usr/bin/mysql_setpermission > +usr/bin/mariadb-show /usr/bin/mysqlshow this is the only line in your whole patch where you symlink by the absolute path. Looks like a typo. > +usr/bin/mariadb-tzinfo-to-sql usr/bin/mysql_tzinfo_to_sql > +usr/bin/mariadbd-multi usr/bin/mysqld_multi > +usr/bin/mariadbd-safe usr/bin/mysqld_safe > +usr/bin/mariadbd-safe-helper usr/bin/mysqld_safe_helper > diff --git a/libmariadb b/libmariadb > index 3be5897c334..ca68b114ba7 160000 > --- a/libmariadb > +++ b/libmariadb > @@ -1 +1 @@ > -Subproject commit 3be5897c3346639fa6d7195480d93108798c4917 > +Subproject commit ca68b114ba7e882e8abe8888204d675c4d8751f8 This is clearly wrong, please make sure you've removed it from your commit before pushing! > diff --git a/scripts/CMakeLists.txt b/scripts/CMakeLists.txt > index 7be46ac1985..350fbd14cf6 100644 > --- a/scripts/CMakeLists.txt > +++ b/scripts/CMakeLists.txt > @@ -244,8 +232,20 @@ SET(mysqlaccess_COMPONENT COMPONENT Client) > SET(mysql_find_rows_COMPONENT COMPONENT Client) > SET(mytop_COMPONENT Mytop) > > +MACRO(INSTALL_LINK old new destination component) > + EXECUTE_PROCESS( > + COMMAND ${CMAKE_COMMAND} -E create_symlink ${old} ${new} > + WORKING_DIRECTORY ${CMAKE_CURRENT_BINARY_DIR} > + ) > + INSTALL( > + PROGRAMS ${CMAKE_CURRENT_BINARY_DIR}/${new} > + DESTINATION ${destination} > + COMPONENT ${component} > + ) > +ENDMACRO() > + > IF(WIN32) > - # On Windows, some .sh and some .pl.in files are configured > + # On Windows, some .sh and some .pl.in files are configured > # The resulting files will have .pl extension (those are perl scripts) > > # Input files with pl.in extension > @@ -296,60 +296,73 @@ ELSE() > # On Unix, most of the files end up in the bin directory > SET(BIN_SCRIPTS > msql2mysql > - mysql_config > - mysql_setpermission > - mysql_secure_installation > - mysqlaccess > - mysql_convert_table_format > - mysql_find_rows > + mariadb-setpermission > + mariadb-secure-installation > + mariadb-access > + mariadb-convert-table-format > + mariadb-find-rows > mytop > - mysqlhotcopy > + mariadb-hotcopy > + mariadb-install-db > ${SERVER_SCRIPTS} > ${WSREP_SCRIPTS} > ${SYSTEMD_SCRIPTS} > ) > + > FOREACH(file ${BIN_SCRIPTS}) > - IF(EXISTS ${CMAKE_CURRENT_SOURCE_DIR}/${file}.sh) > - CONFIGURE_FILE(${CMAKE_CURRENT_SOURCE_DIR}/${file}.sh > - ${CMAKE_CURRENT_BINARY_DIR}/${file} ESCAPE_QUOTES @ONLY) > - ELSEIF(EXISTS ${CMAKE_CURRENT_SOURCE_DIR}/${file}) > - CONFIGURE_FILE(${CMAKE_CURRENT_SOURCE_DIR}/${file} > - ${CMAKE_CURRENT_BINARY_DIR}/${file} COPYONLY) > - ELSE() > - MESSAGE(FATAL_ERROR "Can not find ${file}.sh or ${file} in " > - "${CMAKE_CURRENT_SOURCE_DIR}" ) > - ENDIF() > - # TODO: The following EXECUTE could be redundant as INSTALL_SCRIPT > - # macro does an INSTALL(PROGRAMS ..) that automatically sets +x on > - # the executable. > - EXECUTE_PROCESS(COMMAND chmod +x ${CMAKE_CURRENT_BINARY_DIR}/${file}) > - IF(NOT ${file}_COMPONENT) > - SET(${file}_COMPONENT Server) > - ENDIF() > - INSTALL_SCRIPT( > - ${CMAKE_CURRENT_BINARY_DIR}/${file} > - DESTINATION ${INSTALL_BINDIR} > - COMPONENT ${${file}_COMPONENT} > - ) > + # set name of executable > + list(FIND MARIADB_SYMLINK_FROMS ${file} _index) > + > + if(${_index} GREATER -1) > + list(GET MARIADB_SYMLINK_TOS ${_index} binname) > + else() > + set(binname ${file}) > + endif() 1. why are you not using GET_SYMLINK ? 2. Why INSTALL_SYMLINK here instead of the old CREATE_MARIADB_SYMLINK ? You avoid creating new targets for script symlinks, but who cares about that? > + > + if(NOT "${binname}" STREQUAL "") how can ${binname} be "" ? > + IF(EXISTS ${CMAKE_CURRENT_SOURCE_DIR}/${binname}.sh) > + CONFIGURE_FILE(${CMAKE_CURRENT_SOURCE_DIR}/${binname}.sh > + ${CMAKE_CURRENT_BINARY_DIR}/${file} ESCAPE_QUOTES @ONLY) > + ELSEIF(EXISTS ${CMAKE_CURRENT_SOURCE_DIR}/${binname}) > + CONFIGURE_FILE(${CMAKE_CURRENT_SOURCE_DIR}/${binname} > + ${CMAKE_CURRENT_BINARY_DIR}/${file} COPYONLY) > + ELSE() > + MESSAGE(FATAL_ERROR "Can not find ${binname}.sh or ${binname} in " > + "${CMAKE_CURRENT_SOURCE_DIR}" ) > + ENDIF() > + > + IF(NOT ${file}_COMPONENT) > + SET(${file}_COMPONENT Server) > + ENDIF() > + INSTALL_SCRIPT( > + ${CMAKE_CURRENT_BINARY_DIR}/${file} > + DESTINATION ${INSTALL_BINDIR} > + COMPONENT ${${file}_COMPONENT} > + ) > + > + # Create symlink > + if (NOT ${binname} STREQUAL ${file}) > + INSTALL_LINK(${file} ${binname} ${INSTALL_BINDIR} ${${file}_COMPONENT}) > + > + # Place mysql_install_db symlink also in scripts > + if (${binname} STREQUAL "mysql_install_db") > + SET(bindir ${bindir1}) > + SET(prefix ${prefix1}) > + SET(sbindir ${sbindir1}) > + SET(scriptdir ${scriptdir1}) > + SET(libexecdir ${libexecdir1}) > + SET(pkgdatadir ${pkgdatadir1}) > + SET(pkgplugindir ${pkgplugindir1}) > + SET(localstatedir ${localstatedir1}) > + CONFIGURE_FILE(${CMAKE_CURRENT_SOURCE_DIR}/mysql_install_db.sh ${CMAKE_CURRENT_BINARY_DIR}/mariadb-install-db ESCAPE_QUOTES @ONLY) > + INSTALL(PROGRAMS ${CMAKE_CURRENT_BINARY_DIR}/${file} DESTINATION ${INSTALL_SCRIPTDIR} COMPONENT Server) > + INSTALL(PROGRAMS ${CMAKE_CURRENT_BINARY_DIR}/${binname} DESTINATION ${INSTALL_SCRIPTDIR} COMPONENT Server) I don't understand that at all. 1. What does the comment "Place mysql_install_db symlink also in scripts" even mean? 2. you CONFIGURE_FILE mariadb-install-db twice. 3. Why two INSTALL(PROGRAM ? 4. You set all bindir/etc variables for mysql_install_db and it will affect CONFIGURE_FILE for all following scripts 5. why do you even compare ${binname} with mysql_install_db and not ${file} with mariadb-install-db ? > + endif() > + endif() > + endif() we customary always use uppercase in all cmake keywords. > ENDFOREACH() > > - SET (wsrep_sst_rsync_wan ${CMAKE_CURRENT_BINARY_DIR}/wsrep_sst_rsync_wan) > - ADD_CUSTOM_COMMAND( > - OUTPUT ${wsrep_sst_rsync_wan} > - COMMAND ${CMAKE_COMMAND} ARGS -E create_symlink > - wsrep_sst_rsync > - wsrep_sst_rsync_wan > - ) > - ADD_CUSTOM_TARGET(symlink_wsrep_sst_rsync > - ALL > - DEPENDS ${wsrep_sst_rsync_wan} > - ) > - INSTALL( > - FILES ${wsrep_sst_rsync_wan} > - DESTINATION ${INSTALL_BINDIR} > - COMPONENT Server > - ) > - > + INSTALL_LINK(wsrep_sst_rsync wsrep_sst_rsync_wan ${INSTALL_BINDIR} Server) > FOREACH(file ${WSREP_SOURCE}) > CONFIGURE_FILE(${CMAKE_CURRENT_SOURCE_DIR}/${file}.sh > ${CMAKE_CURRENT_BINARY_DIR}/${file} ESCAPE_QUOTES @ONLY) > diff --git a/win/packaging/ca/CMakeLists.txt b/win/packaging/ca/CMakeLists.txt > index 99dc7da01fb..326bab47de4 100644 > --- a/win/packaging/ca/CMakeLists.txt > +++ b/win/packaging/ca/CMakeLists.txt I didn't review windows packaging, it's between you and wlad. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 5365db70cb8: Improve update handler (long unique keys on blobs)
by Sergei Golubchik 16 Mar '20

16 Mar '20

Hi, Michael! On Feb 29, Michael Widenius wrote: > commit 5365db70cb8 > Author: Michael Widenius <monty(a)mariadb.com> > Date: Mon Jan 13 18:30:13 2020 +0200 > > Improve update handler (long unique keys on blobs) > > MDEV-21606 Improve update handler (long unique keys on blobs) > MDEV-21470 MyISAM and Aria start_bulk_insert doesn't work with long unique > MDEV-21606 Bug fix for previous version of this code > MDEV-21819 2 Assertion `inited == NONE || update_handler != this' > > - Move update_handler from TABLE to handler > - Move out initialization of update handler from ha_write_row() to > prepare_for_insert() > - Fixed that INSERT DELAYED works with update handler > - Give an error if using long unique with an autoincrement column > - Added handler function to check if table has long unique hash indexes > - Disable write cache in MyISAM and Aria when using update_handler as > if cache is used, the row will not be inserted until end of statement > and update_handler would not find conflicting rows. > - Removed not used handler argument from > check_duplicate_long_entries_update() > - Syntax cleanups > - Indentation fixes > - Don't use single character indentifiers for arguments > > squash! 1ca2f0871ecfb550268ffc1a5cc23d7043d6b855 you may want to remove this line from the commit comment > > diff --git a/mysql-test/main/long_unique.result b/mysql-test/main/long_unique.result > index a4955b3e7b5..fee8e7721bf 100644 > --- a/mysql-test/main/long_unique.result > +++ b/mysql-test/main/long_unique.result > @@ -1477,4 +1477,28 @@ id select_type table type possible_keys key key_len ref rows Extra > SELECT t2.b FROM t1 JOIN t2 ON t1.d = t2.f WHERE t2.pk >= 20; > b > drop table t1,t2; > +# > +# MDEV-21470 MyISAM start_bulk_insert doesn't work with long unique > +# > +CREATE TABLE t1 (a INT, b BLOB) ENGINE=MyISAM; > +INSERT INTO t1 VALUES (1,'foo'),(2,'bar'); > +CREATE TABLE t2 (c BIT, d BLOB, UNIQUE(d)) ENGINE=MyISAM; > +INSERT INTO t2 SELECT * FROM t1; > +Warnings: > +Warning 1264 Out of range value for column 'c' at row 2 > +DROP TABLE t1, t2; > +# > +# MDEV-19338 Using AUTO_INCREMENT with long unique > +# > +CREATE TABLE t1 (pk INT, a TEXT NOT NULL DEFAULT '', PRIMARY KEY (pk), b INT AUTO_INCREMENT, UNIQUE(b), UNIQUE (a,b)) ENGINE=myisam; > +ERROR HY000: Function or expression 'AUTO_INCREMENT' cannot be used in the UNIQUE clause of `a` Quite confusing error message :( I understand where it comes from, but for a user, I'm afraid, it just won't make any sense. A reasonable message could be, for example, AUTO_INCREMENT column %`s cannot be used in the UNIQUE index %`s > +# > +# MDEV-21819 Assertion `inited == NONE || update_handler != this' > +# failed in handler::ha_write_row > +# > +CREATE OR REPLACE TABLE t1 (a INT, b BLOB, s DATE, e DATE, PERIOD FOR app(s,e), UNIQUE(b)) ENGINE=MyISAM PARTITION BY HASH(a) PARTITIONS 2; > +INSERT INTO t1 VALUES (1,'foo','2022-01-01', '2025-01-01'); > +DELETE FROM t1 FOR PORTION OF app FROM '2023-01-01' TO '2024-01-01'; > +ERROR 23000: Duplicate entry 'foo' for key 'b' > +DROP TABLE t1; > set @@GLOBAL.max_allowed_packet= @allowed_packet; > diff --git a/mysql-test/suite/versioning/r/long_unique.result b/mysql-test/suite/versioning/r/long_unique.result > new file mode 100644 > index 00000000000..da07bc66e22 > --- /dev/null > +++ b/mysql-test/suite/versioning/r/long_unique.result > @@ -0,0 +1,8 @@ > +# > +# Assertion `inited == NONE || update_handler != this' failed in > +# handler::ha_write_row > +# > +CREATE TABLE t1 (f VARCHAR(4096), s DATE, e DATE, PERIOD FOR app(s,e), UNIQUE(f)) ENGINE=MyISAM; > +INSERT INTO t1 VALUES ('foo', '2023-08-30', '2025-07-09'),('bar', '2021-01-01', '2021-12-31'); > +DELETE FROM t1 FOR PORTION OF app FROM '2023-08-29' TO '2025-07-01'; > +DROP TABLE t1; you've had almost the same test (MDEV-21819) in main/long_unique.test it's a bit strange to have to very similar tests in different suites I suggest to move MDEV-21819 test from main/long_unique.test to this file. and then move this whole file from versioning suite to the period suite. > diff --git a/sql/ha_partition.cc b/sql/ha_partition.cc > index 73191907aac..c7e61864366 100644 > --- a/sql/ha_partition.cc > +++ b/sql/ha_partition.cc > @@ -4340,6 +4339,10 @@ int ha_partition::write_row(const uchar * buf) > } > m_last_part= part_id; > DBUG_PRINT("info", ("Insert in partition %u", part_id)); > + if (table->s->long_unique_table && > + m_file[part_id]->update_handler == m_file[part_id] && inited == RND) > + m_file[part_id]->prepare_for_insert(0); This looks quite incomprehensible. If the table has a long unique key and the update_handler is the handler itself you _prepare for insert_ ??? if it's an insert it should've been prepared earlier, like, normally for an insert, and it don't have to do anything with long uniques. What you actually do here (as I've found out looking firther in the patch) you prepare update_handler. That is "prepare_for_insert" is VERY confusing here, it sounds like it prepares for a normal insert and should be called at the beginning of mysql_insert(). And it's nothing like that. It would remove half of the questions if you'd call it prepare_update_handler() or, better, prepare_auxiliary_handler() (because it'll be used not only for long uniques) or something like that. Still, some questions are left: 1. why partitioning needs special code for that? 2. when a handler itself is its update_handler AND inited==RND? the handler itself can be the update_handler on INSERT, but then inited is NONE Is it UPDATE with system versioning? > + > start_part_bulk_insert(thd, part_id); > > tmp_disable_binlog(thd); /* Do not replicate the low-level changes. */ > diff --git a/sql/handler.cc b/sql/handler.cc > index 0700b1571e5..1e3f987b4e5 100644 > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -2648,6 +2649,39 @@ handler *handler::clone(const char *name, MEM_ROOT *mem_root) > return NULL; > } > > +/** > + @brief clone of current handler. you generally don't have to write @brief in these cases. Just /** clone of current handler. Creates a clone of handler used in update for unique hash key. */ > + Creates a clone of handler used in update for > + unique hash key. > +*/ > +bool handler::clone_handler_for_update() > +{ > + handler *tmp; > + DBUG_ASSERT(table->s->long_unique_table); > + > + if (update_handler != this) > + return 0; // Already done > + if (!(tmp= clone(table->s->normalized_path.str, table->in_use->mem_root))) > + return 1; > + update_handler= tmp; > + update_handler->ha_external_lock(table->in_use, F_RDLCK); Looks strange. Why F_RDLCK if it's an *update* handler? This definitely needs a comment, why you're using read lock for updates. > + return 0; > +} > + > +/** > + @brief Deletes update handler object > +*/ > +void handler::delete_update_handler() > +{ > + if (update_handler != this) > + { > + update_handler->ha_external_lock(table->in_use, F_UNLCK); > + update_handler->ha_close(); > + delete update_handler; > + } > + update_handler= this; > +} > + > LEX_CSTRING *handler::engine_name() > { > return hton_name(ht); > @@ -6481,6 +6515,8 @@ int handler::ha_reset() > DBUG_ASSERT(inited == NONE); > /* reset the bitmaps to point to defaults */ > table->default_column_bitmaps(); > + if (update_handler != this) > + delete_update_handler(); you already have an if() inside the delete_update_handler(). Either remove it from here or from there. > pushed_cond= NULL; > tracker= NULL; > mark_trx_read_write_done= 0; > @@ -6671,10 +6717,35 @@ static int check_duplicate_long_entries_update(TABLE *table, handler *h, uchar * > } > } > } > - exit: > - return error; > + return 0; > +} > + > + > +/** > + Do all initialization needed for insert > + > + @param force_update_handler Set to TRUE if we should always create an > + update handler. Needed if we don't know if we > + are going to do inserted while a scan is in "going to do inserts" > + progress. > +*/ > + > +int handler::prepare_for_insert(bool force_update_handler) > +{ > + /* Preparation for unique of blob's */ > + if (table->s->long_unique_table && (inited == RND || force_update_handler)) > + { > + /* > + When doing a scan we can't use the same handler to check > + duplicate rows. Create a new temporary one > + */ > + if (clone_handler_for_update()) > + return 1; > + } > + return 0; > } > > + > int handler::ha_write_row(const uchar *buf) > { > int error; > @@ -6977,6 +7044,11 @@ void handler::set_lock_type(enum thr_lock_type lock) > table->reginfo.lock_type= lock; > } > > +bool handler::has_long_unique() > +{ > + return table->s->long_unique_table; > +} I don't see why you need it for Aria. It can always check table->s->long_unique_table without an extra call > + > #ifdef WITH_WSREP > /** > @details > diff --git a/sql/sql_delete.cc b/sql/sql_delete.cc > index 6a4ce266af2..e55ae4f97e6 100644 > --- a/sql/sql_delete.cc > +++ b/sql/sql_delete.cc > @@ -752,6 +752,10 @@ bool mysql_delete(THD *thd, TABLE_LIST *table_list, COND *conds, > && !table->versioned() > && table->file->has_transactions(); > > + if (table->versioned(VERS_TIMESTAMP) || > + (table_list->has_period() && !portion_of_time_through_update)) > + table->file->prepare_for_insert(1); Ok, now I think that your idea of the default update_handler=this is rather fragile. If the default is NULL, than we can put in many places DBUG_ASSERT(update_handler != NULL); to make sure it was initialized when needed. Now we cannot do it. And there are many places in the code and non-trivial conditions when the update_handler has to be initialized and these places and conditions will change in the future. An assert would be helpful here. > + > THD_STAGE_INFO(thd, stage_updating); > while (likely(!(error=info.read_record())) && likely(!thd->killed) && > likely(!thd->is_error())) > diff --git a/sql/table.h b/sql/table.h > index 6ce92ee048e..1a0b9514d23 100644 > --- a/sql/table.h > +++ b/sql/table.h > @@ -1182,6 +1181,7 @@ struct TABLE > /* Map of keys dependent on some constraint */ > key_map constraint_dependent_keys; > KEY *key_info; /* data of keys in database */ > + KEY_PART_INFO *base_key_part; /* Where key parts are stored */ again, this is *only* needed for INSERT DELAYED that very few people ever use. It should be in some INSERT DELAYED specific data structure, not in the common TABLE. By the way, why cannot you get the list of keyparts from table->key_info->key_part ? > > Field **field; /* Pointer to fields */ > Field **vfield; /* Pointer to virtual fields*/ > diff --git a/storage/myisam/ha_myisam.cc b/storage/myisam/ha_myisam.cc > index 38091dae0ba..39147396359 100644 > --- a/storage/myisam/ha_myisam.cc > +++ b/storage/myisam/ha_myisam.cc > @@ -1740,7 +1740,7 @@ void ha_myisam::start_bulk_insert(ha_rows rows, uint flags) > enable_indexes() failed, thus it's essential that indexes are > disabled ONLY for an empty table. > */ > - if (file->state->records == 0 && can_enable_indexes && > + if (file->state->records == 0 && can_enable_indexes && !has_long_unique() && > (!rows || rows >= MI_MIN_ROWS_TO_DISABLE_INDEXES)) why do you disable bulk inserts for long uniques? > { > if (file->open_flag & HA_OPEN_INTERNAL_TABLE) > diff --git a/sql/table.cc b/sql/table.cc > index 718efa5767c..18a942964c3 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -1241,31 +1241,46 @@ bool parse_vcol_defs(THD *thd, MEM_ROOT *mem_root, TABLE *table, > Item *list_item; > KEY *key= 0; > uint key_index, parts= 0; > + KEY_PART_INFO *key_part= table->base_key_part; > + > for (key_index= 0; key_index < table->s->keys; key_index++) > { > - key=table->key_info + key_index; > - parts= key->user_defined_key_parts; > - if (key->key_part[parts].fieldnr == field->field_index + 1) > - break; > + /* > + We have to use key from share as this function may have changed > + table->key_info if it was ever invoked before. This could happen > + in case of INSERT DELAYED. > + */ > + key= table->s->key_info + key_index; > + if (key->algorithm == HA_KEY_ALG_LONG_HASH) > + { > + parts= key->user_defined_key_parts; > + if (key_part[parts].fieldnr == field->field_index + 1) > + break; > + key_part++; > + } Here, again, you've basically duplicated the logic of how long uniques are represented in the keys and keyparts. All for the sake of INSERT DELAYED. I'd really prefer INSERT DELAYED problems to be solved inside the Delayed_insert class. And it can use setup_keyinfo_hash() and re_setup_keyinfo_hash() functions to avoid duplicating the logic. > + key_part+= key->ext_key_parts; > } > - if (!key || key->algorithm != HA_KEY_ALG_LONG_HASH) > + if (key_index == table->s->keys) > goto end; > - KEY_PART_INFO *keypart; > - for (uint i=0; i < parts; i++) > + > + /* Correct the key & key_parts if this function has been called before */ > + key= table->key_info + key_index; > + key->key_part= key_part; > + > + for (uint i=0; i < parts; i++, key_part++) > { > - keypart= key->key_part + i; > - if (keypart->key_part_flag & HA_PART_KEY_SEG) > + if (key_part->key_part_flag & HA_PART_KEY_SEG) > { > - int length= keypart->length/keypart->field->charset()->mbmaxlen; > + int length= key_part->length/key_part->field->charset()->mbmaxlen; > list_item= new (mem_root) Item_func_left(thd, > - new (mem_root) Item_field(thd, keypart->field), > + new (mem_root) Item_field(thd, key_part->field), > new (mem_root) Item_int(thd, length)); > list_item->fix_fields(thd, NULL); > - keypart->field->vcol_info= > - table->field[keypart->field->field_index]->vcol_info; > + key_part->field->vcol_info= > + table->field[key_part->field->field_index]->vcol_info; > } > else > - list_item= new (mem_root) Item_field(thd, keypart->field); > + list_item= new (mem_root) Item_field(thd, key_part->field); > field_list->push_back(list_item, mem_root); > } > Item_func_hash *hash_item= new(mem_root)Item_func_hash(thd, *field_list); Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 3

Re: [Maria-developers] [Commits] 70b0c838fad: MDEV-21922: Allow packing addon fields even if they don't honour max_length_for_sort_data
by Sergey Petrunia 12 Mar '20

12 Mar '20

Hi Varun, * The patch needs a testcase * Please check the comment at https://jira.mariadb.org/browse/MDEV-21922 , and add a comment in filesort_use_addons() about the use of upper bound for computations. * Ok to push after the above is addressed. On Thu, Mar 12, 2020 at 05:11:45PM +0530, Varun wrote: > revision-id: 70b0c838fada8e02d4ce9525611d6f4aeadf56d4 (mariadb-10.5.0-367-g70b0c838fad) > parent(s): 9d7ed94f6a526748eff29dae2939a3fd341f118b > author: Varun Gupta > committer: Varun Gupta > timestamp: 2020-03-12 14:41:03 +0530 > message: > > MDEV-21922: Allow packing addon fields even if they don't honour max_length_for_sort_data > > Addon fields will be packed if the length of addon fields is greater > than max_length_for_sort_data. > > --- > sql/filesort.cc | 2 -- > 1 file changed, 2 deletions(-) > > diff --git a/sql/filesort.cc b/sql/filesort.cc > index 1e19d492220..7b4e0bc31cd 100644 > --- a/sql/filesort.cc > +++ b/sql/filesort.cc > @@ -139,8 +139,6 @@ void Sort_param::try_to_pack_addons(ulong max_length_for_sort_data) > return; > > const uint sz= Addon_fields::size_of_length_field; > - if (rec_length + sz > max_length_for_sort_data) > - return; > > // Heuristic: skip packing if potential savings are less than 10 bytes. > if (m_packable_length < (10 + sz)) > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] [Commits] 5d0e4ce: MySQL support added
by andrei.elkin＠pp.inet.fi 12 Mar '20

12 Mar '20

Привет, Сергей! О чем этот "суппорт", не обьяснишь одной фразой? Спасибо! Андрей psergey <sergey(a)mariadb.com> writes: > revision-id: 5d0e4ce291ae940feaa8398435158dc56a3db3c4 () > parent(s): d504887718112e211544beca0e6651d5477466e1 > author: Sergei Petrunia > committer: Sergei Petrunia > timestamp: 2020-03-12 10:26:06 +0300 > message: > > MySQL support added > > --- > .gitignore | 1 + > filesort-bench1/06-make-varchar-bench.sh | 30 +++++++++++++++-- > prepare-server.sh | 11 ++++-- > setup-server/setup-mariadb-current.sh | 6 ++-- > setup-server/setup-mysql-8.0.sh | 58 ++++++++++++++++++++++++++++++++ > 5 files changed, 98 insertions(+), 8 deletions(-) > > diff --git a/.gitignore b/.gitignore > index fcd5175..3bccc18 100644 > --- a/.gitignore > +++ b/.gitignore > @@ -23,3 +23,4 @@ my-10.5-old.cnf > mysql-boost > mysql-8.0 > mysql8-data > +mysql-8.0-data.clean/ > diff --git a/filesort-bench1/06-make-varchar-bench.sh b/filesort-bench1/06-make-varchar-bench.sh > index 85afcfd..21da087 100644 > --- a/filesort-bench1/06-make-varchar-bench.sh > +++ b/filesort-bench1/06-make-varchar-bench.sh > @@ -29,6 +29,17 @@ create table test_run_queries ( > test_time_ms bigint, > sort_merge_passes int > ); > + > +drop view if exists session_status; > + > +set @var= IF(version() like '%8.0%', > + 'create view session_status as select * from performance_schema.session_status', > + 'create view session_status as select * from information_schema.session_status'); > + > +prepare s from @var; > +execute s; > + > + > END > > ### > @@ -55,7 +66,18 @@ create table $test_table_name ( > char_field varchar($varchar_size) character set utf8, b int > ) engine=myisam; > > -insert into $rand_table_name select 1+floor(rand() * @n_countries) from seq_1_to_$table_size; > +drop table if exists ten, one_k; > +create table ten(a int); > +insert into ten values (0),(1),(2),(3),(4),(5),(6),(7),(8),(9); > + > +create table one_k(a int); > +insert into one_k select A.a + B.a* 10 + C.a * 100 from ten A, ten B, ten C; > + > +set @a=0; > +insert into $rand_table_name > +select 1+floor(rand() * @n_countries) > +from > + (select @a:=@a+1 from one_k A, one_k B, one_k C limit $table_size) T; > insert into $test_table_name > select > (select Name from Country where id=T.a), 1234 > @@ -63,13 +85,15 @@ from $rand_table_name T ; > > drop table $rand_table_name; > analyze table $test_table_name; > +select count(*) from $test_table_name; > +show create table $test_table_name; > END > > for i in 1 2 3 4 5 6 7 8 9 10 ; do > > ### query_start.sql here: > cat <<END > -select variable_value into @query_start_smp from information_schema.session_status where variable_name like 'sort_merge_passes'; > +select variable_value into @query_start_smp from session_status where variable_name like 'sort_merge_passes'; > select current_timestamp(6) into @query_start_time; > END > ### > @@ -87,7 +111,7 @@ echo $QUERY > cat << END > set @test_name='$TEST_NAME'; > set @query_time_ms= timestampdiff(microsecond, @query_start_time, current_timestamp(6))/1000; > -select variable_value into @query_end_smp from information_schema.session_status where variable_name like 'sort_merge_passes'; > +select variable_value into @query_end_smp from session_status where variable_name like 'sort_merge_passes'; > set @query_merge_passes = @query_end_smp - @query_start_smp; > insert into test_run_queries > (table_size, varchar_size, test_ts, test_time_ms, sort_merge_passes) > diff --git a/prepare-server.sh b/prepare-server.sh > index 01dbeba..d4da8db 100755 > --- a/prepare-server.sh > +++ b/prepare-server.sh > @@ -38,6 +38,11 @@ if [ ! -d $SERVERNAME ]; then > exit 1 > fi > > +if [ ! -f $SERVERNAME-vars.sh ]; then > + echo "Can't find settings file $SERVERNAME-vars.sh." > + exit 1 > +fi > + > if [[ $USE_RAMDISK ]] ; then > echo " Using /dev/shm for data dir" > fi > @@ -49,6 +54,8 @@ sleep 5 > > DATA_DIR=$SERVERNAME-data > > +source ${SERVERNAME}-vars.sh > + > if [[ $RECOVER ]] ; then > echo "Recovering the existing datadir" > else > @@ -64,7 +71,7 @@ else > fi > > #exit 0; > -./$SERVERNAME/sql/mysqld --defaults-file=./my-${SERVERNAME}.cnf & > +$MYSQLD --defaults-file=./my-${SERVERNAME}.cnf & > > > server_attempts=0 > @@ -72,7 +79,7 @@ server_attempts=0 > while true ; do > client_attempts=0 > while true ; do > - ./$SERVERNAME/client/mysql --defaults-file=./my-${SERVERNAME}.cnf -uroot -e "create database sbtest" > + $MYSQL $MYSQL_ARGS -e "select 1" > > if [ $? -eq 0 ]; then > break > diff --git a/setup-server/setup-mariadb-current.sh b/setup-server/setup-mariadb-current.sh > index 1b49ef0..b6bc417 100755 > --- a/setup-server/setup-mariadb-current.sh > +++ b/setup-server/setup-mariadb-current.sh > @@ -85,16 +85,16 @@ innodb_buffer_pool_size=8G > > EOF > > -cat > mysql-vars.sh <<EOF > +cat > $DIRNAME-vars.sh <<EOF > MYSQL="`pwd`/$DIRNAME/client/mysql" > +MYSQLD="`pwd`/$DIRNAME/sql/mysqld" > MYSQLSLAP="`pwd`/$DIRNAME/client/mysqlslap" > MYSQL_SOCKET="--socket=$SOCKETNAME" > MYSQL_USER="-uroot" > MYSQL_ARGS="\$MYSQL_USER \$MYSQL_SOCKET" > EOF > > -source mysql-vars.sh > -cp mysql-vars.sh $DIRNAME-vars.sh > +source $DIRNAME-vars.sh > > ( > cd $HOMEDIR/$DIRNAME/sql > diff --git a/setup-server/setup-mysql-8.0.sh b/setup-server/setup-mysql-8.0.sh > new file mode 100644 > index 0000000..ffd9987 > --- /dev/null > +++ b/setup-server/setup-mysql-8.0.sh > @@ -0,0 +1,58 @@ > +#!/bin/bash > + > +HOMEDIR=`pwd` > + > +BRANCH=8.0 > +SERVER_VERSION=8.0 > +DIRNAME="mysql-$SERVER_VERSION" > + > +git clone --branch $BRANCH --depth 1 https://github.com/mysql/mysql-server.git $DIRNAME > + > +cd mysql-$SERVER_VERSION > +cmake . -DCMAKE_BUILD_TYPE=RelWithDebInfo -DDOWNLOAD_BOOST=1 -DWITH_BOOST=$HOMEDIR/mysql-boost \ > + -DENABLE_DOWNLOADS=1 -DFORCE_INSOURCE_BUILD=1 -DWITH_UNIT_TESTS=0 > + > +make -j8 > + > +cd mysql-test > +perl ./mysql-test-run alias > +cp -r var/data $HOMEDIR/$DIRNAME-data > +cp -r var/data $HOMEDIR/$DIRNAME-data.clean > +cd .. > + > + > +source_dir=`pwd` > +socket_name="`basename $source_dir`.sock" > +SOCKETNAME="/tmp/$socket_name" > + > +cat > $HOMEDIR/my-$DIRNAME.cnf <<EOF > + > +[mysqld] > +datadir=$HOMEDIR/$DIRNAME-data > + > +tmpdir=/tmp > +port=3320 > +socket=$SOCKETNAME > +#binlog-format=row > +gdb > +lc_messages_dir=../share > +server-id=12 > +bind-address=0.0.0.0 > +log-error > +secure_file_priv= > +innodb_buffer_pool_size=4G > +EOF > + > +cat > $DIRNAME-vars.sh <<EOF > +MYSQL="`pwd`/$DIRNAME/bin/mysql" > +MYSQLD="`pwd`/$DIRNAME/bin/mysqld" > +MYSQLSLAP="`pwd`/$DIRNAME/bin/mysqlslap" > +MYSQL_SOCKET="--socket=$SOCKETNAME" > +MYSQL_USER="-uroot" > +MYSQL_ARGS="\$MYSQL_USER \$MYSQL_SOCKET" > +EOF > + > +source $DIRNAME-vars.sh > + > +$MYSQLD --defaults-file=$HOMEDIR/my-$DIRNAME.cnf & > + > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

[Maria-developers] (no subject)
by Badrul Chowdhury 11 Mar '20

11 Mar '20

-- Cheers, Badrul

1 0

Re: [Maria-developers] bbe056ac3fa: support NULL fields in key
by Sergei Golubchik 11 Mar '20

11 Mar '20

Hi, Nikita! On Mar 10, Nikita Malyavin wrote: > revision-id: bbe056ac3fa (mariadb-10.5.0-273-gbbe056ac3fa) > parent(s): 7a5d3316805 > author: Nikita Malyavin <nikitamalyavin(a)gmail.com> > committer: Nikita Malyavin <nikitamalyavin(a)gmail.com> > timestamp: 2020-03-11 00:46:24 +1000 > message: > > support NULL fields in key > > --- > mysql-test/suite/period/r/overlaps.result | 18 ++++++++++++++++-- > mysql-test/suite/period/t/overlaps.test | 16 +++++++++++++++- > sql/handler.cc | 18 ++++++++++++++---- > sql/sql_table.cc | 29 +++++++---------------------- > 4 files changed, 52 insertions(+), 29 deletions(-) > > diff --git a/mysql-test/suite/period/r/overlaps.result b/mysql-test/suite/period/r/overlaps.result > index cf980afd7f0..e52b21496b5 100644 > --- a/mysql-test/suite/period/r/overlaps.result > +++ b/mysql-test/suite/period/r/overlaps.result > @@ -104,16 +104,30 @@ show create table t; > Table Create Table > t CREATE TABLE `t` ( > `id` int(11) NOT NULL, > - `u` int(11) NOT NULL, > + `u` int(11) DEFAULT NULL, > `s` date NOT NULL, > `e` date NOT NULL, > PERIOD FOR `p` (`s`, `e`), > PRIMARY KEY (`id`,`p` WITHOUT OVERLAPS), > UNIQUE KEY `u` (`u`,`p` WITHOUT OVERLAPS) > ) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 > +insert into t values (2, NULL, '2003-03-01', '2003-05-01'); > +insert into t values (2, NULL, '2003-03-01', '2003-05-01'); > +ERROR 23000: Duplicate entry '2-2003-05-01-2003-03-01' for key 'PRIMARY' This is wrong. Should be no duplicate key error above. > +insert into t values (3, NULL, '2003-03-01', '2003-05-01'); > insert into t values (1, 1, '2003-03-01', '2003-05-01'); > insert into t values (1, 2, '2003-05-01', '2003-07-01'); > -insert into t values (2, 1, '2003-05-01', '2003-07-01'); > +insert into t values (4, NULL, '2003-03-01', '2003-05-01'); > +create sequence seq start=5; > +update t set id= nextval(seq), u= nextval(seq), s='2003-05-01', e='2003-07-01' > + where u is NULL; > +select * from t; > +id u s e > +1 1 2003-03-01 2003-05-01 > +1 2 2003-05-01 2003-07-01 > +5 6 2003-05-01 2003-07-01 > +7 8 2003-05-01 2003-07-01 > +9 10 2003-05-01 2003-07-01 > create or replace table t(id int, s date, e date, > period for p(s,e)); > insert into t values (1, '2003-01-01', '2003-03-01'), > diff --git a/sql/handler.cc b/sql/handler.cc > index 917386f4392..16f57533f27 100644 > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -7015,7 +7017,8 @@ int handler::ha_check_overlaps(const uchar *old_data, const uchar* new_data) > bool key_used= false; > for (uint k= 0; k < key_parts && !key_used; k++) > key_used= bitmap_is_set(table->write_set, > - key_info.key_part[k].fieldnr - 1); > + key_info.key_part[k].fieldnr - 1) > + && !key_info.key_part[k].field->is_null_in_record(new_data); Why is that? > if (!key_used) > continue; > } > @@ -7064,8 +7067,15 @@ int handler::ha_check_overlaps(const uchar *old_data, const uchar* new_data) > error= handler->ha_index_next(record_buffer); > } > > - if (!error && table->check_period_overlaps(key_info, key_info, > - new_data, record_buffer) == 0) > + bool null_in_key= false; > + for (uint k= 0; k < key_parts && !null_in_key; k++) > + { > + null_in_key= key_info.key_part[k].field->is_null_in_record(record_buffer); > + } > + > + if (!null_in_key && !error > + && table->check_period_overlaps(key_info, key_info, > + new_data, record_buffer) == 0) That's strange. You compare keys in key_period_compare_bases(), why do you do NULL values here? > error= HA_ERR_FOUND_DUPP_KEY; > > if (error == HA_ERR_KEY_NOT_FOUND || error == HA_ERR_END_OF_FILE) Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] eea71e8b05a: MDEV-16978 Application-time periods: WITHOUT OVERLAPS
by Sergei Golubchik 10 Mar '20

10 Mar '20

Hi, Nikita! Despite what the subject says, it's the review for eea71e8b05a..7a5d3316805 That is everything in bb-10.5-MDEV-16978-without-overlaps minus the last commit (that came too late and I'll review it separately) In general it looks pretty good, just few minor comments below: On Mar 10, Nikita Malyavin wrote: > revision-id: eea71e8b05a (mariadb-10.5.0-246-geea71e8b05a) > parent(s): 6618fc29749 > author: Nikita Malyavin <nikitamalyavin(a)gmail.com> > committer: Nikita Malyavin <nikitamalyavin(a)gmail.com> > timestamp: 2020-02-21 02:38:57 +1000 > message: > > MDEV-16978 Application-time periods: WITHOUT OVERLAPS > > * The overlaps check is implemented on a handler level per row command. > It creates a separate cursor (actually, another handler instance) and > caches it inside the original handler, when ha_update_row or > ha_insert_row is issued. Cursor closes on unlocking the handler. > > * Containing the same key in index means unique constraint violation > even in usual terms. So we fetch left and right neighbours and check > that they have same key prefix, excluding from the key only the period part. > If it doesnt match, then there's no such neighbour, and the check passes. > Otherwise, we check if this neighbour intersects with the considered key. > > * The check does introduce new error and fails with ER_DUPP_KEY error. "does not introduce new error" you mean? > This might break REPLACE workflow and should be fixed separately > > diff --git a/sql/field.h b/sql/field.h > index 4a8eec35b05..e187ffeb331 100644 > --- a/sql/field.h > +++ b/sql/field.h > @@ -1444,8 +1444,9 @@ class Field: public Value_source > if (null_ptr) > null_ptr=ADD_TO_PTR(null_ptr,ptr_diff,uchar*); > } > - virtual void get_image(uchar *buff, uint length, CHARSET_INFO *cs) > - { memcpy(buff,ptr,length); } > + virtual void get_image(uchar *buff, uint length, > + const uchar *ptr_arg, CHARSET_INFO *cs) const > + { memcpy(buff,ptr_arg,length); } please, add a convenience method. void get_image(uchar *buff, uint length, CHARSET_INFO *cs) { get_image(buff, length, ptr, cs); } and the same below, where you add ptr_arg > virtual void set_image(const uchar *buff,uint length, CHARSET_INFO *cs) > { memcpy(ptr,buff,length); } > > @@ -4056,7 +4066,8 @@ class Field_varstring :public Field_longstr { > using Field_str::store; > double val_real() override; > longlong val_int() override; > - String *val_str(String *, String *) override; > + String *val_str(String *, String *) final; > + virtual String *val_str(String*,String *, const uchar*) const; This means that for the sake of indexes WITHOUT OVERLAPS (that very few people will use) and compressed blobs (that are used even less) you've added a new virtual call to Field_varstring::val_str (that is used awfully a lot) Try to avoid it, please > my_decimal *val_decimal(my_decimal *) override; > int cmp_max(const uchar *, const uchar *, uint max_length) const override; > int cmp(const uchar *a,const uchar *b) const override > diff --git a/sql/field.cc b/sql/field.cc > index 1ce49b0bdfa..82df2784057 100644 > --- a/sql/field.cc > +++ b/sql/field.cc > @@ -9638,11 +9645,12 @@ int Field_bit::cmp_offset(my_ptrdiff_t row_offset) > } > > > -uint Field_bit::get_key_image(uchar *buff, uint length, imagetype type_arg) > +uint Field_bit::get_key_image(uchar *buff, uint length, const uchar *ptr_arg, imagetype type_arg) const > { > if (bit_len) > { > - uchar bits= get_rec_bits(bit_ptr, bit_ofs, bit_len); > + auto *bit_ptr_for_arg= ptr_arg + (bit_ptr - ptr); please, don't use auto in trivial cases like this one. it might be easier to type, but then the reviewer and whoever will in the future will edit this code will have to do type derivation in the head. > + uchar bits= get_rec_bits(bit_ptr_for_arg, bit_ofs, bit_len); > *buff++= bits; > length--; > } > diff --git a/sql/item_buff.cc b/sql/item_buff.cc > index 81949bcdae0..514ac740697 100644 > --- a/sql/item_buff.cc > +++ b/sql/item_buff.cc > @@ -195,7 +195,7 @@ bool Cached_item_field::cmp(void) > becasue of value change), then copy the new value to buffer. > */ > if (! null_value && (tmp || (tmp= (field->cmp(buff) != 0)))) > - field->get_image(buff,length,field->charset()); > + field->get_image(buff,length,field->ptr,field->charset()); not needed if you add convenience methods > return tmp; > } > > diff --git a/sql/sql_class.h b/sql/sql_class.h > index 13b2659789d..2f1b1431dc0 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -357,13 +357,15 @@ class Key :public Sql_alloc, public DDL_options { > engine_option_value *option_list; > bool generated; > bool invisible; > + bool without_overlaps; > + Lex_ident period; strictly speaking, you don't need without_overlaps property, if the period name is set it *has to be* without overlaps. but ok, whatever you prefer > > Key(enum Keytype type_par, const LEX_CSTRING *name_arg, > ha_key_alg algorithm_arg, bool generated_arg, DDL_options_st ddl_options) > :DDL_options(ddl_options), > type(type_par), key_create_info(default_key_create_info), > name(*name_arg), option_list(NULL), generated(generated_arg), > - invisible(false) > + invisible(false), without_overlaps(false) > { > key_create_info.algorithm= algorithm_arg; > } > diff --git a/sql/table.h b/sql/table.h > index 6ce92ee048e..09f03690a8c 100644 > --- a/sql/table.h > +++ b/sql/table.h > @@ -1635,13 +1635,12 @@ struct TABLE > int insert_portion_of_time(THD *thd, const vers_select_conds_t &period_conds, > ha_rows *rows_inserted); > bool vers_check_update(List<Item> &items); > - > + static int check_period_overlaps(const KEY &lhs_key, const KEY &rhs_key, > + const uchar *lhs, const uchar *rhs); why did you make it a static method of TABLE? > int delete_row(); > void vers_update_fields(); > void vers_update_end(); > void find_constraint_correlated_indexes(); > - void clone_handler_for_update(); > - void delete_update_handler(); > > /** Number of additional fields used in versioned tables */ > #define VERSIONING_FIELDS 2 > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index 240f001f7de..74d28ede25f 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -3959,6 +3959,28 @@ mysql_prepare_create_table(THD *thd, HA_CREATE_INFO *create_info, > DBUG_RETURN(TRUE); > } > > + switch (key->type) { > + case Key::UNIQUE: > + if (!key->period) > + break; > + /* Fall through: > + WITHOUT OVERLAPS forces fields to be NOT NULL > + */ Why is that? > + case Key::PRIMARY: > + /* Implicitly set primary key fields to NOT NULL for ISO conf. */ > + if (!(sql_field->flags & NOT_NULL_FLAG)) > + { > + /* Implicitly set primary key fields to NOT NULL for ISO conf. */ duplicated comment > + sql_field->flags|= NOT_NULL_FLAG; > + sql_field->pack_flag&= ~FIELDFLAG_MAYBE_NULL; > + null_fields--; > + } > + break; > + default: > + // Fall through > + break; > + } > + > cols2.rewind(); > switch(key->type) { > > @@ -4536,15 +4556,13 @@ bool Column_definition::sp_prepare_create_field(THD *thd, MEM_ROOT *mem_root) > } > > > -static bool vers_prepare_keys(THD *thd, HA_CREATE_INFO *create_info, > - Alter_info *alter_info, KEY **key_info, uint key_count) > +static bool append_system_key_parts(THD *thd, HA_CREATE_INFO *create_info, > + Alter_info *alter_info, KEY **key_info, > + uint key_count) > { > - DBUG_ASSERT(create_info->versioned()); > - > - const char *row_start_field= create_info->vers_info.as_row.start; > - DBUG_ASSERT(row_start_field); > - const char *row_end_field= create_info->vers_info.as_row.end; > - DBUG_ASSERT(row_end_field); > + const auto &row_start_field= create_info->vers_info.as_row.start; > + const auto &row_end_field= create_info->vers_info.as_row.end; please, don't use auto in trivial cases like this one. it might be easier to type, but then the reviewer and whoever will in the future will edit this code will have to do type derivation in the head. > + DBUG_ASSERT(!create_info->versioned() || (row_start_field && row_end_field)); > > List_iterator<Key> key_it(alter_info->key_list); > Key *key= NULL; > @@ -4553,25 +4571,61 @@ static bool vers_prepare_keys(THD *thd, HA_CREATE_INFO *create_info, > if (key->type != Key::PRIMARY && key->type != Key::UNIQUE) > continue; > > + if (create_info->versioned()) > + { > Key_part_spec *key_part=NULL; > List_iterator<Key_part_spec> part_it(key->columns); > while ((key_part=part_it++)) > { > - if (!my_strcasecmp(system_charset_info, > - row_start_field, > - key_part->field_name.str) || > - > - !my_strcasecmp(system_charset_info, > - row_end_field, > - key_part->field_name.str)) > + if (row_start_field.streq(key_part->field_name) || > + row_end_field.streq(key_part->field_name)) > break; > } > - if (key_part) > - continue; // Key already contains Sys_start or Sys_end > + if (!key_part) > + key->columns.push_back(new Key_part_spec(&row_end_field, 0)); > + } > + } > > - Key_part_spec *key_part_sys_end_col= > - new (thd->mem_root) Key_part_spec(&create_info->vers_info.as_row.end, 0); > - key->columns.push_back(key_part_sys_end_col); > + key_it.rewind(); > + while ((key=key_it++)) please skip this loop if there's no PERIOD and skip the loop above if there's no system versioning. > + { > + if (key->without_overlaps) > + { > + if (key->type != Key::PRIMARY && key->type != Key::UNIQUE) > + { > + my_error(ER_PERIOD_WITHOUT_OVERLAPS_NON_UNIQUE, MYF(0), key->period.str); > + return true; I think this can even be a syntax error in the parser. there's no need to postpone it till here, is there? > + } > + if (!create_info->period_info.is_set() > + || !key->period.streq(create_info->period_info.name)) > + { > + my_error(ER_PERIOD_NOT_FOUND, MYF(0), key->period.str); > + return true; > + } > + if (thd->work_part_info) > + { > + // Unfortunately partitions do not support searching upper/lower bounds > + // (i.e. ha_index_read_map with KEY_OR_PREV, KEY_OR_NEXT) > + my_error(ER_FEATURE_NOT_SUPPORTED_WITH_PARTITIONING, MYF(0), > + "WITHOUT OVERLAPS"); > + return true; > + } > + const auto &period_start= create_info->period_info.period.start; > + const auto &period_end= create_info->period_info.period.end; > + List_iterator<Key_part_spec> part_it(key->columns); > + while (Key_part_spec *key_part= part_it++) > + { > + if (period_start.streq(key_part->field_name) > + || period_end.streq(key_part->field_name)) > + { > + my_error(ER_KEY_CONTAINS_PERIOD_FIELDS, MYF(0), key->name.str, > + key_part->field_name); > + return true; > + } > + } > + key->columns.push_back(new Key_part_spec(&period_end, 0)); > + key->columns.push_back(new Key_part_spec(&period_start, 0)); > + } > } > > return false; > diff --git a/sql/handler.cc b/sql/handler.cc > index 7d61252eea6..917386f4392 100644 > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -4173,6 +4173,11 @@ uint handler::get_dup_key(int error) > if (table->s->long_unique_table && table->file->errkey < table->s->keys) > DBUG_RETURN(table->file->errkey); > table->file->errkey = (uint) -1; > + if (overlaps_error_key != -1) > + { > + table->file->errkey= (uint)overlaps_error_key; > + DBUG_RETURN(table->file->errkey); > + } Why do you need overlaps_error_key? it looks like you can store the conflicting key number directly in errkey and it's somewhat confusing that this method uses table->file->errkey instead of just errkey. It's totally not clear why it does that. Looks like some historical thing from 2000. > if (error == HA_ERR_FOUND_DUPP_KEY || > error == HA_ERR_FOREIGN_DUPLICATE_KEY || > error == HA_ERR_FOUND_DUPP_UNIQUE || error == HA_ERR_NULL_IN_SPATIAL || > @@ -6563,10 +6576,12 @@ static int check_duplicate_long_entry_key(TABLE *table, handler *h, > unique constraint on long columns. > @returns 0 if no duplicate else returns error > */ > -static int check_duplicate_long_entries(TABLE *table, handler *h, > - const uchar *new_rec) > +int handler::check_duplicate_long_entries(const uchar *new_rec) > { > - table->file->errkey= -1; > + if (this->inited == RND) > + create_lookup_handler(); 1. and if inited==INDEX ? 2. why 'this->' ? 3. generally it's not a good idea to check inited==RND and lookup_handler==NULL inside a loop per row, because these values cannot change in the middle of a scan. Better to check them once, when inited is being set to RND. > + handler *h= lookup_handler ? lookup_handler : table->file; why table->file and not this? > + errkey= -1; > int result; > for (uint i= 0; i < table->s->keys; i++) > { > @@ -6935,6 +6956,142 @@ void handler::set_lock_type(enum thr_lock_type lock) > table->reginfo.lock_type= lock; > } > > +/** > + @brief clone of current handler. > + Creates a clone of handler used for unique hash key and WITHOUT OVERLAPS. > + @return error code > +*/ > +int handler::create_lookup_handler() > +{ > + if (lookup_handler) > + return 0; > + lookup_handler= clone(table_share->normalized_path.str, > + table->in_use->mem_root); > + int error= lookup_handler->ha_external_lock(table->in_use, F_RDLCK); > + return error; > +} > + > +int handler::ha_check_overlaps(const uchar *old_data, const uchar* new_data) > +{ > + DBUG_ASSERT(new_data); > + if (!table_share->period.unique_keys) > + return 0; > + if (table->versioned() && !table->vers_end_field()->is_max()) > + return 0; > + > + bool is_update= old_data != NULL; > + if (!check_overlaps_buffer) > + check_overlaps_buffer= (uchar*)alloc_root(&table_share->mem_root, > + table_share->max_unique_length > + + table_share->reclength); check_overlaps_buffer is per handler. It should be allocated in the TABLE::mem_root, not TABLE_SHARE::mem_root Also, it should probably be called lookup_buffer and check_duplicate_long_entry_key() should use it too. > + auto *record_buffer= check_overlaps_buffer + table_share->max_unique_length; > + auto *handler= this; > + // handler->inited can be NONE on INSERT > + if (handler->inited != NONE) > + { > + create_lookup_handler(); > + handler= lookup_handler; > + > + // Needs to compare record refs later is old_row_found() > + if (is_update) > + position(old_data); > + } > + > + // Save and later restore this handler's keyread > + int old_this_keyread= this->keyread; Again, why do you save/restore this->keyread? If you're using lookup_handler below, then this->keyread doesn't matter. And if handler==this below, then this->inited==NONE, which implies no keyread. Either way, handler->keyread_enabled() should always be false here. What about DBUG_ASSERT(!handler->keyread_enabled()) ? > + DBUG_ASSERT(this->ha_end_keyread() == 0); please fix all cases where you used a function with side effects inside an assert. not just the one I've commented about > + > + int error= 0; > + > + for (uint key_nr= 0; key_nr < table_share->keys && !error; key_nr++) > + { > + const KEY &key_info= table->key_info[key_nr]; > + const uint key_parts= key_info.user_defined_key_parts; > + if (!key_info.without_overlaps) > + continue; > + > + if (is_update) > + { > + bool key_used= false; > + for (uint k= 0; k < key_parts && !key_used; k++) > + key_used= bitmap_is_set(table->write_set, > + key_info.key_part[k].fieldnr - 1); > + if (!key_used) > + continue; > + } > + > + error= handler->ha_index_init(key_nr, 0); > + if (error) > + return error; we should try to minimize number of index_init/index_end, it doesn't look like a cheap operation, at least in InnoDB. but in a separate commit, because it should cover long uniques too. > + > + error= handler->ha_start_keyread(key_nr); > + DBUG_ASSERT(!error); > + > + const uint period_field_length= key_info.key_part[key_parts - 1].length; > + const uint key_base_length= key_info.key_length - 2 * period_field_length; > + > + key_copy(check_overlaps_buffer, new_data, &key_info, 0); > + > + /* Copy period_start to period_end. > + the value in period_start field is not significant, but anyway let's leave > + it defined to avoid uninitialized memory access > + */ > + memcpy(check_overlaps_buffer + key_base_length, > + check_overlaps_buffer + key_base_length + period_field_length, > + period_field_length); > + > + /* Find row with period_end > (period_start of new_data) */ > + error = handler->ha_index_read_map(record_buffer, > + check_overlaps_buffer, > + key_part_map((1 << (key_parts - 1)) - 1), > + HA_READ_AFTER_KEY); > + > + if (!error && is_update) > + { > + /* In case of update it could happen that the nearest neighbour is > + a record we are updating. It means, that there are no overlaps > + from this side. > + > + An assumption is made that during update we always have the last > + fetched row in old_data. Therefore, comparing ref's is enough > + */ > + DBUG_ASSERT(handler != this); > + DBUG_ASSERT(inited != NONE); > + DBUG_ASSERT(ref_length == handler->ref_length); > + > + handler->position(record_buffer); > + if (memcmp(ref, handler->ref, ref_length) == 0) > + error= handler->ha_index_next(record_buffer); > + } > + > + if (!error && table->check_period_overlaps(key_info, key_info, > + new_data, record_buffer) == 0) > + error= HA_ERR_FOUND_DUPP_KEY; > + > + if (error == HA_ERR_KEY_NOT_FOUND || error == HA_ERR_END_OF_FILE) > + error= 0; > + > + if (error == HA_ERR_FOUND_DUPP_KEY) > + overlaps_error_key= key_nr; > + > + int end_error= handler->ha_end_keyread(); > + DBUG_ASSERT(!end_error); > + > + end_error= handler->ha_index_end(); > + if (!error && end_error) > + error= end_error; > + } > + > + // Restore keyread of this handler, if it was enabled > + if (old_this_keyread < MAX_KEY) > + { > + error= this->ha_start_keyread(old_this_keyread); > + DBUG_ASSERT(error == 0); > + } > + > + return error; > +} > + > #ifdef WITH_WSREP > /** > @details > diff --git a/sql/table.cc b/sql/table.cc > index 718efa5767c..65fc44458f4 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -1499,6 +1500,14 @@ static size_t extra2_read_len(const uchar **extra2, const uchar *extra2_end) > return length; > } > > +static > +bool fill_unique_extra2(const uchar *extra2, size_t len, LEX_CUSTRING *section) thanks, good point. a bit confusing name, I thought it's something about UNIQUE particularly as this is what the whole patch is about :) What about read_extra2_section_once() or something like that? Or get_ or consume_ or store_ ? > +{ > + if (section->str) > + return true; > + *section= {extra2, len}; > + return false; > +} > > static > bool read_extra2(const uchar *frm_image, size_t len, extra2_fields *fields) > @@ -1725,11 +1725,26 @@ int TABLE_SHARE::init_from_binary_frm_image(THD *thd, bool write, > keyinfo= &first_keyinfo; > thd->mem_root= &share->mem_root; > > + auto err= [thd, share, &handler_file, &se_plugin, old_root](){ > + share->db_plugin= NULL; > + share->error= OPEN_FRM_CORRUPTED; > + share->open_errno= my_errno; > + delete handler_file; > + plugin_unlock(0, se_plugin); > + my_hash_free(&share->name_hash); > + > + if (!thd->is_error()) > + open_table_error(share, OPEN_FRM_CORRUPTED, share->open_errno); > + > + thd->mem_root= old_root; > + return HA_ERR_NOT_A_TABLE; > + }; > + Okay. I kind of see your point, but let's postpone this refactoring until after 10.5.2. I still need some time to think about it. You'll still be able to push it later. > if (write && write_frm_image(frm_image, frm_length)) > - goto err; > + DBUG_RETURN(err()); > > if (frm_length < FRM_HEADER_SIZE + FRM_FORMINFO_SIZE) > - goto err; > + DBUG_RETURN(err()); > > share->frm_version= frm_image[2]; > /* > @@ -8603,6 +8626,21 @@ void TABLE::evaluate_update_default_function() > DBUG_VOID_RETURN; > } > > +/** > + Compare two keys with periods > + @return -1, lhs precedes rhs > + 0, lhs overlaps rhs > + 1, lhs succeeds rhs > + */ > +int TABLE::check_period_overlaps(const KEY &lhs_key, const KEY &rhs_key, > + const uchar *lhs, const uchar *rhs) > +{ > + int cmp_res= key_period_compare_bases(lhs_key, rhs_key, lhs, rhs); you only use key_period_compare_bases here. I just wouldn't create a separate small function for that, but rather inlined it here. > + if (cmp_res) > + return cmp_res; > + > + return key_period_compare_periods(lhs_key, rhs_key, lhs, rhs); same for key_period_compare_periods > +} > > void TABLE::vers_update_fields() > { > diff --git a/sql/key.cc b/sql/key.cc > index 9dbb7a15726..49e97faea22 100644 > --- a/sql/key.cc > +++ b/sql/key.cc > @@ -896,3 +897,51 @@ bool key_buf_cmp(KEY *key_info, uint used_key_parts, > } > return FALSE; > } > + > + > +/** > + Compare base parts (not including the period) of keys with period > + @return -1, lhs less than rhs > + 0, lhs equals rhs > + 1, lhs more than rhs > + */ > +int key_period_compare_bases(const KEY &lhs_key, const KEY &rhs_key, > + const uchar *lhs, const uchar *rhs) > +{ > + uint base_part_nr= lhs_key.user_defined_key_parts - 2; may be, give this '2' a name, like 'period_key_parts' ? and use them everywhere, of course, not just here > + int cmp_res= 0; > + for (uint part_nr= 0; !cmp_res && part_nr < base_part_nr; part_nr++) > + { > + Field *f= lhs_key.key_part[part_nr].field; > + cmp_res= f->cmp(f->ptr_in_record(lhs), > + rhs_key.key_part[part_nr].field->ptr_in_record(rhs)); This doesn't seem to be handling NULLs (see the next review) > + } > + > + return cmp_res; > +} > + > +/** > + Compare periods of two keys > + @return -1, lhs preceeds rhs > + 0, lhs overlaps rhs > + 1, lhs succeeds rhs > + */ > +int key_period_compare_periods(const KEY &lhs_key, const KEY &rhs_key, > + const uchar *lhs, const uchar *rhs) > +{ > + uint period_start= lhs_key.user_defined_key_parts - 1; > + uint period_end= lhs_key.user_defined_key_parts - 2; > + > + const auto *f= lhs_key.key_part[period_start].field; > + const uchar *l[]= {lhs_key.key_part[period_start].field->ptr_in_record(lhs), > + rhs_key.key_part[period_start].field->ptr_in_record(rhs)}; > + > + const uchar *r[]= {lhs_key.key_part[period_end].field->ptr_in_record(lhs), > + rhs_key.key_part[period_end].field->ptr_in_record(rhs)}; I'd still prefer names like 'ls', 'le', 'rs', 're'. Now I need to look up and keep in mind that 'r[0] is left key end period' etc > + > + if (f->cmp(r[0], l[1]) <= 0) > + return -1; > + if (f->cmp(l[0], r[1]) >= 0) > + return 1; > + return 0; > +} Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] 9ae015878f1: MDEV-10047: table-based master info repository
by sujatha 10 Mar '20

10 Mar '20

revision-id: 9ae015878f11be3e3033fd1b35357ea5927c6c51 (mariadb-10.5.0-329-g9ae015878f1) parent(s): b753ac066bc26acda9deb707a31c112f1bbf9ec2 author: Sujatha committer: Sujatha timestamp: 2020-03-10 15:55:50 +0530 message: MDEV-10047: table-based master info repository Problem: ======= When we upgrade from "mysql" to "mariadb" if slave is using repositories as tables their data is completely ignored and no warning is issued in error log. Fix: === "mysql_upgrade" test should check for the presence of data in "mysql.slave_master_info" and "mysql.slave_relay_log_info" tables. When tables have some data the upgrade script should report a warning which hints users that the data in repository tables will be ignored. --- client/mysql_upgrade.c | 61 +++++++++- .../main/rpl_mysql_upgrade_slave_repo_check.result | 33 ++++++ .../main/rpl_mysql_upgrade_slave_repo_check.test | 127 +++++++++++++++++++++ 3 files changed, 220 insertions(+), 1 deletion(-) diff --git a/client/mysql_upgrade.c b/client/mysql_upgrade.c index 4e17089593f..bea82c2a112 100644 --- a/client/mysql_upgrade.c +++ b/client/mysql_upgrade.c @@ -1014,6 +1014,64 @@ static int install_used_engines(void) return 0; } +static int check_slave_repositories(void) +{ + DYNAMIC_STRING ds_result; + int row_count= 0; + int error= 0; + const char *query = "SELECT COUNT(*) AS c1 FROM mysql.slave_master_info"; + + if (init_dynamic_string(&ds_result, "", 512, 512)) + die("Out of memory"); + + run_query(query, &ds_result, TRUE); + + if (ds_result.length) + { + row_count= atoi((char *)ds_result.str); + if (row_count) + { + fprintf(stderr,"Slave info repository compatibility check:" + " Found data in `mysql`.`slave_master_info` table.\n"); + fprintf(stderr,"Warning: Content of `mysql`.`slave_master_info` table" + " will be ignored as MariaDB supports file based info " + "repository.\n"); + error= 1; + } + } + dynstr_free(&ds_result); + + query = "SELECT COUNT(*) AS c1 FROM mysql.slave_relay_log_info"; + + if (init_dynamic_string(&ds_result, "", 512, 512)) + die("Out of memory"); + + run_query(query, &ds_result, TRUE); + + if (ds_result.length) + { + row_count= atoi((char *)ds_result.str); + if (row_count) + { + fprintf(stderr, "Slave info repository compatibility check:" + " Found data in `mysql`.`slave_relay_log_info` table.\n"); + fprintf(stderr, "Warning: Content of `mysql`.`slave_relay_log_info` " + "table will be ignored as MariaDB supports file based " + "repository.\n"); + error= 1; + } + } + dynstr_free(&ds_result); + if (error) + { + fprintf(stderr,"Slave server may not possess the correct replication " + "metadata.\n"); + fprintf(stderr, "Execution of CHANGE MASTER as per " + "`mysql`.`slave_master_info` and `mysql`.`slave_relay_log_info` " + "table content is recommended.\n"); + } + return 0; +} /* Update all system tables in MySQL Server to current @@ -1225,7 +1283,8 @@ int main(int argc, char **argv) run_mysqlcheck_views() || run_sql_fix_privilege_tables() || run_mysqlcheck_fixnames() || - run_mysqlcheck_upgrade(FALSE)) + run_mysqlcheck_upgrade(FALSE) || + check_slave_repositories()) die("Upgrade failed" ); verbose("Phase %d/%d: Running 'FLUSH PRIVILEGES'", ++phase, phases_total); diff --git a/mysql-test/main/rpl_mysql_upgrade_slave_repo_check.result b/mysql-test/main/rpl_mysql_upgrade_slave_repo_check.result new file mode 100644 index 00000000000..87cc9ab5a24 --- /dev/null +++ b/mysql-test/main/rpl_mysql_upgrade_slave_repo_check.result @@ -0,0 +1,33 @@ +include/master-slave.inc +[connection master] +******************************************************************** +* Test case1: Upgrade when repository tables have data. * +* mysql_upgrade script should report warnings. * +******************************************************************** +connection master; +Slave info repository compatibility check: Found data in `mysql`.`slave_master_info` table. +Warning: Content of `mysql`.`slave_master_info` table will be ignored as MariaDB supports file based info repository. +Slave info repository compatibility check: Found data in `mysql`.`slave_relay_log_info` table. +Warning: Content of `mysql`.`slave_relay_log_info` table will be ignored as MariaDB supports file based repository. +Slave server may not possess the correct replication metadata. +Execution of CHANGE MASTER as per `mysql`.`slave_master_info` and `mysql`.`slave_relay_log_info` table content is recommended. +connection slave; +Slave info repository compatibility check: Found data in `mysql`.`slave_master_info` table. +Warning: Content of `mysql`.`slave_master_info` table will be ignored as MariaDB supports file based info repository. +Slave info repository compatibility check: Found data in `mysql`.`slave_relay_log_info` table. +Warning: Content of `mysql`.`slave_relay_log_info` table will be ignored as MariaDB supports file based repository. +Slave server may not possess the correct replication metadata. +Execution of CHANGE MASTER as per `mysql`.`slave_master_info` and `mysql`.`slave_relay_log_info` table content is recommended. +connection master; +TRUNCATE TABLE `mysql`.`slave_master_info`; +TRUNCATE TABLE `mysql`.`slave_relay_log_info`; +******************************************************************** +* Test case2: Upgrade when repository tables are empty. * +* mysql_upgrade script should not report any warning. * +******************************************************************** +connection master; +connection slave; +"====== Clean up ======" +connection master; +DROP TABLE `mysql`.`slave_master_info`, `mysql`.`slave_relay_log_info`; +include/rpl_end.inc diff --git a/mysql-test/main/rpl_mysql_upgrade_slave_repo_check.test b/mysql-test/main/rpl_mysql_upgrade_slave_repo_check.test new file mode 100644 index 00000000000..24b5f029e8d --- /dev/null +++ b/mysql-test/main/rpl_mysql_upgrade_slave_repo_check.test @@ -0,0 +1,127 @@ +# ==== Purpose ==== +# +# While upgrading from "mysql" to "mariadb" if slave info repositories are +# configured to be tables then appropriate warnings should be reported. +# +# ==== Implementation ==== +# +# Steps: +# 1 - On MariaDB server create `mysql`.`slave_master_info` and +# `mysql.slave_relay_log_info` tables to simulate upgrade from "mysql" +# to "mariadb" server. Insert data into these tables. +# 2 - Execute "mysql_upgrade" script and verify that appropriate warning +# is reported. i.e Warning is to alert user that the data present in +# repository tables will be ignored. +# 3 - Truncate these tables. This simulates repositories being file and +# the tables are empty. +# 4 - Execute "mysql_upgrade" script and verify that no warnings are +# reported. +# +# ==== References ==== +# +# MDEV-10047: table-based master info repository +# + +--source include/have_innodb.inc +--source include/mysql_upgrade_preparation.inc +--source include/have_binlog_format_mixed.inc +--source include/master-slave.inc + +--write_file $MYSQLTEST_VARDIR/tmp/slave_table_repo_init.sql +--disable_query_log +--disable_result_log +SET SQL_LOG_BIN=0; +# Table structure extracted from MySQL-5.6.47 +CREATE TABLE `mysql`.`slave_master_info` ( + `Number_of_lines` int(10) unsigned NOT NULL COMMENT 'Number of lines in the file.', + `Master_log_name` text CHARACTER SET utf8 COLLATE utf8_bin NOT NULL COMMENT 'The name of the master binary log currently being read from the master.', + `Master_log_pos` bigint(20) unsigned NOT NULL COMMENT 'The master log position of the last read event.', + `Host` char(64) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL DEFAULT '' COMMENT 'The host name of the master.', + `User_name` text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The user name used to connect to the master.', + `User_password` text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The password used to connect to the master.', + `Port` int(10) unsigned NOT NULL COMMENT 'The network port used to connect to the master.', + `Connect_retry` int(10) unsigned NOT NULL COMMENT 'The period (in seconds) that the slave will wait before trying to reconnect to the master.', + `Enabled_ssl` tinyint(1) NOT NULL COMMENT 'Indicates whether the server supports SSL connections.', + `Ssl_ca` text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The file used for the Certificate Authority (CA) certificate.', + `Ssl_capath` text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The path to the Certificate Authority (CA) certificates.', + `Ssl_cert` text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The name of the SSL certificate file.', + `Ssl_cipher` text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The name of the cipher in use for the SSL connection.', + `Ssl_key` text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The name of the SSL key file.', + `Ssl_verify_server_cert` tinyint(1) NOT NULL COMMENT 'Whether to verify the server certificate.', + `Heartbeat` float NOT NULL, + `Bind` text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'Displays which interface is employed when connecting to the MySQL server', + `Ignored_server_ids` text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The number of server IDs to be ignored, followed by the actual server IDs', + `Uuid` text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The master server uuid.', + `Retry_count` bigint(20) unsigned NOT NULL COMMENT 'Number of reconnect attempts, to the master, before giving up.', + `Ssl_crl` text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The file used for the Certificate Revocation List (CRL)', + `Ssl_crlpath` text CHARACTER SET utf8 COLLATE utf8_bin COMMENT 'The path used for Certificate Revocation List (CRL) files', + `Enabled_auto_position` tinyint(1) NOT NULL COMMENT 'Indicates whether GTIDs will be used to retrieve events from the master.', + PRIMARY KEY (`Host`,`Port`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8 STATS_PERSISTENT=0 COMMENT='Master Information'; + +INSERT INTO `mysql`.`slave_master_info` VALUES (23,'master-bin.000001', 120, 'localhost', 'root'," ", 13000, 60, 0," "," "," "," "," ",0 , 60," ", " ", '28e10fdd-6289-11ea-aab9-207918567a34',10," "," ", 0 ); + +# Table structure extracted from MySQL-5.6.47 +CREATE TABLE `mysql`.`slave_relay_log_info` ( + `Number_of_lines` int(10) unsigned NOT NULL COMMENT 'Number of lines in the file or rows in the table. Used to version table definitions.', + `Relay_log_name` text CHARACTER SET utf8 COLLATE utf8_bin NOT NULL COMMENT 'The name of the current relay log file.', + `Relay_log_pos` bigint(20) unsigned NOT NULL COMMENT 'The relay log position of the last executed event.', + `Master_log_name` text CHARACTER SET utf8 COLLATE utf8_bin NOT NULL COMMENT 'The name of the master binary log file from which the events in the relay log file were read.', + `Master_log_pos` bigint(20) unsigned NOT NULL COMMENT 'The master log position of the last executed event.', + `Sql_delay` int(11) NOT NULL COMMENT 'The number of seconds that the slave must lag behind the master.', + `Number_of_workers` int(10) unsigned NOT NULL, + `Id` int(10) unsigned NOT NULL COMMENT 'Internal Id that uniquely identifies this record.', + PRIMARY KEY (`Id`) +) ENGINE=InnoDB DEFAULT CHARSET=utf8 STATS_PERSISTENT=0 COMMENT='Relay Log Information'; + +INSERT INTO `mysql`.`slave_relay_log_info` VALUES (7,'./slave-relay-bin.000001',4 ," ",0, 0 ,0 , 1); +SET SQL_LOG_BIN=1; +--enable_query_log +--enable_result_log +EOF + +--echo ******************************************************************** +--echo * Test case1: Upgrade when repository tables have data. * +--echo * mysql_upgrade script should report warnings. * +--echo ******************************************************************** +--connection master +--source $MYSQLTEST_VARDIR/tmp/slave_table_repo_init.sql +--exec $MYSQL_UPGRADE --skip-verbose --force --user=root > $MYSQLTEST_VARDIR/log/mysql_upgrade_master.log 2>&1 +--cat_file $MYSQLTEST_VARDIR/log/mysql_upgrade_master.log + +--connection slave +--source $MYSQLTEST_VARDIR/tmp/slave_table_repo_init.sql +--exec $MYSQL_UPGRADE --skip-verbose --force --user=root > $MYSQLTEST_VARDIR/log/mysql_upgrade_slave.log 2>&1 +--cat_file $MYSQLTEST_VARDIR/log/mysql_upgrade_slave.log + +--connection master +let $datadir= `select @@datadir`; +remove_file $datadir/mysql_upgrade_info; +TRUNCATE TABLE `mysql`.`slave_master_info`; +TRUNCATE TABLE `mysql`.`slave_relay_log_info`; +--remove_file $MYSQLTEST_VARDIR/log/mysql_upgrade_master.log +--remove_file $MYSQLTEST_VARDIR/log/mysql_upgrade_slave.log + +--echo ******************************************************************** +--echo * Test case2: Upgrade when repository tables are empty. * +--echo * mysql_upgrade script should not report any warning. * +--echo ******************************************************************** +--connection master +--exec $MYSQL_UPGRADE --skip-verbose --force --user=root > $MYSQLTEST_VARDIR/log/mysql_upgrade_master.log 2>&1 +--cat_file $MYSQLTEST_VARDIR/log/mysql_upgrade_master.log + +--connection slave +--exec $MYSQL_UPGRADE --skip-verbose --force --user=root > $MYSQLTEST_VARDIR/log/mysql_upgrade_slave.log 2>&1 +--cat_file $MYSQLTEST_VARDIR/log/mysql_upgrade_slave.log + +--echo "====== Clean up ======" +--connection master +let $datadir= `select @@datadir`; +remove_file $datadir/mysql_upgrade_info; +DROP TABLE `mysql`.`slave_master_info`, `mysql`.`slave_relay_log_info`; + +--remove_file $MYSQLTEST_VARDIR/tmp/slave_table_repo_init.sql +--remove_file $MYSQLTEST_VARDIR/log/mysql_upgrade_master.log +--remove_file $MYSQLTEST_VARDIR/log/mysql_upgrade_slave.log + +--source include/rpl_end.inc

1 0

[Maria-developers] Please review MDEV-17832 Protocol: extensions for Pluggable types and JSON, GEOMETRY
by Alexander Barkov 06 Mar '20

06 Mar '20

Hi, Sergei, Georg, Please review a fixed version of the patch for MDEV-17832. There are two files attached: - mdev-17821.v18.diff (server changes) - mdev-17821-cli.v06.diff (libmariadb changes) Comparing to the previous version, this version: 1. Adds a new structure MA_FIELD_EXTENSION 2. Moves extended data type information from MYSQL_FIELD to MYSQL_FIELD::extension in the client-server implementation. Note, in case of embedded server, the extended metadata is stored directly to MYSQL_FIELD. 3. Adds a new API function mariadb_field_metadata_attr(), to extract metadata from MYSQL_FIELD. 4. Changes the way how the metadata is packed on the wire from "easily human readable" to "easily parse-able", which: - makes the things faster - allows to transfer arbitrary binary data in the future, if needed. Every metadata chunk is now encoded as: a. chunk type (1 byte) b. chunk data length (1 byte) c. chunk data (according to #b) For now, two chunk types are implemented: - data type name (used for GEOMETRY sub-types, and for INET6) - format name (for JSON) Thanks!

3 4

[Maria-developers] Fwd: [andrei.elkin@mariadb.com] 30a3ee1b8ce: MDEV-21469: Implement crash-safe logging of the user XA
by andrei.elkin＠pp.inet.fi 04 Mar '20

04 Mar '20

Kristian, Fyi, here is the XA replication event recovery part. It's largely coded by Sujatha Sivakumar. The latest patch resides in bb-10.5-MDEV_21469. Cheers, Andrei revision-id: 30a3ee1b8ce98ea33dfc0595207bf467cdb91def (mariadb-10.5.0-285-g30a3ee1b8ce) parent(s): 77f4a1938f2a069174e07b3453fa0f99ba171a2e author: Sujatha committer: Andrei Elkin timestamp: 2020-03-04 14:45:01 +0200 message: MDEV-21469: Implement crash-safe logging of the user XA Description: Make XA PREPARE, XA COMMIT and XA ROLLBACK statements crash-safe. Implementation: In order to ensure consistent replication XA statements like XA PREPARE, XA COMMIT and XA ROLLBACK are firstly written into the binary log and then to storage engine. In a case server crashes after writing to binary log but not in storage engine it will lead to inconsistent state. In order to make both binary log and engine to be consistent crash recovery needs to be initiated. During crash recovery binary log needs to be parsed to identify the transactions which are present only in binary log and not present in engine. These transaction are resubmitted to engine to make it consistent along with binary log. --- .../r/binlog_xa_multi_binlog_crash_recovery.result | 40 +++ .../t/binlog_xa_multi_binlog_crash_recovery.test | 85 +++++ .../suite/rpl/r/rpl_xa_commit_crash_safe.result | 50 +++ .../suite/rpl/r/rpl_xa_event_apply_failure.result | 60 ++++ .../rpl/r/rpl_xa_prepare_commit_prepare.result | 48 +++ .../suite/rpl/r/rpl_xa_prepare_crash_safe.result | 62 ++++ .../rpl/r/rpl_xa_rollback_commit_crash_safe.result | 47 +++ .../suite/rpl/t/rpl_xa_commit_crash_safe.test | 98 ++++++ .../suite/rpl/t/rpl_xa_event_apply_failure.test | 119 +++++++ .../suite/rpl/t/rpl_xa_prepare_commit_prepare.test | 95 ++++++ .../suite/rpl/t/rpl_xa_prepare_crash_safe.test | 117 +++++++ .../rpl/t/rpl_xa_rollback_commit_crash_safe.test | 97 ++++++ sql/handler.cc | 29 +- sql/handler.h | 12 +- sql/log.cc | 379 +++++++++++++++++++-- sql/log.h | 8 +- sql/log_event.cc | 18 +- sql/log_event.h | 15 +- sql/mysqld.cc | 4 +- sql/xa.cc | 4 + 20 files changed, 1353 insertions(+), 34 deletions(-) diff --git a/mysql-test/suite/binlog/r/binlog_xa_multi_binlog_crash_recovery.result b/mysql-test/suite/binlog/r/binlog_xa_multi_binlog_crash_recovery.result new file mode 100644 index 00000000000..a09472aa94c --- /dev/null +++ b/mysql-test/suite/binlog/r/binlog_xa_multi_binlog_crash_recovery.result @@ -0,0 +1,40 @@ +RESET MASTER; +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +CALL mtr.add_suppression("Found 1 prepared XA transactions"); +connect con1,localhost,root,,; +SET DEBUG_SYNC= "simulate_hang_after_binlog_prepare SIGNAL con1_ready WAIT_FOR con1_go"; +SET GLOBAL DEBUG_DBUG="d,simulate_crash_after_binlog_prepare"; +XA START 'xa1'; +INSERT INTO t1 SET a=1; +XA END 'xa1'; +XA PREPARE 'xa1';; +connection default; +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; +FLUSH LOGS; +FLUSH LOGS; +FLUSH LOGS; +show binary logs; +Log_name File_size +master-bin.000001 # +master-bin.000002 # +master-bin.000003 # +master-bin.000004 # +include/show_binlog_events.inc +Log_name Pos Event_type Server_id End_log_pos Info +master-bin.000004 # Format_desc # # SERVER_VERSION, BINLOG_VERSION +master-bin.000004 # Gtid_list # # [#-#-#] +master-bin.000004 # Binlog_checkpoint # # master-bin.000001 +SET DEBUG_SYNC= "now SIGNAL con1_go"; +connection con1; +ERROR HY000: Lost connection to MySQL server during query +connection default; +XA RECOVER; +formatID gtrid_length bqual_length data +1 3 0 xa1 +XA COMMIT 'xa1'; +SELECT * FROM t1; +a b +1 NULL +connection default; +DROP TABLE t1; +SET debug_sync = 'reset'; diff --git a/mysql-test/suite/binlog/t/binlog_xa_multi_binlog_crash_recovery.test b/mysql-test/suite/binlog/t/binlog_xa_multi_binlog_crash_recovery.test new file mode 100644 index 00000000000..aa8d3d04fc7 --- /dev/null +++ b/mysql-test/suite/binlog/t/binlog_xa_multi_binlog_crash_recovery.test @@ -0,0 +1,85 @@ +# ==== Purpose ==== +# +# Test verifies that XA crash recovery works fine across multiple binary logs. +# +# ==== Implementation ==== +# +# Steps: +# 0 - Generate an explicit XA transaction. Using debug simulation hold the +# execution of XA PREPARE statement after the XA PREPARE is written to +# the binary log. With this the prepare will not be done in engine. +# 1 - By executing FLUSH LOGS generate multiple binary logs. +# 2 - Now make the server to disappear at this point. +# 3 - Restart the server. During recovery the XA PREPARE from the binary +# log will be read. It is cross checked with engine. Since it is not +# present in engine it will be executed once again. +# 4 - When server is up execute XA RECOVER to check that the XA is +# prepared in engine as well. +# 5 - XA COMMIT the transaction and check the validity of the data. +# +# ==== References ==== +# +# MDEV-21469: Implement crash-safe logging of the user XA +# + +--source include/have_innodb.inc +--source include/have_debug.inc +--source include/have_debug_sync.inc +--source include/have_log_bin.inc + +RESET MASTER; + +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +CALL mtr.add_suppression("Found 1 prepared XA transactions"); + +connect(con1,localhost,root,,); +SET DEBUG_SYNC= "simulate_hang_after_binlog_prepare SIGNAL con1_ready WAIT_FOR con1_go"; +SET GLOBAL DEBUG_DBUG="d,simulate_crash_after_binlog_prepare"; +XA START 'xa1'; +INSERT INTO t1 SET a=1; +XA END 'xa1'; +--send XA PREPARE 'xa1'; + +connection default; +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; +FLUSH LOGS; +FLUSH LOGS; +FLUSH LOGS; + +--source include/show_binary_logs.inc +--let $binlog_file= master-bin.000004 +--let $binlog_start= 4 +--source include/show_binlog_events.inc + +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +wait +EOF + +SET DEBUG_SYNC= "now SIGNAL con1_go"; +--source include/wait_until_disconnected.inc + +--connection con1 +--error 2013 +--reap +--source include/wait_until_disconnected.inc + +# +# Server restart +# +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart +EOF + +connection default; +--enable_reconnect +--source include/wait_until_connected_again.inc + +XA RECOVER; +XA COMMIT 'xa1'; + +SELECT * FROM t1; + +# Clean up. +connection default; +DROP TABLE t1; +SET debug_sync = 'reset'; diff --git a/mysql-test/suite/rpl/r/rpl_xa_commit_crash_safe.result b/mysql-test/suite/rpl/r/rpl_xa_commit_crash_safe.result new file mode 100644 index 00000000000..27d043270ea --- /dev/null +++ b/mysql-test/suite/rpl/r/rpl_xa_commit_crash_safe.result @@ -0,0 +1,50 @@ +include/master-slave.inc +[connection master] +connect master2,localhost,root,,; +connection master; +CALL mtr.add_suppression("Found 1 prepared XA transactions"); +CREATE TABLE t ( f INT ) ENGINE=INNODB; +XA START 'xa1'; +INSERT INTO t VALUES (20); +XA END 'xa1'; +XA PREPARE 'xa1'; +XA COMMIT 'xa1'; +connection slave; +include/stop_slave.inc +connection master1; +XA START 'xa2'; +INSERT INTO t VALUES (40); +XA END 'xa2'; +XA PREPARE 'xa2'; +SET GLOBAL DEBUG_DBUG="d,simulate_crash_after_binlog_commit"; +XA COMMIT 'xa2'; +ERROR HY000: Lost connection to MySQL server during query +connection master1; +connection master; +connection default; +connection server_1; +connection master; +connection slave; +include/start_slave.inc +connection master; +SELECT * FROM t; +f +20 +40 +XA RECOVER; +formatID gtrid_length bqual_length data +XA COMMIT 'xa2'; +ERROR XAE04: XAER_NOTA: Unknown XID +SELECT * FROM t; +f +20 +40 +connection slave; +SELECT * FROM t; +f +20 +40 +connection master; +DROP TABLE t; +connection slave; +include/rpl_end.inc diff --git a/mysql-test/suite/rpl/r/rpl_xa_event_apply_failure.result b/mysql-test/suite/rpl/r/rpl_xa_event_apply_failure.result new file mode 100644 index 00000000000..547a0aae9a6 --- /dev/null +++ b/mysql-test/suite/rpl/r/rpl_xa_event_apply_failure.result @@ -0,0 +1,60 @@ +include/master-slave.inc +[connection master] +connect master2,localhost,root,,; +connection master; +CALL mtr.add_suppression("Found 1 prepared XA transactions"); +CALL mtr.add_suppression("Failed to execute binlog query event"); +CALL mtr.add_suppression("Recovery: Error .Out of memory.."); +CALL mtr.add_suppression("Crash recovery failed."); +CALL mtr.add_suppression("Can.t init tc log"); +CALL mtr.add_suppression("Aborting"); +CREATE TABLE t ( f INT ) ENGINE=INNODB; +XA START 'xa1'; +INSERT INTO t VALUES (20); +XA END 'xa1'; +XA PREPARE 'xa1'; +XA COMMIT 'xa1'; +connection slave; +include/stop_slave.inc +connection master1; +XA START 'xa2'; +INSERT INTO t VALUES (40); +XA END 'xa2'; +XA PREPARE 'xa2'; +SET GLOBAL DEBUG_DBUG="d,simulate_crash_after_binlog_commit"; +XA COMMIT 'xa2'; +ERROR HY000: Lost connection to MySQL server during query +connection master1; +connection master; +connection default; +connection default; +connection master; +*** must be no 'xa2' commit seen, as it's still prepared: +SELECT * FROM t; +f +20 +XA RECOVER; +formatID gtrid_length bqual_length data +1 3 0 xa2 +SET GLOBAL DEBUG_DBUG=""; +SET SQL_LOG_BIN=0; +XA COMMIT 'xa2'; +SET SQL_LOG_BIN=1; +connection server_1; +connection master; +connection slave; +include/start_slave.inc +connection master; +SELECT * FROM t; +f +20 +40 +connection slave; +SELECT * FROM t; +f +20 +40 +connection master; +DROP TABLE t; +connection slave; +include/rpl_end.inc diff --git a/mysql-test/suite/rpl/r/rpl_xa_prepare_commit_prepare.result b/mysql-test/suite/rpl/r/rpl_xa_prepare_commit_prepare.result new file mode 100644 index 00000000000..9ba24716639 --- /dev/null +++ b/mysql-test/suite/rpl/r/rpl_xa_prepare_commit_prepare.result @@ -0,0 +1,48 @@ +include/master-slave.inc +[connection master] +connect master2,localhost,root,,; +connection master; +CALL mtr.add_suppression("Found 1 prepared XA transactions"); +CREATE TABLE t ( f INT ) ENGINE=INNODB; +XA START 'xa1'; +INSERT INTO t VALUES (20); +XA END 'xa1'; +XA PREPARE 'xa1'; +XA COMMIT 'xa1'; +connection slave; +include/stop_slave.inc +connection master1; +XA START 'xa2'; +INSERT INTO t VALUES (40); +XA END 'xa2'; +SET GLOBAL DEBUG_DBUG="d,simulate_crash_after_binlog_prepare"; +XA PREPARE 'xa2'; +ERROR HY000: Lost connection to MySQL server during query +connection master1; +connection master; +connection default; +connection server_1; +connection master; +connection slave; +include/start_slave.inc +connection master; +SELECT * FROM t; +f +20 +XA RECOVER; +formatID gtrid_length bqual_length data +1 3 0 xa2 +XA COMMIT 'xa2'; +SELECT * FROM t; +f +20 +40 +connection slave; +SELECT * FROM t; +f +20 +40 +connection master; +DROP TABLE t; +connection slave; +include/rpl_end.inc diff --git a/mysql-test/suite/rpl/r/rpl_xa_prepare_crash_safe.result b/mysql-test/suite/rpl/r/rpl_xa_prepare_crash_safe.result new file mode 100644 index 00000000000..99baf59a3c1 --- /dev/null +++ b/mysql-test/suite/rpl/r/rpl_xa_prepare_crash_safe.result @@ -0,0 +1,62 @@ +include/master-slave.inc +[connection master] +connect master2,localhost,root,,; +connection master; +CALL mtr.add_suppression("Found 1 prepared XA transactions"); +CALL mtr.add_suppression("Found 2 prepared XA transactions"); +CALL mtr.add_suppression("Found 3 prepared XA transactions"); +CREATE TABLE t ( f INT ) ENGINE=INNODB; +XA START 'xa1'; +INSERT INTO t VALUES (20); +XA END 'xa1'; +XA PREPARE 'xa1'; +connection slave; +include/stop_slave.inc +connection master1; +use test; +xa start 'xa2'; +insert into t values (30); +xa end 'xa2'; +SET DEBUG_SYNC="simulate_hang_after_binlog_prepare SIGNAL reached WAIT_FOR go"; +xa prepare 'xa2'; +connection master2; +XA START 'xa3'; +INSERT INTO t VALUES (40); +XA END 'xa3'; +SET GLOBAL DEBUG_DBUG="d,simulate_crash_after_binlog_prepare"; +XA PREPARE 'xa3'; +ERROR HY000: Lost connection to MySQL server during query +connection master1; +ERROR HY000: Lost connection to MySQL server during query +connection master; +connection default; +connection server_1; +connection master; +connection slave; +include/start_slave.inc +connection master; +SELECT * FROM t; +f +XA RECOVER; +formatID gtrid_length bqual_length data +1 3 0 xa3 +1 3 0 xa1 +1 3 0 xa2 +XA COMMIT 'xa1'; +XA COMMIT 'xa2'; +XA COMMIT 'xa3'; +SELECT * FROM t; +f +20 +30 +40 +connection slave; +SELECT * FROM t; +f +20 +30 +40 +connection master; +DROP TABLE t; +connection slave; +include/rpl_end.inc diff --git a/mysql-test/suite/rpl/r/rpl_xa_rollback_commit_crash_safe.result b/mysql-test/suite/rpl/r/rpl_xa_rollback_commit_crash_safe.result new file mode 100644 index 00000000000..bc48c84e1c7 --- /dev/null +++ b/mysql-test/suite/rpl/r/rpl_xa_rollback_commit_crash_safe.result @@ -0,0 +1,47 @@ +include/master-slave.inc +[connection master] +connect master2,localhost,root,,; +connection master; +CALL mtr.add_suppression("Found 1 prepared XA transactions"); +CREATE TABLE t ( f INT ) ENGINE=INNODB; +XA START 'xa1'; +INSERT INTO t VALUES (20); +XA END 'xa1'; +XA PREPARE 'xa1'; +XA COMMIT 'xa1'; +connection slave; +include/stop_slave.inc +connection master1; +XA START 'xa2'; +INSERT INTO t VALUES (40); +XA END 'xa2'; +XA PREPARE 'xa2'; +SET GLOBAL DEBUG_DBUG="d,simulate_crash_after_binlog_rollback"; +XA ROLLBACK 'xa2'; +ERROR HY000: Lost connection to MySQL server during query +connection master1; +connection master; +connection default; +connection server_1; +connection master; +connection slave; +include/start_slave.inc +connection master; +SELECT * FROM t; +f +20 +XA RECOVER; +formatID gtrid_length bqual_length data +XA ROLLBACK 'xa2'; +ERROR XAE04: XAER_NOTA: Unknown XID +SELECT * FROM t; +f +20 +connection slave; +SELECT * FROM t; +f +20 +connection master; +DROP TABLE t; +connection slave; +include/rpl_end.inc diff --git a/mysql-test/suite/rpl/t/rpl_xa_commit_crash_safe.test b/mysql-test/suite/rpl/t/rpl_xa_commit_crash_safe.test new file mode 100644 index 00000000000..b9e3b0d3d0d --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_xa_commit_crash_safe.test @@ -0,0 +1,98 @@ +# ==== Purpose ==== +# +# Test verifies that XA COMMIT statements are crash safe. +# +# ==== Implementation ==== +# +# Steps: +# 0 - Generate 2 explicit XA transactions. 'xa1' and 'xa2'. +# 'xa1' will be prepared and committed. +# 1 - For 'xa2' let the XA COMMIT be done in binary log and crash the +# server so that it is not committed in engine. +# 2 - Restart the server. The recovery code should successfully recover +# 'xa2'. The COMMIT should be executed during recovery. +# 3 - Check the data in table. Both rows should be present in table. +# 4 - Trying to commit 'xa2' should report unknown 'XA' error as COMMIT is +# already complete during recovery. +# +# ==== References ==== +# +# MDEV-21469: Implement crash-safe logging of the user XA + + +--source include/have_innodb.inc +--source include/master-slave.inc +--source include/have_debug.inc + +connect (master2,localhost,root,,); +--connection master +CALL mtr.add_suppression("Found 1 prepared XA transactions"); + +CREATE TABLE t ( f INT ) ENGINE=INNODB; +XA START 'xa1'; +INSERT INTO t VALUES (20); +XA END 'xa1'; +XA PREPARE 'xa1'; +XA COMMIT 'xa1'; +--sync_slave_with_master +--source include/stop_slave.inc + +--connection master1 +XA START 'xa2'; +INSERT INTO t VALUES (40); +XA END 'xa2'; +XA PREPARE 'xa2'; + +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +wait +EOF + +SET GLOBAL DEBUG_DBUG="d,simulate_crash_after_binlog_commit"; +--error 2013 # CR_SERVER_LOST +XA COMMIT 'xa2'; +--source include/wait_until_disconnected.inc + +--connection master1 +--source include/wait_until_disconnected.inc + +--connection master +--source include/wait_until_disconnected.inc + +# +# Server restart +# +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart +EOF + +connection default; +--enable_reconnect +--source include/wait_until_connected_again.inc + +# rpl_end.inc needs to use the connection server_1 +connection server_1; +--enable_reconnect +--source include/wait_until_connected_again.inc + +--connection master +--enable_reconnect +--source include/wait_until_connected_again.inc + +--connection slave +--source include/start_slave.inc +--sync_with_master + +--connection master +SELECT * FROM t; +XA RECOVER; +--error 1397 # ER_XAER_NOTA +XA COMMIT 'xa2'; +SELECT * FROM t; +--sync_slave_with_master + +SELECT * FROM t; + +--connection master +DROP TABLE t; +--sync_slave_with_master +--source include/rpl_end.inc diff --git a/mysql-test/suite/rpl/t/rpl_xa_event_apply_failure.test b/mysql-test/suite/rpl/t/rpl_xa_event_apply_failure.test new file mode 100644 index 00000000000..71d0de0fc56 --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_xa_event_apply_failure.test @@ -0,0 +1,119 @@ +# ==== Purpose ==== +# +# Test verifies that if for some reason an event cannot be applied during +# recovery, appropriate error is reported. +# +# ==== Implementation ==== +# +# Steps: +# 0 - Generate 2 explicit XA transactions. 'xa1' and 'xa2'. +# 'xa1' will be prepared and committed. +# 1 - For 'xa2' let the XA COMMIT be done in binary log and crash the +# server so that it is not committed in engine. +# 2 - Restart the server. Using debug simulation point make XA COMMIT 'xa2' +# execution to fail. The server will resume anyway +# to leave the error in the errlog (see "Recovery: Error.."). +# 3 - Work around the simulated failure with Commit once again +# from a connection that turns OFF binlogging. +# Slave must catch up with the master. +# +# ==== References ==== +# +# MDEV-21469: Implement crash-safe logging of the user XA + + +--source include/have_innodb.inc +--source include/master-slave.inc +--source include/have_debug.inc + +connect (master2,localhost,root,,); +--connection master +CALL mtr.add_suppression("Found 1 prepared XA transactions"); +CALL mtr.add_suppression("Failed to execute binlog query event"); +CALL mtr.add_suppression("Recovery: Error .Out of memory.."); +CALL mtr.add_suppression("Crash recovery failed."); +CALL mtr.add_suppression("Can.t init tc log"); +CALL mtr.add_suppression("Aborting"); + +CREATE TABLE t ( f INT ) ENGINE=INNODB; +XA START 'xa1'; +INSERT INTO t VALUES (20); +XA END 'xa1'; +XA PREPARE 'xa1'; +XA COMMIT 'xa1'; +--sync_slave_with_master +--source include/stop_slave.inc + +--connection master1 +XA START 'xa2'; +INSERT INTO t VALUES (40); +XA END 'xa2'; +XA PREPARE 'xa2'; + +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +wait +EOF + +SET GLOBAL DEBUG_DBUG="d,simulate_crash_after_binlog_commit"; +--error 2013 # CR_SERVER_LOST +XA COMMIT 'xa2'; +--source include/wait_until_disconnected.inc + +--connection master1 +--source include/wait_until_disconnected.inc + +--connection master +--source include/wait_until_disconnected.inc + +# +# Server restart +# +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: --debug-dbug=d,trans_xa_commit_fail +EOF + +connection default; +--source include/wait_until_disconnected.inc + +connection default; +--enable_reconnect +--source include/wait_until_connected_again.inc + +--connection master +--enable_reconnect +--echo *** must be no 'xa2' commit seen, as it's still prepared: +SELECT * FROM t; +XA RECOVER; + +# Commit it manually now to work around the extra binlog record +# by turning binlogging OFF by the connection. + +SET GLOBAL DEBUG_DBUG=""; +SET SQL_LOG_BIN=0; +--error 0 +XA COMMIT 'xa2'; +SET SQL_LOG_BIN=1; + + +# rpl_end.inc needs to use the connection server_1 +connection server_1; +--enable_reconnect +--source include/wait_until_connected_again.inc + +--connection master +--source include/wait_until_connected_again.inc + +--connection slave +--source include/start_slave.inc +--sync_with_master + +--connection master +SELECT * FROM t; + +--sync_slave_with_master +SELECT * FROM t; + +--connection master +DROP TABLE t; +--sync_slave_with_master +--source include/rpl_end.inc diff --git a/mysql-test/suite/rpl/t/rpl_xa_prepare_commit_prepare.test b/mysql-test/suite/rpl/t/rpl_xa_prepare_commit_prepare.test new file mode 100644 index 00000000000..7b987c7f29b --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_xa_prepare_commit_prepare.test @@ -0,0 +1,95 @@ +# ==== Purpose ==== +# +# Test verifies that XA PREPARE transactions are crash safe. +# +# ==== Implementation ==== +# +# Steps: +# 0 - Generate 2 explicit XA transactions. 'xa1' and 'xa2'. +# 'xa1' will be prepared and committed. +# 1 - For 'xa2' let the XA PREPARE be done in binary log and crash the +# server so that it is not prepared in engine. +# 2 - Restart the server. The recovery code should successfully recover +# 'xa2'. +# 3 - When server is up, execute XA RECOVER and verify that 'xa2' is +# present. +# 4 - Commit the XA transaction and verify its correctness. +# +# ==== References ==== +# +# MDEV-21469: Implement crash-safe logging of the user XA + +--source include/have_innodb.inc +--source include/master-slave.inc +--source include/have_debug.inc + +connect (master2,localhost,root,,); +--connection master +CALL mtr.add_suppression("Found 1 prepared XA transactions"); + +CREATE TABLE t ( f INT ) ENGINE=INNODB; +XA START 'xa1'; +INSERT INTO t VALUES (20); +XA END 'xa1'; +XA PREPARE 'xa1'; +XA COMMIT 'xa1'; +--sync_slave_with_master +--source include/stop_slave.inc + +--connection master1 +XA START 'xa2'; +INSERT INTO t VALUES (40); +XA END 'xa2'; + +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +wait +EOF + +SET GLOBAL DEBUG_DBUG="d,simulate_crash_after_binlog_prepare"; +--error 2013 # CR_SERVER_LOST +XA PREPARE 'xa2'; +--source include/wait_until_disconnected.inc + +--connection master1 +--source include/wait_until_disconnected.inc + +--connection master +--source include/wait_until_disconnected.inc + +# +# Server restart +# +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart +EOF + +connection default; +--enable_reconnect +--source include/wait_until_connected_again.inc + +# rpl_end.inc needs to use the connection server_1 +connection server_1; +--enable_reconnect +--source include/wait_until_connected_again.inc + +--connection master +--enable_reconnect +--source include/wait_until_connected_again.inc + +--connection slave +--source include/start_slave.inc +--sync_with_master + +--connection master +SELECT * FROM t; +XA RECOVER; +XA COMMIT 'xa2'; +SELECT * FROM t; +--sync_slave_with_master + +SELECT * FROM t; + +--connection master +DROP TABLE t; +--sync_slave_with_master +--source include/rpl_end.inc diff --git a/mysql-test/suite/rpl/t/rpl_xa_prepare_crash_safe.test b/mysql-test/suite/rpl/t/rpl_xa_prepare_crash_safe.test new file mode 100644 index 00000000000..9d2c5cce528 --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_xa_prepare_crash_safe.test @@ -0,0 +1,117 @@ +# ==== Purpose ==== +# +# Test verifies that XA PREPARE transactions are crash safe. +# +# ==== Implementation ==== +# +# Steps: +# 0 - Generate 3 explicit XA transactions. 'xa1', 'xa2' and 'xa3'. +# Using debug simulation hold the execution of second XA PREPARE +# statement after the XA PREPARE is written to the binary log. +# With this the prepare will not be done in engine. +# 1 - For 'xa3' allow the PREPARE statement to be written to binary log and +# simulate server crash. +# 2 - Restart the server. The recovery code should successfully recover +# 'xa2' and 'xa3'. +# 3 - When server is up, execute XA RECOVER and verify that 'xa2' and 'xa3' +# are present along with 'xa1'. +# 4 - Commit all the XA transactions and verify their correctness. +# +# ==== References ==== +# +# MDEV-21469: Implement crash-safe logging of the user XA + + +--source include/have_innodb.inc +--source include/master-slave.inc +--source include/have_debug.inc + +connect (master2,localhost,root,,); +--connection master +CALL mtr.add_suppression("Found 1 prepared XA transactions"); +CALL mtr.add_suppression("Found 2 prepared XA transactions"); +CALL mtr.add_suppression("Found 3 prepared XA transactions"); + +CREATE TABLE t ( f INT ) ENGINE=INNODB; +XA START 'xa1'; +INSERT INTO t VALUES (20); +XA END 'xa1'; +XA PREPARE 'xa1'; +--sync_slave_with_master +--source include/stop_slave.inc + +--connection master1 +use test; +xa start 'xa2'; +insert into t values (30); +xa end 'xa2'; +SET DEBUG_SYNC="simulate_hang_after_binlog_prepare SIGNAL reached WAIT_FOR go"; +send xa prepare 'xa2'; + +--connection master2 +let $wait_condition= + SELECT COUNT(*) = 1 FROM INFORMATION_SCHEMA.PROCESSLIST + WHERE STATE like "debug sync point: simulate_hang_after_binlog_prepare%"; +--source include/wait_condition.inc + +XA START 'xa3'; +INSERT INTO t VALUES (40); +XA END 'xa3'; + +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +wait +EOF + +SET GLOBAL DEBUG_DBUG="d,simulate_crash_after_binlog_prepare"; +--error 2013 # CR_SERVER_LOST +XA PREPARE 'xa3'; +--source include/wait_until_disconnected.inc + +--connection master1 +--error 2013 +--reap +--source include/wait_until_disconnected.inc + +--connection master +--source include/wait_until_disconnected.inc + +# +# Server restart +# +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart +EOF + +connection default; +--enable_reconnect +--source include/wait_until_connected_again.inc + +# rpl_end.inc needs to use the connection server_1 +connection server_1; +--enable_reconnect +--source include/wait_until_connected_again.inc + +--connection master +--enable_reconnect +--source include/wait_until_connected_again.inc + + +--connection slave +--source include/start_slave.inc +--sync_with_master + +--connection master +SELECT * FROM t; +XA RECOVER; +XA COMMIT 'xa1'; +XA COMMIT 'xa2'; +XA COMMIT 'xa3'; +SELECT * FROM t; +--sync_slave_with_master + +SELECT * FROM t; + +--connection master +DROP TABLE t; +--sync_slave_with_master +--source include/rpl_end.inc diff --git a/mysql-test/suite/rpl/t/rpl_xa_rollback_commit_crash_safe.test b/mysql-test/suite/rpl/t/rpl_xa_rollback_commit_crash_safe.test new file mode 100644 index 00000000000..6416602da5e --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_xa_rollback_commit_crash_safe.test @@ -0,0 +1,97 @@ +# ==== Purpose ==== +# +# Test verifies that XA COMMIT statements are crash safe. +# +# ==== Implementation ==== +# +# Steps: +# 0 - Generate 2 explicit XA transactions. 'xa1' and 'xa2'. +# 'xa1' will be prepared and committed. +# 1 - For 'xa2' let the XA ROLLBACK be done in binary log and crash the +# server so that it is not committed in engine. +# 2 - Restart the server. The recovery code should successfully recover +# 'xa2'. The ROLLBACK should be executed during recovery. +# 3 - Check the data in table. Only one row should be present in table. +# 4 - Trying to rollback 'xa2' should report unknown 'XA' error as rollback +# is already complete during recovery. +# +# ==== References ==== +# +# MDEV-21469: Implement crash-safe logging of the user XA + +--source include/have_innodb.inc +--source include/master-slave.inc +--source include/have_debug.inc + +connect (master2,localhost,root,,); +--connection master +CALL mtr.add_suppression("Found 1 prepared XA transactions"); + +CREATE TABLE t ( f INT ) ENGINE=INNODB; +XA START 'xa1'; +INSERT INTO t VALUES (20); +XA END 'xa1'; +XA PREPARE 'xa1'; +XA COMMIT 'xa1'; +--sync_slave_with_master +--source include/stop_slave.inc + +--connection master1 +XA START 'xa2'; +INSERT INTO t VALUES (40); +XA END 'xa2'; +XA PREPARE 'xa2'; + +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +wait +EOF + +SET GLOBAL DEBUG_DBUG="d,simulate_crash_after_binlog_rollback"; +--error 2013 # CR_SERVER_LOST +XA ROLLBACK 'xa2'; +--source include/wait_until_disconnected.inc + +--connection master1 +--source include/wait_until_disconnected.inc + +--connection master +--source include/wait_until_disconnected.inc + +# +# Server restart +# +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart +EOF + +connection default; +--enable_reconnect +--source include/wait_until_connected_again.inc + +# rpl_end.inc needs to use the connection server_1 +connection server_1; +--enable_reconnect +--source include/wait_until_connected_again.inc + +--connection master +--enable_reconnect +--source include/wait_until_connected_again.inc + +--connection slave +--source include/start_slave.inc +--sync_with_master + +--connection master +SELECT * FROM t; +XA RECOVER; +--error 1397 # ER_XAER_NOTA +XA ROLLBACK 'xa2'; +SELECT * FROM t; +--sync_slave_with_master + +SELECT * FROM t; + +--connection master +DROP TABLE t; +--sync_slave_with_master +--source include/rpl_end.inc diff --git a/sql/handler.cc b/sql/handler.cc index a1719f9b922..c997c52e602 100644 --- a/sql/handler.cc +++ b/sql/handler.cc @@ -1290,6 +1290,9 @@ int ha_prepare(THD *thd) error=1; break; } + DEBUG_SYNC(thd, "simulate_hang_after_binlog_prepare"); + DBUG_EXECUTE_IF("simulate_crash_after_binlog_prepare", + DBUG_SUICIDE();); } else { @@ -1795,6 +1798,8 @@ commit_one_phase_2(THD *thd, bool all, THD_TRANS *trans, bool is_real_trans) ++count; ha_info_next= ha_info->next(); ha_info->reset(); /* keep it conveniently zero-filled */ + DBUG_EXECUTE_IF("simulate_crash_after_binlog_commit", + DBUG_SUICIDE();); } trans->ha_list= 0; trans->no_2pc=0; @@ -1908,6 +1913,8 @@ int ha_rollback_trans(THD *thd, bool all) status_var_increment(thd->status_var.ha_rollback_count); ha_info_next= ha_info->next(); ha_info->reset(); /* keep it conveniently zero-filled */ + DBUG_EXECUTE_IF("simulate_crash_after_binlog_rollback", + DBUG_SUICIDE();); } trans->ha_list= 0; trans->no_2pc=0; @@ -2107,6 +2114,7 @@ struct xarecover_st int len, found_foreign_xids, found_my_xids; XID *list; HASH *commit_list; + HASH *xa_prepared_list; bool dry_run; }; @@ -2155,7 +2163,23 @@ static my_bool xarecover_handlerton(THD *unused, plugin_ref plugin, _db_doprnt_("ignore xid %s", xid_to_str(buf, info->list+i)); }); xid_cache_insert(info->list + i, true); + XID *foreign_xid= info->list + i; info->found_foreign_xids++; + + /* + For each foreign xid prepraed in engine, check if it is present in + xa_prepared_list sent by binlog. + */ + if (info->xa_prepared_list) + { + struct xa_recovery_member *member= NULL; + if ((member= (xa_recovery_member *) + my_hash_search(info->xa_prepared_list, foreign_xid->key(), + foreign_xid->key_length()))) + { + member->in_engine_prepare= true; + } + } continue; } if (IF_WSREP(!(wsrep_emulate_bin_log && @@ -2202,12 +2226,13 @@ static my_bool xarecover_handlerton(THD *unused, plugin_ref plugin, return FALSE; } -int ha_recover(HASH *commit_list) +int ha_recover(HASH *commit_list, HASH *xa_prepared_list) { struct xarecover_st info; DBUG_ENTER("ha_recover"); info.found_foreign_xids= info.found_my_xids= 0; info.commit_list= commit_list; + info.xa_prepared_list= xa_prepared_list; info.dry_run= (info.commit_list==0 && tc_heuristic_recover==0); info.list= NULL; @@ -2254,7 +2279,7 @@ int ha_recover(HASH *commit_list) info.found_my_xids, opt_tc_log_file); DBUG_RETURN(1); } - if (info.commit_list) + if (info.commit_list && !info.found_foreign_xids) sql_print_information("Crash recovery finished."); DBUG_RETURN(0); } diff --git a/sql/handler.h b/sql/handler.h index 92c2a61ed0e..4ff08d7bd08 100644 --- a/sql/handler.h +++ b/sql/handler.h @@ -521,6 +521,8 @@ enum legacy_db_type DB_TYPE_FIRST_DYNAMIC=45, DB_TYPE_DEFAULT=127 // Must be last }; + +enum xa_binlog_state {XA_PREPARE=0, XA_COMPLETE}; /* Better name for DB_TYPE_UNKNOWN. Should be used for engines that do not have a hard-coded type value here. @@ -806,7 +808,6 @@ struct st_system_tablename const char *tablename; }; - typedef ulonglong my_xid; // this line is the same as in log_event.h #define MYSQL_XID_PREFIX "MySQLXid" #define MYSQL_XID_PREFIX_LEN 8 // must be a multiple of 8 @@ -898,6 +899,13 @@ struct xid_t { }; typedef struct xid_t XID; +struct xa_recovery_member +{ + XID xid; + enum xa_binlog_state state; + bool in_engine_prepare; +}; + /* for recover() handlerton call */ #define MIN_XID_LIST_SIZE 128 #define MAX_XID_LIST_SIZE (1024*128) @@ -4996,7 +5004,7 @@ int ha_commit_one_phase(THD *thd, bool all); int ha_commit_trans(THD *thd, bool all); int ha_rollback_trans(THD *thd, bool all); int ha_prepare(THD *thd); -int ha_recover(HASH *commit_list); +int ha_recover(HASH *commit_list, HASH *xa_recover_list); /* transactions: these functions never call handlerton functions directly */ int ha_enable_transaction(THD *thd, bool on); diff --git a/sql/log.cc b/sql/log.cc index e13f8fbc88f..ed6dc87b262 100644 --- a/sql/log.cc +++ b/sql/log.cc @@ -38,6 +38,7 @@ #include "log_event.h" // Query_log_event #include "rpl_filter.h" #include "rpl_rli.h" +#include "rpl_mi.h" #include "sql_audit.h" #include "mysqld.h" @@ -3406,6 +3407,8 @@ MYSQL_BIN_LOG::MYSQL_BIN_LOG(uint *sync_period) index_file_name[0] = 0; bzero((char*) &index_file, sizeof(index_file)); bzero((char*) &purge_index_file, sizeof(purge_index_file)); + /* non-zero is a marker to conduct xa recovery and related cleanup */ + xa_recover_list.records= 0; } void MYSQL_BIN_LOG::stop_background_thread() @@ -3467,6 +3470,11 @@ void MYSQL_BIN_LOG::cleanup() mysql_cond_destroy(&COND_xid_list); mysql_cond_destroy(&COND_binlog_background_thread); mysql_cond_destroy(&COND_binlog_background_thread_end); + if (!is_relay_log && xa_recover_list.records) + { + free_root(&mem_root, MYF(0)); + my_hash_free(&xa_recover_list); + } } /* @@ -8028,7 +8036,7 @@ MYSQL_BIN_LOG::trx_group_commit_leader(group_commit_entry *leader) /* Now we have in queue the list of transactions to be committed in order. */ } - + DBUG_ASSERT(is_open()); if (likely(is_open())) // Should always be true { @@ -9717,7 +9725,7 @@ int TC_LOG_MMAP::recover() goto err2; // OOM } - if (ha_recover(&xids)) + if (ha_recover(&xids, 0)) goto err2; my_hash_free(&xids); @@ -9758,7 +9766,7 @@ int TC_LOG::using_heuristic_recover() return 0; sql_print_information("Heuristic crash recovery mode"); - if (ha_recover(0)) + if (ha_recover(0, 0)) sql_print_error("Heuristic crash recovery failed"); sql_print_information("Please restart mysqld without --tc-heuristic-recover"); return 1; @@ -10217,14 +10225,108 @@ start_binlog_background_thread() return 0; } +/** + Auxiliary function for ::recover(). + @returns a successfully created and inserted @c xa_recovery_member + into hash @c hash_arg, + or NULL. +*/ +static xa_recovery_member* +xa_member_insert(HASH *hash_arg, xid_t *xid_arg, xa_binlog_state state_arg, + MEM_ROOT *ptr_mem_root) +{ + xa_recovery_member *member= (xa_recovery_member*) + alloc_root(ptr_mem_root, sizeof(xa_recovery_member)); + if (!member) + return NULL; + + member->xid.set(xid_arg); + member->state= state_arg; + member->in_engine_prepare= false; + return my_hash_insert(hash_arg, (uchar*) member) ? NULL : member; +} +/* Inserts or update an existing hash member with a proper state */ +static bool xa_member_replace(HASH *hash_arg, xid_t *xid_arg, bool is_prepare, + MEM_ROOT *ptr_mem_root) +{ + if(is_prepare) + { + if (!(xa_member_insert(hash_arg, xid_arg, XA_PREPARE, ptr_mem_root))) + return true; + } + else + { + /* + Search if XID is already present in recovery_list. If found + and the state is 'XA_PREPRAED' mark it as XA_COMPLETE. + Effectively, there won't be XA-prepare event group replay. + */ + xa_recovery_member* member; + if ((member= (xa_recovery_member *) + my_hash_search(hash_arg, xid_arg->key(), xid_arg->key_length()))) + { + if (member->state == XA_PREPARE) + member->state= XA_COMPLETE; + } + else // We found only XA COMMIT during recovery insert to list + { + if (!(member= xa_member_insert(hash_arg, + xid_arg, XA_COMPLETE, ptr_mem_root))) + return true; + } + } + return false; +} + +extern "C" uchar *xid_get_var_key(xid_t *entry, size_t *length, + my_bool not_used __attribute__((unused))) +{ + *length= entry->key_length(); + return (uchar*) entry->key(); +} + +/** + Performs recovery based on transaction coordinator log for 2pc. At the + time of crash, if the binary log was in active state, then recovery for + 'xid's and explicit 'XA' transactions is initiated, otherwise the gtid + binlog state is updated. For 'xid' and 'XA' based recovery following steps + are performed. + + Look for latest binlog checkpoint file. There can be two cases. The active + binary log and the latest binlog checkpoint file can be the same. + + Scan the binary log from the beginning. + From GTID_LIST and GTID_EVENTs reconstruct the gtid binlog state. + Prepare a list of 'xid's for recovery. + Prepare a list of explicit 'XA' transactions for recovery. + Recover the 'xid' transactions. + The explicit 'XA' transaction recovery is initiated once all the server + components are initialized. Please check 'execute_xa_for_recovery()'. + + Called from @c MYSQL_BIN_LOG::do_binlog_recovery() + + @param linfo Store here the found log file name and position to + the NEXT log file name in the index file. + + @param last_log_name Name of the last active binary log at the time of + crash. + + @param first_log Pointer to IO_CACHE of active binary log + @param fdle Format_description_log_event of active binary log + @param do_xa Is 2pc recovery needed for 'xid's and explicit XA + transactions. + @return indicates success or failure of recovery. + @retval 0 success + @retval 1 failure + +*/ int TC_LOG_BINLOG::recover(LOG_INFO *linfo, const char *last_log_name, IO_CACHE *first_log, Format_description_log_event *fdle, bool do_xa) { Log_event *ev= NULL; HASH xids; - MEM_ROOT mem_root; char binlog_checkpoint_name[FN_REFLEN]; bool binlog_checkpoint_found; bool first_round; @@ -10237,9 +10339,17 @@ int TC_LOG_BINLOG::recover(LOG_INFO *linfo, const char *last_log_name, bool last_gtid_valid= false; #endif - if (! fdle->is_valid() || - (do_xa && my_hash_init(&xids, &my_charset_bin, TC_LOG_PAGE_SIZE/3, 0, - sizeof(my_xid), 0, 0, MYF(0)))) + binlog_checkpoint_name[0]= 0; + if (!fdle->is_valid() || + (do_xa && + (my_hash_init(&xids, &my_charset_bin, TC_LOG_PAGE_SIZE/3, + 0, + sizeof(my_xid), 0, 0, MYF(0)) || + my_hash_init(&xa_recover_list, + &my_charset_bin, + TC_LOG_PAGE_SIZE/3, + 0, 0, + (my_hash_get_key) xid_get_var_key, 0, MYF(0))))) goto err1; if (do_xa) @@ -10313,21 +10423,29 @@ int TC_LOG_BINLOG::recover(LOG_INFO *linfo, const char *last_log_name, #ifdef HAVE_REPLICATION case GTID_EVENT: - if (first_round) { Gtid_log_event *gev= (Gtid_log_event *)ev; - - /* Update the binlog state with any GTID logged after Gtid_list. */ - last_gtid.domain_id= gev->domain_id; - last_gtid.server_id= gev->server_id; - last_gtid.seq_no= gev->seq_no; - last_gtid_standalone= - ((gev->flags2 & Gtid_log_event::FL_STANDALONE) ? true : false); - last_gtid_valid= true; + if (first_round) + { + /* Update the binlog state with any GTID logged after Gtid_list. */ + last_gtid.domain_id= gev->domain_id; + last_gtid.server_id= gev->server_id; + last_gtid.seq_no= gev->seq_no; + last_gtid_standalone= + ((gev->flags2 & Gtid_log_event::FL_STANDALONE) ? true : false); + last_gtid_valid= true; + } + if (do_xa && + (gev->flags2 & + (Gtid_log_event::FL_PREPARED_XA | + Gtid_log_event::FL_COMPLETED_XA)) && + xa_member_replace(&xa_recover_list, &gev->xid, + gev->flags2 & Gtid_log_event::FL_PREPARED_XA, + &mem_root)) + goto err2; + break; } - break; #endif - case START_ENCRYPTION_EVENT: { if (fdle->start_decryption((Start_encryption_log_event*) ev)) @@ -10417,10 +10535,22 @@ int TC_LOG_BINLOG::recover(LOG_INFO *linfo, const char *last_log_name, if (do_xa) { - if (ha_recover(&xids)) + if (ha_recover(&xids, &xa_recover_list)) goto err2; - free_root(&mem_root, MYF(0)); + DBUG_ASSERT(!xa_recover_list.records || + (binlog_checkpoint_found && binlog_checkpoint_name[0] != 0)); + + if (!xa_recover_list.records) + { + free_root(&mem_root, MYF(0)); + my_hash_free(&xa_recover_list); + } + else + { + xa_binlog_checkpoint_name= strmake_root(&mem_root, binlog_checkpoint_name, + strlen(binlog_checkpoint_name)); + } my_hash_free(&xids); } return 0; @@ -10436,6 +10566,7 @@ int TC_LOG_BINLOG::recover(LOG_INFO *linfo, const char *last_log_name, { free_root(&mem_root, MYF(0)); my_hash_free(&xids); + my_hash_free(&xa_recover_list); } err1: sql_print_error("Crash recovery failed. Either correct the problem " @@ -10445,6 +10576,214 @@ int TC_LOG_BINLOG::recover(LOG_INFO *linfo, const char *last_log_name, return 1; } +void MYSQL_BIN_LOG::execute_xa_for_recovery() +{ + if (xa_recover_list.records) + (void) recover_explicit_xa_prepare(); + free_root(&mem_root, MYF(0)); + my_hash_free(&xa_recover_list); +}; + +/** + Performs recovery of explict XA transactions. + 'xa_recover_list' contains the list of XA transactions to be recovered. + These events are replayed from the binary log to complete the recovery. + + @return indicates success or failure of recovery. + @retval false success + @retval true failure + +*/ +bool MYSQL_BIN_LOG::recover_explicit_xa_prepare() +{ +#ifndef HAVE_REPLICATION + /* Can't be supported without replication applier built in. */ + return false; +#else + bool err= true; + int error=0; + Relay_log_info *rli= NULL; + rpl_group_info *rgi; + THD *thd= new THD(0); /* Needed by start_slave_threads */ + thd->thread_stack= (char*) &thd; + thd->store_globals(); + thd->security_ctx->skip_grants(); + IO_CACHE log; + const char *errmsg; + File file; + bool enable_apply_event= false; + Log_event *ev = 0; + LOG_INFO linfo; + int recover_xa_count= xa_recover_list.records; + xa_recovery_member *member= NULL; + + //DBUG_ASSERT(!thd->rli_fake); + + if (!(rli= thd->rli_fake= new Relay_log_info(FALSE, "Recovery"))) + { + my_error(ER_OUTOFMEMORY, MYF(ME_FATAL), 1); + goto err2; + } + rli->sql_driver_thd= thd; + static LEX_CSTRING connection_name= { STRING_WITH_LEN("Recovery") }; + rli->mi= new Master_info(&connection_name, false); + if (!(rgi= thd->rgi_fake)) + rgi= thd->rgi_fake= new rpl_group_info(rli); + rgi->thd= thd; + thd->system_thread_info.rpl_sql_info= + new rpl_sql_thread_info(rli->mi->rpl_filter); + + if (rli && !rli->relay_log.description_event_for_exec) + { + rli->relay_log.description_event_for_exec= + new Format_description_log_event(4); + } + if (find_log_pos(&linfo, xa_binlog_checkpoint_name, 1)) + { + sql_print_error("Binlog file '%s' not found in binlog index, needed " + "for recovery. Aborting.", xa_binlog_checkpoint_name); + goto err2; + } + + tmp_disable_binlog(thd); + thd->variables.pseudo_slave_mode= TRUE; + for (;;) + { + if ((file= open_binlog(&log, linfo.log_file_name, &errmsg)) < 0) + { + sql_print_error("%s", errmsg); + goto err1; + } + while (recover_xa_count > 0 && + (ev= Log_event::read_log_event(&log, + rli->relay_log.description_event_for_exec, + opt_master_verify_checksum))) + { + if (!ev->is_valid()) + { + sql_print_error("Found invalid binlog query event %s" + " at %s:%lu; error %d %s", ev->get_type_str(), + linfo.log_file_name, + (ev->log_pos - ev->data_written)); + goto err1; + } + enum Log_event_type typ= ev->get_type_code(); + ev->thd= thd; + + if (typ == FORMAT_DESCRIPTION_EVENT) + enable_apply_event= true; + + if (typ == GTID_EVENT) + { + Gtid_log_event *gev= (Gtid_log_event *)ev; + if (gev->flags2 & + (Gtid_log_event::FL_PREPARED_XA | Gtid_log_event::FL_COMPLETED_XA)) + { + if ((member= + (xa_recovery_member*) my_hash_search(&xa_recover_list, + gev->xid.key(), + gev->xid.key_length()))) + { + /* Got XA PREPARE query in binlog but check member->state. If it is + marked as XA_PREPARE then this PREPARE has not seen its end + COMMIT/ROLLBACK. Check if it exists in engine in prepared state. + If so apply. + */ + if (gev->flags2 & Gtid_log_event::FL_PREPARED_XA) + { + if (member->state == XA_PREPARE) + { + // XA is prepared in binlog and not present in engine then apply + if (member->in_engine_prepare == false) + enable_apply_event= true; + else + --recover_xa_count; + } + } + else if (gev->flags2 & Gtid_log_event::FL_COMPLETED_XA) + { + if (member->state == XA_COMPLETE && + member->in_engine_prepare == true) + enable_apply_event= true; + else + --recover_xa_count; + } + } + } + } + + if (enable_apply_event) + { + if (typ == XA_PREPARE_LOG_EVENT) + thd->transaction.xid_state.set_binlogged(); + if ((err= ev->apply_event(rgi))) + { + sql_print_error("Failed to execute binlog query event of type: %s," + " at %s:%lu; error %d %s", ev->get_type_str(), + linfo.log_file_name, + (ev->log_pos - ev->data_written), + thd->get_stmt_da()->sql_errno(), + thd->get_stmt_da()->message()); + delete ev; + goto err1; + } + else if (typ == FORMAT_DESCRIPTION_EVENT) + enable_apply_event=false; + else if (thd->lex->sql_command == SQLCOM_XA_PREPARE || + thd->lex->sql_command == SQLCOM_XA_COMMIT || + thd->lex->sql_command == SQLCOM_XA_ROLLBACK) + { + --recover_xa_count; + enable_apply_event=false; + + sql_print_information("Binlog event %s at %s:%lu" + " successfully applied", + typ == XA_PREPARE_LOG_EVENT ? + static_cast<XA_prepare_log_event *>(ev)->get_query() : + static_cast<Query_log_event *>(ev)->query, + linfo.log_file_name, (ev->log_pos - ev->data_written)); + } + } + if (typ != FORMAT_DESCRIPTION_EVENT) + delete ev; + } + end_io_cache(&log); + mysql_file_close(file, MYF(MY_WME)); + file= -1; + if (unlikely((error= find_next_log(&linfo, 1)))) + { + if (error != LOG_INFO_EOF) + sql_print_error("find_log_pos() failed (error: %d)", error); + else + break; + } + } +err1: + reenable_binlog(thd); + /* + There should be no more XA transactions to recover upon successful + completion. + */ + if (recover_xa_count > 0) + goto err2; + sql_print_information("Crash recovery finished."); + err= false; +err2: + if (file >= 0) + { + end_io_cache(&log); + mysql_file_close(file, MYF(MY_WME)); + } + thd->variables.pseudo_slave_mode= FALSE; + delete rli->mi; + delete thd->system_thread_info.rpl_sql_info; + rgi->slave_close_thread_tables(thd); + thd->reset_globals(); + delete thd; + + return err; +#endif /* !HAVE_REPLICATION */ +} int MYSQL_BIN_LOG::do_binlog_recovery(const char *opt_name, bool do_xa_recovery) diff --git a/sql/log.h b/sql/log.h index 8e70d3c8f4c..9bf3248d4c9 100644 --- a/sql/log.h +++ b/sql/log.h @@ -63,6 +63,7 @@ class TC_LOG virtual int unlog(ulong cookie, my_xid xid)=0; virtual int unlog_xa_prepare(THD *thd, bool all)= 0; virtual void commit_checkpoint_notify(void *cookie)= 0; + virtual void execute_xa_for_recovery() {}; protected: /* @@ -708,6 +709,8 @@ class MYSQL_BIN_LOG: public TC_LOG, private MYSQL_LOG void commit_checkpoint_notify(void *cookie); int recover(LOG_INFO *linfo, const char *last_log_name, IO_CACHE *first_log, Format_description_log_event *fdle, bool do_xa); + bool recover_explicit_xa_prepare(); + int do_binlog_recovery(const char *opt_name, bool do_xa_recovery); #if !defined(MYSQL_CLIENT) @@ -932,7 +935,7 @@ class MYSQL_BIN_LOG: public TC_LOG, private MYSQL_LOG mysql_mutex_t* get_binlog_end_pos_lock() { return &LOCK_binlog_end_pos; } int wait_for_update_binlog_end_pos(THD* thd, struct timespec * timeout); - + void execute_xa_for_recovery(); /* Binlog position of end of the binlog. Access to this is protected by LOCK_binlog_end_pos @@ -945,6 +948,9 @@ class MYSQL_BIN_LOG: public TC_LOG, private MYSQL_LOG */ my_off_t binlog_end_pos; char binlog_end_pos_file[FN_REFLEN]; + MEM_ROOT mem_root; + char *xa_binlog_checkpoint_name; + HASH xa_recover_list; }; class Log_event_handler diff --git a/sql/log_event.cc b/sql/log_event.cc index ee44f7f1da4..9adfefceb97 100644 --- a/sql/log_event.cc +++ b/sql/log_event.cc @@ -18,7 +18,7 @@ #include "mariadb.h" #include "sql_priv.h" - +#include "handler.h" #ifndef MYSQL_CLIENT #include "unireg.h" #include "log_event.h" @@ -2812,9 +2812,25 @@ XA_prepare_log_event(const char* buf, buf += sizeof(temp); memcpy(&temp, buf, sizeof(temp)); m_xid.gtrid_length= uint4korr(&temp); + // Todo: validity here and elsewhere checks to be replaced by MDEV-21839 fixes + if (m_xid.gtrid_length < 0 || m_xid.gtrid_length > MAXGTRIDSIZE) + { + m_xid.formatID= -1; + return; + } buf += sizeof(temp); memcpy(&temp, buf, sizeof(temp)); m_xid.bqual_length= uint4korr(&temp); + if (m_xid.bqual_length < 0 || m_xid.bqual_length > MAXBQUALSIZE) + { + m_xid.formatID= -1; + return; + } + if (m_xid.gtrid_length + m_xid.bqual_length > XIDDATASIZE) + { + m_xid.formatID= -1; + return; + } buf += sizeof(temp); memcpy(m_xid.data, buf, m_xid.gtrid_length + m_xid.bqual_length); diff --git a/sql/log_event.h b/sql/log_event.h index a6543b70eb5..595dc9f6c3c 100644 --- a/sql/log_event.h +++ b/sql/log_event.h @@ -3234,6 +3234,7 @@ class XA_prepare_log_event: public Xid_apply_log_event const Format_description_log_event *description_event); ~XA_prepare_log_event() {} Log_event_type get_type_code() { return XA_PREPARE_LOG_EVENT; } + bool is_valid() const { return m_xid.formatID != -1; } int get_data_size() { return xid_subheader_no_data + m_xid.gtrid_length + m_xid.bqual_length; @@ -3241,12 +3242,7 @@ class XA_prepare_log_event: public Xid_apply_log_event #ifdef MYSQL_SERVER bool write(); -#endif - -private: -#if defined(MYSQL_SERVER) && defined(HAVE_REPLICATION) - char query[sizeof("XA COMMIT ONE PHASE") + 1 + ser_buf_size]; - int do_commit(); +#ifdef HAVE_REPLICATION const char* get_query() { sprintf(query, @@ -3254,6 +3250,13 @@ class XA_prepare_log_event: public Xid_apply_log_event m_xid.serialize()); return query; } +#endif /* HAVE_REPLICATION */ +#endif /* MYSQL_SERVER */ + +private: +#if defined(MYSQL_SERVER) && defined(HAVE_REPLICATION) + char query[sizeof("XA COMMIT ONE PHASE") + 1 + ser_buf_size]; + int do_commit(); #endif }; diff --git a/sql/mysqld.cc b/sql/mysqld.cc index b2f8afca7a6..f669a4ca5d8 100644 --- a/sql/mysqld.cc +++ b/sql/mysqld.cc @@ -5189,7 +5189,7 @@ static int init_server_components() unireg_abort(1); } - if (ha_recover(0)) + if (ha_recover(0, 0)) { unireg_abort(1); } @@ -5606,7 +5606,7 @@ int mysqld_main(int argc, char **argv) initialize_information_schema_acl(); execute_ddl_log_recovery(); - + tc_log->execute_xa_for_recovery(); /* Change EVENTS_ORIGINAL to EVENTS_OFF (the default value) as there is no point in using ORIGINAL during startup diff --git a/sql/xa.cc b/sql/xa.cc index 786d09c2b39..df7d0229157 100644 --- a/sql/xa.cc +++ b/sql/xa.cc @@ -581,6 +581,10 @@ bool trans_xa_commit(THD *thd) XID_STATE &xid_state= thd->transaction.xid_state; DBUG_ENTER("trans_xa_commit"); + DBUG_EXECUTE_IF("trans_xa_commit_fail", + {my_error(ER_OUT_OF_RESOURCES, MYF(0)); + DBUG_RETURN(TRUE);}); + if (!xid_state.is_explicit_XA() || !xid_state.xid_cache_element->xid.eq(thd->lex->xid))

1 0

[Maria-developers] [andrei.elkin@mariadb.com] 25fe744e5c3: MDEV-742 XA PREPAREd transaction survive disconnect/server restart
by andrei.elkin＠pp.inet.fi 04 Mar '20

04 Mar '20

Howdy, Kristian. I am forwarding for your attention a patch that HEADs bb-10.5-mdev_742 branch. While it's of some size maybe you'll find some time to review changes/extension esp done to the pararellel slave. Looking forward to hear from you! Thanks. Andrei revision-id: 25fe744e5c3d2e8f7d3db7eed37d8e412837457b (mariadb-10.5.0-283-g25fe744e5c3) parent(s): 36cebe53a3645bf1e665ffdf5b552cabcc1e8e56 author: Andrei Elkin committer: Andrei Elkin timestamp: 2020-03-04 15:48:43 +0200 message: MDEV-742 XA PREPAREd transaction survive disconnect/server restart Lifted long standing limitation to the XA of rolling it back at the transaction's connection close even if the XA is prepared. Prepared XA-transaction is made to sustain connection close or server restart. The patch consists of - binary logging extension to write prepared XA part of transaction signified with its XID in a new XA_prepare_log_event. The concusion part - with Commit or Rollback decision - is logged separately as Query_log_event. That is in the binlog the XA consists of two separate group of events. That makes the whole XA possibly interweaving in binlog with other XA:s or regular transaction but with no harm to replication and data consistency. Gtid_log_event receives two more flags to identify which of the two XA phases of the transaction it represents. With either flag set also XID info is added to the event. - engines are made aware of the server policy to keep up user prepared XA:s so they (Innodb, rocksdb) don't roll them back anymore at their disconnect methods. - slave applier is refined to cope with two phase logged XA:s including parallel modes of execution. This patch does not address crash-safe logging of the new events which is being addressed by MDEV-21469, its commit is to be published shortly. There's a list of fixes to 10.5 that were required by this MDEV, incl MDEV-21856: XID::formatID is constrained to 4 bytes by requirements of cross-platform replication and Innodb legacy. MDEV-21659 XA rollback 'foreign_xid' is allowed inside active XA MDEV-21766 Forbid XID with empty 'gtrid' MDEV-21854 xa commit 'xid' one phase for already prepared transaction must always error out Many thanks to Alexey Botchkov for driving this work initially! --- mysql-test/include/kill_and_restart_mysqld.inc | 15 + mysql-test/main/flush_read_lock.result | 76 +- mysql-test/main/flush_read_lock.test | 112 +- mysql-test/main/xa.result | 61 + mysql-test/main/xa.test | 49 + mysql-test/main/xa_binlog.result | 11 +- mysql-test/main/xa_binlog.test | 2 +- mysql-test/main/xa_prepared_binlog_off-master.opt | 1 + mysql-test/main/xa_prepared_binlog_off.result | 1044 +++++++++++++++++ mysql-test/main/xa_prepared_binlog_off.test | 11 + mysql-test/main/xa_sync.result | 10 + mysql-test/main/xa_sync.test | 5 + .../include/binlog_xa_prepare_connection.inc | 31 + .../include/binlog_xa_prepare_disconnect.inc | 35 + .../include/binlog_xa_prepared_do_and_restart.inc | 323 ++++++ .../suite/binlog/r/binlog_xa_checkpoint.result | 33 + .../suite/binlog/r/binlog_xa_prepared.result | 1176 ++++++++++++++++++++ .../binlog/r/binlog_xa_prepared_disconnect.result | 1176 ++++++++++++++++++++ .../suite/binlog/t/binlog_xa_checkpoint.test | 57 + mysql-test/suite/binlog/t/binlog_xa_prepared.inc | 102 ++ .../binlog/t/binlog_xa_prepared_disconnect.test | 11 + .../suite/rpl/include/rpl_xa_mixed_engines.inc | 183 +++ .../suite/rpl/r/rpl_parallel_optimistic_xa.result | 51 + .../r/rpl_parallel_optimistic_xa_lsu_off.result | 51 + .../suite/rpl/r/rpl_parallel_xa_same_xid.result | 23 + mysql-test/suite/rpl/r/rpl_temporary_errors.result | 47 +- mysql-test/suite/rpl/r/rpl_xa.result | 48 + mysql-test/suite/rpl/r/rpl_xa_gap_lock.result | 44 + .../suite/rpl/r/rpl_xa_gtid_pos_auto_engine.result | 64 ++ .../suite/rpl/r/rpl_xa_survive_disconnect.result | 319 ++++++ .../rpl/r/rpl_xa_survive_disconnect_lsu_off.result | 319 ++++++ .../rpl_xa_survive_disconnect_mixed_engines.result | 373 +++++++ .../suite/rpl/t/rpl_parallel_optimistic_xa.test | 235 ++++ .../t/rpl_parallel_optimistic_xa_lsu_off-slave.opt | 1 + .../rpl/t/rpl_parallel_optimistic_xa_lsu_off.test | 2 + .../suite/rpl/t/rpl_parallel_xa_same_xid.test | 138 +++ mysql-test/suite/rpl/t/rpl_temporary_errors.test | 82 +- mysql-test/suite/rpl/t/rpl_xa.inc | 73 ++ mysql-test/suite/rpl/t/rpl_xa.test | 5 + mysql-test/suite/rpl/t/rpl_xa_gap_lock-slave.opt | 1 + mysql-test/suite/rpl/t/rpl_xa_gap_lock.test | 137 +++ .../suite/rpl/t/rpl_xa_gtid_pos_auto_engine.test | 29 + .../suite/rpl/t/rpl_xa_survive_disconnect.test | 294 +++++ .../t/rpl_xa_survive_disconnect_lsu_off-slave.opt | 2 + .../rpl/t/rpl_xa_survive_disconnect_lsu_off.test | 8 + .../t/rpl_xa_survive_disconnect_mixed_engines.test | 68 ++ sql/handler.cc | 11 +- sql/log.cc | 248 ++++- sql/log.h | 10 + sql/log_event.cc | 59 +- sql/log_event.h | 213 +++- sql/log_event_client.cc | 37 +- sql/log_event_server.cc | 298 +++-- sql/rpl_parallel.cc | 220 +++- sql/rpl_parallel.h | 28 +- sql/rpl_rli.cc | 14 +- sql/rpl_rli.h | 4 + sql/slave.cc | 10 +- sql/sql_repl.cc | 4 +- sql/xa.cc | 254 ++++- sql/xa.h | 22 +- storage/innobase/handler/ha_innodb.cc | 1 + storage/innobase/trx/trx0trx.cc | 19 +- storage/rocksdb/ha_rocksdb.cc | 18 +- storage/rocksdb/mysql-test/rocksdb/r/xa.result | 41 +- storage/rocksdb/mysql-test/rocksdb/t/xa.test | 43 +- .../rocksdb/mysql-test/rocksdb_rpl/r/rpl_xa.result | 50 + .../rocksdb/mysql-test/rocksdb_rpl/t/rpl_xa.inc | 70 ++ .../rocksdb/mysql-test/rocksdb_rpl/t/rpl_xa.test | 6 + .../tokudb/mysql-test/tokudb_mariadb/r/xa.result | 1 + storage/tokudb/mysql-test/tokudb_mariadb/t/xa.test | 3 + 71 files changed, 8410 insertions(+), 212 deletions(-) diff --git a/mysql-test/include/kill_and_restart_mysqld.inc b/mysql-test/include/kill_and_restart_mysqld.inc new file mode 100644 index 00000000000..b67fb7350b4 --- /dev/null +++ b/mysql-test/include/kill_and_restart_mysqld.inc @@ -0,0 +1,15 @@ +if (!$restart_parameters) +{ + let $restart_parameters = restart; +} + +--let $_server_id= `SELECT @@server_id` +--let $_expect_file_name= $MYSQLTEST_VARDIR/tmp/mysqld.$_server_id.expect + +--echo # Kill and $restart_parameters +--exec echo "$restart_parameters" > $_expect_file_name +--shutdown_server 0 +--source include/wait_until_disconnected.inc +--enable_reconnect +--source include/wait_until_connected_again.inc +--disable_reconnect diff --git a/mysql-test/main/flush_read_lock.result b/mysql-test/main/flush_read_lock.result index 0f8c2ce9fb9..be710050139 100644 --- a/mysql-test/main/flush_read_lock.result +++ b/mysql-test/main/flush_read_lock.result @@ -1310,6 +1310,8 @@ unlock tables; # Check that XA non-COMMIT statements are not and COMMIT is # blocked by active FTWRL in another connection # +# XA COMMIT, XA ROLLBACK and XA PREPARE does take COMMIT lock to ensure +# that nothing is written to bin log and redo log under FTWRL mode. connection con1; flush tables with read lock; connection default; @@ -1322,11 +1324,25 @@ connection con1; flush tables with read lock; connection default; xa end 'test1'; -xa prepare 'test1'; +xa prepare 'test1';; +connection con1; +unlock tables; +# Switching to connection 'default'. +connection default; +# Reap XA PREPARE. +# Switching to connection 'con1'. +connection con1; +flush tables with read lock; +# Switching to connection 'default'. +connection default; +# Send XA ROLLBACK 'test1' xa rollback 'test1'; +# Switching to connection 'con1'. connection con1; +# Wait until XA ROLLBACK is blocked. unlock tables; connection default; +# Reap XA ROLLBACK xa start 'test1'; insert into t3_trans values (1); connection con1; @@ -1334,7 +1350,20 @@ flush tables with read lock; connection default; connection default; xa end 'test1'; +# Send XA PREPARE 'test1' xa prepare 'test1'; +# Switching to connection 'con1'. +connection con1; +# Wait until XA PREPARE is blocked. +unlock tables; +# Switching to connection 'default'. +connection default; +# Reap XA PREPARE. +# Switching to connection 'con1'. +connection con1; +flush tables with read lock; +# Switching to connection 'default'. +connection default; # Send: xa commit 'test1';; connection con1; @@ -1344,6 +1373,51 @@ connection default; # Reap XA COMMIT. delete from t3_trans; # +# Check that XA COMMIT / ROLLBACK for prepared transaction from a +# disconnected session is blocked by active FTWRL in another connection. +# +# Create temporary connection for XA transaction. +connect con_tmp,localhost,root,,; +xa start 'test1'; +insert into t3_trans values (1); +xa end 'test1'; +xa prepare 'test1'; +# Disconnect temporary connection +disconnect con_tmp; +# Create temporary connection for XA transaction. +connect con_tmp,localhost,root,,; +xa start 'test2'; +insert into t3_trans values (2); +xa end 'test2'; +xa prepare 'test2'; +# Disconnect temporary connection +disconnect con_tmp; +# Switching to connection 'con1'. +connection con1; +flush tables with read lock; +# Switching to connection 'default'. +connection default; +# Send XA ROLLBACK 'test1' +xa rollback 'test1'; +# Switching to connection 'con1'. +connection con1; +# Wait until XA ROLLBACK is blocked. +unlock tables; +flush tables with read lock; +# Switching to connection 'default'. +connection default; +# Reap XA ROLLBACK +# Send XA COMMIT +xa commit 'test2';; +# Switching to connection 'con1'. +connection con1; +# Wait until XA COMMIT is blocked. +unlock tables; +# Switching to connection 'default'. +connection default; +# Reap XA COMMIT. +delete from t3_trans; +# # Check that XA COMMIT blocks FTWRL in another connection. xa start 'test1'; insert into t3_trans values (1); diff --git a/mysql-test/main/flush_read_lock.test b/mysql-test/main/flush_read_lock.test index 80512deac4e..4283358770c 100644 --- a/mysql-test/main/flush_read_lock.test +++ b/mysql-test/main/flush_read_lock.test @@ -1592,6 +1592,8 @@ unlock tables; --echo # Check that XA non-COMMIT statements are not and COMMIT is --echo # blocked by active FTWRL in another connection --echo # +--echo # XA COMMIT, XA ROLLBACK and XA PREPARE does take COMMIT lock to ensure +--echo # that nothing is written to bin log and redo log under FTWRL mode. connection $con_aux1; flush tables with read lock; connection default; @@ -1604,11 +1606,37 @@ connection $con_aux1; flush tables with read lock; connection default; xa end 'test1'; -xa prepare 'test1'; -xa rollback 'test1'; +--send xa prepare 'test1'; connection $con_aux1; +let $wait_condition= + select count(*) = 1 from information_schema.processlist + where state = "Waiting for backup lock" and + info = "xa prepare 'test1'"; +--source include/wait_condition.inc unlock tables; +--echo # Switching to connection 'default'. +connection default; +--echo # Reap XA PREPARE. +--reap +--echo # Switching to connection '$con_aux1'. +connection $con_aux1; +flush tables with read lock; +--echo # Switching to connection 'default'. connection default; +--echo # Send XA ROLLBACK 'test1' +--send xa rollback 'test1' +--echo # Switching to connection '$con_aux1'. +connection $con_aux1; +--echo # Wait until XA ROLLBACK is blocked. +let $wait_condition= + select count(*) = 1 from information_schema.processlist + where state = "Waiting for backup lock" and + info = "xa rollback 'test1'"; +--source include/wait_condition.inc +unlock tables; +connection default; +--echo # Reap XA ROLLBACK +--reap xa start 'test1'; insert into t3_trans values (1); connection $con_aux1; @@ -1616,7 +1644,27 @@ flush tables with read lock; connection default; connection default; xa end 'test1'; -xa prepare 'test1'; +--echo # Send XA PREPARE 'test1' +--send xa prepare 'test1' +--echo # Switching to connection '$con_aux1'. +connection $con_aux1; +--echo # Wait until XA PREPARE is blocked. +let $wait_condition= + select count(*) = 1 from information_schema.processlist + where state = "Waiting for backup lock" and + info = "xa prepare 'test1'"; +--source include/wait_condition.inc +unlock tables; +--echo # Switching to connection 'default'. +connection default; +--echo # Reap XA PREPARE. +--reap +--echo # Switching to connection '$con_aux1'. +connection $con_aux1; +flush tables with read lock; +--echo # Switching to connection 'default'. +connection default; + --echo # Send: --send xa commit 'test1'; connection $con_aux1; @@ -1631,6 +1679,64 @@ connection default; --echo # Reap XA COMMIT. --reap delete from t3_trans; +--echo # +--echo # Check that XA COMMIT / ROLLBACK for prepared transaction from a +--echo # disconnected session is blocked by active FTWRL in another connection. +--echo # +--echo # Create temporary connection for XA transaction. +connect (con_tmp,localhost,root,,); +xa start 'test1'; +insert into t3_trans values (1); +xa end 'test1'; +xa prepare 'test1'; +--echo # Disconnect temporary connection +disconnect con_tmp; +--echo # Create temporary connection for XA transaction. +connect (con_tmp,localhost,root,,); +xa start 'test2'; +insert into t3_trans values (2); +xa end 'test2'; +xa prepare 'test2'; +--echo # Disconnect temporary connection +disconnect con_tmp; +--echo # Switching to connection '$con_aux1'. +connection $con_aux1; +flush tables with read lock; +--echo # Switching to connection 'default'. +connection default; +--echo # Send XA ROLLBACK 'test1' +--send xa rollback 'test1' +--echo # Switching to connection '$con_aux1'. +connection $con_aux1; +--echo # Wait until XA ROLLBACK is blocked. +let $wait_condition= + select count(*) = 1 from information_schema.processlist + where state = "Waiting for backup lock" and + info = "xa rollback 'test1'"; +--source include/wait_condition.inc +unlock tables; +flush tables with read lock; +--echo # Switching to connection 'default'. +connection default; +--echo # Reap XA ROLLBACK +--reap +--echo # Send XA COMMIT +--send xa commit 'test2'; +--echo # Switching to connection '$con_aux1'. +connection $con_aux1; +--echo # Wait until XA COMMIT is blocked. +let $wait_condition= + select count(*) = 1 from information_schema.processlist + where state = "Waiting for backup lock" and + info = "xa commit 'test2'"; +--source include/wait_condition.inc +unlock tables; +--echo # Switching to connection 'default'. +connection default; +--echo # Reap XA COMMIT. +--reap +delete from t3_trans; + --echo # --echo # Check that XA COMMIT blocks FTWRL in another connection. xa start 'test1'; diff --git a/mysql-test/main/xa.result b/mysql-test/main/xa.result index bd2946247d8..aa48f6d26c7 100644 --- a/mysql-test/main/xa.result +++ b/mysql-test/main/xa.result @@ -66,6 +66,9 @@ select * from t1; a 20 disconnect con1; +xa rollback 'testb',0x2030405060,11; +xa recover; +formatID gtrid_length bqual_length data connection default; xa start 'tr1'; insert t1 values (40); @@ -376,3 +379,61 @@ XA PREPARE 'Я_упaлa_c_сеновала_тормозила_головой'; XA ROLLBACK 'Я_упaлa_c_сеновала_тормозила_головой'; SET NAMES default; DROP TABLE t1; +# MDEV-7974 related +# Check XA state when lock_wait_timeout happens +# More tests added to flush_read_lock.test +connect con_tmp,localhost,root,,; +set session lock_wait_timeout=1; +create table asd (a int) engine=innodb; +xa start 'test1'; +insert into asd values(1); +xa end 'test1'; +connection default; +flush table with read lock; +connection con_tmp; +# PREPARE error will do auto rollback. +xa prepare 'test1'; +ERROR HY000: Lock wait timeout exceeded; try restarting transaction +show errors; +Level Code Message +Error 1205 Lock wait timeout exceeded; try restarting transaction +Error 1402 XA_RBROLLBACK: Transaction branch was rolled back +connection default; +unlock tables; +connection con_tmp; +xa start 'test1'; +insert into asd values(1); +xa end 'test1'; +xa prepare 'test1'; +connection default; +flush tables with read lock; +connection con_tmp; +# LOCK error during ROLLBACK will not alter transaction state. +xa rollback 'test1'; +ERROR HY000: Lock wait timeout exceeded; try restarting transaction +show errors; +Level Code Message +Error 1205 Lock wait timeout exceeded; try restarting transaction +Error 1401 XAER_RMERR: Fatal error occurred in the transaction branch - check your data for consistency +xa recover; +formatID gtrid_length bqual_length data +1 5 0 test1 +# LOCK error during COMMIT will not alter transaction state. +xa commit 'test1'; +ERROR HY000: Lock wait timeout exceeded; try restarting transaction +show errors; +Level Code Message +Error 1205 Lock wait timeout exceeded; try restarting transaction +Error 1401 XAER_RMERR: Fatal error occurred in the transaction branch - check your data for consistency +xa recover; +formatID gtrid_length bqual_length data +1 5 0 test1 +connection default; +unlock tables; +connection con_tmp; +xa rollback 'test1'; +xa recover; +formatID gtrid_length bqual_length data +drop table asd; +disconnect con_tmp; +connection default; diff --git a/mysql-test/main/xa.test b/mysql-test/main/xa.test index 55c41452635..ca09989dfc8 100644 --- a/mysql-test/main/xa.test +++ b/mysql-test/main/xa.test @@ -98,6 +98,8 @@ xa start 'zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz'; select * from t1; disconnect con1; --source include/wait_until_count_sessions.inc +xa rollback 'testb',0x2030405060,11; +xa recover; connection default; @@ -523,3 +525,50 @@ SET NAMES default; DROP TABLE t1; --source include/wait_until_count_sessions.inc + +--echo # MDEV-7974 related +--echo # Check XA state when lock_wait_timeout happens +--echo # More tests added to flush_read_lock.test +connect (con_tmp,localhost,root,,); +set session lock_wait_timeout=1; +create table asd (a int) engine=innodb; +xa start 'test1'; +insert into asd values(1); +xa end 'test1'; +connection default; +flush table with read lock; +connection con_tmp; +--echo # PREPARE error will do auto rollback. +--ERROR ER_LOCK_WAIT_TIMEOUT +xa prepare 'test1'; +show errors; +connection default; +unlock tables; + +connection con_tmp; +xa start 'test1'; +insert into asd values(1); +xa end 'test1'; +xa prepare 'test1'; +connection default; +flush tables with read lock; +connection con_tmp; +--echo # LOCK error during ROLLBACK will not alter transaction state. +--ERROR ER_LOCK_WAIT_TIMEOUT +xa rollback 'test1'; +show errors; +xa recover; +--echo # LOCK error during COMMIT will not alter transaction state. +--ERROR ER_LOCK_WAIT_TIMEOUT +xa commit 'test1'; +show errors; +xa recover; +connection default; +unlock tables; +connection con_tmp; +xa rollback 'test1'; +xa recover; +drop table asd; +disconnect con_tmp; +--source include/wait_until_disconnected.inc +connection default; diff --git a/mysql-test/main/xa_binlog.result b/mysql-test/main/xa_binlog.result index 619a6e08b20..c45749d500f 100644 --- a/mysql-test/main/xa_binlog.result +++ b/mysql-test/main/xa_binlog.result @@ -18,14 +18,17 @@ a 1 2 3 -SHOW BINLOG EVENTS LIMIT 3,9; +SHOW BINLOG EVENTS LIMIT 3,12; Log_name Pos Event_type Server_id End_log_pos Info -master-bin.000001 # Gtid 1 # BEGIN GTID #-#-# +master-bin.000001 # Gtid 1 # XA START X'786174657374',X'',1 GTID #-#-# master-bin.000001 # Query 1 # use `test`; INSERT INTO t1 VALUES (1) -master-bin.000001 # Query 1 # COMMIT +master-bin.000001 # Query 1 # XA END X'786174657374',X'',1 +master-bin.000001 # XA_prepare 1 # XA PREPARE X'786174657374',X'',1 +master-bin.000001 # Gtid 1 # GTID #-#-# +master-bin.000001 # Query 1 # XA COMMIT X'786174657374',X'',1 master-bin.000001 # Gtid 1 # BEGIN GTID #-#-# master-bin.000001 # Query 1 # use `test`; INSERT INTO t1 VALUES (2) -master-bin.000001 # Query 1 # COMMIT +master-bin.000001 # Xid 1 # COMMIT /* xid=XX */ master-bin.000001 # Gtid 1 # BEGIN GTID #-#-# master-bin.000001 # Query 1 # use `test`; INSERT INTO t1 VALUES (3) master-bin.000001 # Xid 1 # COMMIT /* xid=XX */ diff --git a/mysql-test/main/xa_binlog.test b/mysql-test/main/xa_binlog.test index ecbf1f4f066..91bca2ac8cb 100644 --- a/mysql-test/main/xa_binlog.test +++ b/mysql-test/main/xa_binlog.test @@ -27,6 +27,6 @@ SELECT * FROM t1 ORDER BY a; --replace_column 2 # 5 # --replace_regex /xid=[0-9]+/xid=XX/ /GTID [0-9]+-[0-9]+-[0-9]+/GTID #-#-#/ -SHOW BINLOG EVENTS LIMIT 3,9; +SHOW BINLOG EVENTS LIMIT 3,12; DROP TABLE t1; diff --git a/mysql-test/main/xa_prepared_binlog_off-master.opt b/mysql-test/main/xa_prepared_binlog_off-master.opt new file mode 100644 index 00000000000..789275fa25e --- /dev/null +++ b/mysql-test/main/xa_prepared_binlog_off-master.opt @@ -0,0 +1 @@ +--skip-log-bin diff --git a/mysql-test/main/xa_prepared_binlog_off.result b/mysql-test/main/xa_prepared_binlog_off.result new file mode 100644 index 00000000000..ca19f6cdfaf --- /dev/null +++ b/mysql-test/main/xa_prepared_binlog_off.result @@ -0,0 +1,1044 @@ +call mtr.add_suppression("You need to use --log-bin to make --log-slave-updates work."); +connection default; +CREATE VIEW v_processlist as SELECT * FROM performance_schema.threads where type = 'FOREGROUND'; +call mtr.add_suppression("Found 10 prepared XA transactions"); +CREATE TABLE t (a INT) ENGINE=innodb; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@sql_log_bin = OFF; +CREATE TEMPORARY TABLE tmp1 (a int) ENGINE=innodb; +XA START 'trx1tmp'; +INSERT INTO tmp1 SET a=1; +XA END 'trx1tmp'; +XA PREPARE 'trx1tmp'; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@sql_log_bin = OFF; +CREATE TEMPORARY TABLE tmp1 (a int) ENGINE=innodb; +XA START 'trx2tmp'; +INSERT INTO tmp1 SET a=1; +XA END 'trx2tmp'; +XA PREPARE 'trx2tmp'; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@sql_log_bin = OFF; +CREATE TEMPORARY TABLE tmp1 (a int) ENGINE=innodb; +XA START 'trx3tmp'; +INSERT INTO tmp1 SET a=1; +XA END 'trx3tmp'; +XA PREPARE 'trx3tmp'; +connection default; +XA COMMIT 'trx1tmp'; +ERROR XAE04: XAER_NOTA: Unknown XID +XA ROLLBACK 'trx1tmp'; +ERROR XAE04: XAER_NOTA: Unknown XID +XA START 'trx1tmp'; +ERROR XAE08: XAER_DUPID: The XID already exists +connection default; +*** 3 prepared transactions must be in the list *** +XA RECOVER; +formatID gtrid_length bqual_length data +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +connection conn1tmp; +disconnect conn1tmp; +connection default; +XA COMMIT 'trx1tmp'; +KILL connection CONN_ID; +XA COMMIT 'trx3tmp'; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'trx1ro'; +SELECT * from t ORDER BY a; +a +XA END 'trx1ro'; +XA PREPARE 'trx1ro'; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'trx2ro'; +SELECT * from t ORDER BY a; +a +XA END 'trx2ro'; +XA PREPARE 'trx2ro'; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'trx3ro'; +SELECT * from t ORDER BY a; +a +XA END 'trx3ro'; +XA PREPARE 'trx3ro'; +connection default; +*** 4 prepared transactions must be in the list *** +XA RECOVER; +formatID gtrid_length bqual_length data +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +connection conn1ro; +disconnect conn1ro; +connection default; +XA ROLLBACK 'trx1ro'; +KILL connection CONN_ID; +XA ROLLBACK 'trx3ro'; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'trx1empty'; +XA END 'trx1empty'; +XA PREPARE 'trx1empty'; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'trx2empty'; +XA END 'trx2empty'; +XA PREPARE 'trx2empty'; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'trx3empty'; +XA END 'trx3empty'; +XA PREPARE 'trx3empty'; +connection default; +*** 5 prepared transactions must be in the list *** +XA RECOVER; +formatID gtrid_length bqual_length data +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +connection conn1empty; +disconnect conn1empty; +connection default; +XA COMMIT 'trx1empty'; +KILL connection CONN_ID; +XA COMMIT 'trx3empty'; +connect conn1$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'trx1unprepared'; +INSERT INTO t set a=0; +XA END 'trx1unprepared'; +INSERT INTO t set a=0; +ERROR XAE07: XAER_RMFAIL: The command cannot be executed when global transaction is in the IDLE state +XA START 'trx1unprepared'; +ERROR XAE07: XAER_RMFAIL: The command cannot be executed when global transaction is in the IDLE state +XA START 'trx1unprepared'; +ERROR XAE07: XAER_RMFAIL: The command cannot be executed when global transaction is in the IDLE state +disconnect conn1unprepared; +connection default; +XA COMMIT 'trx1unprepared'; +ERROR XAE04: XAER_NOTA: Unknown XID +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_0'; +INSERT INTO t SET a=0; +XA END 'trx_0'; +XA PREPARE 'trx_0'; +disconnect conn0; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_1'; +INSERT INTO t SET a=1; +XA END 'trx_1'; +XA PREPARE 'trx_1'; +disconnect conn1; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_2'; +INSERT INTO t SET a=2; +XA END 'trx_2'; +XA PREPARE 'trx_2'; +disconnect conn2; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_3'; +INSERT INTO t SET a=3; +XA END 'trx_3'; +XA PREPARE 'trx_3'; +disconnect conn3; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_4'; +INSERT INTO t SET a=4; +XA END 'trx_4'; +XA PREPARE 'trx_4'; +disconnect conn4; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_5'; +INSERT INTO t SET a=5; +XA END 'trx_5'; +XA PREPARE 'trx_5'; +disconnect conn5; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_6'; +INSERT INTO t SET a=6; +XA END 'trx_6'; +XA PREPARE 'trx_6'; +disconnect conn6; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_7'; +INSERT INTO t SET a=7; +XA END 'trx_7'; +XA PREPARE 'trx_7'; +disconnect conn7; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_8'; +INSERT INTO t SET a=8; +XA END 'trx_8'; +XA PREPARE 'trx_8'; +disconnect conn8; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_9'; +INSERT INTO t SET a=9; +XA END 'trx_9'; +XA PREPARE 'trx_9'; +disconnect conn9; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_10'; +INSERT INTO t SET a=10; +XA END 'trx_10'; +XA PREPARE 'trx_10'; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_11'; +INSERT INTO t SET a=11; +XA END 'trx_11'; +XA PREPARE 'trx_11'; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_12'; +INSERT INTO t SET a=12; +XA END 'trx_12'; +XA PREPARE 'trx_12'; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_13'; +INSERT INTO t SET a=13; +XA END 'trx_13'; +XA PREPARE 'trx_13'; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_14'; +INSERT INTO t SET a=14; +XA END 'trx_14'; +XA PREPARE 'trx_14'; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_15'; +INSERT INTO t SET a=15; +XA END 'trx_15'; +XA PREPARE 'trx_15'; +connection default; +KILL CONNECTION CONN_ID; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_16'; +INSERT INTO t SET a=16; +XA END 'trx_16'; +XA PREPARE 'trx_16'; +connection default; +KILL CONNECTION CONN_ID; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_17'; +INSERT INTO t SET a=17; +XA END 'trx_17'; +XA PREPARE 'trx_17'; +connection default; +KILL CONNECTION CONN_ID; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_18'; +INSERT INTO t SET a=18; +XA END 'trx_18'; +XA PREPARE 'trx_18'; +connection default; +KILL CONNECTION CONN_ID; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_19'; +INSERT INTO t SET a=19; +XA END 'trx_19'; +XA PREPARE 'trx_19'; +connection default; +KILL CONNECTION CONN_ID; +connection default; +XA ROLLBACK 'trx_0'; +XA ROLLBACK 'trx_1'; +XA ROLLBACK 'trx_2'; +XA ROLLBACK 'trx_3'; +XA ROLLBACK 'trx_4'; +XA COMMIT 'trx_5'; +XA COMMIT 'trx_6'; +XA COMMIT 'trx_7'; +XA COMMIT 'trx_8'; +XA COMMIT 'trx_9'; +# restart +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_0'; +INSERT INTO t SET a=0; +XA END 'new_trx_0'; +XA PREPARE 'new_trx_0'; +disconnect conn_restart_0; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_1'; +INSERT INTO t SET a=1; +XA END 'new_trx_1'; +XA PREPARE 'new_trx_1'; +disconnect conn_restart_1; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_2'; +INSERT INTO t SET a=2; +XA END 'new_trx_2'; +XA PREPARE 'new_trx_2'; +disconnect conn_restart_2; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_3'; +INSERT INTO t SET a=3; +XA END 'new_trx_3'; +XA PREPARE 'new_trx_3'; +disconnect conn_restart_3; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_4'; +INSERT INTO t SET a=4; +XA END 'new_trx_4'; +XA PREPARE 'new_trx_4'; +disconnect conn_restart_4; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_5'; +INSERT INTO t SET a=5; +XA END 'new_trx_5'; +XA PREPARE 'new_trx_5'; +disconnect conn_restart_5; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_6'; +INSERT INTO t SET a=6; +XA END 'new_trx_6'; +XA PREPARE 'new_trx_6'; +disconnect conn_restart_6; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_7'; +INSERT INTO t SET a=7; +XA END 'new_trx_7'; +XA PREPARE 'new_trx_7'; +disconnect conn_restart_7; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_8'; +INSERT INTO t SET a=8; +XA END 'new_trx_8'; +XA PREPARE 'new_trx_8'; +disconnect conn_restart_8; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_9'; +INSERT INTO t SET a=9; +XA END 'new_trx_9'; +XA PREPARE 'new_trx_9'; +disconnect conn_restart_9; +connection default; +connection default; +XA COMMIT 'new_trx_0'; +XA COMMIT 'new_trx_1'; +XA COMMIT 'new_trx_2'; +XA COMMIT 'new_trx_3'; +XA COMMIT 'new_trx_4'; +XA COMMIT 'new_trx_5'; +XA COMMIT 'new_trx_6'; +XA COMMIT 'new_trx_7'; +XA COMMIT 'new_trx_8'; +XA COMMIT 'new_trx_9'; +XA START 'trx_10'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA COMMIT 'trx_10'; +XA START 'trx_11'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA COMMIT 'trx_11'; +XA START 'trx_12'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA COMMIT 'trx_12'; +XA START 'trx_13'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA COMMIT 'trx_13'; +XA START 'trx_14'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA COMMIT 'trx_14'; +XA START 'trx_15'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA ROLLBACK 'trx_15'; +XA START 'trx_16'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA ROLLBACK 'trx_16'; +XA START 'trx_17'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA ROLLBACK 'trx_17'; +XA START 'trx_18'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA ROLLBACK 'trx_18'; +XA START 'trx_19'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA ROLLBACK 'trx_19'; +SELECT * FROM t; +a +5 +6 +7 +8 +9 +10 +11 +12 +13 +14 +0 +1 +2 +3 +4 +5 +6 +7 +8 +9 +disconnect conn2tmp; +disconnect conn3tmp; +disconnect conn2ro; +disconnect conn3ro; +disconnect conn2empty; +disconnect conn3empty; +connection default; +XA ROLLBACK 'trx_20'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn19; +connection default; +XA ROLLBACK 'trx_19'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn18; +connection default; +XA ROLLBACK 'trx_18'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn17; +connection default; +XA ROLLBACK 'trx_17'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn16; +connection default; +XA ROLLBACK 'trx_16'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn15; +connection default; +XA ROLLBACK 'trx_15'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn14; +connection default; +XA ROLLBACK 'trx_14'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn13; +connection default; +XA ROLLBACK 'trx_13'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn12; +connection default; +XA ROLLBACK 'trx_12'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn11; +connection default; +XA ROLLBACK 'trx_11'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn10; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@sql_log_bin = OFF; +CREATE TEMPORARY TABLE tmp1 (a int) ENGINE=innodb; +XA START 'trx1tmp'; +INSERT INTO tmp1 SET a=1; +XA END 'trx1tmp'; +XA PREPARE 'trx1tmp'; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@sql_log_bin = OFF; +CREATE TEMPORARY TABLE tmp1 (a int) ENGINE=innodb; +XA START 'trx2tmp'; +INSERT INTO tmp1 SET a=1; +XA END 'trx2tmp'; +XA PREPARE 'trx2tmp'; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@sql_log_bin = OFF; +CREATE TEMPORARY TABLE tmp1 (a int) ENGINE=innodb; +XA START 'trx3tmp'; +INSERT INTO tmp1 SET a=1; +XA END 'trx3tmp'; +XA PREPARE 'trx3tmp'; +connection default; +XA COMMIT 'trx1tmp'; +ERROR XAE04: XAER_NOTA: Unknown XID +XA ROLLBACK 'trx1tmp'; +ERROR XAE04: XAER_NOTA: Unknown XID +XA START 'trx1tmp'; +ERROR XAE08: XAER_DUPID: The XID already exists +connection default; +*** 3 prepared transactions must be in the list *** +XA RECOVER; +formatID gtrid_length bqual_length data +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +connection conn1tmp; +disconnect conn1tmp; +connection default; +XA COMMIT 'trx1tmp'; +KILL connection CONN_ID; +XA COMMIT 'trx3tmp'; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'trx1ro'; +SELECT * from t ORDER BY a; +a +0 +1 +2 +3 +4 +5 +5 +6 +6 +7 +7 +8 +8 +9 +9 +10 +11 +12 +13 +14 +XA END 'trx1ro'; +XA PREPARE 'trx1ro'; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'trx2ro'; +SELECT * from t ORDER BY a; +a +0 +1 +2 +3 +4 +5 +5 +6 +6 +7 +7 +8 +8 +9 +9 +10 +11 +12 +13 +14 +XA END 'trx2ro'; +XA PREPARE 'trx2ro'; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'trx3ro'; +SELECT * from t ORDER BY a; +a +0 +1 +2 +3 +4 +5 +5 +6 +6 +7 +7 +8 +8 +9 +9 +10 +11 +12 +13 +14 +XA END 'trx3ro'; +XA PREPARE 'trx3ro'; +connection default; +*** 4 prepared transactions must be in the list *** +XA RECOVER; +formatID gtrid_length bqual_length data +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +connection conn1ro; +disconnect conn1ro; +connection default; +XA ROLLBACK 'trx1ro'; +KILL connection CONN_ID; +XA ROLLBACK 'trx3ro'; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'trx1empty'; +XA END 'trx1empty'; +XA PREPARE 'trx1empty'; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'trx2empty'; +XA END 'trx2empty'; +XA PREPARE 'trx2empty'; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'trx3empty'; +XA END 'trx3empty'; +XA PREPARE 'trx3empty'; +connection default; +*** 5 prepared transactions must be in the list *** +XA RECOVER; +formatID gtrid_length bqual_length data +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +connection conn1empty; +disconnect conn1empty; +connection default; +XA COMMIT 'trx1empty'; +KILL connection CONN_ID; +XA COMMIT 'trx3empty'; +connect conn1$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'trx1unprepared'; +INSERT INTO t set a=0; +XA END 'trx1unprepared'; +INSERT INTO t set a=0; +ERROR XAE07: XAER_RMFAIL: The command cannot be executed when global transaction is in the IDLE state +XA START 'trx1unprepared'; +ERROR XAE07: XAER_RMFAIL: The command cannot be executed when global transaction is in the IDLE state +XA START 'trx1unprepared'; +ERROR XAE07: XAER_RMFAIL: The command cannot be executed when global transaction is in the IDLE state +disconnect conn1unprepared; +connection default; +XA COMMIT 'trx1unprepared'; +ERROR XAE04: XAER_NOTA: Unknown XID +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_0'; +INSERT INTO t SET a=0; +XA END 'trx_0'; +XA PREPARE 'trx_0'; +disconnect conn0; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_1'; +INSERT INTO t SET a=1; +XA END 'trx_1'; +XA PREPARE 'trx_1'; +disconnect conn1; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_2'; +INSERT INTO t SET a=2; +XA END 'trx_2'; +XA PREPARE 'trx_2'; +disconnect conn2; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_3'; +INSERT INTO t SET a=3; +XA END 'trx_3'; +XA PREPARE 'trx_3'; +disconnect conn3; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_4'; +INSERT INTO t SET a=4; +XA END 'trx_4'; +XA PREPARE 'trx_4'; +disconnect conn4; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_5'; +INSERT INTO t SET a=5; +XA END 'trx_5'; +XA PREPARE 'trx_5'; +disconnect conn5; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_6'; +INSERT INTO t SET a=6; +XA END 'trx_6'; +XA PREPARE 'trx_6'; +disconnect conn6; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_7'; +INSERT INTO t SET a=7; +XA END 'trx_7'; +XA PREPARE 'trx_7'; +disconnect conn7; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_8'; +INSERT INTO t SET a=8; +XA END 'trx_8'; +XA PREPARE 'trx_8'; +disconnect conn8; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_9'; +INSERT INTO t SET a=9; +XA END 'trx_9'; +XA PREPARE 'trx_9'; +disconnect conn9; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_10'; +INSERT INTO t SET a=10; +XA END 'trx_10'; +XA PREPARE 'trx_10'; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_11'; +INSERT INTO t SET a=11; +XA END 'trx_11'; +XA PREPARE 'trx_11'; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_12'; +INSERT INTO t SET a=12; +XA END 'trx_12'; +XA PREPARE 'trx_12'; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_13'; +INSERT INTO t SET a=13; +XA END 'trx_13'; +XA PREPARE 'trx_13'; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_14'; +INSERT INTO t SET a=14; +XA END 'trx_14'; +XA PREPARE 'trx_14'; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_15'; +INSERT INTO t SET a=15; +XA END 'trx_15'; +XA PREPARE 'trx_15'; +connection default; +KILL CONNECTION CONN_ID; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_16'; +INSERT INTO t SET a=16; +XA END 'trx_16'; +XA PREPARE 'trx_16'; +connection default; +KILL CONNECTION CONN_ID; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_17'; +INSERT INTO t SET a=17; +XA END 'trx_17'; +XA PREPARE 'trx_17'; +connection default; +KILL CONNECTION CONN_ID; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_18'; +INSERT INTO t SET a=18; +XA END 'trx_18'; +XA PREPARE 'trx_18'; +connection default; +KILL CONNECTION CONN_ID; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_19'; +INSERT INTO t SET a=19; +XA END 'trx_19'; +XA PREPARE 'trx_19'; +connection default; +KILL CONNECTION CONN_ID; +connection default; +XA ROLLBACK 'trx_0'; +XA ROLLBACK 'trx_1'; +XA ROLLBACK 'trx_2'; +XA ROLLBACK 'trx_3'; +XA ROLLBACK 'trx_4'; +XA COMMIT 'trx_5'; +XA COMMIT 'trx_6'; +XA COMMIT 'trx_7'; +XA COMMIT 'trx_8'; +XA COMMIT 'trx_9'; +# Kill and restart +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_0'; +INSERT INTO t SET a=0; +XA END 'new_trx_0'; +XA PREPARE 'new_trx_0'; +disconnect conn_restart_0; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_1'; +INSERT INTO t SET a=1; +XA END 'new_trx_1'; +XA PREPARE 'new_trx_1'; +disconnect conn_restart_1; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_2'; +INSERT INTO t SET a=2; +XA END 'new_trx_2'; +XA PREPARE 'new_trx_2'; +disconnect conn_restart_2; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_3'; +INSERT INTO t SET a=3; +XA END 'new_trx_3'; +XA PREPARE 'new_trx_3'; +disconnect conn_restart_3; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_4'; +INSERT INTO t SET a=4; +XA END 'new_trx_4'; +XA PREPARE 'new_trx_4'; +disconnect conn_restart_4; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_5'; +INSERT INTO t SET a=5; +XA END 'new_trx_5'; +XA PREPARE 'new_trx_5'; +disconnect conn_restart_5; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_6'; +INSERT INTO t SET a=6; +XA END 'new_trx_6'; +XA PREPARE 'new_trx_6'; +disconnect conn_restart_6; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_7'; +INSERT INTO t SET a=7; +XA END 'new_trx_7'; +XA PREPARE 'new_trx_7'; +disconnect conn_restart_7; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_8'; +INSERT INTO t SET a=8; +XA END 'new_trx_8'; +XA PREPARE 'new_trx_8'; +disconnect conn_restart_8; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_9'; +INSERT INTO t SET a=9; +XA END 'new_trx_9'; +XA PREPARE 'new_trx_9'; +disconnect conn_restart_9; +connection default; +connection default; +XA COMMIT 'new_trx_0'; +XA COMMIT 'new_trx_1'; +XA COMMIT 'new_trx_2'; +XA COMMIT 'new_trx_3'; +XA COMMIT 'new_trx_4'; +XA COMMIT 'new_trx_5'; +XA COMMIT 'new_trx_6'; +XA COMMIT 'new_trx_7'; +XA COMMIT 'new_trx_8'; +XA COMMIT 'new_trx_9'; +XA START 'trx_10'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA COMMIT 'trx_10'; +XA START 'trx_11'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA COMMIT 'trx_11'; +XA START 'trx_12'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA COMMIT 'trx_12'; +XA START 'trx_13'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA COMMIT 'trx_13'; +XA START 'trx_14'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA COMMIT 'trx_14'; +XA START 'trx_15'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA ROLLBACK 'trx_15'; +XA START 'trx_16'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA ROLLBACK 'trx_16'; +XA START 'trx_17'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA ROLLBACK 'trx_17'; +XA START 'trx_18'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA ROLLBACK 'trx_18'; +XA START 'trx_19'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA ROLLBACK 'trx_19'; +SELECT * FROM t; +a +5 +6 +7 +8 +9 +10 +11 +12 +13 +14 +0 +1 +2 +3 +4 +5 +6 +7 +8 +9 +5 +6 +7 +8 +9 +10 +11 +12 +13 +14 +0 +1 +2 +3 +4 +5 +6 +7 +8 +9 +disconnect conn2tmp; +disconnect conn3tmp; +disconnect conn2ro; +disconnect conn3ro; +disconnect conn2empty; +disconnect conn3empty; +connection default; +XA ROLLBACK 'trx_20'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn19; +connection default; +XA ROLLBACK 'trx_19'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn18; +connection default; +XA ROLLBACK 'trx_18'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn17; +connection default; +XA ROLLBACK 'trx_17'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn16; +connection default; +XA ROLLBACK 'trx_16'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn15; +connection default; +XA ROLLBACK 'trx_15'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn14; +connection default; +XA ROLLBACK 'trx_14'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn13; +connection default; +XA ROLLBACK 'trx_13'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn12; +connection default; +XA ROLLBACK 'trx_12'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn11; +connection default; +XA ROLLBACK 'trx_11'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn10; +connection default; +XA START 'one_phase_trx_0'; +INSERT INTO t SET a=0; +XA END 'one_phase_trx_0'; +XA COMMIT 'one_phase_trx_0' ONE PHASE; +XA START 'one_phase_trx_1'; +INSERT INTO t SET a=1; +XA END 'one_phase_trx_1'; +XA COMMIT 'one_phase_trx_1' ONE PHASE; +XA START 'one_phase_trx_2'; +INSERT INTO t SET a=2; +XA END 'one_phase_trx_2'; +XA COMMIT 'one_phase_trx_2' ONE PHASE; +XA START 'one_phase_trx_3'; +INSERT INTO t SET a=3; +XA END 'one_phase_trx_3'; +XA COMMIT 'one_phase_trx_3' ONE PHASE; +XA START 'one_phase_trx_4'; +INSERT INTO t SET a=4; +XA END 'one_phase_trx_4'; +XA COMMIT 'one_phase_trx_4' ONE PHASE; +SELECT SUM(a) FROM t; +SUM(a) +290 +DROP TABLE t; +DROP VIEW v_processlist; +All transactions must be completed, to empty-list the following: +XA RECOVER; +formatID gtrid_length bqual_length data diff --git a/mysql-test/main/xa_prepared_binlog_off.test b/mysql-test/main/xa_prepared_binlog_off.test new file mode 100644 index 00000000000..edbfa7c2825 --- /dev/null +++ b/mysql-test/main/xa_prepared_binlog_off.test @@ -0,0 +1,11 @@ +############################################################################### +# MDEV-7974 (bug#12161 Xa recovery and client disconnection) +# Testing XA behaviour with binlog turned off. +############################################################################### + +--source include/not_valgrind.inc +--source include/not_embedded.inc + +# Common part with XA binlogging testing +call mtr.add_suppression("You need to use --log-bin to make --log-slave-updates work."); +--source suite/binlog/t/binlog_xa_prepared.inc diff --git a/mysql-test/main/xa_sync.result b/mysql-test/main/xa_sync.result index 1482ff5cacf..e7dd9b02847 100644 --- a/mysql-test/main/xa_sync.result +++ b/mysql-test/main/xa_sync.result @@ -18,6 +18,11 @@ disconnect con1; SET debug_sync='now SIGNAL go'; connection con2; ERROR XAE04: XAER_NOTA: Unknown XID +*** Must have 'xatest' in the list +XA RECOVER; +formatID gtrid_length bqual_length data +1 6 0 xatest +XA COMMIT 'xatest'; disconnect con2; connection default; SET debug_sync='RESET'; @@ -37,6 +42,11 @@ disconnect con1; SET debug_sync='now SIGNAL go'; connection con2; ERROR XAE04: XAER_NOTA: Unknown XID +*** Must have 'xatest' in the list +XA RECOVER; +formatID gtrid_length bqual_length data +1 6 0 xatest +XA ROLLBACK 'xatest'; disconnect con2; connection default; SET debug_sync='RESET'; diff --git a/mysql-test/main/xa_sync.test b/mysql-test/main/xa_sync.test index bb95af7c0ba..2fe7337501e 100644 --- a/mysql-test/main/xa_sync.test +++ b/mysql-test/main/xa_sync.test @@ -35,6 +35,11 @@ while ($i) connection con2; --error ER_XAER_NOTA reap; + --echo *** Must have 'xatest' in the list + XA RECOVER; + # second time yields no error + --error 0 + --eval $op disconnect con2; connection default; diff --git a/mysql-test/suite/binlog/include/binlog_xa_prepare_connection.inc b/mysql-test/suite/binlog/include/binlog_xa_prepare_connection.inc new file mode 100644 index 00000000000..c0041af1e7f --- /dev/null +++ b/mysql-test/suite/binlog/include/binlog_xa_prepare_connection.inc @@ -0,0 +1,31 @@ +# +# This file initiate connections to run XA transactions up to +# their prepare. +# Connection name, transaction name and its content depends on +# supplied parameters. +# +# param $type type of transaction +# param $index index identifies the connection with those of type $type +# param $sql_init1 a query to execute once connection is established +# param $sql_init2 a query to execute once connection is established +# param $sql_doit a query to execute inside transaction +# Note, the query may depend on tables created by caller +# + +--connect (conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,) +if ($sql_init1) +{ + --eval $sql_init1 +} +if ($sql_init2) +{ + --eval $sql_init2 +} + +--eval XA START 'trx$index$type' +if ($sql_doit) +{ + --eval $sql_doit +} +--eval XA END 'trx$index$type' +--eval XA PREPARE 'trx$index$type' diff --git a/mysql-test/suite/binlog/include/binlog_xa_prepare_disconnect.inc b/mysql-test/suite/binlog/include/binlog_xa_prepare_disconnect.inc new file mode 100644 index 00000000000..1f6ce713cc9 --- /dev/null +++ b/mysql-test/suite/binlog/include/binlog_xa_prepare_disconnect.inc @@ -0,0 +1,35 @@ +# +# This file disconnects two connections. One actively and one through +# kill. It is included by binlog_xa_prepared_do_and_restart. +# +# param $type type of transaction +# param $terminate_with how to conclude actively disconnecte: +# XA COMMIT or XA ROLLBACK +# param $conn3_id connection id of the being killed. +# param $num_trx_prepared number of transactions prepared so far +# +--connection default + +--echo *** $num_trx_prepared prepared transactions must be in the list *** +--replace_column 2 LEN1 3 LEN2 4 TRX_N +XA RECOVER; + +--connection conn1$type +--let $conn1_id=`SELECT connection_id()` +--disconnect conn1$type + +--connection default +--let $wait_condition= SELECT count(*) = 0 FROM v_processlist WHERE PROCESSLIST_ID = $conn1_id +--source include/wait_condition.inc + +# It will conclude now +--eval $terminate_with 'trx1$type' + +--replace_result $conn3_id CONN_ID +--eval KILL connection $conn3_id + +--let $wait_condition= SELECT count(*) = 0 FROM v_processlist WHERE PROCESSLIST_ID = $conn3_id +--source include/wait_condition.inc + +# It will conclude now +--eval $terminate_with 'trx3$type' diff --git a/mysql-test/suite/binlog/include/binlog_xa_prepared_do_and_restart.inc b/mysql-test/suite/binlog/include/binlog_xa_prepared_do_and_restart.inc new file mode 100644 index 00000000000..cbd740fdae4 --- /dev/null +++ b/mysql-test/suite/binlog/include/binlog_xa_prepared_do_and_restart.inc @@ -0,0 +1,323 @@ +# +# This file creates various kinds of prepared XA transactions, +# manipulates their connection state and examines how their prepared +# status behave while the transaction is disconnected, killed or +# the server kisses it shutdown. +# The file can be sourced multiple times +# param $restart_number (as the number of inclusion) adjusts +# verification logics. +# +# param [in] $conn_number Total number of connection each performing +# one insert into table. +# param [in] $commit_number Number of commits from either. +# side of the server restart. +# param [in] $rollback_number The same as the above just for rollback. +# param [in] $term_number Number of transaction that are terminated +# before server restarts +# param [in] $killed_number Instead of disconnect make some +# connections killed when their +# transactions got prepared. +# param [in] $server_disconn_number Make some connections disconnected +# by shutdown rather than actively +# param [in] $post_restart_conn_number Number a "warmup" connection +# after server restart, they all commit +# param [out] restart_number Counter to be incremented at the end of the test +# + +# The test consists of three sections: +# I. Corner cases check +# II. Regular case check +# III. Post server-restart verification + + +# +# I. Corner cases of +# +# A. XA with an update to a temp table +# B. XA with SELECT +# C. XA empty +# Demonstrate their XA status upon prepare and how they react on disconnect and +# shutdown. +# In each of A,B,C three prepared transactions are set up. +# trx1 is for disconnection, trx2 for shutdown, trx3 for being killed. +# The A case additionally contains some XA prohibited state transaction check. +# +# D. Prove that not prepared XA remains to be cleared out by disconnection. +# + +# +# A. The temp table only prepared XA recovers only formally to +# let post recovery XA COMMIT or XA ROLLBACK with no effect. + +--let $type = tmp +--let $index = 1 +--let $sql_init1 = SET @@sql_log_bin = OFF +--let $sql_init2 = CREATE TEMPORARY TABLE tmp$index (a int) ENGINE=innodb +--let $sql_doit = INSERT INTO tmp$index SET a=$index +--source suite/binlog/include/binlog_xa_prepare_connection.inc + +--let $index = 2 +--source suite/binlog/include/binlog_xa_prepare_connection.inc + +--let $index = 3 +--source suite/binlog/include/binlog_xa_prepare_connection.inc +--let $conn3_id=`SELECT connection_id()` + +# +# Various prohibited XA state changes to test here: +# + +--connection default +# Stealing is not allowed +--error ER_XAER_NOTA +--eval XA COMMIT 'trx1$type' +--error ER_XAER_NOTA +--eval XA ROLLBACK 'trx1$type' + +# Before disconnect: creating a duplicate is not allowed +--error ER_XAER_DUPID +--eval XA START 'trx1$type' + +# Manipulate now the prepared transactions. +# Two to terminate, one to leave out. +--let $terminate_with = XA COMMIT +--let $num_trx_prepared = $index +--source suite/binlog/include/binlog_xa_prepare_disconnect.inc + +# +# B. "Read-only" (select) prepared XA recovers only formally to +# let post recovery XA COMMIT or XA ROLLBACK with no effect. +# +--let $type=ro +--let $index = 1 +--let $sql_init1 = +--let $sql_init2 = +--let $sql_doit = SELECT * from t ORDER BY a +--source suite/binlog/include/binlog_xa_prepare_connection.inc + +--let $index = 2 +--source suite/binlog/include/binlog_xa_prepare_connection.inc + +--let $index = 3 +--source suite/binlog/include/binlog_xa_prepare_connection.inc +--let $conn3_id=`SELECT connection_id()` + +--let $terminate_with = XA ROLLBACK +# two three above section prepared transaction were terminated. +--inc $num_trx_prepared +--source suite/binlog/include/binlog_xa_prepare_disconnect.inc + +# +# C. Empty prepared XA recovers only formally to +# let post recovery XA COMMIT or XA ROLLBACK with no effect. +# +--let $type=empty +--let $index = 1 +--let $sql_init1 = +--let $sql_init2 = +--let $sql_doit = +--source suite/binlog/include/binlog_xa_prepare_connection.inc + +--let $index = 2 +--source suite/binlog/include/binlog_xa_prepare_connection.inc + +--let $index = 3 +--source suite/binlog/include/binlog_xa_prepare_connection.inc +--let $conn3_id=`SELECT connection_id()` + +--let $terminate_with = XA COMMIT +--inc $num_trx_prepared +--source suite/binlog/include/binlog_xa_prepare_disconnect.inc + +# +# D. Not prepared XA disconnects to be cleared out, +# no effect on data left as well. +# Few more prohibited XA state transactions is checked out. +# +--let $type=unprepared +--let $prev_count=`SELECT count(*) from t` + +--connect(conn1$type, 127.0.0.1,root,,test,$MASTER_MYPORT,) +--eval XA START 'trx1$type' +INSERT INTO t set a=0; +--eval XA END 'trx1$type' + +--error ER_XAER_RMFAIL +INSERT INTO t set a=0; +--error ER_XAER_RMFAIL +--eval XA START 'trx1$type' +--error ER_XAER_RMFAIL +--eval XA START 'trx1$type' + +--disconnect conn1$type + +--connection default +# No such transactions +--error ER_XAER_NOTA +--eval XA COMMIT 'trx1$type' +if (`SELECT count(*) > $prev_count from t`) +{ + --echo *** Unexpected commit to the table. *** + --die +} + +# +# II. Regular case. +# +# Prepared transactions get disconnected in three ways: +# actively, being killed and by the server shutdown. +# +--let $i=0 +while ($i < $conn_number) +{ + --connect (conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,) + --let $conn_id=`SELECT connection_id()` + --disable_reconnect + SET @@binlog_format = STATEMENT; + if (`SELECT $i % 2`) + { + SET @@binlog_format = ROW; + } + --eval XA START 'trx_$i' + --eval INSERT INTO t SET a=$i + --eval XA END 'trx_$i' + --eval XA PREPARE 'trx_$i' + + --let $disc_via_kill=`SELECT $conn_number - $i <= $killed_number` + if (!$disc_via_kill) + { + --let $disc_via_shutdown=`SELECT $conn_number - $i <= $killed_number + $server_disconn_number` + if (!$disc_via_shutdown) + { + --disconnect conn$i + } + } + if ($disc_via_kill) + { + --connection default + --replace_result $conn_id CONN_ID + --eval KILL CONNECTION $conn_id + } + + if (!$disc_via_shutdown) + { + --connection default + --let $wait_condition= SELECT count(*) = 0 FROM v_processlist WHERE PROCESSLIST_ID = $conn_id + --source include/wait_condition.inc + } + --inc $i +} + +# [0, $rollback_number - 1] are rolled back now +--connection default + +--let $i=0 +while ($i < $rollback_number) +{ + --eval XA ROLLBACK 'trx_$i' + + --inc $i +} + +# [$rollback_number, $rollback_number + $commit_number - 1] get committed +while ($i < $term_number) +{ + --eval XA COMMIT 'trx_$i' + + --inc $i +} + +--source include/$how_to_restart + +# +# III. Post server-restart verification. +# It concludes survived XA:s with a number of commits and rollbacks +# as configured in the 1st part to check expected results in the end. +# Cleanup section consists of explicit disconnect (for killed, or +# not disconnected before shutdown). +# + +# New XA can be prepared and committed +--let $k = 0 +while ($k < $post_restart_conn_number) +{ + --connect (conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,) + --let $conn_id=`SELECT connection_id()` + --eval XA START 'new_trx_$k' + --eval INSERT INTO t SET a=$k + --eval XA END 'new_trx_$k' + --eval XA PREPARE 'new_trx_$k' + + --disconnect conn_restart_$k + + --connection default + --let $wait_condition= SELECT count(*) = 0 FROM v_processlist WHERE PROCESSLIST_ID = $conn_id + --source include/wait_condition.inc + + --inc $k +} + +--connection default +--let $k = 0 +while ($k < $post_restart_conn_number) +{ + --eval XA COMMIT 'new_trx_$k' + --inc $k +} + +# +# Symmetrically to the pre-restart, the resurrected trx:s are committed +# [$term_number, $term_number + $commit_number - 1] +# and the rest is rolled back. +# +--let $i = $term_number + +while ($i < `SELECT $term_number + $commit_number`) +{ + # Expected to fail + --error ER_XAER_DUPID + --eval XA START 'trx_$i' + --eval XA COMMIT 'trx_$i' + --inc $i +} + +while ($i < $conn_number) +{ + # Expected to fail + --error ER_XAER_DUPID + --eval XA START 'trx_$i' + --eval XA ROLLBACK 'trx_$i' + --inc $i +} + +# +# Verification of correct results of recovered XA transaction handling: +# +SELECT * FROM t; + +--let $type=tmp +--disconnect conn2$type +--disconnect conn3$type +--let $type=ro +--disconnect conn2$type +--disconnect conn3$type +--let $type=empty +--disconnect conn2$type +--disconnect conn3$type + +--let $i= $conn_number +--let $k= 0 +--let $expl_disconn_number = `SELECT $killed_number + $server_disconn_number` +while ($k < $expl_disconn_number) +{ + --connection default + --error ER_XAER_NOTA + --eval XA ROLLBACK 'trx_$i' + + --dec $i + --disconnect conn$i + + --inc $k +} + +--inc $restart_number diff --git a/mysql-test/suite/binlog/r/binlog_xa_checkpoint.result b/mysql-test/suite/binlog/r/binlog_xa_checkpoint.result new file mode 100644 index 00000000000..d8a5818674f --- /dev/null +++ b/mysql-test/suite/binlog/r/binlog_xa_checkpoint.result @@ -0,0 +1,33 @@ +RESET MASTER; +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +connect con1,localhost,root,,; +SET DEBUG_SYNC= "at_unlog_xa_prepare SIGNAL con1_ready WAIT_FOR con1_go"; +XA START '1'; +INSERT INTO t1 SET a=1; +XA END '1'; +XA PREPARE '1';; +connection default; +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; +FLUSH LOGS; +FLUSH LOGS; +FLUSH LOGS; +show binary logs; +Log_name File_size +master-bin.000001 # +master-bin.000002 # +master-bin.000003 # +master-bin.000004 # +include/show_binlog_events.inc +Log_name Pos Event_type Server_id End_log_pos Info +master-bin.000004 # Format_desc # # SERVER_VERSION, BINLOG_VERSION +master-bin.000004 # Gtid_list # # [#-#-#] +master-bin.000004 # Binlog_checkpoint # # master-bin.000001 +SET DEBUG_SYNC= "now SIGNAL con1_go"; +connection con1; +*** master-bin.000004 checkpoint must show up now *** +connection con1; +XA ROLLBACK '1'; +SET debug_sync = 'reset'; +connection default; +DROP TABLE t1; +SET debug_sync = 'reset'; diff --git a/mysql-test/suite/binlog/r/binlog_xa_prepared.result b/mysql-test/suite/binlog/r/binlog_xa_prepared.result new file mode 100644 index 00000000000..9fda8ab3143 --- /dev/null +++ b/mysql-test/suite/binlog/r/binlog_xa_prepared.result @@ -0,0 +1,1176 @@ +connection default; +RESET MASTER; +CREATE VIEW v_processlist as SELECT * FROM performance_schema.threads where type = 'FOREGROUND'; +call mtr.add_suppression("Found 10 prepared XA transactions"); +CREATE TABLE t (a INT) ENGINE=innodb; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@sql_log_bin = OFF; +CREATE TEMPORARY TABLE tmp1 (a int) ENGINE=innodb; +XA START 'trx1tmp'; +INSERT INTO tmp1 SET a=1; +XA END 'trx1tmp'; +XA PREPARE 'trx1tmp'; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@sql_log_bin = OFF; +CREATE TEMPORARY TABLE tmp1 (a int) ENGINE=innodb; +XA START 'trx2tmp'; +INSERT INTO tmp1 SET a=1; +XA END 'trx2tmp'; +XA PREPARE 'trx2tmp'; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@sql_log_bin = OFF; +CREATE TEMPORARY TABLE tmp1 (a int) ENGINE=innodb; +XA START 'trx3tmp'; +INSERT INTO tmp1 SET a=1; +XA END 'trx3tmp'; +XA PREPARE 'trx3tmp'; +connection default; +XA COMMIT 'trx1tmp'; +ERROR XAE04: XAER_NOTA: Unknown XID +XA ROLLBACK 'trx1tmp'; +ERROR XAE04: XAER_NOTA: Unknown XID +XA START 'trx1tmp'; +ERROR XAE08: XAER_DUPID: The XID already exists +connection default; +*** 3 prepared transactions must be in the list *** +XA RECOVER; +formatID gtrid_length bqual_length data +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +connection conn1tmp; +disconnect conn1tmp; +connection default; +XA COMMIT 'trx1tmp'; +KILL connection CONN_ID; +XA COMMIT 'trx3tmp'; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'trx1ro'; +SELECT * from t ORDER BY a; +a +XA END 'trx1ro'; +XA PREPARE 'trx1ro'; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'trx2ro'; +SELECT * from t ORDER BY a; +a +XA END 'trx2ro'; +XA PREPARE 'trx2ro'; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'trx3ro'; +SELECT * from t ORDER BY a; +a +XA END 'trx3ro'; +XA PREPARE 'trx3ro'; +connection default; +*** 4 prepared transactions must be in the list *** +XA RECOVER; +formatID gtrid_length bqual_length data +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +connection conn1ro; +disconnect conn1ro; +connection default; +XA ROLLBACK 'trx1ro'; +KILL connection CONN_ID; +XA ROLLBACK 'trx3ro'; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'trx1empty'; +XA END 'trx1empty'; +XA PREPARE 'trx1empty'; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'trx2empty'; +XA END 'trx2empty'; +XA PREPARE 'trx2empty'; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'trx3empty'; +XA END 'trx3empty'; +XA PREPARE 'trx3empty'; +connection default; +*** 5 prepared transactions must be in the list *** +XA RECOVER; +formatID gtrid_length bqual_length data +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +connection conn1empty; +disconnect conn1empty; +connection default; +XA COMMIT 'trx1empty'; +KILL connection CONN_ID; +XA COMMIT 'trx3empty'; +connect conn1$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'trx1unprepared'; +INSERT INTO t set a=0; +XA END 'trx1unprepared'; +INSERT INTO t set a=0; +ERROR XAE07: XAER_RMFAIL: The command cannot be executed when global transaction is in the IDLE state +XA START 'trx1unprepared'; +ERROR XAE07: XAER_RMFAIL: The command cannot be executed when global transaction is in the IDLE state +XA START 'trx1unprepared'; +ERROR XAE07: XAER_RMFAIL: The command cannot be executed when global transaction is in the IDLE state +disconnect conn1unprepared; +connection default; +XA COMMIT 'trx1unprepared'; +ERROR XAE04: XAER_NOTA: Unknown XID +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_0'; +INSERT INTO t SET a=0; +XA END 'trx_0'; +XA PREPARE 'trx_0'; +disconnect conn0; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_1'; +INSERT INTO t SET a=1; +XA END 'trx_1'; +XA PREPARE 'trx_1'; +disconnect conn1; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_2'; +INSERT INTO t SET a=2; +XA END 'trx_2'; +XA PREPARE 'trx_2'; +disconnect conn2; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_3'; +INSERT INTO t SET a=3; +XA END 'trx_3'; +XA PREPARE 'trx_3'; +disconnect conn3; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_4'; +INSERT INTO t SET a=4; +XA END 'trx_4'; +XA PREPARE 'trx_4'; +disconnect conn4; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_5'; +INSERT INTO t SET a=5; +XA END 'trx_5'; +XA PREPARE 'trx_5'; +disconnect conn5; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_6'; +INSERT INTO t SET a=6; +XA END 'trx_6'; +XA PREPARE 'trx_6'; +disconnect conn6; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_7'; +INSERT INTO t SET a=7; +XA END 'trx_7'; +XA PREPARE 'trx_7'; +disconnect conn7; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_8'; +INSERT INTO t SET a=8; +XA END 'trx_8'; +XA PREPARE 'trx_8'; +disconnect conn8; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_9'; +INSERT INTO t SET a=9; +XA END 'trx_9'; +XA PREPARE 'trx_9'; +disconnect conn9; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_10'; +INSERT INTO t SET a=10; +XA END 'trx_10'; +XA PREPARE 'trx_10'; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_11'; +INSERT INTO t SET a=11; +XA END 'trx_11'; +XA PREPARE 'trx_11'; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_12'; +INSERT INTO t SET a=12; +XA END 'trx_12'; +XA PREPARE 'trx_12'; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_13'; +INSERT INTO t SET a=13; +XA END 'trx_13'; +XA PREPARE 'trx_13'; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_14'; +INSERT INTO t SET a=14; +XA END 'trx_14'; +XA PREPARE 'trx_14'; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_15'; +INSERT INTO t SET a=15; +XA END 'trx_15'; +XA PREPARE 'trx_15'; +connection default; +KILL CONNECTION CONN_ID; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_16'; +INSERT INTO t SET a=16; +XA END 'trx_16'; +XA PREPARE 'trx_16'; +connection default; +KILL CONNECTION CONN_ID; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_17'; +INSERT INTO t SET a=17; +XA END 'trx_17'; +XA PREPARE 'trx_17'; +connection default; +KILL CONNECTION CONN_ID; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_18'; +INSERT INTO t SET a=18; +XA END 'trx_18'; +XA PREPARE 'trx_18'; +connection default; +KILL CONNECTION CONN_ID; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_19'; +INSERT INTO t SET a=19; +XA END 'trx_19'; +XA PREPARE 'trx_19'; +connection default; +KILL CONNECTION CONN_ID; +connection default; +XA ROLLBACK 'trx_0'; +XA ROLLBACK 'trx_1'; +XA ROLLBACK 'trx_2'; +XA ROLLBACK 'trx_3'; +XA ROLLBACK 'trx_4'; +XA COMMIT 'trx_5'; +XA COMMIT 'trx_6'; +XA COMMIT 'trx_7'; +XA COMMIT 'trx_8'; +XA COMMIT 'trx_9'; +# restart +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_0'; +INSERT INTO t SET a=0; +XA END 'new_trx_0'; +XA PREPARE 'new_trx_0'; +disconnect conn_restart_0; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_1'; +INSERT INTO t SET a=1; +XA END 'new_trx_1'; +XA PREPARE 'new_trx_1'; +disconnect conn_restart_1; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_2'; +INSERT INTO t SET a=2; +XA END 'new_trx_2'; +XA PREPARE 'new_trx_2'; +disconnect conn_restart_2; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_3'; +INSERT INTO t SET a=3; +XA END 'new_trx_3'; +XA PREPARE 'new_trx_3'; +disconnect conn_restart_3; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_4'; +INSERT INTO t SET a=4; +XA END 'new_trx_4'; +XA PREPARE 'new_trx_4'; +disconnect conn_restart_4; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_5'; +INSERT INTO t SET a=5; +XA END 'new_trx_5'; +XA PREPARE 'new_trx_5'; +disconnect conn_restart_5; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_6'; +INSERT INTO t SET a=6; +XA END 'new_trx_6'; +XA PREPARE 'new_trx_6'; +disconnect conn_restart_6; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_7'; +INSERT INTO t SET a=7; +XA END 'new_trx_7'; +XA PREPARE 'new_trx_7'; +disconnect conn_restart_7; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_8'; +INSERT INTO t SET a=8; +XA END 'new_trx_8'; +XA PREPARE 'new_trx_8'; +disconnect conn_restart_8; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_9'; +INSERT INTO t SET a=9; +XA END 'new_trx_9'; +XA PREPARE 'new_trx_9'; +disconnect conn_restart_9; +connection default; +connection default; +XA COMMIT 'new_trx_0'; +XA COMMIT 'new_trx_1'; +XA COMMIT 'new_trx_2'; +XA COMMIT 'new_trx_3'; +XA COMMIT 'new_trx_4'; +XA COMMIT 'new_trx_5'; +XA COMMIT 'new_trx_6'; +XA COMMIT 'new_trx_7'; +XA COMMIT 'new_trx_8'; +XA COMMIT 'new_trx_9'; +XA START 'trx_10'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA COMMIT 'trx_10'; +XA START 'trx_11'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA COMMIT 'trx_11'; +XA START 'trx_12'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA COMMIT 'trx_12'; +XA START 'trx_13'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA COMMIT 'trx_13'; +XA START 'trx_14'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA COMMIT 'trx_14'; +XA START 'trx_15'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA ROLLBACK 'trx_15'; +XA START 'trx_16'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA ROLLBACK 'trx_16'; +XA START 'trx_17'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA ROLLBACK 'trx_17'; +XA START 'trx_18'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA ROLLBACK 'trx_18'; +XA START 'trx_19'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA ROLLBACK 'trx_19'; +SELECT * FROM t; +a +5 +6 +7 +8 +9 +10 +11 +12 +13 +14 +0 +1 +2 +3 +4 +5 +6 +7 +8 +9 +disconnect conn2tmp; +disconnect conn3tmp; +disconnect conn2ro; +disconnect conn3ro; +disconnect conn2empty; +disconnect conn3empty; +connection default; +XA ROLLBACK 'trx_20'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn19; +connection default; +XA ROLLBACK 'trx_19'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn18; +connection default; +XA ROLLBACK 'trx_18'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn17; +connection default; +XA ROLLBACK 'trx_17'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn16; +connection default; +XA ROLLBACK 'trx_16'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn15; +connection default; +XA ROLLBACK 'trx_15'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn14; +connection default; +XA ROLLBACK 'trx_14'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn13; +connection default; +XA ROLLBACK 'trx_13'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn12; +connection default; +XA ROLLBACK 'trx_12'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn11; +connection default; +XA ROLLBACK 'trx_11'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn10; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@sql_log_bin = OFF; +CREATE TEMPORARY TABLE tmp1 (a int) ENGINE=innodb; +XA START 'trx1tmp'; +INSERT INTO tmp1 SET a=1; +XA END 'trx1tmp'; +XA PREPARE 'trx1tmp'; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@sql_log_bin = OFF; +CREATE TEMPORARY TABLE tmp1 (a int) ENGINE=innodb; +XA START 'trx2tmp'; +INSERT INTO tmp1 SET a=1; +XA END 'trx2tmp'; +XA PREPARE 'trx2tmp'; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@sql_log_bin = OFF; +CREATE TEMPORARY TABLE tmp1 (a int) ENGINE=innodb; +XA START 'trx3tmp'; +INSERT INTO tmp1 SET a=1; +XA END 'trx3tmp'; +XA PREPARE 'trx3tmp'; +connection default; +XA COMMIT 'trx1tmp'; +ERROR XAE04: XAER_NOTA: Unknown XID +XA ROLLBACK 'trx1tmp'; +ERROR XAE04: XAER_NOTA: Unknown XID +XA START 'trx1tmp'; +ERROR XAE08: XAER_DUPID: The XID already exists +connection default; +*** 3 prepared transactions must be in the list *** +XA RECOVER; +formatID gtrid_length bqual_length data +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +connection conn1tmp; +disconnect conn1tmp; +connection default; +XA COMMIT 'trx1tmp'; +KILL connection CONN_ID; +XA COMMIT 'trx3tmp'; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'trx1ro'; +SELECT * from t ORDER BY a; +a +0 +1 +2 +3 +4 +5 +5 +6 +6 +7 +7 +8 +8 +9 +9 +10 +11 +12 +13 +14 +XA END 'trx1ro'; +XA PREPARE 'trx1ro'; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'trx2ro'; +SELECT * from t ORDER BY a; +a +0 +1 +2 +3 +4 +5 +5 +6 +6 +7 +7 +8 +8 +9 +9 +10 +11 +12 +13 +14 +XA END 'trx2ro'; +XA PREPARE 'trx2ro'; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'trx3ro'; +SELECT * from t ORDER BY a; +a +0 +1 +2 +3 +4 +5 +5 +6 +6 +7 +7 +8 +8 +9 +9 +10 +11 +12 +13 +14 +XA END 'trx3ro'; +XA PREPARE 'trx3ro'; +connection default; +*** 4 prepared transactions must be in the list *** +XA RECOVER; +formatID gtrid_length bqual_length data +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +connection conn1ro; +disconnect conn1ro; +connection default; +XA ROLLBACK 'trx1ro'; +KILL connection CONN_ID; +XA ROLLBACK 'trx3ro'; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'trx1empty'; +XA END 'trx1empty'; +XA PREPARE 'trx1empty'; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'trx2empty'; +XA END 'trx2empty'; +XA PREPARE 'trx2empty'; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'trx3empty'; +XA END 'trx3empty'; +XA PREPARE 'trx3empty'; +connection default; +*** 5 prepared transactions must be in the list *** +XA RECOVER; +formatID gtrid_length bqual_length data +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +connection conn1empty; +disconnect conn1empty; +connection default; +XA COMMIT 'trx1empty'; +KILL connection CONN_ID; +XA COMMIT 'trx3empty'; +connect conn1$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'trx1unprepared'; +INSERT INTO t set a=0; +XA END 'trx1unprepared'; +INSERT INTO t set a=0; +ERROR XAE07: XAER_RMFAIL: The command cannot be executed when global transaction is in the IDLE state +XA START 'trx1unprepared'; +ERROR XAE07: XAER_RMFAIL: The command cannot be executed when global transaction is in the IDLE state +XA START 'trx1unprepared'; +ERROR XAE07: XAER_RMFAIL: The command cannot be executed when global transaction is in the IDLE state +disconnect conn1unprepared; +connection default; +XA COMMIT 'trx1unprepared'; +ERROR XAE04: XAER_NOTA: Unknown XID +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_0'; +INSERT INTO t SET a=0; +XA END 'trx_0'; +XA PREPARE 'trx_0'; +disconnect conn0; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_1'; +INSERT INTO t SET a=1; +XA END 'trx_1'; +XA PREPARE 'trx_1'; +disconnect conn1; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_2'; +INSERT INTO t SET a=2; +XA END 'trx_2'; +XA PREPARE 'trx_2'; +disconnect conn2; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_3'; +INSERT INTO t SET a=3; +XA END 'trx_3'; +XA PREPARE 'trx_3'; +disconnect conn3; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_4'; +INSERT INTO t SET a=4; +XA END 'trx_4'; +XA PREPARE 'trx_4'; +disconnect conn4; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_5'; +INSERT INTO t SET a=5; +XA END 'trx_5'; +XA PREPARE 'trx_5'; +disconnect conn5; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_6'; +INSERT INTO t SET a=6; +XA END 'trx_6'; +XA PREPARE 'trx_6'; +disconnect conn6; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_7'; +INSERT INTO t SET a=7; +XA END 'trx_7'; +XA PREPARE 'trx_7'; +disconnect conn7; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_8'; +INSERT INTO t SET a=8; +XA END 'trx_8'; +XA PREPARE 'trx_8'; +disconnect conn8; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_9'; +INSERT INTO t SET a=9; +XA END 'trx_9'; +XA PREPARE 'trx_9'; +disconnect conn9; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_10'; +INSERT INTO t SET a=10; +XA END 'trx_10'; +XA PREPARE 'trx_10'; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_11'; +INSERT INTO t SET a=11; +XA END 'trx_11'; +XA PREPARE 'trx_11'; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_12'; +INSERT INTO t SET a=12; +XA END 'trx_12'; +XA PREPARE 'trx_12'; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_13'; +INSERT INTO t SET a=13; +XA END 'trx_13'; +XA PREPARE 'trx_13'; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_14'; +INSERT INTO t SET a=14; +XA END 'trx_14'; +XA PREPARE 'trx_14'; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_15'; +INSERT INTO t SET a=15; +XA END 'trx_15'; +XA PREPARE 'trx_15'; +connection default; +KILL CONNECTION CONN_ID; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_16'; +INSERT INTO t SET a=16; +XA END 'trx_16'; +XA PREPARE 'trx_16'; +connection default; +KILL CONNECTION CONN_ID; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_17'; +INSERT INTO t SET a=17; +XA END 'trx_17'; +XA PREPARE 'trx_17'; +connection default; +KILL CONNECTION CONN_ID; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_18'; +INSERT INTO t SET a=18; +XA END 'trx_18'; +XA PREPARE 'trx_18'; +connection default; +KILL CONNECTION CONN_ID; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_19'; +INSERT INTO t SET a=19; +XA END 'trx_19'; +XA PREPARE 'trx_19'; +connection default; +KILL CONNECTION CONN_ID; +connection default; +XA ROLLBACK 'trx_0'; +XA ROLLBACK 'trx_1'; +XA ROLLBACK 'trx_2'; +XA ROLLBACK 'trx_3'; +XA ROLLBACK 'trx_4'; +XA COMMIT 'trx_5'; +XA COMMIT 'trx_6'; +XA COMMIT 'trx_7'; +XA COMMIT 'trx_8'; +XA COMMIT 'trx_9'; +# Kill and restart +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_0'; +INSERT INTO t SET a=0; +XA END 'new_trx_0'; +XA PREPARE 'new_trx_0'; +disconnect conn_restart_0; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_1'; +INSERT INTO t SET a=1; +XA END 'new_trx_1'; +XA PREPARE 'new_trx_1'; +disconnect conn_restart_1; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_2'; +INSERT INTO t SET a=2; +XA END 'new_trx_2'; +XA PREPARE 'new_trx_2'; +disconnect conn_restart_2; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_3'; +INSERT INTO t SET a=3; +XA END 'new_trx_3'; +XA PREPARE 'new_trx_3'; +disconnect conn_restart_3; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_4'; +INSERT INTO t SET a=4; +XA END 'new_trx_4'; +XA PREPARE 'new_trx_4'; +disconnect conn_restart_4; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_5'; +INSERT INTO t SET a=5; +XA END 'new_trx_5'; +XA PREPARE 'new_trx_5'; +disconnect conn_restart_5; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_6'; +INSERT INTO t SET a=6; +XA END 'new_trx_6'; +XA PREPARE 'new_trx_6'; +disconnect conn_restart_6; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_7'; +INSERT INTO t SET a=7; +XA END 'new_trx_7'; +XA PREPARE 'new_trx_7'; +disconnect conn_restart_7; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_8'; +INSERT INTO t SET a=8; +XA END 'new_trx_8'; +XA PREPARE 'new_trx_8'; +disconnect conn_restart_8; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_9'; +INSERT INTO t SET a=9; +XA END 'new_trx_9'; +XA PREPARE 'new_trx_9'; +disconnect conn_restart_9; +connection default; +connection default; +XA COMMIT 'new_trx_0'; +XA COMMIT 'new_trx_1'; +XA COMMIT 'new_trx_2'; +XA COMMIT 'new_trx_3'; +XA COMMIT 'new_trx_4'; +XA COMMIT 'new_trx_5'; +XA COMMIT 'new_trx_6'; +XA COMMIT 'new_trx_7'; +XA COMMIT 'new_trx_8'; +XA COMMIT 'new_trx_9'; +XA START 'trx_10'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA COMMIT 'trx_10'; +XA START 'trx_11'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA COMMIT 'trx_11'; +XA START 'trx_12'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA COMMIT 'trx_12'; +XA START 'trx_13'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA COMMIT 'trx_13'; +XA START 'trx_14'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA COMMIT 'trx_14'; +XA START 'trx_15'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA ROLLBACK 'trx_15'; +XA START 'trx_16'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA ROLLBACK 'trx_16'; +XA START 'trx_17'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA ROLLBACK 'trx_17'; +XA START 'trx_18'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA ROLLBACK 'trx_18'; +XA START 'trx_19'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA ROLLBACK 'trx_19'; +SELECT * FROM t; +a +5 +6 +7 +8 +9 +10 +11 +12 +13 +14 +0 +1 +2 +3 +4 +5 +6 +7 +8 +9 +5 +6 +7 +8 +9 +10 +11 +12 +13 +14 +0 +1 +2 +3 +4 +5 +6 +7 +8 +9 +disconnect conn2tmp; +disconnect conn3tmp; +disconnect conn2ro; +disconnect conn3ro; +disconnect conn2empty; +disconnect conn3empty; +connection default; +XA ROLLBACK 'trx_20'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn19; +connection default; +XA ROLLBACK 'trx_19'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn18; +connection default; +XA ROLLBACK 'trx_18'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn17; +connection default; +XA ROLLBACK 'trx_17'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn16; +connection default; +XA ROLLBACK 'trx_16'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn15; +connection default; +XA ROLLBACK 'trx_15'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn14; +connection default; +XA ROLLBACK 'trx_14'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn13; +connection default; +XA ROLLBACK 'trx_13'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn12; +connection default; +XA ROLLBACK 'trx_12'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn11; +connection default; +XA ROLLBACK 'trx_11'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn10; +connection default; +XA START 'one_phase_trx_0'; +INSERT INTO t SET a=0; +XA END 'one_phase_trx_0'; +XA COMMIT 'one_phase_trx_0' ONE PHASE; +XA START 'one_phase_trx_1'; +INSERT INTO t SET a=1; +XA END 'one_phase_trx_1'; +XA COMMIT 'one_phase_trx_1' ONE PHASE; +XA START 'one_phase_trx_2'; +INSERT INTO t SET a=2; +XA END 'one_phase_trx_2'; +XA COMMIT 'one_phase_trx_2' ONE PHASE; +XA START 'one_phase_trx_3'; +INSERT INTO t SET a=3; +XA END 'one_phase_trx_3'; +XA COMMIT 'one_phase_trx_3' ONE PHASE; +XA START 'one_phase_trx_4'; +INSERT INTO t SET a=4; +XA END 'one_phase_trx_4'; +XA COMMIT 'one_phase_trx_4' ONE PHASE; +SELECT SUM(a) FROM t; +SUM(a) +290 +DROP TABLE t; +DROP VIEW v_processlist; +include/show_binlog_events.inc +Log_name Pos Event_type Server_id End_log_pos Info +master-bin.000001 # Gtid # # GTID #-#-# +master-bin.000001 # Query # # use `test`; CREATE ALGORITHM=UNDEFINED DEFINER=`root`@`localhost` SQL SECURITY DEFINER VIEW `v_processlist` AS SELECT * FROM performance_schema.threads where type = 'FOREGROUND' +master-bin.000001 # Gtid # # BEGIN GTID #-#-# +master-bin.000001 # Query # # use `mtr`; INSERT INTO test_suppressions (pattern) VALUES ( NAME_CONST('pattern',_latin1'Found 10 prepared XA transactions' COLLATE 'latin1_swedish_ci')) +master-bin.000001 # Query # # COMMIT +master-bin.000001 # Gtid # # GTID #-#-# +master-bin.000001 # Query # # use `test`; CREATE TABLE t (a INT) ENGINE=innodb +master-bin.000001 # Gtid # # XA START X'7472785f30',X'',1 GTID #-#-# +master-bin.000001 # Query # # use `test`; INSERT INTO t SET a=0 +master-bin.000001 # Query # # XA END X'7472785f30',X'',1 +master-bin.000001 # XA_prepare # # XA PREPARE X'7472785f30',X'',1 +master-bin.000001 # Gtid # # XA START X'7472785f31',X'',1 GTID #-#-# +master-bin.000001 # Annotate_rows # # INSERT INTO t SET a=1 +master-bin.000001 # Table_map # # table_id: # (test.t) +master-bin.000001 # Write_rows_v1 # # table_id: # flags: STMT_END_F +master-bin.000001 # Query # # XA END X'7472785f31',X'',1 +master-bin.000001 # XA_prepare # # XA PREPARE X'7472785f31',X'',1 +master-bin.000001 # Gtid # # XA START X'7472785f32',X'',1 GTID #-#-# +master-bin.000001 # Query # # use `test`; INSERT INTO t SET a=2 +master-bin.000001 # Query # # XA END X'7472785f32',X'',1 +master-bin.000001 # XA_prepare # # XA PREPARE X'7472785f32',X'',1 +master-bin.000001 # Gtid # # XA START X'7472785f33',X'',1 GTID #-#-# +master-bin.000001 # Annotate_rows # # INSERT INTO t SET a=3 +master-bin.000001 # Table_map # # table_id: # (test.t) +master-bin.000001 # Write_rows_v1 # # table_id: # flags: STMT_END_F +master-bin.000001 # Query # # XA END X'7472785f33',X'',1 +master-bin.000001 # XA_prepare # # XA PREPARE X'7472785f33',X'',1 +master-bin.000001 # Gtid # # XA START X'7472785f34',X'',1 GTID #-#-# +master-bin.000001 # Query # # use `test`; INSERT INTO t SET a=4 +master-bin.000001 # Query # # XA END X'7472785f34',X'',1 +master-bin.000001 # XA_prepare # # XA PREPARE X'7472785f34',X'',1 +master-bin.000001 # Gtid # # XA START X'7472785f35',X'',1 GTID #-#-# +master-bin.000001 # Annotate_rows # # INSERT INTO t SET a=5 +master-bin.000001 # Table_map # # table_id: # (test.t) +master-bin.000001 # Write_rows_v1 # # table_id: # flags: STMT_END_F +master-bin.000001 # Query # # XA END X'7472785f35',X'',1 +master-bin.000001 # XA_prepare # # XA PREPARE X'7472785f35',X'',1 +master-bin.000001 # Gtid # # XA START X'7472785f36',X'',1 GTID #-#-# +master-bin.000001 # Query # # use `test`; INSERT INTO t SET a=6 +master-bin.000001 # Query # # XA END X'7472785f36',X'',1 +master-bin.000001 # XA_prepare # # XA PREPARE X'7472785f36',X'',1 +master-bin.000001 # Gtid # # XA START X'7472785f37',X'',1 GTID #-#-# +master-bin.000001 # Annotate_rows # # INSERT INTO t SET a=7 +master-bin.000001 # Table_map # # table_id: # (test.t) +master-bin.000001 # Write_rows_v1 # # table_id: # flags: STMT_END_F +master-bin.000001 # Query # # XA END X'7472785f37',X'',1 +master-bin.000001 # XA_prepare # # XA PREPARE X'7472785f37',X'',1 +master-bin.000001 # Gtid # # XA START X'7472785f38',X'',1 GTID #-#-# +master-bin.000001 # Query # # use `test`; INSERT INTO t SET a=8 +master-bin.000001 # Query # # XA END X'7472785f38',X'',1 +master-bin.000001 # XA_prepare # # XA PREPARE X'7472785f38',X'',1 +master-bin.000001 # Gtid # # XA START X'7472785f39',X'',1 GTID #-#-# +master-bin.000001 # Annotate_rows # # INSERT INTO t SET a=9 +master-bin.000001 # Table_map # # table_id: # (test.t) +master-bin.000001 # Write_rows_v1 # # table_id: # flags: STMT_END_F +master-bin.000001 # Query # # XA END X'7472785f39',X'',1 +master-bin.000001 # XA_prepare # # XA PREPARE X'7472785f39',X'',1 +master-bin.000001 # Gtid # # XA START X'7472785f3130',X'',1 GTID #-#-# +master-bin.000001 # Query # # use `test`; INSERT INTO t SET a=10 +master-bin.000001 # Query # # XA END X'7472785f3130',X'',1 +master-bin.000001 # XA_prepare # # XA PREPARE X'7472785f3130',X'',1 +master-bin.000001 # Gtid # # XA START X'7472785f3131',X'',1 GTID #-#-# +master-bin.000001 # Annotate_rows # # INSERT INTO t SET a=11 +master-bin.000001 # Table_map # # table_id: # (test.t) +master-bin.000001 # Write_rows_v1 # # table_id: # flags: STMT_END_F +master-bin.000001 # Query # # XA END X'7472785f3131',X'',1 +master-bin.000001 # XA_prepare # # XA PREPARE X'7472785f3131',X'',1 +master-bin.000001 # Gtid # # XA START X'7472785f3132',X'',1 GTID #-#-# +master-bin.000001 # Query # # use `test`; INSERT INTO t SET a=12 +master-bin.000001 # Query # # XA END X'7472785f3132',X'',1 +master-bin.000001 # XA_prepare # # XA PREPARE X'7472785f3132',X'',1 +master-bin.000001 # Gtid # # XA START X'7472785f3133',X'',1 GTID #-#-# +master-bin.000001 # Annotate_rows # # INSERT INTO t SET a=13 +master-bin.000001 # Table_map # # table_id: # (test.t) +master-bin.000001 # Write_rows_v1 # # table_id: # flags: STMT_END_F +master-bin.000001 # Query # # XA END X'7472785f3133',X'',1 +master-bin.000001 # XA_prepare # # XA PREPARE X'7472785f3133',X'',1 +master-bin.000001 # Gtid # # XA START X'7472785f3134',X'',1 GTID #-#-# +master-bin.000001 # Query # # use `test`; INSERT INTO t SET a=14 +master-bin.000001 # Query # # XA END X'7472785f3134',X'',1 +master-bin.000001 # XA_prepare # # XA PREPARE X'7472785f3134',X'',1 +master-bin.000001 # Gtid # # XA START X'7472785f3135',X'',1 GTID #-#-# +master-bin.000001 # Annotate_rows # # INSERT INTO t SET a=15 +master-bin.000001 # Table_map # # table_id: # (test.t) +master-bin.000001 # Write_rows_v1 # # table_id: # flags: STMT_END_F +master-bin.000001 # Query # # XA END X'7472785f3135',X'',1 +master-bin.000001 # XA_prepare # # XA PREPARE X'7472785f3135',X'',1 +master-bin.000001 # Gtid # # XA START X'7472785f3136',X'',1 GTID #-#-# +master-bin.000001 # Query # # use `test`; INSERT INTO t SET a=16 +master-bin.000001 # Query # # XA END X'7472785f3136',X'',1 +master-bin.000001 # XA_prepare # # XA PREPARE X'7472785f3136',X'',1 +master-bin.000001 # Gtid # # XA START X'7472785f3137',X'',1 GTID #-#-# +master-bin.000001 # Annotate_rows # # INSERT INTO t SET a=17 +master-bin.000001 # Table_map # # table_id: # (test.t) +master-bin.000001 # Write_rows_v1 # # table_id: # flags: STMT_END_F +master-bin.000001 # Query # # XA END X'7472785f3137',X'',1 +master-bin.000001 # XA_prepare # # XA PREPARE X'7472785f3137',X'',1 +master-bin.000001 # Gtid # # XA START X'7472785f3138',X'',1 GTID #-#-# +master-bin.000001 # Query # # use `test`; INSERT INTO t SET a=18 +master-bin.000001 # Query # # XA END X'7472785f3138',X'',1 +master-bin.000001 # XA_prepare # # XA PREPARE X'7472785f3138',X'',1 +master-bin.000001 # Gtid # # XA START X'7472785f3139',X'',1 GTID #-#-# +master-bin.000001 # Annotate_rows # # INSERT INTO t SET a=19 +master-bin.000001 # Table_map # # table_id: # (test.t) +master-bin.000001 # Write_rows_v1 # # table_id: # flags: STMT_END_F +master-bin.000001 # Query # # XA END X'7472785f3139',X'',1 +master-bin.000001 # XA_prepare # # XA PREPARE X'7472785f3139',X'',1 +master-bin.000001 # Gtid # # GTID #-#-# +master-bin.000001 # Query # # XA ROLLBACK X'7472785f30',X'',1 +master-bin.000001 # Gtid # # GTID #-#-# +master-bin.000001 # Query # # XA ROLLBACK X'7472785f31',X'',1 +master-bin.000001 # Gtid # # GTID #-#-# +master-bin.000001 # Query # # XA ROLLBACK X'7472785f32',X'',1 +master-bin.000001 # Gtid # # GTID #-#-# +master-bin.000001 # Query # # XA ROLLBACK X'7472785f33',X'',1 +master-bin.000001 # Gtid # # GTID #-#-# +master-bin.000001 # Query # # XA ROLLBACK X'7472785f34',X'',1 +master-bin.000001 # Gtid # # GTID #-#-# +master-bin.000001 # Query # # XA COMMIT X'7472785f35',X'',1 +master-bin.000001 # Gtid # # GTID #-#-# +master-bin.000001 # Query # # XA COMMIT X'7472785f36',X'',1 +master-bin.000001 # Gtid # # GTID #-#-# +master-bin.000001 # Query # # XA COMMIT X'7472785f37',X'',1 +master-bin.000001 # Gtid # # GTID #-#-# +master-bin.000001 # Query # # XA COMMIT X'7472785f38',X'',1 +master-bin.000001 # Gtid # # GTID #-#-# +master-bin.000001 # Query # # XA COMMIT X'7472785f39',X'',1 +master-bin.000001 # Stop # # +All transactions must be completed, to empty-list the following: +XA RECOVER; +formatID gtrid_length bqual_length data +XA RECOVER; +formatID gtrid_length bqual_length data diff --git a/mysql-test/suite/binlog/r/binlog_xa_prepared_disconnect.result b/mysql-test/suite/binlog/r/binlog_xa_prepared_disconnect.result new file mode 100644 index 00000000000..9fda8ab3143 --- /dev/null +++ b/mysql-test/suite/binlog/r/binlog_xa_prepared_disconnect.result @@ -0,0 +1,1176 @@ +connection default; +RESET MASTER; +CREATE VIEW v_processlist as SELECT * FROM performance_schema.threads where type = 'FOREGROUND'; +call mtr.add_suppression("Found 10 prepared XA transactions"); +CREATE TABLE t (a INT) ENGINE=innodb; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@sql_log_bin = OFF; +CREATE TEMPORARY TABLE tmp1 (a int) ENGINE=innodb; +XA START 'trx1tmp'; +INSERT INTO tmp1 SET a=1; +XA END 'trx1tmp'; +XA PREPARE 'trx1tmp'; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@sql_log_bin = OFF; +CREATE TEMPORARY TABLE tmp1 (a int) ENGINE=innodb; +XA START 'trx2tmp'; +INSERT INTO tmp1 SET a=1; +XA END 'trx2tmp'; +XA PREPARE 'trx2tmp'; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@sql_log_bin = OFF; +CREATE TEMPORARY TABLE tmp1 (a int) ENGINE=innodb; +XA START 'trx3tmp'; +INSERT INTO tmp1 SET a=1; +XA END 'trx3tmp'; +XA PREPARE 'trx3tmp'; +connection default; +XA COMMIT 'trx1tmp'; +ERROR XAE04: XAER_NOTA: Unknown XID +XA ROLLBACK 'trx1tmp'; +ERROR XAE04: XAER_NOTA: Unknown XID +XA START 'trx1tmp'; +ERROR XAE08: XAER_DUPID: The XID already exists +connection default; +*** 3 prepared transactions must be in the list *** +XA RECOVER; +formatID gtrid_length bqual_length data +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +connection conn1tmp; +disconnect conn1tmp; +connection default; +XA COMMIT 'trx1tmp'; +KILL connection CONN_ID; +XA COMMIT 'trx3tmp'; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'trx1ro'; +SELECT * from t ORDER BY a; +a +XA END 'trx1ro'; +XA PREPARE 'trx1ro'; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'trx2ro'; +SELECT * from t ORDER BY a; +a +XA END 'trx2ro'; +XA PREPARE 'trx2ro'; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'trx3ro'; +SELECT * from t ORDER BY a; +a +XA END 'trx3ro'; +XA PREPARE 'trx3ro'; +connection default; +*** 4 prepared transactions must be in the list *** +XA RECOVER; +formatID gtrid_length bqual_length data +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +connection conn1ro; +disconnect conn1ro; +connection default; +XA ROLLBACK 'trx1ro'; +KILL connection CONN_ID; +XA ROLLBACK 'trx3ro'; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'trx1empty'; +XA END 'trx1empty'; +XA PREPARE 'trx1empty'; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'trx2empty'; +XA END 'trx2empty'; +XA PREPARE 'trx2empty'; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'trx3empty'; +XA END 'trx3empty'; +XA PREPARE 'trx3empty'; +connection default; +*** 5 prepared transactions must be in the list *** +XA RECOVER; +formatID gtrid_length bqual_length data +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +connection conn1empty; +disconnect conn1empty; +connection default; +XA COMMIT 'trx1empty'; +KILL connection CONN_ID; +XA COMMIT 'trx3empty'; +connect conn1$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'trx1unprepared'; +INSERT INTO t set a=0; +XA END 'trx1unprepared'; +INSERT INTO t set a=0; +ERROR XAE07: XAER_RMFAIL: The command cannot be executed when global transaction is in the IDLE state +XA START 'trx1unprepared'; +ERROR XAE07: XAER_RMFAIL: The command cannot be executed when global transaction is in the IDLE state +XA START 'trx1unprepared'; +ERROR XAE07: XAER_RMFAIL: The command cannot be executed when global transaction is in the IDLE state +disconnect conn1unprepared; +connection default; +XA COMMIT 'trx1unprepared'; +ERROR XAE04: XAER_NOTA: Unknown XID +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_0'; +INSERT INTO t SET a=0; +XA END 'trx_0'; +XA PREPARE 'trx_0'; +disconnect conn0; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_1'; +INSERT INTO t SET a=1; +XA END 'trx_1'; +XA PREPARE 'trx_1'; +disconnect conn1; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_2'; +INSERT INTO t SET a=2; +XA END 'trx_2'; +XA PREPARE 'trx_2'; +disconnect conn2; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_3'; +INSERT INTO t SET a=3; +XA END 'trx_3'; +XA PREPARE 'trx_3'; +disconnect conn3; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_4'; +INSERT INTO t SET a=4; +XA END 'trx_4'; +XA PREPARE 'trx_4'; +disconnect conn4; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_5'; +INSERT INTO t SET a=5; +XA END 'trx_5'; +XA PREPARE 'trx_5'; +disconnect conn5; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_6'; +INSERT INTO t SET a=6; +XA END 'trx_6'; +XA PREPARE 'trx_6'; +disconnect conn6; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_7'; +INSERT INTO t SET a=7; +XA END 'trx_7'; +XA PREPARE 'trx_7'; +disconnect conn7; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_8'; +INSERT INTO t SET a=8; +XA END 'trx_8'; +XA PREPARE 'trx_8'; +disconnect conn8; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_9'; +INSERT INTO t SET a=9; +XA END 'trx_9'; +XA PREPARE 'trx_9'; +disconnect conn9; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_10'; +INSERT INTO t SET a=10; +XA END 'trx_10'; +XA PREPARE 'trx_10'; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_11'; +INSERT INTO t SET a=11; +XA END 'trx_11'; +XA PREPARE 'trx_11'; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_12'; +INSERT INTO t SET a=12; +XA END 'trx_12'; +XA PREPARE 'trx_12'; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_13'; +INSERT INTO t SET a=13; +XA END 'trx_13'; +XA PREPARE 'trx_13'; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_14'; +INSERT INTO t SET a=14; +XA END 'trx_14'; +XA PREPARE 'trx_14'; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_15'; +INSERT INTO t SET a=15; +XA END 'trx_15'; +XA PREPARE 'trx_15'; +connection default; +KILL CONNECTION CONN_ID; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_16'; +INSERT INTO t SET a=16; +XA END 'trx_16'; +XA PREPARE 'trx_16'; +connection default; +KILL CONNECTION CONN_ID; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_17'; +INSERT INTO t SET a=17; +XA END 'trx_17'; +XA PREPARE 'trx_17'; +connection default; +KILL CONNECTION CONN_ID; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_18'; +INSERT INTO t SET a=18; +XA END 'trx_18'; +XA PREPARE 'trx_18'; +connection default; +KILL CONNECTION CONN_ID; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_19'; +INSERT INTO t SET a=19; +XA END 'trx_19'; +XA PREPARE 'trx_19'; +connection default; +KILL CONNECTION CONN_ID; +connection default; +XA ROLLBACK 'trx_0'; +XA ROLLBACK 'trx_1'; +XA ROLLBACK 'trx_2'; +XA ROLLBACK 'trx_3'; +XA ROLLBACK 'trx_4'; +XA COMMIT 'trx_5'; +XA COMMIT 'trx_6'; +XA COMMIT 'trx_7'; +XA COMMIT 'trx_8'; +XA COMMIT 'trx_9'; +# restart +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_0'; +INSERT INTO t SET a=0; +XA END 'new_trx_0'; +XA PREPARE 'new_trx_0'; +disconnect conn_restart_0; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_1'; +INSERT INTO t SET a=1; +XA END 'new_trx_1'; +XA PREPARE 'new_trx_1'; +disconnect conn_restart_1; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_2'; +INSERT INTO t SET a=2; +XA END 'new_trx_2'; +XA PREPARE 'new_trx_2'; +disconnect conn_restart_2; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_3'; +INSERT INTO t SET a=3; +XA END 'new_trx_3'; +XA PREPARE 'new_trx_3'; +disconnect conn_restart_3; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_4'; +INSERT INTO t SET a=4; +XA END 'new_trx_4'; +XA PREPARE 'new_trx_4'; +disconnect conn_restart_4; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_5'; +INSERT INTO t SET a=5; +XA END 'new_trx_5'; +XA PREPARE 'new_trx_5'; +disconnect conn_restart_5; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_6'; +INSERT INTO t SET a=6; +XA END 'new_trx_6'; +XA PREPARE 'new_trx_6'; +disconnect conn_restart_6; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_7'; +INSERT INTO t SET a=7; +XA END 'new_trx_7'; +XA PREPARE 'new_trx_7'; +disconnect conn_restart_7; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_8'; +INSERT INTO t SET a=8; +XA END 'new_trx_8'; +XA PREPARE 'new_trx_8'; +disconnect conn_restart_8; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_9'; +INSERT INTO t SET a=9; +XA END 'new_trx_9'; +XA PREPARE 'new_trx_9'; +disconnect conn_restart_9; +connection default; +connection default; +XA COMMIT 'new_trx_0'; +XA COMMIT 'new_trx_1'; +XA COMMIT 'new_trx_2'; +XA COMMIT 'new_trx_3'; +XA COMMIT 'new_trx_4'; +XA COMMIT 'new_trx_5'; +XA COMMIT 'new_trx_6'; +XA COMMIT 'new_trx_7'; +XA COMMIT 'new_trx_8'; +XA COMMIT 'new_trx_9'; +XA START 'trx_10'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA COMMIT 'trx_10'; +XA START 'trx_11'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA COMMIT 'trx_11'; +XA START 'trx_12'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA COMMIT 'trx_12'; +XA START 'trx_13'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA COMMIT 'trx_13'; +XA START 'trx_14'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA COMMIT 'trx_14'; +XA START 'trx_15'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA ROLLBACK 'trx_15'; +XA START 'trx_16'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA ROLLBACK 'trx_16'; +XA START 'trx_17'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA ROLLBACK 'trx_17'; +XA START 'trx_18'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA ROLLBACK 'trx_18'; +XA START 'trx_19'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA ROLLBACK 'trx_19'; +SELECT * FROM t; +a +5 +6 +7 +8 +9 +10 +11 +12 +13 +14 +0 +1 +2 +3 +4 +5 +6 +7 +8 +9 +disconnect conn2tmp; +disconnect conn3tmp; +disconnect conn2ro; +disconnect conn3ro; +disconnect conn2empty; +disconnect conn3empty; +connection default; +XA ROLLBACK 'trx_20'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn19; +connection default; +XA ROLLBACK 'trx_19'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn18; +connection default; +XA ROLLBACK 'trx_18'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn17; +connection default; +XA ROLLBACK 'trx_17'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn16; +connection default; +XA ROLLBACK 'trx_16'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn15; +connection default; +XA ROLLBACK 'trx_15'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn14; +connection default; +XA ROLLBACK 'trx_14'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn13; +connection default; +XA ROLLBACK 'trx_13'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn12; +connection default; +XA ROLLBACK 'trx_12'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn11; +connection default; +XA ROLLBACK 'trx_11'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn10; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@sql_log_bin = OFF; +CREATE TEMPORARY TABLE tmp1 (a int) ENGINE=innodb; +XA START 'trx1tmp'; +INSERT INTO tmp1 SET a=1; +XA END 'trx1tmp'; +XA PREPARE 'trx1tmp'; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@sql_log_bin = OFF; +CREATE TEMPORARY TABLE tmp1 (a int) ENGINE=innodb; +XA START 'trx2tmp'; +INSERT INTO tmp1 SET a=1; +XA END 'trx2tmp'; +XA PREPARE 'trx2tmp'; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@sql_log_bin = OFF; +CREATE TEMPORARY TABLE tmp1 (a int) ENGINE=innodb; +XA START 'trx3tmp'; +INSERT INTO tmp1 SET a=1; +XA END 'trx3tmp'; +XA PREPARE 'trx3tmp'; +connection default; +XA COMMIT 'trx1tmp'; +ERROR XAE04: XAER_NOTA: Unknown XID +XA ROLLBACK 'trx1tmp'; +ERROR XAE04: XAER_NOTA: Unknown XID +XA START 'trx1tmp'; +ERROR XAE08: XAER_DUPID: The XID already exists +connection default; +*** 3 prepared transactions must be in the list *** +XA RECOVER; +formatID gtrid_length bqual_length data +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +connection conn1tmp; +disconnect conn1tmp; +connection default; +XA COMMIT 'trx1tmp'; +KILL connection CONN_ID; +XA COMMIT 'trx3tmp'; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'trx1ro'; +SELECT * from t ORDER BY a; +a +0 +1 +2 +3 +4 +5 +5 +6 +6 +7 +7 +8 +8 +9 +9 +10 +11 +12 +13 +14 +XA END 'trx1ro'; +XA PREPARE 'trx1ro'; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'trx2ro'; +SELECT * from t ORDER BY a; +a +0 +1 +2 +3 +4 +5 +5 +6 +6 +7 +7 +8 +8 +9 +9 +10 +11 +12 +13 +14 +XA END 'trx2ro'; +XA PREPARE 'trx2ro'; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'trx3ro'; +SELECT * from t ORDER BY a; +a +0 +1 +2 +3 +4 +5 +5 +6 +6 +7 +7 +8 +8 +9 +9 +10 +11 +12 +13 +14 +XA END 'trx3ro'; +XA PREPARE 'trx3ro'; +connection default; +*** 4 prepared transactions must be in the list *** +XA RECOVER; +formatID gtrid_length bqual_length data +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +connection conn1ro; +disconnect conn1ro; +connection default; +XA ROLLBACK 'trx1ro'; +KILL connection CONN_ID; +XA ROLLBACK 'trx3ro'; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'trx1empty'; +XA END 'trx1empty'; +XA PREPARE 'trx1empty'; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'trx2empty'; +XA END 'trx2empty'; +XA PREPARE 'trx2empty'; +connect conn$index$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'trx3empty'; +XA END 'trx3empty'; +XA PREPARE 'trx3empty'; +connection default; +*** 5 prepared transactions must be in the list *** +XA RECOVER; +formatID gtrid_length bqual_length data +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +1 LEN1 LEN2 TRX_N +connection conn1empty; +disconnect conn1empty; +connection default; +XA COMMIT 'trx1empty'; +KILL connection CONN_ID; +XA COMMIT 'trx3empty'; +connect conn1$type, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'trx1unprepared'; +INSERT INTO t set a=0; +XA END 'trx1unprepared'; +INSERT INTO t set a=0; +ERROR XAE07: XAER_RMFAIL: The command cannot be executed when global transaction is in the IDLE state +XA START 'trx1unprepared'; +ERROR XAE07: XAER_RMFAIL: The command cannot be executed when global transaction is in the IDLE state +XA START 'trx1unprepared'; +ERROR XAE07: XAER_RMFAIL: The command cannot be executed when global transaction is in the IDLE state +disconnect conn1unprepared; +connection default; +XA COMMIT 'trx1unprepared'; +ERROR XAE04: XAER_NOTA: Unknown XID +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_0'; +INSERT INTO t SET a=0; +XA END 'trx_0'; +XA PREPARE 'trx_0'; +disconnect conn0; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_1'; +INSERT INTO t SET a=1; +XA END 'trx_1'; +XA PREPARE 'trx_1'; +disconnect conn1; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_2'; +INSERT INTO t SET a=2; +XA END 'trx_2'; +XA PREPARE 'trx_2'; +disconnect conn2; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_3'; +INSERT INTO t SET a=3; +XA END 'trx_3'; +XA PREPARE 'trx_3'; +disconnect conn3; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_4'; +INSERT INTO t SET a=4; +XA END 'trx_4'; +XA PREPARE 'trx_4'; +disconnect conn4; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_5'; +INSERT INTO t SET a=5; +XA END 'trx_5'; +XA PREPARE 'trx_5'; +disconnect conn5; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_6'; +INSERT INTO t SET a=6; +XA END 'trx_6'; +XA PREPARE 'trx_6'; +disconnect conn6; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_7'; +INSERT INTO t SET a=7; +XA END 'trx_7'; +XA PREPARE 'trx_7'; +disconnect conn7; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_8'; +INSERT INTO t SET a=8; +XA END 'trx_8'; +XA PREPARE 'trx_8'; +disconnect conn8; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_9'; +INSERT INTO t SET a=9; +XA END 'trx_9'; +XA PREPARE 'trx_9'; +disconnect conn9; +connection default; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_10'; +INSERT INTO t SET a=10; +XA END 'trx_10'; +XA PREPARE 'trx_10'; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_11'; +INSERT INTO t SET a=11; +XA END 'trx_11'; +XA PREPARE 'trx_11'; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_12'; +INSERT INTO t SET a=12; +XA END 'trx_12'; +XA PREPARE 'trx_12'; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_13'; +INSERT INTO t SET a=13; +XA END 'trx_13'; +XA PREPARE 'trx_13'; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_14'; +INSERT INTO t SET a=14; +XA END 'trx_14'; +XA PREPARE 'trx_14'; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_15'; +INSERT INTO t SET a=15; +XA END 'trx_15'; +XA PREPARE 'trx_15'; +connection default; +KILL CONNECTION CONN_ID; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_16'; +INSERT INTO t SET a=16; +XA END 'trx_16'; +XA PREPARE 'trx_16'; +connection default; +KILL CONNECTION CONN_ID; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_17'; +INSERT INTO t SET a=17; +XA END 'trx_17'; +XA PREPARE 'trx_17'; +connection default; +KILL CONNECTION CONN_ID; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +XA START 'trx_18'; +INSERT INTO t SET a=18; +XA END 'trx_18'; +XA PREPARE 'trx_18'; +connection default; +KILL CONNECTION CONN_ID; +connect conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@binlog_format = STATEMENT; +SET @@binlog_format = ROW; +XA START 'trx_19'; +INSERT INTO t SET a=19; +XA END 'trx_19'; +XA PREPARE 'trx_19'; +connection default; +KILL CONNECTION CONN_ID; +connection default; +XA ROLLBACK 'trx_0'; +XA ROLLBACK 'trx_1'; +XA ROLLBACK 'trx_2'; +XA ROLLBACK 'trx_3'; +XA ROLLBACK 'trx_4'; +XA COMMIT 'trx_5'; +XA COMMIT 'trx_6'; +XA COMMIT 'trx_7'; +XA COMMIT 'trx_8'; +XA COMMIT 'trx_9'; +# Kill and restart +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_0'; +INSERT INTO t SET a=0; +XA END 'new_trx_0'; +XA PREPARE 'new_trx_0'; +disconnect conn_restart_0; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_1'; +INSERT INTO t SET a=1; +XA END 'new_trx_1'; +XA PREPARE 'new_trx_1'; +disconnect conn_restart_1; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_2'; +INSERT INTO t SET a=2; +XA END 'new_trx_2'; +XA PREPARE 'new_trx_2'; +disconnect conn_restart_2; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_3'; +INSERT INTO t SET a=3; +XA END 'new_trx_3'; +XA PREPARE 'new_trx_3'; +disconnect conn_restart_3; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_4'; +INSERT INTO t SET a=4; +XA END 'new_trx_4'; +XA PREPARE 'new_trx_4'; +disconnect conn_restart_4; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_5'; +INSERT INTO t SET a=5; +XA END 'new_trx_5'; +XA PREPARE 'new_trx_5'; +disconnect conn_restart_5; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_6'; +INSERT INTO t SET a=6; +XA END 'new_trx_6'; +XA PREPARE 'new_trx_6'; +disconnect conn_restart_6; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_7'; +INSERT INTO t SET a=7; +XA END 'new_trx_7'; +XA PREPARE 'new_trx_7'; +disconnect conn_restart_7; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_8'; +INSERT INTO t SET a=8; +XA END 'new_trx_8'; +XA PREPARE 'new_trx_8'; +disconnect conn_restart_8; +connection default; +connect conn_restart_$k, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'new_trx_9'; +INSERT INTO t SET a=9; +XA END 'new_trx_9'; +XA PREPARE 'new_trx_9'; +disconnect conn_restart_9; +connection default; +connection default; +XA COMMIT 'new_trx_0'; +XA COMMIT 'new_trx_1'; +XA COMMIT 'new_trx_2'; +XA COMMIT 'new_trx_3'; +XA COMMIT 'new_trx_4'; +XA COMMIT 'new_trx_5'; +XA COMMIT 'new_trx_6'; +XA COMMIT 'new_trx_7'; +XA COMMIT 'new_trx_8'; +XA COMMIT 'new_trx_9'; +XA START 'trx_10'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA COMMIT 'trx_10'; +XA START 'trx_11'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA COMMIT 'trx_11'; +XA START 'trx_12'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA COMMIT 'trx_12'; +XA START 'trx_13'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA COMMIT 'trx_13'; +XA START 'trx_14'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA COMMIT 'trx_14'; +XA START 'trx_15'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA ROLLBACK 'trx_15'; +XA START 'trx_16'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA ROLLBACK 'trx_16'; +XA START 'trx_17'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA ROLLBACK 'trx_17'; +XA START 'trx_18'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA ROLLBACK 'trx_18'; +XA START 'trx_19'; +ERROR XAE08: XAER_DUPID: The XID already exists +XA ROLLBACK 'trx_19'; +SELECT * FROM t; +a +5 +6 +7 +8 +9 +10 +11 +12 +13 +14 +0 +1 +2 +3 +4 +5 +6 +7 +8 +9 +5 +6 +7 +8 +9 +10 +11 +12 +13 +14 +0 +1 +2 +3 +4 +5 +6 +7 +8 +9 +disconnect conn2tmp; +disconnect conn3tmp; +disconnect conn2ro; +disconnect conn3ro; +disconnect conn2empty; +disconnect conn3empty; +connection default; +XA ROLLBACK 'trx_20'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn19; +connection default; +XA ROLLBACK 'trx_19'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn18; +connection default; +XA ROLLBACK 'trx_18'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn17; +connection default; +XA ROLLBACK 'trx_17'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn16; +connection default; +XA ROLLBACK 'trx_16'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn15; +connection default; +XA ROLLBACK 'trx_15'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn14; +connection default; +XA ROLLBACK 'trx_14'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn13; +connection default; +XA ROLLBACK 'trx_13'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn12; +connection default; +XA ROLLBACK 'trx_12'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn11; +connection default; +XA ROLLBACK 'trx_11'; +ERROR XAE04: XAER_NOTA: Unknown XID +disconnect conn10; +connection default; +XA START 'one_phase_trx_0'; +INSERT INTO t SET a=0; +XA END 'one_phase_trx_0'; +XA COMMIT 'one_phase_trx_0' ONE PHASE; +XA START 'one_phase_trx_1'; +INSERT INTO t SET a=1; +XA END 'one_phase_trx_1'; +XA COMMIT 'one_phase_trx_1' ONE PHASE; +XA START 'one_phase_trx_2'; +INSERT INTO t SET a=2; +XA END 'one_phase_trx_2'; +XA COMMIT 'one_phase_trx_2' ONE PHASE; +XA START 'one_phase_trx_3'; +INSERT INTO t SET a=3; +XA END 'one_phase_trx_3'; +XA COMMIT 'one_phase_trx_3' ONE PHASE; +XA START 'one_phase_trx_4'; +INSERT INTO t SET a=4; +XA END 'one_phase_trx_4'; +XA COMMIT 'one_phase_trx_4' ONE PHASE; +SELECT SUM(a) FROM t; +SUM(a) +290 +DROP TABLE t; +DROP VIEW v_processlist; +include/show_binlog_events.inc +Log_name Pos Event_type Server_id End_log_pos Info +master-bin.000001 # Gtid # # GTID #-#-# +master-bin.000001 # Query # # use `test`; CREATE ALGORITHM=UNDEFINED DEFINER=`root`@`localhost` SQL SECURITY DEFINER VIEW `v_processlist` AS SELECT * FROM performance_schema.threads where type = 'FOREGROUND' +master-bin.000001 # Gtid # # BEGIN GTID #-#-# +master-bin.000001 # Query # # use `mtr`; INSERT INTO test_suppressions (pattern) VALUES ( NAME_CONST('pattern',_latin1'Found 10 prepared XA transactions' COLLATE 'latin1_swedish_ci')) +master-bin.000001 # Query # # COMMIT +master-bin.000001 # Gtid # # GTID #-#-# +master-bin.000001 # Query # # use `test`; CREATE TABLE t (a INT) ENGINE=innodb +master-bin.000001 # Gtid # # XA START X'7472785f30',X'',1 GTID #-#-# +master-bin.000001 # Query # # use `test`; INSERT INTO t SET a=0 +master-bin.000001 # Query # # XA END X'7472785f30',X'',1 +master-bin.000001 # XA_prepare # # XA PREPARE X'7472785f30',X'',1 +master-bin.000001 # Gtid # # XA START X'7472785f31',X'',1 GTID #-#-# +master-bin.000001 # Annotate_rows # # INSERT INTO t SET a=1 +master-bin.000001 # Table_map # # table_id: # (test.t) +master-bin.000001 # Write_rows_v1 # # table_id: # flags: STMT_END_F +master-bin.000001 # Query # # XA END X'7472785f31',X'',1 +master-bin.000001 # XA_prepare # # XA PREPARE X'7472785f31',X'',1 +master-bin.000001 # Gtid # # XA START X'7472785f32',X'',1 GTID #-#-# +master-bin.000001 # Query # # use `test`; INSERT INTO t SET a=2 +master-bin.000001 # Query # # XA END X'7472785f32',X'',1 +master-bin.000001 # XA_prepare # # XA PREPARE X'7472785f32',X'',1 +master-bin.000001 # Gtid # # XA START X'7472785f33',X'',1 GTID #-#-# +master-bin.000001 # Annotate_rows # # INSERT INTO t SET a=3 +master-bin.000001 # Table_map # # table_id: # (test.t) +master-bin.000001 # Write_rows_v1 # # table_id: # flags: STMT_END_F +master-bin.000001 # Query # # XA END X'7472785f33',X'',1 +master-bin.000001 # XA_prepare # # XA PREPARE X'7472785f33',X'',1 +master-bin.000001 # Gtid # # XA START X'7472785f34',X'',1 GTID #-#-# +master-bin.000001 # Query # # use `test`; INSERT INTO t SET a=4 +master-bin.000001 # Query # # XA END X'7472785f34',X'',1 +master-bin.000001 # XA_prepare # # XA PREPARE X'7472785f34',X'',1 +master-bin.000001 # Gtid # # XA START X'7472785f35',X'',1 GTID #-#-# +master-bin.000001 # Annotate_rows # # INSERT INTO t SET a=5 +master-bin.000001 # Table_map # # table_id: # (test.t) +master-bin.000001 # Write_rows_v1 # # table_id: # flags: STMT_END_F +master-bin.000001 # Query # # XA END X'7472785f35',X'',1 +master-bin.000001 # XA_prepare # # XA PREPARE X'7472785f35',X'',1 +master-bin.000001 # Gtid # # XA START X'7472785f36',X'',1 GTID #-#-# +master-bin.000001 # Query # # use `test`; INSERT INTO t SET a=6 +master-bin.000001 # Query # # XA END X'7472785f36',X'',1 +master-bin.000001 # XA_prepare # # XA PREPARE X'7472785f36',X'',1 +master-bin.000001 # Gtid # # XA START X'7472785f37',X'',1 GTID #-#-# +master-bin.000001 # Annotate_rows # # INSERT INTO t SET a=7 +master-bin.000001 # Table_map # # table_id: # (test.t) +master-bin.000001 # Write_rows_v1 # # table_id: # flags: STMT_END_F +master-bin.000001 # Query # # XA END X'7472785f37',X'',1 +master-bin.000001 # XA_prepare # # XA PREPARE X'7472785f37',X'',1 +master-bin.000001 # Gtid # # XA START X'7472785f38',X'',1 GTID #-#-# +master-bin.000001 # Query # # use `test`; INSERT INTO t SET a=8 +master-bin.000001 # Query # # XA END X'7472785f38',X'',1 +master-bin.000001 # XA_prepare # # XA PREPARE X'7472785f38',X'',1 +master-bin.000001 # Gtid # # XA START X'7472785f39',X'',1 GTID #-#-# +master-bin.000001 # Annotate_rows # # INSERT INTO t SET a=9 +master-bin.000001 # Table_map # # table_id: # (test.t) +master-bin.000001 # Write_rows_v1 # # table_id: # flags: STMT_END_F +master-bin.000001 # Query # # XA END X'7472785f39',X'',1 +master-bin.000001 # XA_prepare # # XA PREPARE X'7472785f39',X'',1 +master-bin.000001 # Gtid # # XA START X'7472785f3130',X'',1 GTID #-#-# +master-bin.000001 # Query # # use `test`; INSERT INTO t SET a=10 +master-bin.000001 # Query # # XA END X'7472785f3130',X'',1 +master-bin.000001 # XA_prepare # # XA PREPARE X'7472785f3130',X'',1 +master-bin.000001 # Gtid # # XA START X'7472785f3131',X'',1 GTID #-#-# +master-bin.000001 # Annotate_rows # # INSERT INTO t SET a=11 +master-bin.000001 # Table_map # # table_id: # (test.t) +master-bin.000001 # Write_rows_v1 # # table_id: # flags: STMT_END_F +master-bin.000001 # Query # # XA END X'7472785f3131',X'',1 +master-bin.000001 # XA_prepare # # XA PREPARE X'7472785f3131',X'',1 +master-bin.000001 # Gtid # # XA START X'7472785f3132',X'',1 GTID #-#-# +master-bin.000001 # Query # # use `test`; INSERT INTO t SET a=12 +master-bin.000001 # Query # # XA END X'7472785f3132',X'',1 +master-bin.000001 # XA_prepare # # XA PREPARE X'7472785f3132',X'',1 +master-bin.000001 # Gtid # # XA START X'7472785f3133',X'',1 GTID #-#-# +master-bin.000001 # Annotate_rows # # INSERT INTO t SET a=13 +master-bin.000001 # Table_map # # table_id: # (test.t) +master-bin.000001 # Write_rows_v1 # # table_id: # flags: STMT_END_F +master-bin.000001 # Query # # XA END X'7472785f3133',X'',1 +master-bin.000001 # XA_prepare # # XA PREPARE X'7472785f3133',X'',1 +master-bin.000001 # Gtid # # XA START X'7472785f3134',X'',1 GTID #-#-# +master-bin.000001 # Query # # use `test`; INSERT INTO t SET a=14 +master-bin.000001 # Query # # XA END X'7472785f3134',X'',1 +master-bin.000001 # XA_prepare # # XA PREPARE X'7472785f3134',X'',1 +master-bin.000001 # Gtid # # XA START X'7472785f3135',X'',1 GTID #-#-# +master-bin.000001 # Annotate_rows # # INSERT INTO t SET a=15 +master-bin.000001 # Table_map # # table_id: # (test.t) +master-bin.000001 # Write_rows_v1 # # table_id: # flags: STMT_END_F +master-bin.000001 # Query # # XA END X'7472785f3135',X'',1 +master-bin.000001 # XA_prepare # # XA PREPARE X'7472785f3135',X'',1 +master-bin.000001 # Gtid # # XA START X'7472785f3136',X'',1 GTID #-#-# +master-bin.000001 # Query # # use `test`; INSERT INTO t SET a=16 +master-bin.000001 # Query # # XA END X'7472785f3136',X'',1 +master-bin.000001 # XA_prepare # # XA PREPARE X'7472785f3136',X'',1 +master-bin.000001 # Gtid # # XA START X'7472785f3137',X'',1 GTID #-#-# +master-bin.000001 # Annotate_rows # # INSERT INTO t SET a=17 +master-bin.000001 # Table_map # # table_id: # (test.t) +master-bin.000001 # Write_rows_v1 # # table_id: # flags: STMT_END_F +master-bin.000001 # Query # # XA END X'7472785f3137',X'',1 +master-bin.000001 # XA_prepare # # XA PREPARE X'7472785f3137',X'',1 +master-bin.000001 # Gtid # # XA START X'7472785f3138',X'',1 GTID #-#-# +master-bin.000001 # Query # # use `test`; INSERT INTO t SET a=18 +master-bin.000001 # Query # # XA END X'7472785f3138',X'',1 +master-bin.000001 # XA_prepare # # XA PREPARE X'7472785f3138',X'',1 +master-bin.000001 # Gtid # # XA START X'7472785f3139',X'',1 GTID #-#-# +master-bin.000001 # Annotate_rows # # INSERT INTO t SET a=19 +master-bin.000001 # Table_map # # table_id: # (test.t) +master-bin.000001 # Write_rows_v1 # # table_id: # flags: STMT_END_F +master-bin.000001 # Query # # XA END X'7472785f3139',X'',1 +master-bin.000001 # XA_prepare # # XA PREPARE X'7472785f3139',X'',1 +master-bin.000001 # Gtid # # GTID #-#-# +master-bin.000001 # Query # # XA ROLLBACK X'7472785f30',X'',1 +master-bin.000001 # Gtid # # GTID #-#-# +master-bin.000001 # Query # # XA ROLLBACK X'7472785f31',X'',1 +master-bin.000001 # Gtid # # GTID #-#-# +master-bin.000001 # Query # # XA ROLLBACK X'7472785f32',X'',1 +master-bin.000001 # Gtid # # GTID #-#-# +master-bin.000001 # Query # # XA ROLLBACK X'7472785f33',X'',1 +master-bin.000001 # Gtid # # GTID #-#-# +master-bin.000001 # Query # # XA ROLLBACK X'7472785f34',X'',1 +master-bin.000001 # Gtid # # GTID #-#-# +master-bin.000001 # Query # # XA COMMIT X'7472785f35',X'',1 +master-bin.000001 # Gtid # # GTID #-#-# +master-bin.000001 # Query # # XA COMMIT X'7472785f36',X'',1 +master-bin.000001 # Gtid # # GTID #-#-# +master-bin.000001 # Query # # XA COMMIT X'7472785f37',X'',1 +master-bin.000001 # Gtid # # GTID #-#-# +master-bin.000001 # Query # # XA COMMIT X'7472785f38',X'',1 +master-bin.000001 # Gtid # # GTID #-#-# +master-bin.000001 # Query # # XA COMMIT X'7472785f39',X'',1 +master-bin.000001 # Stop # # +All transactions must be completed, to empty-list the following: +XA RECOVER; +formatID gtrid_length bqual_length data +XA RECOVER; +formatID gtrid_length bqual_length data diff --git a/mysql-test/suite/binlog/t/binlog_xa_checkpoint.test b/mysql-test/suite/binlog/t/binlog_xa_checkpoint.test new file mode 100644 index 00000000000..b208d02cf2a --- /dev/null +++ b/mysql-test/suite/binlog/t/binlog_xa_checkpoint.test @@ -0,0 +1,57 @@ +--source include/have_innodb.inc +--source include/have_debug.inc +--source include/have_debug_sync.inc +--source include/have_binlog_format_row.inc + +RESET MASTER; + +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; + +# Test that +# 1. XA PREPARE is binlogged before the XA has been prepared in Engine +# 2. While XA PREPARE already binlogged in an old binlog file which has been rotated, +# Binlog checkpoint is not generated for the latest log until +# XA PREPARE returns, e.g OK to the client. + + +# con1 will hang before doing commit checkpoint, blocking RESET MASTER. +connect(con1,localhost,root,,); +SET DEBUG_SYNC= "at_unlog_xa_prepare SIGNAL con1_ready WAIT_FOR con1_go"; +XA START '1'; +INSERT INTO t1 SET a=1; +XA END '1'; +--send XA PREPARE '1'; + + +connection default; +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; +FLUSH LOGS; +FLUSH LOGS; +FLUSH LOGS; + +--source include/show_binary_logs.inc +--let $binlog_file= master-bin.000004 +--let $binlog_start= 4 +--source include/show_binlog_events.inc + +SET DEBUG_SYNC= "now SIGNAL con1_go"; + +connection con1; +reap; +--echo *** master-bin.000004 checkpoint must show up now *** +--source include/wait_for_binlog_checkpoint.inc + +# Todo: think about the error code returned, move to an appropriate test, or remove +# connection default; +#--error 1399 +# DROP TABLE t1; + +connection con1; +XA ROLLBACK '1'; +SET debug_sync = 'reset'; + +# Clean up. +connection default; + +DROP TABLE t1; +SET debug_sync = 'reset'; diff --git a/mysql-test/suite/binlog/t/binlog_xa_prepared.inc b/mysql-test/suite/binlog/t/binlog_xa_prepared.inc new file mode 100644 index 00000000000..b6306791cf4 --- /dev/null +++ b/mysql-test/suite/binlog/t/binlog_xa_prepared.inc @@ -0,0 +1,102 @@ +--source include/have_innodb.inc +--source include/have_perfschema.inc +# +# The test verifies binlogging of XA transaction and state of prepared XA +# as far as binlog is concerned. +# +# The prepared XA transactions can be disconnected from the client, +# discovered from another connection and commited or rolled back +# later. They also survive the server restart. The test runs two +# loops each consisting of prepared XA:s generation, their +# manipulation and a server restart followed with survived XA:s +# completion. +# +# Prepared XA can't get available to an external connection +# until connection that either leaves actively or is killed +# has completed a necessary part of its cleanup. +# Selecting from P_S.threads provides a method to learn that. +# +# Total number of connection each performing one insert into table +--let $conn_number=20 +# Number of rollbacks and commits from either side of the server restart +--let $rollback_number=5 +--let $commit_number=5 +# Number of transactions that are terminated before server restarts +--let $term_number=`SELECT $rollback_number + $commit_number` +# Instead of disconnect make some connections killed when their +# transactions got prepared. +--let $killed_number=5 +# make some connections disconnected by shutdown rather than actively +--let $server_disconn_number=5 +--let $prepared_at_server_restart = `SELECT $conn_number - $term_number` +# number a "warmup" connection after server restart, they all commit +--let $post_restart_conn_number=10 + +# Counter to be used in GTID consistency check. +# It's incremented per each non-XA transaction commit. +# Local to this file variable to control one-phase commit loop +--let $one_phase_number = 5 + +--connection default + +# Remove possibly preceeding binlogs and clear initialization time +# GTID executed info. In the following all transactions are counted +# to conduct verification at the end of the test. +if (`SELECT @@global.log_bin`) +{ + RESET MASTER; +} + +# Disconected and follower threads need synchronization +CREATE VIEW v_processlist as SELECT * FROM performance_schema.threads where type = 'FOREGROUND'; + +--eval call mtr.add_suppression("Found $prepared_at_server_restart prepared XA transactions") + +CREATE TABLE t (a INT) ENGINE=innodb; + +# Counter is incremented at the end of post restart to +# reflect number of loops done in correctness computation. +--let $restart_number = 0 +--let $how_to_restart=restart_mysqld.inc +--source suite/binlog/include/binlog_xa_prepared_do_and_restart.inc + +--let $how_to_restart=kill_and_restart_mysqld.inc +--source suite/binlog/include/binlog_xa_prepared_do_and_restart.inc + +--connection default + +# Few xs that commit in one phase, not subject to the server restart +# nor reconnect. +# This piece of test is related to mysqlbinlog recovery examine below. +--let $k = 0 +while ($k < $one_phase_number) +{ + --eval XA START 'one_phase_trx_$k' + --eval INSERT INTO t SET a=$k + --eval XA END 'one_phase_trx_$k' + --eval XA COMMIT 'one_phase_trx_$k' ONE PHASE + + --inc $k +} + +SELECT SUM(a) FROM t; +DROP TABLE t; +DROP VIEW v_processlist; + +let $outfile= $MYSQLTEST_VARDIR/tmp/mysqlbinlog.sql; +if (`SELECT @@global.log_bin`) +{ + # Recording proper samples of binlogged prepared XA:s + --source include/show_binlog_events.inc + --exec $MYSQL_BINLOG -R --to-last-log master-bin.000001 > $outfile +} + +--echo All transactions must be completed, to empty-list the following: +XA RECOVER; + +if (`SELECT @@global.log_bin`) +{ + --exec $MYSQL test < $outfile + --remove_file $outfile + XA RECOVER; +} diff --git a/mysql-test/suite/binlog/t/binlog_xa_prepared_disconnect.test b/mysql-test/suite/binlog/t/binlog_xa_prepared_disconnect.test new file mode 100644 index 00000000000..2a3184030cf --- /dev/null +++ b/mysql-test/suite/binlog/t/binlog_xa_prepared_disconnect.test @@ -0,0 +1,11 @@ +############################################################################### +# Bug#12161 Xa recovery and client disconnection +# Testing new server options and binary logging prepared XA transaction. +############################################################################### + +# +# MIXED mode is chosen because formats are varied inside the sourced tests. +# +--source include/have_binlog_format_mixed.inc + +--source suite/binlog/t/binlog_xa_prepared.inc diff --git a/mysql-test/suite/rpl/include/rpl_xa_mixed_engines.inc b/mysql-test/suite/rpl/include/rpl_xa_mixed_engines.inc new file mode 100644 index 00000000000..0707a04090a --- /dev/null +++ b/mysql-test/suite/rpl/include/rpl_xa_mixed_engines.inc @@ -0,0 +1,183 @@ +# +# The test file is invoked from rpl.rpl_xa_survive_disconnect_mixed_engines +# +# The test file is orginized as three sections: setup, run and cleanup. +# The main logics is resided in the run section which generates +# three types of XA transaction: two kinds of mixed and one on non-transactional +# table. +# +# param $command one of three of: 'setup', 'run' or 'cleanup' +# param $xa_terminate how to conclude: 'XA COMMIT' or 'XA ROLLBACK' +# param $one_phase 'one_phase' can be opted with XA COMMIT above +# param $xa_prepare_opt '1' or empty can be opted to test with and without XA PREPARE +# param $xid arbitrary name for xa trx, defaults to 'xa_trx' +# Note '' is merely to underline, not a part of the value. +# + +if ($command == setup) +{ + # Test randomizes the following variable's value: + SET @@session.binlog_direct_non_transactional_updates := if(floor(rand()*10)%2,'ON','OFF'); + CREATE TABLE t (a INT) ENGINE=innodb; + CREATE TABLE tm (a INT) ENGINE=myisam; +} +if (!$xid) +{ + --let $xid=xa_trx +} +if ($command == run) +{ + ## Non-temporary table cases + # Non transactional table goes first + --eval XA START '$xid' + --disable_warnings + INSERT INTO tm VALUES (1); + INSERT INTO t VALUES (1); + --enable_warnings + --eval XA END '$xid' + if ($xa_prepare_opt) + { + --eval XA PREPARE '$xid' + } + --eval $xa_terminate '$xid' $one_phase + + # Transactional table goes first + --eval XA START '$xid' + --disable_warnings + INSERT INTO t VALUES (2); + INSERT INTO tm VALUES (2); + --enable_warnings + --eval XA END '$xid' + if ($xa_prepare_opt) + { + --eval XA PREPARE '$xid' + } + --eval $xa_terminate '$xid' $one_phase + + # The pure non-transactional table + --eval XA START '$xid' + --disable_warnings + INSERT INTO tm VALUES (3); + --enable_warnings + --eval XA END '$xid' + if ($xa_prepare_opt) + { + --eval XA PREPARE '$xid' + } + --eval $xa_terminate '$xid' $one_phase + + ## Temporary tables + # create outside xa use at the tail + CREATE TEMPORARY TABLE tmp_i LIKE t; + CREATE TEMPORARY TABLE tmp_m LIKE tm; + --eval XA START '$xid' + --disable_warnings + INSERT INTO t VALUES (4); + INSERT INTO tm VALUES (4); + INSERT INTO tmp_i VALUES (4); + INSERT INTO tmp_m VALUES (4); + INSERT INTO t SELECT * FROM tmp_i; + INSERT INTO tm SELECT * FROM tmp_m; + --enable_warnings + --eval XA END '$xid' + if ($xa_prepare_opt) + { + --eval XA PREPARE '$xid' + } + --eval $xa_terminate '$xid' $one_phase + + # temporary tables at the head + --eval XA START '$xid' + --disable_warnings + INSERT INTO tmp_i VALUES (5); + INSERT INTO tmp_m VALUES (5); + INSERT INTO t SELECT * FROM tmp_i; + INSERT INTO tm SELECT * FROM tmp_m; + INSERT INTO t VALUES (5); + INSERT INTO tm VALUES (5); + --enable_warnings + --eval XA END '$xid' + if ($xa_prepare_opt) + { + --eval XA PREPARE '$xid' + } + --eval $xa_terminate '$xid' $one_phase + + # create inside xa use at the tail + DROP TEMPORARY TABLE tmp_i; + DROP TEMPORARY TABLE tmp_m; + + --eval XA START '$xid' + --disable_warnings + INSERT INTO t VALUES (6); + INSERT INTO tm VALUES (6); + CREATE TEMPORARY TABLE tmp_i LIKE t; + CREATE TEMPORARY TABLE tmp_m LIKE tm; + INSERT INTO tmp_i VALUES (6); + INSERT INTO tmp_m VALUES (6); + INSERT INTO t SELECT * FROM tmp_i; + INSERT INTO tm SELECT * FROM tmp_m; + --enable_warnings + --eval XA END '$xid' + if ($xa_prepare_opt) + { + --eval XA PREPARE '$xid' + } + --eval $xa_terminate '$xid' $one_phase + + # use at the head + DROP TEMPORARY TABLE tmp_i; + DROP TEMPORARY TABLE tmp_m; + --eval XA START '$xid' + --disable_warnings + CREATE TEMPORARY TABLE tmp_i LIKE t; + CREATE TEMPORARY TABLE tmp_m LIKE tm; + INSERT INTO tmp_i VALUES (7); + INSERT INTO tmp_m VALUES (7); + INSERT INTO t SELECT * FROM tmp_i; + INSERT INTO tm SELECT * FROM tmp_m; + INSERT INTO t VALUES (7); + INSERT INTO tm VALUES (7); + --enable_warnings + --eval XA END '$xid' + if ($xa_prepare_opt) + { + --eval XA PREPARE '$xid' + } + --eval $xa_terminate '$xid' $one_phase + + # use at the tail and drop + --eval XA START '$xid' + --disable_warnings + INSERT INTO t VALUES (8); + INSERT INTO tm VALUES (8); + INSERT INTO tmp_i VALUES (8); + INSERT INTO tmp_m VALUES (8); + INSERT INTO t SELECT * FROM tmp_i; + INSERT INTO tm SELECT * FROM tmp_m; + DROP TEMPORARY TABLE tmp_i; + DROP TEMPORARY TABLE tmp_m; + --enable_warnings + --eval XA END '$xid' + if ($xa_prepare_opt) + { + --eval XA PREPARE '$xid' + } + --eval $xa_terminate '$xid' $one_phase + + ## Ineffective transactional table operation case + + --eval XA START '$xid' + UPDATE t SET a = 99 where a = -1; + --eval XA END '$xid' + if ($xa_prepare_opt) + { + --eval XA PREPARE '$xid' + } + --eval $xa_terminate '$xid' $one_phase +} + +if ($command == cleanup) +{ + DROP TABLE t, tm; +} diff --git a/mysql-test/suite/rpl/r/rpl_parallel_optimistic_xa.result b/mysql-test/suite/rpl/r/rpl_parallel_optimistic_xa.result new file mode 100644 index 00000000000..4136f1885db --- /dev/null +++ b/mysql-test/suite/rpl/r/rpl_parallel_optimistic_xa.result @@ -0,0 +1,51 @@ +include/master-slave.inc +[connection master] +call mtr.add_suppression("Deadlock found when trying to get lock; try restarting transaction"); +call mtr.add_suppression("WSREP: handlerton rollback failed"); +CREATE VIEW v_processlist as SELECT * FROM performance_schema.threads where type = 'FOREGROUND'; +connection master; +ALTER TABLE mysql.gtid_slave_pos ENGINE=InnoDB; +connection slave; +include/stop_slave.inc +SET @old_parallel_threads = @@GLOBAL.slave_parallel_threads; +SET @@global.slave_parallel_threads = 7; +SET @old_parallel_mode = @@GLOBAL.slave_parallel_mode; +SET @@global.slave_parallel_mode ='optimistic'; +SET @old_gtid_cleanup_batch_size = @@GLOBAL.gtid_cleanup_batch_size; +SET @@global.gtid_cleanup_batch_size = 1000000; +CHANGE MASTER TO master_use_gtid=slave_pos; +connection master; +CREATE TABLE t0 (a int, b INT) ENGINE=InnoDB; +CREATE TABLE t1 (a int PRIMARY KEY, b INT) ENGINE=InnoDB; +INSERT INTO t1 VALUES (1, 0); +include/save_master_gtid.inc +connection slave; +include/start_slave.inc +include/sync_with_master_gtid.inc +include/stop_slave.inc +connection master; +include/save_master_gtid.inc +connection slave; +include/start_slave.inc +include/sync_with_master_gtid.inc +include/diff_tables.inc [master:t0, slave:t0] +include/diff_tables.inc [master:t1, slave:t1] +connection slave; +include/stop_slave.inc +set global log_warnings=default; +SET GLOBAL slave_parallel_mode=@old_parallel_mode; +SET GLOBAL slave_parallel_threads=@old_parallel_threads; +include/start_slave.inc +connection master; +DROP VIEW v_processlist; +DROP TABLE t0, t1; +include/save_master_gtid.inc +connection slave; +include/sync_with_master_gtid.inc +SELECT COUNT(*) <= 5*@@GLOBAL.gtid_cleanup_batch_size +FROM mysql.gtid_slave_pos; +COUNT(*) <= 5*@@GLOBAL.gtid_cleanup_batch_size +1 +SET GLOBAL gtid_cleanup_batch_size= @old_gtid_cleanup_batch_size; +connection master; +include/rpl_end.inc diff --git a/mysql-test/suite/rpl/r/rpl_parallel_optimistic_xa_lsu_off.result b/mysql-test/suite/rpl/r/rpl_parallel_optimistic_xa_lsu_off.result new file mode 100644 index 00000000000..4136f1885db --- /dev/null +++ b/mysql-test/suite/rpl/r/rpl_parallel_optimistic_xa_lsu_off.result @@ -0,0 +1,51 @@ +include/master-slave.inc +[connection master] +call mtr.add_suppression("Deadlock found when trying to get lock; try restarting transaction"); +call mtr.add_suppression("WSREP: handlerton rollback failed"); +CREATE VIEW v_processlist as SELECT * FROM performance_schema.threads where type = 'FOREGROUND'; +connection master; +ALTER TABLE mysql.gtid_slave_pos ENGINE=InnoDB; +connection slave; +include/stop_slave.inc +SET @old_parallel_threads = @@GLOBAL.slave_parallel_threads; +SET @@global.slave_parallel_threads = 7; +SET @old_parallel_mode = @@GLOBAL.slave_parallel_mode; +SET @@global.slave_parallel_mode ='optimistic'; +SET @old_gtid_cleanup_batch_size = @@GLOBAL.gtid_cleanup_batch_size; +SET @@global.gtid_cleanup_batch_size = 1000000; +CHANGE MASTER TO master_use_gtid=slave_pos; +connection master; +CREATE TABLE t0 (a int, b INT) ENGINE=InnoDB; +CREATE TABLE t1 (a int PRIMARY KEY, b INT) ENGINE=InnoDB; +INSERT INTO t1 VALUES (1, 0); +include/save_master_gtid.inc +connection slave; +include/start_slave.inc +include/sync_with_master_gtid.inc +include/stop_slave.inc +connection master; +include/save_master_gtid.inc +connection slave; +include/start_slave.inc +include/sync_with_master_gtid.inc +include/diff_tables.inc [master:t0, slave:t0] +include/diff_tables.inc [master:t1, slave:t1] +connection slave; +include/stop_slave.inc +set global log_warnings=default; +SET GLOBAL slave_parallel_mode=@old_parallel_mode; +SET GLOBAL slave_parallel_threads=@old_parallel_threads; +include/start_slave.inc +connection master; +DROP VIEW v_processlist; +DROP TABLE t0, t1; +include/save_master_gtid.inc +connection slave; +include/sync_with_master_gtid.inc +SELECT COUNT(*) <= 5*@@GLOBAL.gtid_cleanup_batch_size +FROM mysql.gtid_slave_pos; +COUNT(*) <= 5*@@GLOBAL.gtid_cleanup_batch_size +1 +SET GLOBAL gtid_cleanup_batch_size= @old_gtid_cleanup_batch_size; +connection master; +include/rpl_end.inc diff --git a/mysql-test/suite/rpl/r/rpl_parallel_xa_same_xid.result b/mysql-test/suite/rpl/r/rpl_parallel_xa_same_xid.result new file mode 100644 index 00000000000..03fe5157623 --- /dev/null +++ b/mysql-test/suite/rpl/r/rpl_parallel_xa_same_xid.result @@ -0,0 +1,23 @@ +include/master-slave.inc +[connection master] +connection slave; +call mtr.add_suppression("WSREP: handlerton rollback failed"); +include/stop_slave.inc +ALTER TABLE mysql.gtid_slave_pos ENGINE=InnoDB; +SET @old_parallel_threads = @@GLOBAL.slave_parallel_threads; +SET @old_parallel_mode = @@GLOBAL.slave_parallel_mode; +SET @@global.slave_parallel_mode ='optimistic'; +include/start_slave.inc +connection master; +CREATE TABLE t1 (a INT, b INT) ENGINE=InnoDB; +CREATE TABLE t2 (a INT AUTO_INCREMENT PRIMARY KEY, b INT) ENGINE=InnoDB; +include/sync_slave_sql_with_master.inc +include/diff_tables.inc [master:t1, slave:t1] +connection slave; +include/stop_slave.inc +SET GLOBAL slave_parallel_threads=@old_parallel_threads; +SET GLOBAL slave_parallel_mode=@old_parallel_mode; +include/start_slave.inc +connection master; +DROP TABLE t1, t2; +include/rpl_end.inc diff --git a/mysql-test/suite/rpl/r/rpl_temporary_errors.result b/mysql-test/suite/rpl/r/rpl_temporary_errors.result index 8654fe218dc..c126871e460 100644 --- a/mysql-test/suite/rpl/r/rpl_temporary_errors.result +++ b/mysql-test/suite/rpl/r/rpl_temporary_errors.result @@ -3,7 +3,7 @@ include/master-slave.inc call mtr.add_suppression("Deadlock found"); call mtr.add_suppression("Can't find record in 't.'"); connection master; -CREATE TABLE t1 (a INT PRIMARY KEY, b INT); +CREATE TABLE t1 (a INT PRIMARY KEY, b INT) ENGINE=innodb; INSERT INTO t1 VALUES (1,1), (2,2), (3,3), (4,4); connection slave; SHOW STATUS LIKE 'Slave_retried_transactions'; @@ -11,34 +11,67 @@ Variable_name Value Slave_retried_transactions 0 set @@global.slave_exec_mode= 'IDEMPOTENT'; UPDATE t1 SET a = 5, b = 47 WHERE a = 1; -SELECT * FROM t1; +SELECT * FROM t1 ORDER BY a; a b -5 47 2 2 3 3 4 4 +5 47 connection master; UPDATE t1 SET a = 5, b = 5 WHERE a = 1; -SELECT * FROM t1; +SELECT * FROM t1 ORDER BY a; a b -5 5 2 2 3 3 4 4 +5 5 connection slave; set @@global.slave_exec_mode= default; SHOW STATUS LIKE 'Slave_retried_transactions'; Variable_name Value Slave_retried_transactions 0 -SELECT * FROM t1; +SELECT * FROM t1 ORDER BY a; a b -5 47 2 2 3 3 4 4 +5 47 include/check_slave_is_running.inc connection slave; call mtr.add_suppression("Slave SQL.*Could not execute Update_rows event on table test.t1"); +call mtr.add_suppression("Slave SQL for channel '': worker thread retried transaction"); +call mtr.add_suppression("The slave coordinator and worker threads are stopped"); +connection slave; +set @save_innodb_lock_wait_timeout=@@global.innodb_lock_wait_timeout; +set @save_slave_transaction_retries=@@global.slave_transaction_retries; +set @@global.innodb_lock_wait_timeout=1; +set @@global.slave_transaction_retries=2; +include/restart_slave.inc +connection slave1; +BEGIN; +INSERT INTO t1 SET a = 6, b = 7; +connection master; +INSERT INTO t1 SET a = 99, b = 99; +XA START 'xa1'; +INSERT INTO t1 SET a = 6, b = 6; +XA END 'xa1'; +XA PREPARE 'xa1'; +connection slave; +include/wait_for_slave_sql_error.inc [errno=1213,1205] +set @@global.innodb_lock_wait_timeout=1; +set @@global.slave_transaction_retries=100; +include/restart_slave.inc +Warnings: +Note 1255 Slave already has been stopped +connection slave1; +ROLLBACK; +connection master; +XA COMMIT 'xa1'; +include/sync_slave_sql_with_master.inc +connection slave; +include/assert.inc [XA transaction record must be in the table] +set @@global.innodb_lock_wait_timeout=@save_innodb_lock_wait_timeout; +set @@global.slave_transaction_retries= @save_slave_transaction_retries; connection master; DROP TABLE t1; connection slave; diff --git a/mysql-test/suite/rpl/r/rpl_xa.result b/mysql-test/suite/rpl/r/rpl_xa.result new file mode 100644 index 00000000000..3420f2348e2 --- /dev/null +++ b/mysql-test/suite/rpl/r/rpl_xa.result @@ -0,0 +1,48 @@ +include/master-slave.inc +[connection master] +connection master; +create table t1 (a int, b int) engine=InnoDB; +insert into t1 values(0, 0); +xa start 't'; +insert into t1 values(1, 2); +xa end 't'; +xa prepare 't'; +xa commit 't'; +connection slave; +include/diff_tables.inc [master:t1, slave:t1] +connection master; +xa start 't'; +insert into t1 values(3, 4); +xa end 't'; +xa prepare 't'; +xa rollback 't'; +connection slave; +include/diff_tables.inc [master:t1, slave:t1] +connection master; +SET pseudo_slave_mode=1; +create table t2 (a int) engine=InnoDB; +xa start 't'; +insert into t1 values (5, 6); +xa end 't'; +xa prepare 't'; +xa start 's'; +insert into t2 values (0); +xa end 's'; +xa prepare 's'; +include/save_master_gtid.inc +connection slave; +include/sync_with_master_gtid.inc +xa recover; +formatID gtrid_length bqual_length data +1 1 0 t +1 1 0 s +connection master; +xa commit 't'; +xa commit 's'; +SET pseudo_slave_mode=0; +connection slave; +include/diff_tables.inc [master:t1, slave:t1] +include/diff_tables.inc [master:t2, slave:t2] +connection master; +drop table t1, t2; +include/rpl_end.inc diff --git a/mysql-test/suite/rpl/r/rpl_xa_gap_lock.result b/mysql-test/suite/rpl/r/rpl_xa_gap_lock.result new file mode 100644 index 00000000000..cb760abe2d2 --- /dev/null +++ b/mysql-test/suite/rpl/r/rpl_xa_gap_lock.result @@ -0,0 +1,44 @@ +include/master-slave.inc +[connection master] +connection slave; +SET @saved_innodb_limit_optimistic_insert_debug = @@GLOBAL.innodb_limit_optimistic_insert_debug; +SET @@GLOBAL.innodb_limit_optimistic_insert_debug = 2; +connection master; +CREATE TABLE t1 ( +c1 INT NOT NULL, +KEY(c1) +) ENGINE=InnoDB; +CREATE TABLE t2 ( +c1 INT NOT NULL, +FOREIGN KEY(c1) REFERENCES t1(c1) +) ENGINE=InnoDB; +INSERT INTO t1 VALUES (1), (3), (4); +connection master1; +XA START 'XA1'; +INSERT INTO t1 values(2); +XA END 'XA1'; +connection master; +XA START 'XA2'; +INSERT INTO t2 values(3); +XA END 'XA2'; +XA PREPARE 'XA2'; +connection master1; +XA PREPARE 'XA1'; +XA COMMIT 'XA1'; +connection master; +XA COMMIT 'XA2'; +include/sync_slave_sql_with_master.inc +include/stop_slave.inc +DROP TABLE t2, t1; +RESET SLAVE; +RESET MASTER; +connection master; +Restore binary log from the master into the slave +include/diff_tables.inc [master:test.t1, slave:test.t1] +include/diff_tables.inc [master:test.t2, slave:test.t2] +DROP TABLE t2, t1; +connection slave; +CHANGE MASTER TO MASTER_LOG_FILE='LOG_FILE', MASTER_LOG_POS=LOG_POS; +SET @@GLOBAL.innodb_limit_optimistic_insert_debug = @saved_innodb_limit_optimistic_insert_debug; +include/start_slave.inc +include/rpl_end.inc diff --git a/mysql-test/suite/rpl/r/rpl_xa_gtid_pos_auto_engine.result b/mysql-test/suite/rpl/r/rpl_xa_gtid_pos_auto_engine.result new file mode 100644 index 00000000000..a7ed0f97ea2 --- /dev/null +++ b/mysql-test/suite/rpl/r/rpl_xa_gtid_pos_auto_engine.result @@ -0,0 +1,64 @@ +include/master-slave.inc +[connection master] +connection slave; +call mtr.add_suppression("The automatically created table.*name may not be entirely in lowercase"); +include/stop_slave.inc +CHANGE MASTER TO master_use_gtid=slave_pos; +SET @@global.gtid_pos_auto_engines="innodb"; +include/start_slave.inc +connection master; +create table t1 (a int, b int) engine=InnoDB; +insert into t1 values(0, 0); +xa start 't'; +insert into t1 values(1, 2); +xa end 't'; +xa prepare 't'; +xa commit 't'; +connection slave; +include/diff_tables.inc [master:t1, slave:t1] +connection master; +xa start 't'; +insert into t1 values(3, 4); +xa end 't'; +xa prepare 't'; +xa rollback 't'; +connection slave; +include/diff_tables.inc [master:t1, slave:t1] +connection master; +SET pseudo_slave_mode=1; +create table t2 (a int) engine=InnoDB; +xa start 't'; +insert into t1 values (5, 6); +xa end 't'; +xa prepare 't'; +xa start 's'; +insert into t2 values (0); +xa end 's'; +xa prepare 's'; +include/save_master_gtid.inc +connection slave; +include/sync_with_master_gtid.inc +SELECT @@global.gtid_slave_pos = CONCAT(domain_id,"-",server_id,"-",seq_no) FROM mysql.gtid_slave_pos WHERE seq_no = (SELECT DISTINCT max(seq_no) FROM mysql.gtid_slave_pos); +@@global.gtid_slave_pos = CONCAT(domain_id,"-",server_id,"-",seq_no) +1 +xa recover; +formatID gtrid_length bqual_length data +1 1 0 t +1 1 0 s +connection master; +xa commit 't'; +xa commit 's'; +SET pseudo_slave_mode=0; +connection slave; +include/diff_tables.inc [master:t1, slave:t1] +include/diff_tables.inc [master:t2, slave:t2] +connection master; +drop table t1, t2; +connection slave; +include/stop_slave.inc +SET @@global.gtid_pos_auto_engines=""; +SET @@session.sql_log_bin=0; +DROP TABLE mysql.gtid_slave_pos_InnoDB; +SET @@session.sql_log_bin=1; +include/start_slave.inc +include/rpl_end.inc diff --git a/mysql-test/suite/rpl/r/rpl_xa_survive_disconnect.result b/mysql-test/suite/rpl/r/rpl_xa_survive_disconnect.result new file mode 100644 index 00000000000..d2ed1e2c235 --- /dev/null +++ b/mysql-test/suite/rpl/r/rpl_xa_survive_disconnect.result @@ -0,0 +1,319 @@ +include/master-slave.inc +[connection master] +connection master; +call mtr.add_suppression("Found 2 prepared XA transactions"); +CREATE VIEW v_processlist as SELECT * FROM performance_schema.threads where type = 'FOREGROUND'; +CREATE DATABASE d1; +CREATE DATABASE d2; +CREATE TABLE d1.t (a INT) ENGINE=innodb; +CREATE TABLE d2.t (a INT) ENGINE=innodb; +connect master_conn1, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@session.binlog_format= statement; +XA START '1-stmt'; +INSERT INTO d1.t VALUES (1); +XA END '1-stmt'; +XA PREPARE '1-stmt'; +disconnect master_conn1; +connection master; +connect master_conn2, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@session.binlog_format= row; +XA START '1-row'; +INSERT INTO d2.t VALUES (1); +XA END '1-row'; +XA PREPARE '1-row'; +disconnect master_conn2; +connection master; +XA START '2'; +INSERT INTO d1.t VALUES (2); +XA END '2'; +XA PREPARE '2'; +XA COMMIT '2'; +XA COMMIT '1-row'; +XA COMMIT '1-stmt'; +include/show_binlog_events.inc +Log_name Pos Event_type Server_id End_log_pos Info +master-bin.000001 # Gtid # # BEGIN GTID #-#-# +master-bin.000001 # Query # # use `mtr`; INSERT INTO test_suppressions (pattern) VALUES ( NAME_CONST('pattern',_latin1'Found 2 prepared XA transactions' COLLATE 'latin1_swedish_ci')) +master-bin.000001 # Query # # COMMIT +master-bin.000001 # Gtid # # GTID #-#-# +master-bin.000001 # Query # # use `test`; CREATE ALGORITHM=UNDEFINED DEFINER=`root`@`localhost` SQL SECURITY DEFINER VIEW `v_processlist` AS SELECT * FROM performance_schema.threads where type = 'FOREGROUND' +master-bin.000001 # Gtid # # GTID #-#-# +master-bin.000001 # Query # # CREATE DATABASE d1 +master-bin.000001 # Gtid # # GTID #-#-# +master-bin.000001 # Query # # CREATE DATABASE d2 +master-bin.000001 # Gtid # # GTID #-#-# +master-bin.000001 # Query # # use `test`; CREATE TABLE d1.t (a INT) ENGINE=innodb +master-bin.000001 # Gtid # # GTID #-#-# +master-bin.000001 # Query # # use `test`; CREATE TABLE d2.t (a INT) ENGINE=innodb +master-bin.000001 # Gtid # # XA START X'312d73746d74',X'',1 GTID #-#-# +master-bin.000001 # Query # # use `test`; INSERT INTO d1.t VALUES (1) +master-bin.000001 # Query # # XA END X'312d73746d74',X'',1 +master-bin.000001 # XA_prepare # # XA PREPARE X'312d73746d74',X'',1 +master-bin.000001 # Gtid # # XA START X'312d726f77',X'',1 GTID #-#-# +master-bin.000001 # Annotate_rows # # INSERT INTO d2.t VALUES (1) +master-bin.000001 # Table_map # # table_id: # (d2.t) +master-bin.000001 # Write_rows_v1 # # table_id: # flags: STMT_END_F +master-bin.000001 # Query # # XA END X'312d726f77',X'',1 +master-bin.000001 # XA_prepare # # XA PREPARE X'312d726f77',X'',1 +master-bin.000001 # Gtid # # XA START X'32',X'',1 GTID #-#-# +master-bin.000001 # Query # # use `test`; INSERT INTO d1.t VALUES (2) +master-bin.000001 # Query # # XA END X'32',X'',1 +master-bin.000001 # XA_prepare # # XA PREPARE X'32',X'',1 +master-bin.000001 # Gtid # # GTID #-#-# +master-bin.000001 # Query # # XA COMMIT X'32',X'',1 +master-bin.000001 # Gtid # # GTID #-#-# +master-bin.000001 # Query # # XA COMMIT X'312d726f77',X'',1 +master-bin.000001 # Gtid # # GTID #-#-# +master-bin.000001 # Query # # XA COMMIT X'312d73746d74',X'',1 +include/sync_slave_sql_with_master.inc +include/stop_slave.inc +connection master; +connect master2, 127.0.0.1,root,,test,$MASTER_MYPORT,; +connection master2; +SET @@session.binlog_format= statement; +XA START '3-stmt'; +INSERT INTO d1.t VALUES (3); +XA END '3-stmt'; +XA PREPARE '3-stmt'; +disconnect master2; +connect master2, 127.0.0.1,root,,test,$MASTER_MYPORT,; +connection master2; +SET @@session.binlog_format= row; +XA START '3-row'; +INSERT INTO d2.t VALUES (4); +XA END '3-row'; +XA PREPARE '3-row'; +disconnect master2; +connection master; +connect master2, 127.0.0.1,root,,test,$MASTER_MYPORT,; +connection master2; +XA START '4'; +SELECT * FROM d1.t; +a +1 +2 +XA END '4'; +XA PREPARE '4'; +disconnect master2; +connect master_bulk_conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'bulk_trx_10'; +INSERT INTO d1.t VALUES (10); +INSERT INTO d2.t VALUES (10); +XA END 'bulk_trx_10'; +XA PREPARE 'bulk_trx_10'; +disconnect master_bulk_conn10; +connection master; +connect master_bulk_conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'bulk_trx_9'; +INSERT INTO d1.t VALUES (9); +INSERT INTO d2.t VALUES (9); +XA END 'bulk_trx_9'; +XA PREPARE 'bulk_trx_9'; +disconnect master_bulk_conn9; +connection master; +connect master_bulk_conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'bulk_trx_8'; +INSERT INTO d1.t VALUES (8); +INSERT INTO d2.t VALUES (8); +XA END 'bulk_trx_8'; +XA PREPARE 'bulk_trx_8'; +disconnect master_bulk_conn8; +connection master; +connect master_bulk_conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'bulk_trx_7'; +INSERT INTO d1.t VALUES (7); +INSERT INTO d2.t VALUES (7); +XA END 'bulk_trx_7'; +XA PREPARE 'bulk_trx_7'; +disconnect master_bulk_conn7; +connection master; +connect master_bulk_conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'bulk_trx_6'; +INSERT INTO d1.t VALUES (6); +INSERT INTO d2.t VALUES (6); +XA END 'bulk_trx_6'; +XA PREPARE 'bulk_trx_6'; +disconnect master_bulk_conn6; +connection master; +connect master_bulk_conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'bulk_trx_5'; +INSERT INTO d1.t VALUES (5); +INSERT INTO d2.t VALUES (5); +XA END 'bulk_trx_5'; +XA PREPARE 'bulk_trx_5'; +disconnect master_bulk_conn5; +connection master; +connect master_bulk_conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'bulk_trx_4'; +INSERT INTO d1.t VALUES (4); +INSERT INTO d2.t VALUES (4); +XA END 'bulk_trx_4'; +XA PREPARE 'bulk_trx_4'; +disconnect master_bulk_conn4; +connection master; +connect master_bulk_conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'bulk_trx_3'; +INSERT INTO d1.t VALUES (3); +INSERT INTO d2.t VALUES (3); +XA END 'bulk_trx_3'; +XA PREPARE 'bulk_trx_3'; +disconnect master_bulk_conn3; +connection master; +connect master_bulk_conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'bulk_trx_2'; +INSERT INTO d1.t VALUES (2); +INSERT INTO d2.t VALUES (2); +XA END 'bulk_trx_2'; +XA PREPARE 'bulk_trx_2'; +disconnect master_bulk_conn2; +connection master; +connect master_bulk_conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'bulk_trx_1'; +INSERT INTO d1.t VALUES (1); +INSERT INTO d2.t VALUES (1); +XA END 'bulk_trx_1'; +XA PREPARE 'bulk_trx_1'; +disconnect master_bulk_conn1; +connection master; +connection slave; +include/start_slave.inc +connection master; +include/sync_slave_sql_with_master.inc +include/stop_slave.inc +connection master; +XA COMMIT 'bulk_trx_10'; +XA ROLLBACK 'bulk_trx_9'; +XA COMMIT 'bulk_trx_8'; +XA ROLLBACK 'bulk_trx_7'; +XA COMMIT 'bulk_trx_6'; +XA ROLLBACK 'bulk_trx_5'; +XA COMMIT 'bulk_trx_4'; +XA ROLLBACK 'bulk_trx_3'; +XA COMMIT 'bulk_trx_2'; +XA ROLLBACK 'bulk_trx_1'; +include/rpl_restart_server.inc [server_number=1] +connection slave; +include/start_slave.inc +connection master; +*** '3-stmt','3-row' xa-transactions must be in the list *** +XA RECOVER; +formatID gtrid_length bqual_length data +1 6 0 3-stmt +1 5 0 3-row +XA COMMIT '3-stmt'; +XA ROLLBACK '3-row'; +include/sync_slave_sql_with_master.inc +connection master; +connect master_conn2, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START '0123456789012345678901234567890123456789012345678901234567890124','0123456789012345678901234567890123456789012345678901234567890124',4294967292; +INSERT INTO d1.t VALUES (64); +XA END '0123456789012345678901234567890123456789012345678901234567890124','0123456789012345678901234567890123456789012345678901234567890124',4294967292; +XA PREPARE '0123456789012345678901234567890123456789012345678901234567890124','0123456789012345678901234567890123456789012345678901234567890124',4294967292; +disconnect master_conn2; +connection master; +connect master_conn3, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START X'FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF',X'00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000',0; +INSERT INTO d1.t VALUES (0); +XA END X'FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF',X'00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000',0; +XA PREPARE X'FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF',X'00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000',0; +disconnect master_conn3; +connection master; +disconnect master_conn4; +connection master; +XA COMMIT '0123456789012345678901234567890123456789012345678901234567890124','0123456789012345678901234567890123456789012345678901234567890124',4294967292; +XA COMMIT X'FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF',X'00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000',0; +XA COMMIT 'RANDOM XID' +include/sync_slave_sql_with_master.inc +connect master_bulk_conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +connection master_bulk_conn10; +XA START 'one_phase_10'; +INSERT INTO d1.t VALUES (10); +INSERT INTO d2.t VALUES (10); +XA END 'one_phase_10'; +XA COMMIT 'one_phase_10' ONE PHASE; +disconnect master_bulk_conn10; +connect master_bulk_conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +connection master_bulk_conn9; +XA START 'one_phase_9'; +INSERT INTO d1.t VALUES (9); +INSERT INTO d2.t VALUES (9); +XA END 'one_phase_9'; +XA COMMIT 'one_phase_9' ONE PHASE; +disconnect master_bulk_conn9; +connect master_bulk_conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +connection master_bulk_conn8; +XA START 'one_phase_8'; +INSERT INTO d1.t VALUES (8); +INSERT INTO d2.t VALUES (8); +XA END 'one_phase_8'; +XA COMMIT 'one_phase_8' ONE PHASE; +disconnect master_bulk_conn8; +connect master_bulk_conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +connection master_bulk_conn7; +XA START 'one_phase_7'; +INSERT INTO d1.t VALUES (7); +INSERT INTO d2.t VALUES (7); +XA END 'one_phase_7'; +XA COMMIT 'one_phase_7' ONE PHASE; +disconnect master_bulk_conn7; +connect master_bulk_conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +connection master_bulk_conn6; +XA START 'one_phase_6'; +INSERT INTO d1.t VALUES (6); +INSERT INTO d2.t VALUES (6); +XA END 'one_phase_6'; +XA COMMIT 'one_phase_6' ONE PHASE; +disconnect master_bulk_conn6; +connect master_bulk_conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +connection master_bulk_conn5; +XA START 'one_phase_5'; +INSERT INTO d1.t VALUES (5); +INSERT INTO d2.t VALUES (5); +XA END 'one_phase_5'; +XA COMMIT 'one_phase_5' ONE PHASE; +disconnect master_bulk_conn5; +connect master_bulk_conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +connection master_bulk_conn4; +XA START 'one_phase_4'; +INSERT INTO d1.t VALUES (4); +INSERT INTO d2.t VALUES (4); +XA END 'one_phase_4'; +XA COMMIT 'one_phase_4' ONE PHASE; +disconnect master_bulk_conn4; +connect master_bulk_conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +connection master_bulk_conn3; +XA START 'one_phase_3'; +INSERT INTO d1.t VALUES (3); +INSERT INTO d2.t VALUES (3); +XA END 'one_phase_3'; +XA COMMIT 'one_phase_3' ONE PHASE; +disconnect master_bulk_conn3; +connect master_bulk_conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +connection master_bulk_conn2; +XA START 'one_phase_2'; +INSERT INTO d1.t VALUES (2); +INSERT INTO d2.t VALUES (2); +XA END 'one_phase_2'; +XA COMMIT 'one_phase_2' ONE PHASE; +disconnect master_bulk_conn2; +connect master_bulk_conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +connection master_bulk_conn1; +XA START 'one_phase_1'; +INSERT INTO d1.t VALUES (1); +INSERT INTO d2.t VALUES (1); +XA END 'one_phase_1'; +XA COMMIT 'one_phase_1' ONE PHASE; +disconnect master_bulk_conn1; +connection master; +include/sync_slave_sql_with_master.inc +include/diff_tables.inc [master:d1.t, slave:d1.t] +include/diff_tables.inc [master:d2.t, slave:d2.t] +connection master; +DELETE FROM d1.t; +DELETE FROM d2.t; +DROP TABLE d1.t, d2.t; +DROP DATABASE d1; +DROP DATABASE d2; +DROP VIEW v_processlist; +include/sync_slave_sql_with_master.inc +include/rpl_end.inc diff --git a/mysql-test/suite/rpl/r/rpl_xa_survive_disconnect_lsu_off.result b/mysql-test/suite/rpl/r/rpl_xa_survive_disconnect_lsu_off.result new file mode 100644 index 00000000000..d2ed1e2c235 --- /dev/null +++ b/mysql-test/suite/rpl/r/rpl_xa_survive_disconnect_lsu_off.result @@ -0,0 +1,319 @@ +include/master-slave.inc +[connection master] +connection master; +call mtr.add_suppression("Found 2 prepared XA transactions"); +CREATE VIEW v_processlist as SELECT * FROM performance_schema.threads where type = 'FOREGROUND'; +CREATE DATABASE d1; +CREATE DATABASE d2; +CREATE TABLE d1.t (a INT) ENGINE=innodb; +CREATE TABLE d2.t (a INT) ENGINE=innodb; +connect master_conn1, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@session.binlog_format= statement; +XA START '1-stmt'; +INSERT INTO d1.t VALUES (1); +XA END '1-stmt'; +XA PREPARE '1-stmt'; +disconnect master_conn1; +connection master; +connect master_conn2, 127.0.0.1,root,,test,$MASTER_MYPORT,; +SET @@session.binlog_format= row; +XA START '1-row'; +INSERT INTO d2.t VALUES (1); +XA END '1-row'; +XA PREPARE '1-row'; +disconnect master_conn2; +connection master; +XA START '2'; +INSERT INTO d1.t VALUES (2); +XA END '2'; +XA PREPARE '2'; +XA COMMIT '2'; +XA COMMIT '1-row'; +XA COMMIT '1-stmt'; +include/show_binlog_events.inc +Log_name Pos Event_type Server_id End_log_pos Info +master-bin.000001 # Gtid # # BEGIN GTID #-#-# +master-bin.000001 # Query # # use `mtr`; INSERT INTO test_suppressions (pattern) VALUES ( NAME_CONST('pattern',_latin1'Found 2 prepared XA transactions' COLLATE 'latin1_swedish_ci')) +master-bin.000001 # Query # # COMMIT +master-bin.000001 # Gtid # # GTID #-#-# +master-bin.000001 # Query # # use `test`; CREATE ALGORITHM=UNDEFINED DEFINER=`root`@`localhost` SQL SECURITY DEFINER VIEW `v_processlist` AS SELECT * FROM performance_schema.threads where type = 'FOREGROUND' +master-bin.000001 # Gtid # # GTID #-#-# +master-bin.000001 # Query # # CREATE DATABASE d1 +master-bin.000001 # Gtid # # GTID #-#-# +master-bin.000001 # Query # # CREATE DATABASE d2 +master-bin.000001 # Gtid # # GTID #-#-# +master-bin.000001 # Query # # use `test`; CREATE TABLE d1.t (a INT) ENGINE=innodb +master-bin.000001 # Gtid # # GTID #-#-# +master-bin.000001 # Query # # use `test`; CREATE TABLE d2.t (a INT) ENGINE=innodb +master-bin.000001 # Gtid # # XA START X'312d73746d74',X'',1 GTID #-#-# +master-bin.000001 # Query # # use `test`; INSERT INTO d1.t VALUES (1) +master-bin.000001 # Query # # XA END X'312d73746d74',X'',1 +master-bin.000001 # XA_prepare # # XA PREPARE X'312d73746d74',X'',1 +master-bin.000001 # Gtid # # XA START X'312d726f77',X'',1 GTID #-#-# +master-bin.000001 # Annotate_rows # # INSERT INTO d2.t VALUES (1) +master-bin.000001 # Table_map # # table_id: # (d2.t) +master-bin.000001 # Write_rows_v1 # # table_id: # flags: STMT_END_F +master-bin.000001 # Query # # XA END X'312d726f77',X'',1 +master-bin.000001 # XA_prepare # # XA PREPARE X'312d726f77',X'',1 +master-bin.000001 # Gtid # # XA START X'32',X'',1 GTID #-#-# +master-bin.000001 # Query # # use `test`; INSERT INTO d1.t VALUES (2) +master-bin.000001 # Query # # XA END X'32',X'',1 +master-bin.000001 # XA_prepare # # XA PREPARE X'32',X'',1 +master-bin.000001 # Gtid # # GTID #-#-# +master-bin.000001 # Query # # XA COMMIT X'32',X'',1 +master-bin.000001 # Gtid # # GTID #-#-# +master-bin.000001 # Query # # XA COMMIT X'312d726f77',X'',1 +master-bin.000001 # Gtid # # GTID #-#-# +master-bin.000001 # Query # # XA COMMIT X'312d73746d74',X'',1 +include/sync_slave_sql_with_master.inc +include/stop_slave.inc +connection master; +connect master2, 127.0.0.1,root,,test,$MASTER_MYPORT,; +connection master2; +SET @@session.binlog_format= statement; +XA START '3-stmt'; +INSERT INTO d1.t VALUES (3); +XA END '3-stmt'; +XA PREPARE '3-stmt'; +disconnect master2; +connect master2, 127.0.0.1,root,,test,$MASTER_MYPORT,; +connection master2; +SET @@session.binlog_format= row; +XA START '3-row'; +INSERT INTO d2.t VALUES (4); +XA END '3-row'; +XA PREPARE '3-row'; +disconnect master2; +connection master; +connect master2, 127.0.0.1,root,,test,$MASTER_MYPORT,; +connection master2; +XA START '4'; +SELECT * FROM d1.t; +a +1 +2 +XA END '4'; +XA PREPARE '4'; +disconnect master2; +connect master_bulk_conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'bulk_trx_10'; +INSERT INTO d1.t VALUES (10); +INSERT INTO d2.t VALUES (10); +XA END 'bulk_trx_10'; +XA PREPARE 'bulk_trx_10'; +disconnect master_bulk_conn10; +connection master; +connect master_bulk_conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'bulk_trx_9'; +INSERT INTO d1.t VALUES (9); +INSERT INTO d2.t VALUES (9); +XA END 'bulk_trx_9'; +XA PREPARE 'bulk_trx_9'; +disconnect master_bulk_conn9; +connection master; +connect master_bulk_conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'bulk_trx_8'; +INSERT INTO d1.t VALUES (8); +INSERT INTO d2.t VALUES (8); +XA END 'bulk_trx_8'; +XA PREPARE 'bulk_trx_8'; +disconnect master_bulk_conn8; +connection master; +connect master_bulk_conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'bulk_trx_7'; +INSERT INTO d1.t VALUES (7); +INSERT INTO d2.t VALUES (7); +XA END 'bulk_trx_7'; +XA PREPARE 'bulk_trx_7'; +disconnect master_bulk_conn7; +connection master; +connect master_bulk_conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'bulk_trx_6'; +INSERT INTO d1.t VALUES (6); +INSERT INTO d2.t VALUES (6); +XA END 'bulk_trx_6'; +XA PREPARE 'bulk_trx_6'; +disconnect master_bulk_conn6; +connection master; +connect master_bulk_conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'bulk_trx_5'; +INSERT INTO d1.t VALUES (5); +INSERT INTO d2.t VALUES (5); +XA END 'bulk_trx_5'; +XA PREPARE 'bulk_trx_5'; +disconnect master_bulk_conn5; +connection master; +connect master_bulk_conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'bulk_trx_4'; +INSERT INTO d1.t VALUES (4); +INSERT INTO d2.t VALUES (4); +XA END 'bulk_trx_4'; +XA PREPARE 'bulk_trx_4'; +disconnect master_bulk_conn4; +connection master; +connect master_bulk_conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'bulk_trx_3'; +INSERT INTO d1.t VALUES (3); +INSERT INTO d2.t VALUES (3); +XA END 'bulk_trx_3'; +XA PREPARE 'bulk_trx_3'; +disconnect master_bulk_conn3; +connection master; +connect master_bulk_conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'bulk_trx_2'; +INSERT INTO d1.t VALUES (2); +INSERT INTO d2.t VALUES (2); +XA END 'bulk_trx_2'; +XA PREPARE 'bulk_trx_2'; +disconnect master_bulk_conn2; +connection master; +connect master_bulk_conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START 'bulk_trx_1'; +INSERT INTO d1.t VALUES (1); +INSERT INTO d2.t VALUES (1); +XA END 'bulk_trx_1'; +XA PREPARE 'bulk_trx_1'; +disconnect master_bulk_conn1; +connection master; +connection slave; +include/start_slave.inc +connection master; +include/sync_slave_sql_with_master.inc +include/stop_slave.inc +connection master; +XA COMMIT 'bulk_trx_10'; +XA ROLLBACK 'bulk_trx_9'; +XA COMMIT 'bulk_trx_8'; +XA ROLLBACK 'bulk_trx_7'; +XA COMMIT 'bulk_trx_6'; +XA ROLLBACK 'bulk_trx_5'; +XA COMMIT 'bulk_trx_4'; +XA ROLLBACK 'bulk_trx_3'; +XA COMMIT 'bulk_trx_2'; +XA ROLLBACK 'bulk_trx_1'; +include/rpl_restart_server.inc [server_number=1] +connection slave; +include/start_slave.inc +connection master; +*** '3-stmt','3-row' xa-transactions must be in the list *** +XA RECOVER; +formatID gtrid_length bqual_length data +1 6 0 3-stmt +1 5 0 3-row +XA COMMIT '3-stmt'; +XA ROLLBACK '3-row'; +include/sync_slave_sql_with_master.inc +connection master; +connect master_conn2, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START '0123456789012345678901234567890123456789012345678901234567890124','0123456789012345678901234567890123456789012345678901234567890124',4294967292; +INSERT INTO d1.t VALUES (64); +XA END '0123456789012345678901234567890123456789012345678901234567890124','0123456789012345678901234567890123456789012345678901234567890124',4294967292; +XA PREPARE '0123456789012345678901234567890123456789012345678901234567890124','0123456789012345678901234567890123456789012345678901234567890124',4294967292; +disconnect master_conn2; +connection master; +connect master_conn3, 127.0.0.1,root,,test,$MASTER_MYPORT,; +XA START X'FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF',X'00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000',0; +INSERT INTO d1.t VALUES (0); +XA END X'FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF',X'00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000',0; +XA PREPARE X'FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF',X'00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000',0; +disconnect master_conn3; +connection master; +disconnect master_conn4; +connection master; +XA COMMIT '0123456789012345678901234567890123456789012345678901234567890124','0123456789012345678901234567890123456789012345678901234567890124',4294967292; +XA COMMIT X'FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF',X'00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000',0; +XA COMMIT 'RANDOM XID' +include/sync_slave_sql_with_master.inc +connect master_bulk_conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +connection master_bulk_conn10; +XA START 'one_phase_10'; +INSERT INTO d1.t VALUES (10); +INSERT INTO d2.t VALUES (10); +XA END 'one_phase_10'; +XA COMMIT 'one_phase_10' ONE PHASE; +disconnect master_bulk_conn10; +connect master_bulk_conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +connection master_bulk_conn9; +XA START 'one_phase_9'; +INSERT INTO d1.t VALUES (9); +INSERT INTO d2.t VALUES (9); +XA END 'one_phase_9'; +XA COMMIT 'one_phase_9' ONE PHASE; +disconnect master_bulk_conn9; +connect master_bulk_conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +connection master_bulk_conn8; +XA START 'one_phase_8'; +INSERT INTO d1.t VALUES (8); +INSERT INTO d2.t VALUES (8); +XA END 'one_phase_8'; +XA COMMIT 'one_phase_8' ONE PHASE; +disconnect master_bulk_conn8; +connect master_bulk_conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +connection master_bulk_conn7; +XA START 'one_phase_7'; +INSERT INTO d1.t VALUES (7); +INSERT INTO d2.t VALUES (7); +XA END 'one_phase_7'; +XA COMMIT 'one_phase_7' ONE PHASE; +disconnect master_bulk_conn7; +connect master_bulk_conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +connection master_bulk_conn6; +XA START 'one_phase_6'; +INSERT INTO d1.t VALUES (6); +INSERT INTO d2.t VALUES (6); +XA END 'one_phase_6'; +XA COMMIT 'one_phase_6' ONE PHASE; +disconnect master_bulk_conn6; +connect master_bulk_conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +connection master_bulk_conn5; +XA START 'one_phase_5'; +INSERT INTO d1.t VALUES (5); +INSERT INTO d2.t VALUES (5); +XA END 'one_phase_5'; +XA COMMIT 'one_phase_5' ONE PHASE; +disconnect master_bulk_conn5; +connect master_bulk_conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +connection master_bulk_conn4; +XA START 'one_phase_4'; +INSERT INTO d1.t VALUES (4); +INSERT INTO d2.t VALUES (4); +XA END 'one_phase_4'; +XA COMMIT 'one_phase_4' ONE PHASE; +disconnect master_bulk_conn4; +connect master_bulk_conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +connection master_bulk_conn3; +XA START 'one_phase_3'; +INSERT INTO d1.t VALUES (3); +INSERT INTO d2.t VALUES (3); +XA END 'one_phase_3'; +XA COMMIT 'one_phase_3' ONE PHASE; +disconnect master_bulk_conn3; +connect master_bulk_conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +connection master_bulk_conn2; +XA START 'one_phase_2'; +INSERT INTO d1.t VALUES (2); +INSERT INTO d2.t VALUES (2); +XA END 'one_phase_2'; +XA COMMIT 'one_phase_2' ONE PHASE; +disconnect master_bulk_conn2; +connect master_bulk_conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,; +connection master_bulk_conn1; +XA START 'one_phase_1'; +INSERT INTO d1.t VALUES (1); +INSERT INTO d2.t VALUES (1); +XA END 'one_phase_1'; +XA COMMIT 'one_phase_1' ONE PHASE; +disconnect master_bulk_conn1; +connection master; +include/sync_slave_sql_with_master.inc +include/diff_tables.inc [master:d1.t, slave:d1.t] +include/diff_tables.inc [master:d2.t, slave:d2.t] +connection master; +DELETE FROM d1.t; +DELETE FROM d2.t; +DROP TABLE d1.t, d2.t; +DROP DATABASE d1; +DROP DATABASE d2; +DROP VIEW v_processlist; +include/sync_slave_sql_with_master.inc +include/rpl_end.inc diff --git a/mysql-test/suite/rpl/r/rpl_xa_survive_disconnect_mixed_engines.result b/mysql-test/suite/rpl/r/rpl_xa_survive_disconnect_mixed_engines.result new file mode 100644 index 00000000000..09bfffc0da4 --- /dev/null +++ b/mysql-test/suite/rpl/r/rpl_xa_survive_disconnect_mixed_engines.result @@ -0,0 +1,373 @@ +include/master-slave.inc +[connection master] +connection master; +CALL mtr.add_suppression("Unsafe statement written to the binary log using statement format since BINLOG_FORMAT = STATEMENT"); +SET @@session.binlog_direct_non_transactional_updates := if(floor(rand()*10)%2,'ON','OFF'); +CREATE TABLE t (a INT) ENGINE=innodb; +CREATE TABLE tm (a INT) ENGINE=myisam; +=== COMMIT === +XA START 'xa_trx'; +INSERT INTO tm VALUES (1); +INSERT INTO t VALUES (1); +XA END 'xa_trx'; +XA PREPARE 'xa_trx'; +XA COMMIT 'xa_trx' ; +XA START 'xa_trx'; +INSERT INTO t VALUES (2); +INSERT INTO tm VALUES (2); +XA END 'xa_trx'; +XA PREPARE 'xa_trx'; +XA COMMIT 'xa_trx' ; +XA START 'xa_trx'; +INSERT INTO tm VALUES (3); +XA END 'xa_trx'; +XA PREPARE 'xa_trx'; +XA COMMIT 'xa_trx' ; +CREATE TEMPORARY TABLE tmp_i LIKE t; +CREATE TEMPORARY TABLE tmp_m LIKE tm; +XA START 'xa_trx'; +INSERT INTO t VALUES (4); +INSERT INTO tm VALUES (4); +INSERT INTO tmp_i VALUES (4); +INSERT INTO tmp_m VALUES (4); +INSERT INTO t SELECT * FROM tmp_i; +INSERT INTO tm SELECT * FROM tmp_m; +XA END 'xa_trx'; +XA PREPARE 'xa_trx'; +XA COMMIT 'xa_trx' ; +XA START 'xa_trx'; +INSERT INTO tmp_i VALUES (5); +INSERT INTO tmp_m VALUES (5); +INSERT INTO t SELECT * FROM tmp_i; +INSERT INTO tm SELECT * FROM tmp_m; +INSERT INTO t VALUES (5); +INSERT INTO tm VALUES (5); +XA END 'xa_trx'; +XA PREPARE 'xa_trx'; +XA COMMIT 'xa_trx' ; +DROP TEMPORARY TABLE tmp_i; +DROP TEMPORARY TABLE tmp_m; +XA START 'xa_trx'; +INSERT INTO t VALUES (6); +INSERT INTO tm VALUES (6); +CREATE TEMPORARY TABLE tmp_i LIKE t; +CREATE TEMPORARY TABLE tmp_m LIKE tm; +INSERT INTO tmp_i VALUES (6); +INSERT INTO tmp_m VALUES (6); +INSERT INTO t SELECT * FROM tmp_i; +INSERT INTO tm SELECT * FROM tmp_m; +XA END 'xa_trx'; +XA PREPARE 'xa_trx'; +XA COMMIT 'xa_trx' ; +DROP TEMPORARY TABLE tmp_i; +DROP TEMPORARY TABLE tmp_m; +XA START 'xa_trx'; +CREATE TEMPORARY TABLE tmp_i LIKE t; +CREATE TEMPORARY TABLE tmp_m LIKE tm; +INSERT INTO tmp_i VALUES (7); +INSERT INTO tmp_m VALUES (7); +INSERT INTO t SELECT * FROM tmp_i; +INSERT INTO tm SELECT * FROM tmp_m; +INSERT INTO t VALUES (7); +INSERT INTO tm VALUES (7); +XA END 'xa_trx'; +XA PREPARE 'xa_trx'; +XA COMMIT 'xa_trx' ; +XA START 'xa_trx'; +INSERT INTO t VALUES (8); +INSERT INTO tm VALUES (8); +INSERT INTO tmp_i VALUES (8); +INSERT INTO tmp_m VALUES (8); +INSERT INTO t SELECT * FROM tmp_i; +INSERT INTO tm SELECT * FROM tmp_m; +DROP TEMPORARY TABLE tmp_i; +DROP TEMPORARY TABLE tmp_m; +XA END 'xa_trx'; +XA PREPARE 'xa_trx'; +XA COMMIT 'xa_trx' ; +XA START 'xa_trx'; +UPDATE t SET a = 99 where a = -1; +XA END 'xa_trx'; +XA PREPARE 'xa_trx'; +XA COMMIT 'xa_trx' ; +include/sync_slave_sql_with_master.inc +connection master; +=== COMMIT ONE PHASE === +XA START 'xa_trx'; +INSERT INTO tm VALUES (1); +INSERT INTO t VALUES (1); +XA END 'xa_trx'; +XA COMMIT 'xa_trx' ONE PHASE; +XA START 'xa_trx'; +INSERT INTO t VALUES (2); +INSERT INTO tm VALUES (2); +XA END 'xa_trx'; +XA COMMIT 'xa_trx' ONE PHASE; +XA START 'xa_trx'; +INSERT INTO tm VALUES (3); +XA END 'xa_trx'; +XA COMMIT 'xa_trx' ONE PHASE; +CREATE TEMPORARY TABLE tmp_i LIKE t; +CREATE TEMPORARY TABLE tmp_m LIKE tm; +XA START 'xa_trx'; +INSERT INTO t VALUES (4); +INSERT INTO tm VALUES (4); +INSERT INTO tmp_i VALUES (4); +INSERT INTO tmp_m VALUES (4); +INSERT INTO t SELECT * FROM tmp_i; +INSERT INTO tm SELECT * FROM tmp_m; +XA END 'xa_trx'; +XA COMMIT 'xa_trx' ONE PHASE; +XA START 'xa_trx'; +INSERT INTO tmp_i VALUES (5); +INSERT INTO tmp_m VALUES (5); +INSERT INTO t SELECT * FROM tmp_i; +INSERT INTO tm SELECT * FROM tmp_m; +INSERT INTO t VALUES (5); +INSERT INTO tm VALUES (5); +XA END 'xa_trx'; +XA COMMIT 'xa_trx' ONE PHASE; +DROP TEMPORARY TABLE tmp_i; +DROP TEMPORARY TABLE tmp_m; +XA START 'xa_trx'; +INSERT INTO t VALUES (6); +INSERT INTO tm VALUES (6); +CREATE TEMPORARY TABLE tmp_i LIKE t; +CREATE TEMPORARY TABLE tmp_m LIKE tm; +INSERT INTO tmp_i VALUES (6); +INSERT INTO tmp_m VALUES (6); +INSERT INTO t SELECT * FROM tmp_i; +INSERT INTO tm SELECT * FROM tmp_m; +XA END 'xa_trx'; +XA COMMIT 'xa_trx' ONE PHASE; +DROP TEMPORARY TABLE tmp_i; +DROP TEMPORARY TABLE tmp_m; +XA START 'xa_trx'; +CREATE TEMPORARY TABLE tmp_i LIKE t; +CREATE TEMPORARY TABLE tmp_m LIKE tm; +INSERT INTO tmp_i VALUES (7); +INSERT INTO tmp_m VALUES (7); +INSERT INTO t SELECT * FROM tmp_i; +INSERT INTO tm SELECT * FROM tmp_m; +INSERT INTO t VALUES (7); +INSERT INTO tm VALUES (7); +XA END 'xa_trx'; +XA COMMIT 'xa_trx' ONE PHASE; +XA START 'xa_trx'; +INSERT INTO t VALUES (8); +INSERT INTO tm VALUES (8); +INSERT INTO tmp_i VALUES (8); +INSERT INTO tmp_m VALUES (8); +INSERT INTO t SELECT * FROM tmp_i; +INSERT INTO tm SELECT * FROM tmp_m; +DROP TEMPORARY TABLE tmp_i; +DROP TEMPORARY TABLE tmp_m; +XA END 'xa_trx'; +XA COMMIT 'xa_trx' ONE PHASE; +XA START 'xa_trx'; +UPDATE t SET a = 99 where a = -1; +XA END 'xa_trx'; +XA COMMIT 'xa_trx' ONE PHASE; +include/sync_slave_sql_with_master.inc +connection master; +=== ROLLBACK with PREPARE === +XA START 'xa_trx'; +INSERT INTO tm VALUES (1); +INSERT INTO t VALUES (1); +XA END 'xa_trx'; +XA PREPARE 'xa_trx'; +xa rollback 'xa_trx' ; +Warnings: +Warning 1196 Some non-transactional changed tables couldn't be rolled back +XA START 'xa_trx'; +INSERT INTO t VALUES (2); +INSERT INTO tm VALUES (2); +XA END 'xa_trx'; +XA PREPARE 'xa_trx'; +xa rollback 'xa_trx' ; +Warnings: +Warning 1196 Some non-transactional changed tables couldn't be rolled back +XA START 'xa_trx'; +INSERT INTO tm VALUES (3); +XA END 'xa_trx'; +XA PREPARE 'xa_trx'; +xa rollback 'xa_trx' ; +Warnings: +Warning 1196 Some non-transactional changed tables couldn't be rolled back +CREATE TEMPORARY TABLE tmp_i LIKE t; +CREATE TEMPORARY TABLE tmp_m LIKE tm; +XA START 'xa_trx'; +INSERT INTO t VALUES (4); +INSERT INTO tm VALUES (4); +INSERT INTO tmp_i VALUES (4); +INSERT INTO tmp_m VALUES (4); +INSERT INTO t SELECT * FROM tmp_i; +INSERT INTO tm SELECT * FROM tmp_m; +XA END 'xa_trx'; +XA PREPARE 'xa_trx'; +xa rollback 'xa_trx' ; +Warnings: +Warning 1196 Some non-transactional changed tables couldn't be rolled back +XA START 'xa_trx'; +INSERT INTO tmp_i VALUES (5); +INSERT INTO tmp_m VALUES (5); +INSERT INTO t SELECT * FROM tmp_i; +INSERT INTO tm SELECT * FROM tmp_m; +INSERT INTO t VALUES (5); +INSERT INTO tm VALUES (5); +XA END 'xa_trx'; +XA PREPARE 'xa_trx'; +xa rollback 'xa_trx' ; +Warnings: +Warning 1196 Some non-transactional changed tables couldn't be rolled back +DROP TEMPORARY TABLE tmp_i; +DROP TEMPORARY TABLE tmp_m; +XA START 'xa_trx'; +INSERT INTO t VALUES (6); +INSERT INTO tm VALUES (6); +CREATE TEMPORARY TABLE tmp_i LIKE t; +CREATE TEMPORARY TABLE tmp_m LIKE tm; +INSERT INTO tmp_i VALUES (6); +INSERT INTO tmp_m VALUES (6); +INSERT INTO t SELECT * FROM tmp_i; +INSERT INTO tm SELECT * FROM tmp_m; +XA END 'xa_trx'; +XA PREPARE 'xa_trx'; +xa rollback 'xa_trx' ; +Warnings: +Warning 1196 Some non-transactional changed tables couldn't be rolled back +DROP TEMPORARY TABLE tmp_i; +DROP TEMPORARY TABLE tmp_m; +XA START 'xa_trx'; +CREATE TEMPORARY TABLE tmp_i LIKE t; +CREATE TEMPORARY TABLE tmp_m LIKE tm; +INSERT INTO tmp_i VALUES (7); +INSERT INTO tmp_m VALUES (7); +INSERT INTO t SELECT * FROM tmp_i; +INSERT INTO tm SELECT * FROM tmp_m; +INSERT INTO t VALUES (7); +INSERT INTO tm VALUES (7); +XA END 'xa_trx'; +XA PREPARE 'xa_trx'; +xa rollback 'xa_trx' ; +Warnings: +Warning 1196 Some non-transactional changed tables couldn't be rolled back +XA START 'xa_trx'; +INSERT INTO t VALUES (8); +INSERT INTO tm VALUES (8); +INSERT INTO tmp_i VALUES (8); +INSERT INTO tmp_m VALUES (8); +INSERT INTO t SELECT * FROM tmp_i; +INSERT INTO tm SELECT * FROM tmp_m; +DROP TEMPORARY TABLE tmp_i; +DROP TEMPORARY TABLE tmp_m; +XA END 'xa_trx'; +XA PREPARE 'xa_trx'; +xa rollback 'xa_trx' ; +Warnings: +Warning 1196 Some non-transactional changed tables couldn't be rolled back +XA START 'xa_trx'; +UPDATE t SET a = 99 where a = -1; +XA END 'xa_trx'; +XA PREPARE 'xa_trx'; +xa rollback 'xa_trx' ; +include/sync_slave_sql_with_master.inc +connection master; +=== ROLLBACK with no PREPARE === +XA START 'xa_trx'; +INSERT INTO tm VALUES (1); +INSERT INTO t VALUES (1); +XA END 'xa_trx'; +xa rollback 'xa_trx' ; +Warnings: +Warning 1196 Some non-transactional changed tables couldn't be rolled back +XA START 'xa_trx'; +INSERT INTO t VALUES (2); +INSERT INTO tm VALUES (2); +XA END 'xa_trx'; +xa rollback 'xa_trx' ; +Warnings: +Warning 1196 Some non-transactional changed tables couldn't be rolled back +XA START 'xa_trx'; +INSERT INTO tm VALUES (3); +XA END 'xa_trx'; +xa rollback 'xa_trx' ; +Warnings: +Warning 1196 Some non-transactional changed tables couldn't be rolled back +CREATE TEMPORARY TABLE tmp_i LIKE t; +CREATE TEMPORARY TABLE tmp_m LIKE tm; +XA START 'xa_trx'; +INSERT INTO t VALUES (4); +INSERT INTO tm VALUES (4); +INSERT INTO tmp_i VALUES (4); +INSERT INTO tmp_m VALUES (4); +INSERT INTO t SELECT * FROM tmp_i; +INSERT INTO tm SELECT * FROM tmp_m; +XA END 'xa_trx'; +xa rollback 'xa_trx' ; +Warnings: +Warning 1196 Some non-transactional changed tables couldn't be rolled back +XA START 'xa_trx'; +INSERT INTO tmp_i VALUES (5); +INSERT INTO tmp_m VALUES (5); +INSERT INTO t SELECT * FROM tmp_i; +INSERT INTO tm SELECT * FROM tmp_m; +INSERT INTO t VALUES (5); +INSERT INTO tm VALUES (5); +XA END 'xa_trx'; +xa rollback 'xa_trx' ; +Warnings: +Warning 1196 Some non-transactional changed tables couldn't be rolled back +DROP TEMPORARY TABLE tmp_i; +DROP TEMPORARY TABLE tmp_m; +XA START 'xa_trx'; +INSERT INTO t VALUES (6); +INSERT INTO tm VALUES (6); +CREATE TEMPORARY TABLE tmp_i LIKE t; +CREATE TEMPORARY TABLE tmp_m LIKE tm; +INSERT INTO tmp_i VALUES (6); +INSERT INTO tmp_m VALUES (6); +INSERT INTO t SELECT * FROM tmp_i; +INSERT INTO tm SELECT * FROM tmp_m; +XA END 'xa_trx'; +xa rollback 'xa_trx' ; +Warnings: +Warning 1196 Some non-transactional changed tables couldn't be rolled back +DROP TEMPORARY TABLE tmp_i; +DROP TEMPORARY TABLE tmp_m; +XA START 'xa_trx'; +CREATE TEMPORARY TABLE tmp_i LIKE t; +CREATE TEMPORARY TABLE tmp_m LIKE tm; +INSERT INTO tmp_i VALUES (7); +INSERT INTO tmp_m VALUES (7); +INSERT INTO t SELECT * FROM tmp_i; +INSERT INTO tm SELECT * FROM tmp_m; +INSERT INTO t VALUES (7); +INSERT INTO tm VALUES (7); +XA END 'xa_trx'; +xa rollback 'xa_trx' ; +Warnings: +Warning 1196 Some non-transactional changed tables couldn't be rolled back +XA START 'xa_trx'; +INSERT INTO t VALUES (8); +INSERT INTO tm VALUES (8); +INSERT INTO tmp_i VALUES (8); +INSERT INTO tmp_m VALUES (8); +INSERT INTO t SELECT * FROM tmp_i; +INSERT INTO tm SELECT * FROM tmp_m; +DROP TEMPORARY TABLE tmp_i; +DROP TEMPORARY TABLE tmp_m; +XA END 'xa_trx'; +xa rollback 'xa_trx' ; +Warnings: +Warning 1196 Some non-transactional changed tables couldn't be rolled back +XA START 'xa_trx'; +UPDATE t SET a = 99 where a = -1; +XA END 'xa_trx'; +xa rollback 'xa_trx' ; +include/sync_slave_sql_with_master.inc +include/diff_tables.inc [master:tm, slave:tm] +connection master; +DROP TABLE t, tm; +include/sync_slave_sql_with_master.inc +include/rpl_end.inc diff --git a/mysql-test/suite/rpl/t/rpl_parallel_optimistic_xa.test b/mysql-test/suite/rpl/t/rpl_parallel_optimistic_xa.test new file mode 100644 index 00000000000..35c22d1e92e --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_parallel_optimistic_xa.test @@ -0,0 +1,235 @@ +# The tests verify concurrent execution of replicated (MDEV-742) +# XA transactions in the parallel optimistic mode. + +--source include/have_innodb.inc +--source include/have_perfschema.inc +--source include/master-slave.inc + +# Tests' global declarations +--let $trx = _trx_ + +call mtr.add_suppression("Deadlock found when trying to get lock; try restarting transaction"); +call mtr.add_suppression("WSREP: handlerton rollback failed"); +#call mtr.add_suppression("Can't find record in 't1'"); +CREATE VIEW v_processlist as SELECT * FROM performance_schema.threads where type = 'FOREGROUND'; + +--connection master +ALTER TABLE mysql.gtid_slave_pos ENGINE=InnoDB; +--save_master_pos + +# Prepare to restart slave into optimistic parallel mode +--connection slave +--sync_with_master +--source include/stop_slave.inc +SET @old_parallel_threads = @@GLOBAL.slave_parallel_threads; +SET @@global.slave_parallel_threads = 7; +SET @old_parallel_mode = @@GLOBAL.slave_parallel_mode; +SET @@global.slave_parallel_mode ='optimistic'; +# Run the first part of the test with high batch size and see that +# old rows remain in the table. +SET @old_gtid_cleanup_batch_size = @@GLOBAL.gtid_cleanup_batch_size; +SET @@global.gtid_cleanup_batch_size = 1000000; + +CHANGE MASTER TO master_use_gtid=slave_pos; + +# LOAD GENERATOR creates XA:s interleaved in binlog when they are from +# different connections. All the following block XA:s of the same connection +# update the same data which challenges slave optimistic scheduler's correctness. +# Slave must eventually apply such load, and correctly (checked). + +--connection master +CREATE TABLE t0 (a int, b INT) ENGINE=InnoDB; +CREATE TABLE t1 (a int PRIMARY KEY, b INT) ENGINE=InnoDB; +INSERT INTO t1 VALUES (1, 0); + + +# I. Logging some sequence of XA:s by one connection. +# +# The slave applier's task is to successfully execute a series of +# Prepare and Complete parts of a sequence of XA:s + +--let $trx_num = 300 +--let $i = $trx_num +--let $conn = master +--disable_query_log +while($i > 0) +{ + # 'decision' to commit 0, or rollback 1 + --let $decision = `SELECT $i % 2` + --eval XA START '$conn$trx$i' + --eval UPDATE t1 SET b = 1 - 2 * $decision WHERE a = 1 + --eval XA END '$conn$trx$i' + --let $one_phase = `SELECT IF(floor(rand()*10)%2, "ONE PHASE", 0)` + if (!$one_phase) + { + --eval XA PREPARE '$conn$trx$i' + --let $one_phase = + } + + --let $term = COMMIT + if ($decision) + { + --let $term = ROLLBACK + --let $one_phase = + } + --eval XA $term '$conn$trx$i' $one_phase + + --dec $i +} +--enable_query_log +--source include/save_master_gtid.inc + +--connection slave +--source include/start_slave.inc +--source include/sync_with_master_gtid.inc +--source include/stop_slave.inc + + +# II. Logging XS:s from multiple connections in random interweaving manner: +# +# in a loop ($i) per connection +# arrange an inner ($k) loop where +# start and prepare an XA; +# decide whether to terminate it and then continue to loop innerly +# OR disconnect to break the inner loop; +# the disconnected one's XA is taken care by 'master' connection +# +# Effectively binlog must collect a well mixed XA- prepared and terminated +# groups for slave to handle. + +--connection master +# Total # of connections +--let $conn_num=53 + +--let $i = $conn_num +--disable_query_log +while($i > 0) +{ + --connect (master_conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,) +--dec $i +} +--enable_query_log + +--let $i = $conn_num +while($i > 0) +{ + --let $conn_i = conn$i + # $i2 indexes the current connection's "own" row + --let $i2 = `SELECT $i + 2` +--disable_query_log + --connection master_conn$i +--enable_query_log + --disable_query_log + --let $i_conn_id = `SELECT connection_id()` + + --let $decision = 0 + # the row id of the last connection that committed its XA + --let $c_max = 1 + --let $k = 0 + while ($decision < 3) + { + --inc $k + --eval XA START '$conn_i$trx$k' + # UPDATE depends on previously *committed* transactions + --eval UPDATE t1 SET b = b + $k + 1 WHERE a = $c_max + if (`SELECT $k % 2 = 1`) + { + --eval REPLACE INTO t1 VALUES ($i2, $k) + } + if (`SELECT $k % 2 = 0`) + { + --eval DELETE FROM t1 WHERE a = $i2 + } + CREATE TEMPORARY TABLE tmp LIKE t0; + --eval INSERT INTO tmp SET a=$i, b= $k + INSERT INTO t0 SELECT * FROM tmp; + DROP TEMPORARY TABLE tmp; + --eval XA END '$conn_i$trx$k' + + --let $term = COMMIT + --let $decision = `SELECT (floor(rand()*10 % 10) + ($i+$k)) % 4` + if ($decision == 1) + { + --let $term = ROLLBACK + } + if ($decision < 2) + { + --eval XA PREPARE '$conn_i$trx$k' + --eval XA $term '$conn_i$trx$k' + # Iteration counter is taken care *now* + } + if ($decision == 2) + { + --eval XA COMMIT '$conn_i$trx$k' ONE PHASE + } + } + + # $decision = 3 + --eval XA PREPARE '$conn_i$trx$k' + # disconnect now + --disconnect master_conn$i + --connection master + + --let $wait_condition= SELECT count(*) = 0 FROM v_processlist WHERE PROCESSLIST_ID = $i_conn_id + --source include/wait_condition.inc + + --disable_query_log + --let $decision = `SELECT ($i+$k) % 2` + --let $term = COMMIT + if ($decision == 1) + { + --let $term = ROLLBACK + } + --eval XA $term '$conn_i$trx$k' + --let $c_max = $i2 + +--dec $i +} +--enable_query_log +--source include/save_master_gtid.inc + +--connection slave +--source include/start_slave.inc +--source include/sync_with_master_gtid.inc + +# +# Overall consistency check +# +--let $diff_tables= master:t0, slave:t0 +--source include/diff_tables.inc +--let $diff_tables= master:t1, slave:t1 +--source include/diff_tables.inc + + +# +# Clean up. +# +--connection slave +--source include/stop_slave.inc +set global log_warnings=default; +SET GLOBAL slave_parallel_mode=@old_parallel_mode; +SET GLOBAL slave_parallel_threads=@old_parallel_threads; +--source include/start_slave.inc + +--connection master +DROP VIEW v_processlist; +DROP TABLE t0, t1; +--source include/save_master_gtid.inc + +--connection slave +--source include/sync_with_master_gtid.inc +# Check that old rows are deleted from mysql.gtid_slave_pos. +# Deletion is asynchronous, so use wait_condition.inc. +# Also, there is a small amount of non-determinism in the deletion of old +# rows, so it is not guaranteed that there can never be more than +# @@gtid_cleanup_batch_size rows in the table; so allow a bit of slack +# here. +let $wait_condition= + SELECT COUNT(*) <= 5*@@GLOBAL.gtid_cleanup_batch_size + FROM mysql.gtid_slave_pos; +--source include/wait_condition.inc +eval $wait_condition; +SET GLOBAL gtid_cleanup_batch_size= @old_gtid_cleanup_batch_size; + +--connection master +--source include/rpl_end.inc diff --git a/mysql-test/suite/rpl/t/rpl_parallel_optimistic_xa_lsu_off-slave.opt b/mysql-test/suite/rpl/t/rpl_parallel_optimistic_xa_lsu_off-slave.opt new file mode 100644 index 00000000000..88cf77fd281 --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_parallel_optimistic_xa_lsu_off-slave.opt @@ -0,0 +1 @@ +--log-slave-updates=OFF diff --git a/mysql-test/suite/rpl/t/rpl_parallel_optimistic_xa_lsu_off.test b/mysql-test/suite/rpl/t/rpl_parallel_optimistic_xa_lsu_off.test new file mode 100644 index 00000000000..f82b522eefe --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_parallel_optimistic_xa_lsu_off.test @@ -0,0 +1,2 @@ +# --log-slave-updates OFF version of rpl_parallel_optimistic_xa +--source rpl_parallel_optimistic_xa.test diff --git a/mysql-test/suite/rpl/t/rpl_parallel_xa_same_xid.test b/mysql-test/suite/rpl/t/rpl_parallel_xa_same_xid.test new file mode 100644 index 00000000000..888dd2f177b --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_parallel_xa_same_xid.test @@ -0,0 +1,138 @@ +# The tests verify concurrent execution of replicated (MDEV-742) +# XA transactions in the parallel optimistic mode. +# Prove optimistic scheduler handles xid-namesake XA:s. +# That is despite running in parallel there must be no conflicts +# caused by multiple transactions' same xid. + +--source include/have_binlog_format_mixed_or_row.inc +--source include/have_innodb.inc +--source include/have_perfschema.inc +--source include/master-slave.inc + +--let $xid_num = 19 +--let $repeat = 17 +--let $workers = 7 +--connection slave +call mtr.add_suppression("WSREP: handlerton rollback failed"); + +--source include/stop_slave.inc +# a measure against MDEV-20605 +ALTER TABLE mysql.gtid_slave_pos ENGINE=InnoDB; + +SET @old_parallel_threads = @@GLOBAL.slave_parallel_threads; +--disable_query_log +--eval SET @@global.slave_parallel_threads = $workers +--enable_query_log +SET @old_parallel_mode = @@GLOBAL.slave_parallel_mode; +SET @@global.slave_parallel_mode ='optimistic'; +--source include/start_slave.inc + +--connection master +CREATE TABLE t1 (a INT, b INT) ENGINE=InnoDB; + +--let $i = $xid_num +--let $t = t1 +--disable_query_log +while ($i) +{ +--let $k = $repeat +while ($k) +{ +--eval XA START 'xid_$i' +--eval INSERT INTO $t SET a=$i, b=$k +--eval XA END 'xid_$i' +--let $one_phase = `SELECT IF(floor(rand()*10)%2, "ONE PHASE", 0)` + if (!$one_phase) + { + --eval XA PREPARE 'xid_$i' + --eval XA COMMIT 'xid_$i' + } + if ($one_phase) + { + --eval XA COMMIT 'xid_$i' ONE PHASE + } + + if (!$one_phase) + { + --eval XA START 'xid_$i' + --eval INSERT INTO $t SET a=$i, b=$k + --eval XA END 'xid_$i' + --eval XA PREPARE 'xid_$i' + --eval XA ROLLBACK 'xid_$i' + } + +--dec $k +} + +--dec $i +} +--enable_query_log + + + +# Above-like test complicates execution env to create +# data conflicts as well. They will be resolved by the optmistic +# scheduler as usual. + +CREATE TABLE t2 (a INT AUTO_INCREMENT PRIMARY KEY, b INT) ENGINE=InnoDB; + +--let $i = $xid_num +--let $t = t2 +--disable_query_log +while ($i) +{ +--let $k = $repeat +while ($k) +{ +--eval XA START 'xid_$i' +--eval INSERT INTO $t SET a=NULL, b=$k +--eval UPDATE $t SET b=$k + 1 WHERE a=last_insert_id() % $workers +--eval XA END 'xid_$i' +--let $one_phase = `SELECT IF(floor(rand()*10)%2, "ONE PHASE", 0)` + if (!$one_phase) + { + --eval XA PREPARE 'xid_$i' + --eval XA COMMIT 'xid_$i' + } + if ($one_phase) + { + --eval XA COMMIT 'xid_$i' ONE PHASE + } + +--eval XA START 'xid_$i' +--eval UPDATE $t SET b=$k + 1 WHERE a=last_insert_id() % $workers +--eval DELETE FROM $t WHERE a=last_insert_id() +--eval XA END 'xid_$i' +--eval XA PREPARE 'xid_$i' +--eval XA ROLLBACK 'xid_$i' + +--let $do_drop_create = `SELECT IF(floor(rand()*10)%100, 1, 0)` +if ($do_drop_create) +{ + DROP TABLE t1; + CREATE TABLE t1 (a INT, b INT) ENGINE=InnoDB; +} +--dec $k +} + +--dec $i +} +--enable_query_log + +--source include/sync_slave_sql_with_master.inc +--let $diff_tables= master:t1, slave:t1 +--source include/diff_tables.inc + +# +# Clean up. +# +--connection slave +--source include/stop_slave.inc +SET GLOBAL slave_parallel_threads=@old_parallel_threads; +SET GLOBAL slave_parallel_mode=@old_parallel_mode; +--source include/start_slave.inc + +--connection master +DROP TABLE t1, t2; + +--source include/rpl_end.inc diff --git a/mysql-test/suite/rpl/t/rpl_temporary_errors.test b/mysql-test/suite/rpl/t/rpl_temporary_errors.test index 6392fb90b9b..85e16afa270 100644 --- a/mysql-test/suite/rpl/t/rpl_temporary_errors.test +++ b/mysql-test/suite/rpl/t/rpl_temporary_errors.test @@ -6,7 +6,7 @@ call mtr.add_suppression("Deadlock found"); call mtr.add_suppression("Can't find record in 't.'"); connection master; -CREATE TABLE t1 (a INT PRIMARY KEY, b INT); +CREATE TABLE t1 (a INT PRIMARY KEY, b INT) ENGINE=innodb; INSERT INTO t1 VALUES (1,1), (2,2), (3,3), (4,4); sync_slave_with_master; SHOW STATUS LIKE 'Slave_retried_transactions'; @@ -14,20 +14,94 @@ SHOW STATUS LIKE 'Slave_retried_transactions'; # the following UPDATE t1 to pass the mode is switched temprorarily set @@global.slave_exec_mode= 'IDEMPOTENT'; UPDATE t1 SET a = 5, b = 47 WHERE a = 1; -SELECT * FROM t1; +SELECT * FROM t1 ORDER BY a; connection master; UPDATE t1 SET a = 5, b = 5 WHERE a = 1; -SELECT * FROM t1; +SELECT * FROM t1 ORDER BY a; #SHOW BINLOG EVENTS; sync_slave_with_master; set @@global.slave_exec_mode= default; SHOW STATUS LIKE 'Slave_retried_transactions'; -SELECT * FROM t1; +SELECT * FROM t1 ORDER BY a; source include/check_slave_is_running.inc; connection slave; call mtr.add_suppression("Slave SQL.*Could not execute Update_rows event on table test.t1"); +call mtr.add_suppression("Slave SQL for channel '': worker thread retried transaction"); +call mtr.add_suppression("The slave coordinator and worker threads are stopped"); +# +# Bug#24764800 REPLICATION FAILING ON SLAVE WITH XAER_RMFAIL ERROR +# +# Verify that a temporary failing replicated xa transaction completes +# upon slave applier restart after previous +# @@global.slave_transaction_retries number of retries in vain. +# +connection slave; + +set @save_innodb_lock_wait_timeout=@@global.innodb_lock_wait_timeout; +set @save_slave_transaction_retries=@@global.slave_transaction_retries; + +# Slave applier parameters for the failed retry +set @@global.innodb_lock_wait_timeout=1; +set @@global.slave_transaction_retries=2; +--source include/restart_slave_sql.inc + +# Temporary error implement: a record is blocked by slave local trx +connection slave1; +BEGIN; +INSERT INTO t1 SET a = 6, b = 7; + +connection master; +INSERT INTO t1 SET a = 99, b = 99; # slave applier warm up trx +XA START 'xa1'; +INSERT INTO t1 SET a = 6, b = 6; # this record eventually must be found on slave +XA END 'xa1'; +XA PREPARE 'xa1'; + +connection slave; +# convert_error(ER_LOCK_WAIT_TIMEOUT) +--let $err_timeout= 1205 +# convert_error(ER_LOCK_DEADLOCK) +--let $err_deadlock= 1213 +--let $slave_sql_errno=$err_deadlock,$err_timeout +--let $show_slave_sql_error= +--source include/wait_for_slave_sql_error.inc + +# b. Slave applier parameters for successful retry after restart +set @@global.innodb_lock_wait_timeout=1; +set @@global.slave_transaction_retries=100; + +--source include/restart_slave_sql.inc + +--let $last_retries= query_get_value(SHOW GLOBAL STATUS LIKE 'Slave_retried_transactions', Value, 1) +--let $status_type=GLOBAL +--let $status_var=Slave_retried_transactions +--let $status_var_value=`SELECT 1 + $last_retries` +--let $$status_var_comparsion= > +--source include/wait_for_status_var.inc + +# Release the record after just one retry +connection slave1; +ROLLBACK; + +connection master; +XA COMMIT 'xa1'; + +--source include/sync_slave_sql_with_master.inc + +# Proof of correctness: the committed XA is on the slave +connection slave; +--let $assert_text=XA transaction record must be in the table +--let $assert_cond=count(*)=1 FROM t1 WHERE a=6 AND b=6 +--source include/assert.inc + +# Bug#24764800 cleanup: +set @@global.innodb_lock_wait_timeout=@save_innodb_lock_wait_timeout; +set @@global.slave_transaction_retries= @save_slave_transaction_retries; +# +# Total cleanup: +# connection master; DROP TABLE t1; --sync_slave_with_master diff --git a/mysql-test/suite/rpl/t/rpl_xa.inc b/mysql-test/suite/rpl/t/rpl_xa.inc new file mode 100644 index 00000000000..f1ba4cf8557 --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_xa.inc @@ -0,0 +1,73 @@ +# +# This "body" file checks general properties of XA transaction replication +# as of MDEV-7974. +# Parameters: +# --let rpl_xa_check= SELECT ... +# +connection master; +create table t1 (a int, b int) engine=InnoDB; +insert into t1 values(0, 0); +xa start 't'; +insert into t1 values(1, 2); +xa end 't'; +xa prepare 't'; +xa commit 't'; + +sync_slave_with_master; +let $diff_tables= master:t1, slave:t1; +source include/diff_tables.inc; + +connection master; + +xa start 't'; +insert into t1 values(3, 4); +xa end 't'; +xa prepare 't'; +xa rollback 't'; + +sync_slave_with_master; +let $diff_tables= master:t1, slave:t1; +source include/diff_tables.inc; + +connection master; +--disable_warnings +SET pseudo_slave_mode=1; +--enable_warnings +create table t2 (a int) engine=InnoDB; +xa start 't'; +insert into t1 values (5, 6); +xa end 't'; +xa prepare 't'; +xa start 's'; +insert into t2 values (0); +xa end 's'; +xa prepare 's'; +--source include/save_master_gtid.inc + +connection slave; +source include/sync_with_master_gtid.inc; +if ($rpl_xa_check) +{ + --eval $rpl_xa_check + if ($rpl_xa_verbose) + { + --eval SELECT $rpl_xa_check_lhs + --eval SELECT $rpl_xa_check_rhs + } +} +xa recover; + +connection master; +xa commit 't'; +xa commit 's'; +--disable_warnings +SET pseudo_slave_mode=0; +--enable_warnings +sync_slave_with_master; +let $diff_tables= master:t1, slave:t1; +source include/diff_tables.inc; +let $diff_tables= master:t2, slave:t2; +source include/diff_tables.inc; + +connection master; +drop table t1, t2; diff --git a/mysql-test/suite/rpl/t/rpl_xa.test b/mysql-test/suite/rpl/t/rpl_xa.test new file mode 100644 index 00000000000..05a1abe59ae --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_xa.test @@ -0,0 +1,5 @@ +source include/have_innodb.inc; +source include/master-slave.inc; + +source rpl_xa.inc; +source include/rpl_end.inc; diff --git a/mysql-test/suite/rpl/t/rpl_xa_gap_lock-slave.opt b/mysql-test/suite/rpl/t/rpl_xa_gap_lock-slave.opt new file mode 100644 index 00000000000..4602a43ce25 --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_xa_gap_lock-slave.opt @@ -0,0 +1 @@ +--transaction-isolation=READ-COMMITTED diff --git a/mysql-test/suite/rpl/t/rpl_xa_gap_lock.test b/mysql-test/suite/rpl/t/rpl_xa_gap_lock.test new file mode 100644 index 00000000000..9c48891b889 --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_xa_gap_lock.test @@ -0,0 +1,137 @@ +# ==== Purpose ==== +# +# This test will generate two XA transactions on the master in a way that +# they will block each other on the slave if the transaction isolation level +# used by the slave applier is more restrictive than the READ COMMITTED one. +# +# Consider: +# E=execute, P=prepare, C=commit; +# 1=first transaction, 2=second transaction; +# +# Master does: E1, E2, P2, P1, C1, C2 +# Slave does: E2, P2, E1, P1, C1, C2 +# +# The transactions are designed so that, if the applier transaction isolation +# level is more restrictive than the READ COMMITTED, E1 will be blocked on +# the slave waiting for gap locks to be released. +# +# Step 1 +# +# The test will verify that the transactions don't block each other because +# the applier thread automatically changed the isolation level. +# +# Step 2 +# +# The test will verify that applying master's binary log dump in slave doesn't +# block because mysqlbinlog is informing the isolation level to be used. +# +# ==== Related Bugs and Worklogs ==== +# +# BUG#25040331: INTERLEAVED XA TRANSACTIONS MAY DEADLOCK SLAVE APPLIER WITH +# REPEATABLE READ +# +--source include/have_debug.inc +--source include/have_innodb.inc +# The test case only make sense for RBR +--source include/have_binlog_format_row.inc +--source include/master-slave.inc + +--connection slave +# To hit the issue, we need to split the data in two pages. +# This global variable will help us. +SET @saved_innodb_limit_optimistic_insert_debug = @@GLOBAL.innodb_limit_optimistic_insert_debug; +SET @@GLOBAL.innodb_limit_optimistic_insert_debug = 2; + +# +# Step 1 - Using async replication +# + +# Let's generate the workload on the master +--connection master +CREATE TABLE t1 ( + c1 INT NOT NULL, + KEY(c1) +) ENGINE=InnoDB; + +CREATE TABLE t2 ( + c1 INT NOT NULL, + FOREIGN KEY(c1) REFERENCES t1(c1) +) ENGINE=InnoDB; + +INSERT INTO t1 VALUES (1), (3), (4); + +--connection master1 +XA START 'XA1'; +INSERT INTO t1 values(2); +XA END 'XA1'; + +# This transaction will reference the gap where XA1 +# was inserted, and will be prepared and committed +# before XA1, so the slave will prepare it (but will +# not commit it) before preparing XA1. +--connection master +XA START 'XA2'; +INSERT INTO t2 values(3); +XA END 'XA2'; + +# The XA2 prepare should be binary logged first +XA PREPARE 'XA2'; + +# The XA1 prepare should be binary logged +# after XA2 prepare and before XA2 commit. +--connection master1 +XA PREPARE 'XA1'; + +# The commit order doesn't matter much for the issue being tested. +XA COMMIT 'XA1'; +--connection master +XA COMMIT 'XA2'; + +# Everything is fine if the slave can sync with the master. +--source include/sync_slave_sql_with_master.inc + +# +# Step 2 - Using mysqlbinlog dump to restore the salve +# +--source include/stop_slave.inc +DROP TABLE t2, t1; +RESET SLAVE; +RESET MASTER; + +--connection master +--let $master_data_dir= `SELECT @@datadir` +--let $master_log_file= query_get_value(SHOW MASTER STATUS, File, 1) +--let $mysql_server= $MYSQL --defaults-group-suffix=.2 +--echo Restore binary log from the master into the slave +--exec $MYSQL_BINLOG --force-if-open $master_data_dir/$master_log_file | $mysql_server + +--let $diff_tables= master:test.t1, slave:test.t1 +--source include/diff_tables.inc +--let $diff_tables= master:test.t2, slave:test.t2 +--source include/diff_tables.inc + +# +# Cleanup +# +--let $master_file= query_get_value(SHOW MASTER STATUS, File, 1) +--let $master_pos= query_get_value(SHOW MASTER STATUS, Position, 1) +DROP TABLE t2, t1; + +## When GTID_MODE=OFF, we need to skip already applied transactions +--connection slave +#--let $gtid_mode= `SELECT @@GTID_MODE` +#if ($gtid_mode == OFF) +#{ +# --disable_query_log +# --disable_result_log +# --eval CHANGE MASTER TO MASTER_LOG_FILE='$master_file', MASTER_LOG_POS=$master_pos +# --enable_result_log +# --enable_query_log +#} +--replace_result $master_file LOG_FILE $master_pos LOG_POS +--eval CHANGE MASTER TO MASTER_LOG_FILE='$master_file', MASTER_LOG_POS=$master_pos + +SET @@GLOBAL.innodb_limit_optimistic_insert_debug = @saved_innodb_limit_optimistic_insert_debug; +--source include/start_slave.inc + +--source include/rpl_end.inc diff --git a/mysql-test/suite/rpl/t/rpl_xa_gtid_pos_auto_engine.test b/mysql-test/suite/rpl/t/rpl_xa_gtid_pos_auto_engine.test new file mode 100644 index 00000000000..b83493762c3 --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_xa_gtid_pos_auto_engine.test @@ -0,0 +1,29 @@ +--source include/have_innodb.inc +--source include/master-slave.inc + +--connection slave +call mtr.add_suppression("The automatically created table.*name may not be entirely in lowercase"); + +--source include/stop_slave.inc +CHANGE MASTER TO master_use_gtid=slave_pos; + +SET @@global.gtid_pos_auto_engines="innodb"; +--source include/start_slave.inc +--let $rpl_xa_check_lhs= @@global.gtid_slave_pos +--let $rpl_xa_check_rhs= CONCAT(domain_id,"-",server_id,"-",seq_no) FROM mysql.gtid_slave_pos WHERE seq_no = (SELECT DISTINCT max(seq_no) FROM mysql.gtid_slave_pos) +--let $rpl_xa_check=SELECT $rpl_xa_check_lhs = $rpl_xa_check_rhs +--source rpl_xa.inc + +--connection slave +--source include/stop_slave.inc +SET @@global.gtid_pos_auto_engines=""; +SET @@session.sql_log_bin=0; +DROP TABLE mysql.gtid_slave_pos_InnoDB; +if (`SHOW COUNT(*) WARNINGS`) +{ + show tables in mysql like 'gtid_slave_pos%'; +} +SET @@session.sql_log_bin=1; +--source include/start_slave.inc + +--source include/rpl_end.inc diff --git a/mysql-test/suite/rpl/t/rpl_xa_survive_disconnect.test b/mysql-test/suite/rpl/t/rpl_xa_survive_disconnect.test new file mode 100644 index 00000000000..1c33435473e --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_xa_survive_disconnect.test @@ -0,0 +1,294 @@ +# BUG #12161 Xa recovery and client disconnection +# the test verifies that +# a. disconnection does not lose a prepared transaction +# so it can be committed from another connection +# c. the prepared transaction is logged +# d. interleaved prepared transactions are correctly applied on the slave. + +# +# Both replication format are checked through explict +# set @@binlog_format in the test. +# +--source include/have_innodb.inc +--source include/have_binlog_format_mixed.inc +# +# Prepared XA can't get available to an external connection +# until a connection, that either leaves actively or is killed, +# has completed a necessary part of its cleanup. +# Selecting from P_S.threads provides a method to learn that. +# +--source include/have_perfschema.inc +--source include/master-slave.inc + +--connection master +call mtr.add_suppression("Found 2 prepared XA transactions"); +CREATE VIEW v_processlist as SELECT * FROM performance_schema.threads where type = 'FOREGROUND'; + +CREATE DATABASE d1; +CREATE DATABASE d2; + +CREATE TABLE d1.t (a INT) ENGINE=innodb; +CREATE TABLE d2.t (a INT) ENGINE=innodb; + +connect (master_conn1, 127.0.0.1,root,,test,$MASTER_MYPORT,); +--let $conn_id=`SELECT connection_id()` +SET @@session.binlog_format= statement; +XA START '1-stmt'; +INSERT INTO d1.t VALUES (1); +XA END '1-stmt'; +XA PREPARE '1-stmt'; + +--disconnect master_conn1 + +--connection master + +--let $wait_condition= SELECT count(*) = 0 FROM v_processlist WHERE PROCESSLIST_ID = $conn_id +--source include/wait_condition.inc + +connect (master_conn2, 127.0.0.1,root,,test,$MASTER_MYPORT,); +--let $conn_id=`SELECT connection_id()` +SET @@session.binlog_format= row; +XA START '1-row'; +INSERT INTO d2.t VALUES (1); +XA END '1-row'; +XA PREPARE '1-row'; + +--disconnect master_conn2 + +--connection master +--let $wait_condition= SELECT count(*) = 0 FROM v_processlist WHERE PROCESSLIST_ID = $conn_id +--source include/wait_condition.inc + +XA START '2'; +INSERT INTO d1.t VALUES (2); +XA END '2'; +XA PREPARE '2'; +XA COMMIT '2'; + +XA COMMIT '1-row'; +XA COMMIT '1-stmt'; +source include/show_binlog_events.inc; + +# the proof: slave is in sync with the table updated by the prepared transactions. +--source include/sync_slave_sql_with_master.inc + +--source include/stop_slave.inc + +# +# Recover with Master server restart +# +--connection master + +connect (master2, 127.0.0.1,root,,test,$MASTER_MYPORT,); +--connection master2 +SET @@session.binlog_format= statement; +XA START '3-stmt'; +INSERT INTO d1.t VALUES (3); +XA END '3-stmt'; +XA PREPARE '3-stmt'; +--disconnect master2 + +connect (master2, 127.0.0.1,root,,test,$MASTER_MYPORT,); +--connection master2 +SET @@session.binlog_format= row; +XA START '3-row'; +INSERT INTO d2.t VALUES (4); +XA END '3-row'; +XA PREPARE '3-row'; +--disconnect master2 + +--connection master + +# +# Testing read-only +# +connect (master2, 127.0.0.1,root,,test,$MASTER_MYPORT,); +--connection master2 +XA START '4'; +SELECT * FROM d1.t; +XA END '4'; +XA PREPARE '4'; +--disconnect master2 + +# +# Logging few disconnected XA:s for replication. +# +--let $bulk_trx_num=10 +--let $i = $bulk_trx_num + +while($i > 0) +{ + --connect (master_bulk_conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,) + --let $conn_id=`SELECT connection_id()` + + --eval XA START 'bulk_trx_$i' + --eval INSERT INTO d1.t VALUES ($i) + --eval INSERT INTO d2.t VALUES ($i) + --eval XA END 'bulk_trx_$i' + --eval XA PREPARE 'bulk_trx_$i' + + --disconnect master_bulk_conn$i + + --connection master + --let $wait_condition= SELECT count(*) = 0 FROM v_processlist WHERE PROCESSLIST_ID = $conn_id + --source include/wait_condition.inc + + --dec $i +} + +# +# Prove the slave applier is capable to resume the prepared XA:s +# upon its restart. +# +--connection slave +--source include/start_slave.inc +--connection master +--source include/sync_slave_sql_with_master.inc +--source include/stop_slave.inc + +--connection master +--let $i = $bulk_trx_num +while($i > 0) +{ + --let $command=COMMIT + if (`SELECT $i % 2`) + { + --let $command=ROLLBACK + } + --eval XA $command 'bulk_trx_$i' + --dec $i +} + +--let $rpl_server_number= 1 +--source include/rpl_restart_server.inc + +--connection slave +--source include/start_slave.inc + +--connection master +--echo *** '3-stmt','3-row' xa-transactions must be in the list *** +XA RECOVER; +XA COMMIT '3-stmt'; +XA ROLLBACK '3-row'; + +--source include/sync_slave_sql_with_master.inc + +# +# Testing replication with marginal XID values and in two formats. +# + +--connection master +--let $wait_condition= SELECT count(*) = 0 FROM v_processlist WHERE PROCESSLIST_ID = $conn_id +--source include/wait_condition.inc + +# Max size XID incl max value of formatID +connect (master_conn2, 127.0.0.1,root,,test,$MASTER_MYPORT,); +--let $conn_id=`SELECT connection_id()` + +--let $gtrid=0123456789012345678901234567890123456789012345678901234567890124 +--let $bqual=0123456789012345678901234567890123456789012345678901234567890124 +--eval XA START '$gtrid','$bqual',4294967292 + INSERT INTO d1.t VALUES (64); +--eval XA END '$gtrid','$bqual',4294967292 +--eval XA PREPARE '$gtrid','$bqual',4294967292 + +--disconnect master_conn2 + +--connection master +--let $wait_condition= SELECT count(*) = 0 FROM v_processlist WHERE PROCESSLIST_ID = $conn_id +--source include/wait_condition.inc + +# Max size XID with non-ascii chars +connect (master_conn3, 127.0.0.1,root,,test,$MASTER_MYPORT,); +--let $conn_id=`SELECT connection_id()` + +--let $gtrid_hex=FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF +--let $bqual_hex=00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 +--eval XA START X'$gtrid_hex',X'$bqual_hex',0 + INSERT INTO d1.t VALUES (0); +--eval XA END X'$gtrid_hex',X'$bqual_hex',0 +--eval XA PREPARE X'$gtrid_hex',X'$bqual_hex',0 + +--disconnect master_conn3 + +--connection master +--let $wait_condition= SELECT count(*) = 0 FROM v_processlist WHERE PROCESSLIST_ID = $conn_id +--source include/wait_condition.inc + +# Random XID +--disable_query_log + +connect (master_conn4, 127.0.0.1,root,,test,$MASTER_MYPORT,); +--let $conn_id=`SELECT connection_id()` + +--let $gtridlen=`SELECT 2*(1 + round(rand()*100) % 31)` +--let $bquallen=`SELECT 2*(1 + round(rand()*100) % 31)` +--let $gtrid_rand=`SELECT substring(concat(MD5(rand()), MD5(rand())), 1, $gtridlen)` +--let $bqual_rand=`SELECT substring(concat(MD5(rand()), MD5(rand())), 1, $bquallen)` +--let $formt_rand=`SELECT floor((rand()*10000000000) % 4294967293)` +--eval XA START X'$gtrid_rand',X'$bqual_rand',$formt_rand + INSERT INTO d1.t VALUES (0); +--eval XA END X'$gtrid_rand',X'$bqual_rand',$formt_rand +--eval XA PREPARE X'$gtrid_rand',X'$bqual_rand',$formt_rand + +--enable_query_log + +--disconnect master_conn4 + +--connection master +--let $wait_condition= SELECT count(*) = 0 FROM v_processlist WHERE PROCESSLIST_ID = $conn_id +--source include/wait_condition.inc + +--eval XA COMMIT '$gtrid','$bqual',4294967292 +--eval XA COMMIT X'$gtrid_hex',X'$bqual_hex',0 +--disable_query_log +--echo XA COMMIT 'RANDOM XID' +--eval XA COMMIT X'$gtrid_rand',X'$bqual_rand',$formt_rand +--enable_query_log + +--source include/sync_slave_sql_with_master.inc + +# +# Testing ONE PHASE +# +--let $onephase_trx_num=10 +--let $i = $onephase_trx_num +while($i > 0) +{ + --connect (master_bulk_conn$i, 127.0.0.1,root,,test,$MASTER_MYPORT,) + + --connection master_bulk_conn$i + --eval XA START 'one_phase_$i' + --eval INSERT INTO d1.t VALUES ($i) + --eval INSERT INTO d2.t VALUES ($i) + --eval XA END 'one_phase_$i' + --eval XA COMMIT 'one_phase_$i' ONE PHASE + + --disconnect master_bulk_conn$i + --dec $i +} +--connection master +--source include/sync_slave_sql_with_master.inc + +# +# Overall consistency check +# +--let $diff_tables= master:d1.t, slave:d1.t +--source include/diff_tables.inc +--let $diff_tables= master:d2.t, slave:d2.t +--source include/diff_tables.inc +# +# cleanup +# +--connection master + +DELETE FROM d1.t; +DELETE FROM d2.t; +DROP TABLE d1.t, d2.t; +DROP DATABASE d1; +DROP DATABASE d2; +DROP VIEW v_processlist; + +--source include/sync_slave_sql_with_master.inc + +--source include/rpl_end.inc diff --git a/mysql-test/suite/rpl/t/rpl_xa_survive_disconnect_lsu_off-slave.opt b/mysql-test/suite/rpl/t/rpl_xa_survive_disconnect_lsu_off-slave.opt new file mode 100644 index 00000000000..94c3650024f --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_xa_survive_disconnect_lsu_off-slave.opt @@ -0,0 +1,2 @@ +--log-slave-updates=off + diff --git a/mysql-test/suite/rpl/t/rpl_xa_survive_disconnect_lsu_off.test b/mysql-test/suite/rpl/t/rpl_xa_survive_disconnect_lsu_off.test new file mode 100644 index 00000000000..df3811df6ae --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_xa_survive_disconnect_lsu_off.test @@ -0,0 +1,8 @@ +# ==== Purpose ==== +# 'rpl_xa_survive_disconnect_lsu_off' verifies the same properties as the sourced file +# in conditions of the slave does not log own updates +# (lsu in the name stands for log_slave_updates). +# Specifically this mode aims at proving correct operations on the slave +# mysql.gtid_executed. + +--source ./rpl_xa_survive_disconnect.test diff --git a/mysql-test/suite/rpl/t/rpl_xa_survive_disconnect_mixed_engines.test b/mysql-test/suite/rpl/t/rpl_xa_survive_disconnect_mixed_engines.test new file mode 100644 index 00000000000..f52a9630a87 --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_xa_survive_disconnect_mixed_engines.test @@ -0,0 +1,68 @@ +# BUG#12161 Xa recovery and client disconnection +# +# The test verifies correct XA transaction two phase logging and its applying +# in a case the transaction updates transactional and non-transactional tables. +# Transactions are terminated according to specfied parameters to +# a sourced inc-file. + +--source include/have_innodb.inc +--source include/master-slave.inc + +--connection master +CALL mtr.add_suppression("Unsafe statement written to the binary log using statement format since BINLOG_FORMAT = STATEMENT"); + +--let $command=setup +--source include/rpl_xa_mixed_engines.inc + +--echo === COMMIT === +--let $command=run +--let $xa_terminate=XA COMMIT +--let $xa_prepare_opt=1 +--source include/rpl_xa_mixed_engines.inc + +--source include/sync_slave_sql_with_master.inc +--connection master + +--echo === COMMIT ONE PHASE === + +--let $command=run +--let $xa_terminate=XA COMMIT +--let $one_phase=ONE PHASE +--let $xa_prepare_opt= +--source include/rpl_xa_mixed_engines.inc +--let $one_phase= +--source include/sync_slave_sql_with_master.inc +--connection master + +--echo === ROLLBACK with PREPARE === + +--let $command=run +--let $xa_terminate=xa rollback +--let $xa_prepare_opt=1 +--source include/rpl_xa_mixed_engines.inc + +--source include/sync_slave_sql_with_master.inc +--connection master + +--echo === ROLLBACK with no PREPARE === + +--let $command=run +--let $xa_terminate=xa rollback +--let $xa_prepare_opt= +--source include/rpl_xa_mixed_engines.inc +--let $xa_rollback_only= + +--source include/sync_slave_sql_with_master.inc + +--let $diff_tables= master:tm, slave:tm +--source include/diff_tables.inc + +# Cleanup + +--connection master +--let $command=cleanup +--source include/rpl_xa_mixed_engines.inc + +--source include/sync_slave_sql_with_master.inc + +--source include/rpl_end.inc diff --git a/sql/handler.cc b/sql/handler.cc index 7d61252eea6..c72386c7e99 100644 --- a/sql/handler.cc +++ b/sql/handler.cc @@ -1300,6 +1300,14 @@ int ha_prepare(THD *thd) } } + + DEBUG_SYNC(thd, "at_unlog_xa_prepare"); + + if (tc_log->unlog_xa_prepare(thd, all)) + { + ha_rollback_trans(thd, all); + error=1; + } } DBUG_RETURN(error); @@ -1853,7 +1861,8 @@ int ha_rollback_trans(THD *thd, bool all) rollback without signalling following transactions. And in release builds, we explicitly do the signalling before rolling back. */ - DBUG_ASSERT(!(thd->rgi_slave && thd->rgi_slave->did_mark_start_commit)); + DBUG_ASSERT(!(thd->rgi_slave && thd->rgi_slave->did_mark_start_commit) || + thd->transaction.xid_state.is_explicit_XA()); if (thd->rgi_slave && thd->rgi_slave->did_mark_start_commit) thd->rgi_slave->unmark_start_commit(); } diff --git a/sql/log.cc b/sql/log.cc index 56e83bf2448..e13f8fbc88f 100644 --- a/sql/log.cc +++ b/sql/log.cc @@ -91,7 +91,13 @@ static bool binlog_savepoint_rollback_can_release_mdl(handlerton *hton, static int binlog_commit(handlerton *hton, THD *thd, bool all); static int binlog_rollback(handlerton *hton, THD *thd, bool all); static int binlog_prepare(handlerton *hton, THD *thd, bool all); +static int binlog_xa_recover_dummy(handlerton *hton, XID *xid_list, uint len); +static int binlog_commit_by_xid(handlerton *hton, XID *xid); +static int binlog_rollback_by_xid(handlerton *hton, XID *xid); static int binlog_start_consistent_snapshot(handlerton *hton, THD *thd); +static int binlog_flush_cache(THD *thd, binlog_cache_mngr *cache_mngr, + Log_event *end_ev, bool all, bool using_stmt, + bool using_trx); static const LEX_CSTRING write_error_msg= { STRING_WITH_LEN("error writing to the binary log") }; @@ -1693,6 +1699,10 @@ int binlog_init(void *p) { binlog_hton->prepare= binlog_prepare; binlog_hton->start_consistent_snapshot= binlog_start_consistent_snapshot; + binlog_hton->commit_by_xid= binlog_commit_by_xid; + binlog_hton->rollback_by_xid= binlog_rollback_by_xid; + // recover needs to be set to make xa{commit,rollback}_handlerton effective + binlog_hton->recover= binlog_xa_recover_dummy; } binlog_hton->flags= HTON_NOT_USER_SELECTABLE | HTON_HIDDEN; return 0; @@ -1765,7 +1775,8 @@ binlog_flush_cache(THD *thd, binlog_cache_mngr *cache_mngr, DBUG_PRINT("enter", ("end_ev: %p", end_ev)); if ((using_stmt && !cache_mngr->stmt_cache.empty()) || - (using_trx && !cache_mngr->trx_cache.empty())) + (using_trx && !cache_mngr->trx_cache.empty()) || + thd->transaction.xid_state.is_explicit_XA()) { if (using_stmt && thd->binlog_flush_pending_rows_event(TRUE, FALSE)) DBUG_RETURN(1); @@ -1837,6 +1848,17 @@ binlog_commit_flush_stmt_cache(THD *thd, bool all, DBUG_RETURN(binlog_flush_cache(thd, cache_mngr, &end_evt, all, TRUE, FALSE)); } + +inline size_t serialize_with_xid(XID *xid, char *buf, + const char *query, size_t q_len) +{ + memcpy(buf, query, q_len); + + return + q_len + strlen(static_cast<event_xid_t*>(xid)->serialize(buf + q_len)); +} + + /** This function flushes the trx-cache upon commit. @@ -1850,11 +1872,28 @@ static inline int binlog_commit_flush_trx_cache(THD *thd, bool all, binlog_cache_mngr *cache_mngr) { DBUG_ENTER("binlog_commit_flush_trx_cache"); - Query_log_event end_evt(thd, STRING_WITH_LEN("COMMIT"), - TRUE, TRUE, TRUE, 0); + + const char query[]= "XA COMMIT "; + const size_t q_len= sizeof(query) - 1; // do not count trailing 0 + char buf[q_len + ser_buf_size]= "COMMIT"; + size_t buflen= sizeof("COMMIT") - 1; + + if (thd->lex->sql_command == SQLCOM_XA_COMMIT && + thd->lex->xa_opt != XA_ONE_PHASE) + { + DBUG_ASSERT(thd->transaction.xid_state.is_explicit_XA()); + DBUG_ASSERT(thd->transaction.xid_state.get_state_code() == + XA_PREPARED); + + buflen= serialize_with_xid(thd->transaction.xid_state.get_xid(), + buf, query, q_len); + } + Query_log_event end_evt(thd, buf, buflen, TRUE, TRUE, TRUE, 0); + DBUG_RETURN(binlog_flush_cache(thd, cache_mngr, &end_evt, all, FALSE, TRUE)); } + /** This function flushes the trx-cache upon rollback. @@ -1868,8 +1907,20 @@ static inline int binlog_rollback_flush_trx_cache(THD *thd, bool all, binlog_cache_mngr *cache_mngr) { - Query_log_event end_evt(thd, STRING_WITH_LEN("ROLLBACK"), - TRUE, TRUE, TRUE, 0); + const char query[]= "XA ROLLBACK "; + const size_t q_len= sizeof(query) - 1; // do not count trailing 0 + char buf[q_len + ser_buf_size]= "ROLLBACK"; + size_t buflen= sizeof("ROLLBACK") - 1; + + if (thd->transaction.xid_state.is_explicit_XA()) + { + /* for not prepared use plain ROLLBACK */ + if (thd->transaction.xid_state.get_state_code() == XA_PREPARED) + buflen= serialize_with_xid(thd->transaction.xid_state.get_xid(), + buf, query, q_len); + } + Query_log_event end_evt(thd, buf, buflen, TRUE, TRUE, TRUE, 0); + return (binlog_flush_cache(thd, cache_mngr, &end_evt, all, FALSE, TRUE)); } @@ -1887,23 +1938,10 @@ static inline int binlog_commit_flush_xid_caches(THD *thd, binlog_cache_mngr *cache_mngr, bool all, my_xid xid) { - if (xid) - { - Xid_log_event end_evt(thd, xid, TRUE); - return (binlog_flush_cache(thd, cache_mngr, &end_evt, all, TRUE, TRUE)); - } - else - { - /* - Empty xid occurs in XA COMMIT ... ONE PHASE. - In this case, we do not have a MySQL xid for the transaction, and the - external XA transaction coordinator will have to handle recovery if - needed. So we end the transaction with a plain COMMIT query event. - */ - Query_log_event end_evt(thd, STRING_WITH_LEN("COMMIT"), - TRUE, TRUE, TRUE, 0); - return (binlog_flush_cache(thd, cache_mngr, &end_evt, all, TRUE, TRUE)); - } + DBUG_ASSERT(xid); // replaced former treatment of ONE-PHASE XA + + Xid_log_event end_evt(thd, xid, TRUE); + return (binlog_flush_cache(thd, cache_mngr, &end_evt, all, TRUE, TRUE)); } /** @@ -1959,17 +1997,67 @@ binlog_truncate_trx_cache(THD *thd, binlog_cache_mngr *cache_mngr, bool all) DBUG_RETURN(error); } + +inline bool is_preparing_xa(THD *thd) +{ + return + thd->transaction.xid_state.is_explicit_XA() && + thd->lex->sql_command == SQLCOM_XA_PREPARE; +} + + static int binlog_prepare(handlerton *hton, THD *thd, bool all) { /* - do nothing. - just pretend we can do 2pc, so that MySQL won't - switch to 1pc. - real work will be done in MYSQL_BIN_LOG::log_and_order() + Do nothing unless the transaction is a user XA. */ + return + !is_preparing_xa(thd) ? 0 : binlog_commit(NULL, thd, all); +} + + +static int binlog_xa_recover_dummy(handlerton *hton __attribute__((unused)), + XID *xid_list __attribute__((unused)), + uint len __attribute__((unused))) +{ + /* Does nothing. */ return 0; } + +static int binlog_commit_by_xid(handlerton *hton, XID *xid) +{ + THD *thd= current_thd; + + if (thd->transaction.xid_state.is_binlogged()) + (void) thd->binlog_setup_trx_data(); + + DBUG_ASSERT(thd->lex->sql_command == SQLCOM_XA_COMMIT); + + return binlog_commit(hton, thd, TRUE); +} + + +static int binlog_rollback_by_xid(handlerton *hton, XID *xid) +{ + THD *thd= current_thd; + + if (thd->transaction.xid_state.is_binlogged()) + (void) thd->binlog_setup_trx_data(); + + DBUG_ASSERT(thd->lex->sql_command == SQLCOM_XA_ROLLBACK); + + return binlog_rollback(hton, thd, TRUE); +} + + +inline bool is_prepared_xa(THD *thd) +{ + return thd->transaction.xid_state.is_explicit_XA() && + thd->transaction.xid_state.get_state_code() == XA_PREPARED; +} + + /* We flush the cache wrapped in a beging/rollback if: . aborting a single or multi-statement transaction and; @@ -1992,7 +2080,55 @@ static bool trans_cannot_safely_rollback(THD *thd, bool all) thd->wsrep_binlog_format() == BINLOG_FORMAT_MIXED) || (trans_has_updated_non_trans_table(thd) && ending_single_stmt_trans(thd,all) && - thd->wsrep_binlog_format() == BINLOG_FORMAT_MIXED)); + thd->wsrep_binlog_format() == BINLOG_FORMAT_MIXED) || + is_prepared_xa(thd)); +} + + +/** + Specific log flusher invoked through log_xa_prepare(). +*/ +static int binlog_commit_flush_xa_prepare(THD *thd, bool all, + binlog_cache_mngr *cache_mngr) +{ + XID *xid= thd->transaction.xid_state.get_xid(); + { + // todo assert wsrep_simulate || is_open() + + /* + Log the XA END event first. + We don't do that in trans_xa_end() as XA COMMIT ONE PHASE + is logged as simple BEGIN/COMMIT so the XA END should + not get to the log. + */ + const char query[]= "XA END "; + const size_t q_len= sizeof(query) - 1; // do not count trailing 0 + char buf[q_len + ser_buf_size]; + size_t buflen; + binlog_cache_data *cache_data; + IO_CACHE *file; + + memcpy(buf, query, q_len); + buflen= q_len + + strlen(static_cast<event_xid_t*>(xid)->serialize(buf + q_len)); + cache_data= cache_mngr->get_binlog_cache_data(true); + file= &cache_data->cache_log; + thd->lex->sql_command= SQLCOM_XA_END; + Query_log_event xa_end(thd, buf, buflen, true, false, true, 0); + if (mysql_bin_log.write_event(&xa_end, cache_data, file)) + return 1; + thd->lex->sql_command= SQLCOM_XA_PREPARE; + } + + cache_mngr->using_xa= FALSE; + XA_prepare_log_event end_evt(thd, xid, FALSE); + /* + Memorize the fact of prepare-logging to recall at commit + possibly from another session. + */ + if (thd->variables.option_bits & OPTION_BIN_LOG) + thd->transaction.xid_state.set_binlogged(); + return (binlog_flush_cache(thd, cache_mngr, &end_evt, all, TRUE, TRUE)); } @@ -2019,7 +2155,9 @@ static int binlog_commit(handlerton *hton, THD *thd, bool all) if (!cache_mngr) { - DBUG_ASSERT(WSREP(thd)); + DBUG_ASSERT(WSREP(thd) || + (thd->transaction.xid_state.is_explicit_XA() && + !thd->transaction.xid_state.is_binlogged())); DBUG_RETURN(0); } @@ -2038,7 +2176,8 @@ static int binlog_commit(handlerton *hton, THD *thd, bool all) error= binlog_commit_flush_stmt_cache(thd, all, cache_mngr); } - if (cache_mngr->trx_cache.empty()) + if (cache_mngr->trx_cache.empty() && + !thd->transaction.xid_state.is_binlogged()) { /* we're here because cache_log was flushed in MYSQL_BIN_LOG::log_xid() @@ -2055,8 +2194,11 @@ static int binlog_commit(handlerton *hton, THD *thd, bool all) Otherwise, we accumulate the changes. */ if (likely(!error) && ending_trans(thd, all)) - error= binlog_commit_flush_trx_cache(thd, all, cache_mngr); - + { + error= !is_preparing_xa(thd) ? + binlog_commit_flush_trx_cache (thd, all, cache_mngr) : + binlog_commit_flush_xa_prepare(thd, all, cache_mngr); + } /* This is part of the stmt rollback. */ @@ -2080,13 +2222,16 @@ static int binlog_commit(handlerton *hton, THD *thd, bool all) static int binlog_rollback(handlerton *hton, THD *thd, bool all) { DBUG_ENTER("binlog_rollback"); + int error= 0; binlog_cache_mngr *const cache_mngr= (binlog_cache_mngr*) thd_get_ha_data(thd, binlog_hton); if (!cache_mngr) { - DBUG_ASSERT(WSREP(thd)); + DBUG_ASSERT(WSREP(thd) || + (is_prepared_xa(thd) || + !thd->transaction.xid_state.is_binlogged())); DBUG_RETURN(0); } @@ -2101,15 +2246,16 @@ static int binlog_rollback(handlerton *hton, THD *thd, bool all) */ if (cache_mngr->stmt_cache.has_incident()) { - error= mysql_bin_log.write_incident(thd); + error |= static_cast<int>(mysql_bin_log.write_incident(thd)); cache_mngr->reset(true, false); } else if (!cache_mngr->stmt_cache.empty()) { - error= binlog_commit_flush_stmt_cache(thd, all, cache_mngr); + error |= binlog_commit_flush_stmt_cache(thd, all, cache_mngr); } - if (cache_mngr->trx_cache.empty()) + if (cache_mngr->trx_cache.empty() && + !thd->transaction.xid_state.is_binlogged()) { /* we're here because cache_log was flushed in MYSQL_BIN_LOG::log_xid() @@ -7340,10 +7486,10 @@ MYSQL_BIN_LOG::write_transaction_to_binlog(THD *thd, entry.all= all; entry.using_stmt_cache= using_stmt_cache; entry.using_trx_cache= using_trx_cache; - entry.need_unlog= false; + entry.need_unlog= is_preparing_xa(thd); ha_info= all ? thd->transaction.all.ha_list : thd->transaction.stmt.ha_list; - for (; ha_info; ha_info= ha_info->next()) + for (; !entry.need_unlog && ha_info; ha_info= ha_info->next()) { if (ha_info->is_started() && ha_info->ht() != binlog_hton && !ha_info->ht()->commit_checkpoint_request) @@ -7916,7 +8062,9 @@ MYSQL_BIN_LOG::trx_group_commit_leader(group_commit_entry *leader) We already checked before that at least one cache is non-empty; if both are empty we would have skipped calling into here. */ - DBUG_ASSERT(!cache_mngr->stmt_cache.empty() || !cache_mngr->trx_cache.empty()); + DBUG_ASSERT(!cache_mngr->stmt_cache.empty() || + !cache_mngr->trx_cache.empty() || + current->thd->transaction.xid_state.is_explicit_XA()); if (unlikely((current->error= write_transaction_or_stmt(current, commit_id)))) @@ -7925,7 +8073,7 @@ MYSQL_BIN_LOG::trx_group_commit_leader(group_commit_entry *leader) strmake_buf(cache_mngr->last_commit_pos_file, log_file_name); commit_offset= my_b_write_tell(&log_file); cache_mngr->last_commit_pos_offset= commit_offset; - if (cache_mngr->using_xa && cache_mngr->xa_xid) + if ((cache_mngr->using_xa && cache_mngr->xa_xid) || current->need_unlog) { /* If all storage engines support commit_checkpoint_request(), then we @@ -8160,7 +8308,8 @@ MYSQL_BIN_LOG::write_transaction_or_stmt(group_commit_entry *entry, binlog_cache_mngr *mngr= entry->cache_mngr; DBUG_ENTER("MYSQL_BIN_LOG::write_transaction_or_stmt"); - if (write_gtid_event(entry->thd, false, entry->using_trx_cache, commit_id)) + if (write_gtid_event(entry->thd, is_prepared_xa(entry->thd), + entry->using_trx_cache, commit_id)) DBUG_RETURN(ER_ERROR_ON_WRITE); if (entry->using_stmt_cache && !mngr->stmt_cache.empty() && @@ -9862,6 +10011,24 @@ int TC_LOG_BINLOG::unlog(ulong cookie, my_xid xid) DBUG_RETURN(BINLOG_COOKIE_GET_ERROR_FLAG(cookie)); } + +int TC_LOG_BINLOG::unlog_xa_prepare(THD *thd, bool all) +{ + DBUG_ASSERT(is_preparing_xa(thd)); + + binlog_cache_mngr *cache_mngr= thd->binlog_setup_trx_data(); + int cookie= 0; + + if (!cache_mngr || !cache_mngr->need_unlog) + return 0; + else + cookie= BINLOG_COOKIE_MAKE(cache_mngr->binlog_id, cache_mngr->delayed_error); + cache_mngr->need_unlog= false; + + return unlog(cookie, 1); +} + + void TC_LOG_BINLOG::commit_checkpoint_notify(void *cookie) { @@ -10178,6 +10345,7 @@ int TC_LOG_BINLOG::recover(LOG_INFO *linfo, const char *last_log_name, ((last_gtid_standalone && !ev->is_part_of_group(typ)) || (!last_gtid_standalone && (typ == XID_EVENT || + typ == XA_PREPARE_LOG_EVENT || (LOG_EVENT_IS_QUERY(typ) && (((Query_log_event *)ev)->is_commit() || ((Query_log_event *)ev)->is_rollback())))))) diff --git a/sql/log.h b/sql/log.h index 8684eaba786..8e70d3c8f4c 100644 --- a/sql/log.h +++ b/sql/log.h @@ -61,6 +61,7 @@ class TC_LOG bool need_prepare_ordered, bool need_commit_ordered) = 0; virtual int unlog(ulong cookie, my_xid xid)=0; + virtual int unlog_xa_prepare(THD *thd, bool all)= 0; virtual void commit_checkpoint_notify(void *cookie)= 0; protected: @@ -115,6 +116,10 @@ class TC_LOG_DUMMY: public TC_LOG // use it to disable the logging return 1; } int unlog(ulong cookie, my_xid xid) { return 0; } + int unlog_xa_prepare(THD *thd, bool all) + { + return 0; + } void commit_checkpoint_notify(void *cookie) { DBUG_ASSERT(0); }; }; @@ -198,6 +203,10 @@ class TC_LOG_MMAP: public TC_LOG int log_and_order(THD *thd, my_xid xid, bool all, bool need_prepare_ordered, bool need_commit_ordered); int unlog(ulong cookie, my_xid xid); + int unlog_xa_prepare(THD *thd, bool all) + { + return 0; + } void commit_checkpoint_notify(void *cookie); int recover(); @@ -695,6 +704,7 @@ class MYSQL_BIN_LOG: public TC_LOG, private MYSQL_LOG int log_and_order(THD *thd, my_xid xid, bool all, bool need_prepare_ordered, bool need_commit_ordered); int unlog(ulong cookie, my_xid xid); + int unlog_xa_prepare(THD *thd, bool all); void commit_checkpoint_notify(void *cookie); int recover(LOG_INFO *linfo, const char *last_log_name, IO_CACHE *first_log, Format_description_log_event *fdle, bool do_xa); diff --git a/sql/log_event.cc b/sql/log_event.cc index 4c1c18fffff..ee44f7f1da4 100644 --- a/sql/log_event.cc +++ b/sql/log_event.cc @@ -1178,6 +1178,9 @@ Log_event* Log_event::read_log_event(const char* buf, uint event_len, case XID_EVENT: ev = new Xid_log_event(buf, fdle); break; + case XA_PREPARE_LOG_EVENT: + ev = new XA_prepare_log_event(buf, fdle); + break; case RAND_EVENT: ev = new Rand_log_event(buf, fdle); break; @@ -1229,7 +1232,6 @@ Log_event* Log_event::read_log_event(const char* buf, uint event_len, case PREVIOUS_GTIDS_LOG_EVENT: case TRANSACTION_CONTEXT_EVENT: case VIEW_CHANGE_EVENT: - case XA_PREPARE_LOG_EVENT: ev= new Ignorable_log_event(buf, fdle, get_type_str((Log_event_type) event_type)); break; @@ -2066,6 +2068,7 @@ Format_description_log_event(uint8 binlog_ver, const char* server_ver) post_header_len[USER_VAR_EVENT-1]= USER_VAR_HEADER_LEN; post_header_len[FORMAT_DESCRIPTION_EVENT-1]= FORMAT_DESCRIPTION_HEADER_LEN; post_header_len[XID_EVENT-1]= XID_HEADER_LEN; + post_header_len[XA_PREPARE_LOG_EVENT-1]= XA_PREPARE_HEADER_LEN; post_header_len[BEGIN_LOAD_QUERY_EVENT-1]= BEGIN_LOAD_QUERY_HEADER_LEN; post_header_len[EXECUTE_LOAD_QUERY_EVENT-1]= EXECUTE_LOAD_QUERY_HEADER_LEN; /* @@ -2556,7 +2559,7 @@ Binlog_checkpoint_log_event::Binlog_checkpoint_log_event( Gtid_log_event::Gtid_log_event(const char *buf, uint event_len, const Format_description_log_event *description_event) - : Log_event(buf, description_event), seq_no(0), commit_id(0) + : Log_event(buf, description_event), seq_no(0), commit_id(0), xid_pins_idx(0) { uint8 header_size= description_event->common_header_len; uint8 post_header_len= description_event->post_header_len[GTID_EVENT-1]; @@ -2569,7 +2572,7 @@ Gtid_log_event::Gtid_log_event(const char *buf, uint event_len, buf+= 8; domain_id= uint4korr(buf); buf+= 4; - flags2= *buf; + flags2= *(buf++); if (flags2 & FL_GROUP_COMMIT_ID) { if (event_len < (uint)header_size + GTID_HEADER_LEN + 2) @@ -2577,8 +2580,24 @@ Gtid_log_event::Gtid_log_event(const char *buf, uint event_len, seq_no= 0; // So is_valid() returns false return; } - ++buf; commit_id= uint8korr(buf); + buf+= 8; + } + if (flags2 & (FL_PREPARED_XA | FL_COMPLETED_XA)) + { + uint32 temp= 0; + + memcpy(&temp, buf, sizeof(temp)); + xid.formatID= uint4korr(&temp); + buf += sizeof(temp); + + xid.gtrid_length= (long) buf[0]; + xid.bqual_length= (long) buf[1]; + buf+= 2; + + long data_length= xid.bqual_length + xid.gtrid_length; + memcpy(xid.data, buf, data_length); + buf+= data_length; } } @@ -2764,7 +2783,7 @@ Rand_log_event::Rand_log_event(const char* buf, Xid_log_event:: Xid_log_event(const char* buf, const Format_description_log_event *description_event) - :Log_event(buf, description_event) + :Xid_apply_log_event(buf, description_event) { /* The Post-Header is empty. The Variable Data part begins immediately. */ buf+= description_event->common_header_len + @@ -2772,6 +2791,36 @@ Xid_log_event(const char* buf, memcpy((char*) &xid, buf, sizeof(xid)); } +/************************************************************************** + XA_prepare_log_event methods +**************************************************************************/ +XA_prepare_log_event:: +XA_prepare_log_event(const char* buf, + const Format_description_log_event *description_event) + :Xid_apply_log_event(buf, description_event) +{ + uint32 temp= 0; + uint8 temp_byte; + + buf+= description_event->common_header_len + + description_event->post_header_len[XA_PREPARE_LOG_EVENT-1]; + memcpy(&temp_byte, buf, 1); + one_phase= (bool) temp_byte; + buf += sizeof(temp_byte); + memcpy(&temp, buf, sizeof(temp)); + m_xid.formatID= uint4korr(&temp); + buf += sizeof(temp); + memcpy(&temp, buf, sizeof(temp)); + m_xid.gtrid_length= uint4korr(&temp); + buf += sizeof(temp); + memcpy(&temp, buf, sizeof(temp)); + m_xid.bqual_length= uint4korr(&temp); + buf += sizeof(temp); + memcpy(m_xid.data, buf, m_xid.gtrid_length + m_xid.bqual_length); + + xid= NULL; +} + /************************************************************************** User_var_log_event methods diff --git a/sql/log_event.h b/sql/log_event.h index 15442bd5a97..a6543b70eb5 100644 --- a/sql/log_event.h +++ b/sql/log_event.h @@ -222,6 +222,7 @@ class String; #define GTID_HEADER_LEN 19 #define GTID_LIST_HEADER_LEN 4 #define START_ENCRYPTION_HEADER_LEN 0 +#define XA_PREPARE_HEADER_LEN 0 /* Max number of possible extra bytes in a replication event compared to a @@ -664,6 +665,7 @@ enum Log_event_type /* MySQL 5.7 events, ignored by MariaDB */ TRANSACTION_CONTEXT_EVENT= 36, VIEW_CHANGE_EVENT= 37, + /* not ignored */ XA_PREPARE_LOG_EVENT= 38, /* @@ -3022,6 +3024,32 @@ class Rand_log_event: public Log_event #endif }; + +class Xid_apply_log_event: public Log_event +{ +public: +#ifdef MYSQL_SERVER + Xid_apply_log_event(THD* thd_arg): + Log_event(thd_arg, 0, TRUE) {} +#endif + Xid_apply_log_event(const char* buf, + const Format_description_log_event *description_event): + Log_event(buf, description_event) {} + + ~Xid_apply_log_event() {} + bool is_valid() const { return 1; } +private: +#if defined(MYSQL_SERVER) && defined(HAVE_REPLICATION) + virtual int do_commit()= 0; + virtual int do_apply_event(rpl_group_info *rgi); + int do_record_gtid(THD *thd, rpl_group_info *rgi, bool in_trans, + void **out_hton); + enum_skip_reason do_shall_skip(rpl_group_info *rgi); + virtual const char* get_query()= 0; +#endif +}; + + /** @class Xid_log_event @@ -3034,18 +3062,22 @@ class Rand_log_event: public Log_event typedef ulonglong my_xid; // this line is the same as in handler.h #endif -class Xid_log_event: public Log_event +class Xid_log_event: public Xid_apply_log_event { - public: - my_xid xid; +public: + my_xid xid; #ifdef MYSQL_SERVER Xid_log_event(THD* thd_arg, my_xid x, bool direct): - Log_event(thd_arg, 0, TRUE), xid(x) + Xid_apply_log_event(thd_arg), xid(x) { if (direct) cache_type= Log_event::EVENT_NO_CACHE; } + const char* get_query() + { + return "COMMIT /* implicit, from Xid_log_event */"; + } #ifdef HAVE_REPLICATION void pack_info(Protocol* protocol); #endif /* HAVE_REPLICATION */ @@ -3061,15 +3093,171 @@ class Xid_log_event: public Log_event #ifdef MYSQL_SERVER bool write(); #endif - bool is_valid() const { return 1; } private: #if defined(MYSQL_SERVER) && defined(HAVE_REPLICATION) - virtual int do_apply_event(rpl_group_info *rgi); - enum_skip_reason do_shall_skip(rpl_group_info *rgi); + int do_commit(); #endif }; + +/** + @class XA_prepare_log_event + + Similar to Xid_log_event except that + - it is specific to XA transaction + - it carries out the prepare logics rather than the final committing + when @c one_phase member is off. The latter option is only for + compatibility with the upstream. + + From the groupping perspective the event finalizes the current + "prepare" group that is started with Gtid_log_event similarly to the + regular replicated transaction. +*/ + +/** + Function serializes XID which is characterized by by four last arguments + of the function. + Serialized XID is presented in valid hex format and is returned to + the caller in a buffer pointed by the first argument. + The buffer size provived by the caller must be not less than + 8 + 2 * XIDDATASIZE + 4 * sizeof(XID::formatID) + 1, see + {MYSQL_,}XID definitions. + + @param buf pointer to a buffer allocated for storing serialized data + @param fmt formatID value + @param gln gtrid_length value + @param bln bqual_length value + @param dat data value + + @return the value of the buffer pointer +*/ + +inline char *serialize_xid(char *buf, long fmt, long gln, long bln, + const char *dat) +{ + int i; + char *c= buf; + /* + Build a string consisting of the hex format representation of XID + as passed through fmt,gln,bln,dat argument: + X'hex11hex12...hex1m',X'hex21hex22...hex2n',11 + and store it into buf. + */ + c[0]= 'X'; + c[1]= '\''; + c+= 2; + for (i= 0; i < gln; i++) + { + c[0]=_dig_vec_lower[((uchar*) dat)[i] >> 4]; + c[1]=_dig_vec_lower[((uchar*) dat)[i] & 0x0f]; + c+= 2; + } + c[0]= '\''; + c[1]= ','; + c[2]= 'X'; + c[3]= '\''; + c+= 4; + + for (; i < gln + bln; i++) + { + c[0]=_dig_vec_lower[((uchar*) dat)[i] >> 4]; + c[1]=_dig_vec_lower[((uchar*) dat)[i] & 0x0f]; + c+= 2; + } + c[0]= '\''; + sprintf(c+1, ",%lu", fmt); + + return buf; +} + +/* + The size of the string containing serialized Xid representation + is computed as a sum of + eight as the number of formatting symbols (X'',X'',) + plus 2 x XIDDATASIZE (2 due to hex format), + plus space for decimal digits of XID::formatID, + plus one for 0x0. +*/ +static const uint ser_buf_size= + 8 + 2 * MYSQL_XIDDATASIZE + 4 * sizeof(long) + 1; + +struct event_mysql_xid_t : MYSQL_XID +{ + char buf[ser_buf_size]; + char *serialize() + { + return serialize_xid(buf, formatID, gtrid_length, bqual_length, data); + } +}; + +#ifndef MYSQL_CLIENT +struct event_xid_t : XID +{ + char buf[ser_buf_size]; + + char *serialize(char *buf_arg) + { + return serialize_xid(buf_arg, formatID, gtrid_length, bqual_length, data); + } + char *serialize() + { + return serialize(buf); + } +}; +#endif + +class XA_prepare_log_event: public Xid_apply_log_event +{ +protected: + + /* Constant contributor to subheader in write() by members of XID struct. */ + static const int xid_subheader_no_data= 12; + event_mysql_xid_t m_xid; + void *xid; + bool one_phase; + +public: +#ifdef MYSQL_SERVER + XA_prepare_log_event(THD* thd_arg, XID *xid_arg, bool one_phase_arg): + Xid_apply_log_event(thd_arg), xid(xid_arg), one_phase(one_phase_arg) + { + cache_type= Log_event::EVENT_NO_CACHE; + } +#ifdef HAVE_REPLICATION + void pack_info(Protocol* protocol); +#endif /* HAVE_REPLICATION */ +#else + bool print(FILE* file, PRINT_EVENT_INFO* print_event_info); +#endif + XA_prepare_log_event(const char* buf, + const Format_description_log_event *description_event); + ~XA_prepare_log_event() {} + Log_event_type get_type_code() { return XA_PREPARE_LOG_EVENT; } + int get_data_size() + { + return xid_subheader_no_data + m_xid.gtrid_length + m_xid.bqual_length; + } + +#ifdef MYSQL_SERVER + bool write(); +#endif + +private: +#if defined(MYSQL_SERVER) && defined(HAVE_REPLICATION) + char query[sizeof("XA COMMIT ONE PHASE") + 1 + ser_buf_size]; + int do_commit(); + const char* get_query() + { + sprintf(query, + (one_phase ? "XA COMMIT %s ONE PHASE" : "XA PREPARE %s"), + m_xid.serialize()); + return query; + } +#endif +}; + + /** @class User_var_log_event @@ -3382,8 +3570,13 @@ class Gtid_log_event: public Log_event uint64 seq_no; uint64 commit_id; uint32 domain_id; +#ifdef MYSQL_SERVER + event_xid_t xid; +#else + event_mysql_xid_t xid; +#endif uchar flags2; - + uint32 xid_pins_idx; /* Flags2. */ /* FL_STANDALONE is set when there is no terminating COMMIT event. */ @@ -3410,6 +3603,10 @@ class Gtid_log_event: public Log_event static const uchar FL_WAITED= 16; /* FL_DDL is set for event group containing DDL. */ static const uchar FL_DDL= 32; + /* FL_PREPARED_XA is set for XA transaction. */ + static const uchar FL_PREPARED_XA= 64; + /* FL_"COMMITTED or ROLLED-BACK"_XA is set for XA transaction. */ + static const uchar FL_COMPLETED_XA= 128; #ifdef MYSQL_SERVER Gtid_log_event(THD *thd_arg, uint64 seq_no, uint32 domain_id, bool standalone, diff --git a/sql/log_event_client.cc b/sql/log_event_client.cc index cae4842355a..0e57b82476b 100644 --- a/sql/log_event_client.cc +++ b/sql/log_event_client.cc @@ -3892,11 +3892,44 @@ Gtid_log_event::print(FILE *file, PRINT_EVENT_INFO *print_event_info) buf, print_event_info->delimiter)) goto err; } - if (!(flags2 & FL_STANDALONE)) - if (my_b_printf(&cache, is_flashback ? "COMMIT\n%s\n" : "BEGIN\n%s\n", print_event_info->delimiter)) + if ((flags2 & FL_PREPARED_XA) && !is_flashback) + { + my_b_write_string(&cache, "XA START "); + xid.serialize(); + my_b_write(&cache, (uchar*) xid.buf, strlen(xid.buf)); + if (my_b_printf(&cache, "%s\n", print_event_info->delimiter)) + goto err; + } + else if (!(flags2 & FL_STANDALONE)) + { + if (my_b_printf(&cache, is_flashback ? "COMMIT\n%s\n" : "BEGIN\n%s\n", + print_event_info->delimiter)) goto err; + } return cache.flush_data(); err: return 1; } + +bool XA_prepare_log_event::print(FILE* file, PRINT_EVENT_INFO* print_event_info) +{ + Write_on_release_cache cache(&print_event_info->head_cache, file, + Write_on_release_cache::FLUSH_F, this); + m_xid.serialize(); + + if (!print_event_info->short_form) + { + print_header(&cache, print_event_info, FALSE); + if (my_b_printf(&cache, "\tXID = %s\n", m_xid.buf)) + goto error; + } + + if (my_b_printf(&cache, "XA PREPARE %s\n%s\n", + m_xid.buf, print_event_info->delimiter)) + goto error; + + return cache.flush_data(); +error: + return TRUE; +} diff --git a/sql/log_event_server.cc b/sql/log_event_server.cc index 202a41c2837..210d321d2cd 100644 --- a/sql/log_event_server.cc +++ b/sql/log_event_server.cc @@ -1487,6 +1487,7 @@ Query_log_event::Query_log_event(THD* thd_arg, const char* query_arg, size_t que case SQLCOM_RELEASE_SAVEPOINT: case SQLCOM_ROLLBACK_TO_SAVEPOINT: case SQLCOM_SAVEPOINT: + case SQLCOM_XA_END: use_cache= trx_cache= TRUE; break; default: @@ -3218,6 +3219,26 @@ Gtid_log_event::Gtid_log_event(THD *thd_arg, uint64 seq_no_arg, /* Preserve any DDL or WAITED flag in the slave's binlog. */ if (thd_arg->rgi_slave) flags2|= (thd_arg->rgi_slave->gtid_ev_flags2 & (FL_DDL|FL_WAITED)); + + XID_STATE &xid_state= thd->transaction.xid_state; + if (is_transactional && xid_state.is_explicit_XA() && + ((xid_state.get_state_code() == XA_IDLE && + thd->lex->sql_command == SQLCOM_XA_PREPARE) || + xid_state.get_state_code() == XA_PREPARED)) + { + DBUG_ASSERT(thd->lex->xa_opt != XA_ONE_PHASE); + DBUG_ASSERT(xid_state.get_state_code() == XA_IDLE || + xid_state.is_binlogged()); + + flags2|= xid_state.get_state_code() == XA_IDLE ? + FL_PREPARED_XA : FL_COMPLETED_XA; + + xid.formatID= xid_state.get_xid()->formatID; + xid.gtrid_length= xid_state.get_xid()->gtrid_length; + xid.bqual_length= xid_state.get_xid()->bqual_length; + long data_length= xid.bqual_length + xid.gtrid_length; + memcpy(xid.data, xid_state.get_xid()->data, data_length); + } } @@ -3260,7 +3281,7 @@ Gtid_log_event::peek(const char *event_start, size_t event_len, bool Gtid_log_event::write() { - uchar buf[GTID_HEADER_LEN+2]; + uchar buf[GTID_HEADER_LEN+2+sizeof(XID)]; size_t write_len; int8store(buf, seq_no); @@ -3272,8 +3293,22 @@ Gtid_log_event::write() write_len= GTID_HEADER_LEN + 2; } else + write_len= 13; + + if (flags2 & (FL_PREPARED_XA | FL_COMPLETED_XA)) + { + int4store(&buf[write_len], xid.formatID); + buf[write_len +4]= (uchar) xid.gtrid_length; + buf[write_len +4+1]= (uchar) xid.bqual_length; + write_len+= 6; + long data_length= xid.bqual_length + xid.gtrid_length; + memcpy(buf+write_len, xid.data, data_length); + write_len+= data_length; + } + + if (write_len < GTID_HEADER_LEN) { - bzero(buf+13, GTID_HEADER_LEN-13); + bzero(buf+write_len, GTID_HEADER_LEN-write_len); write_len= GTID_HEADER_LEN; } return write_header(write_len) || @@ -3316,9 +3351,14 @@ Gtid_log_event::make_compatible_event(String *packet, bool *need_dummy_event, void Gtid_log_event::pack_info(Protocol *protocol) { - char buf[6+5+10+1+10+1+20+1+4+20+1]; + char buf[6+5+10+1+10+1+20+1+4+20+1+ ser_buf_size+5 /* sprintf */]; char *p; - p = strmov(buf, (flags2 & FL_STANDALONE ? "GTID " : "BEGIN GTID ")); + p = strmov(buf, (flags2 & FL_STANDALONE ? "GTID " : + flags2 & FL_PREPARED_XA ? "XA START " : "BEGIN GTID ")); + if (flags2 & FL_PREPARED_XA) + { + p += sprintf(p, "%s GTID ", xid.serialize()); + } p= longlong10_to_str(domain_id, p, 10); *p++= '-'; p= longlong10_to_str(server_id, p, 10); @@ -3378,16 +3418,37 @@ Gtid_log_event::do_apply_event(rpl_group_info *rgi) bits|= (ulonglong)OPTION_RPL_SKIP_PARALLEL; thd->variables.option_bits= bits; DBUG_PRINT("info", ("Set OPTION_GTID_BEGIN")); - thd->set_query_and_id(gtid_begin_string, sizeof(gtid_begin_string)-1, - &my_charset_bin, next_query_id()); - thd->lex->sql_command= SQLCOM_BEGIN; thd->is_slave_error= 0; - status_var_increment(thd->status_var.com_stat[thd->lex->sql_command]); - if (trans_begin(thd, 0)) + + char buf_xa[sizeof("XA START") + 1 + ser_buf_size]; + if (flags2 & FL_PREPARED_XA) { - DBUG_PRINT("error", ("trans_begin() failed")); - thd->is_slave_error= 1; + const char fmt[]= "XA START %s"; + + thd->lex->xid= &xid; + thd->lex->xa_opt= XA_NONE; + sprintf(buf_xa, fmt, xid.serialize()); + thd->set_query_and_id(buf_xa, static_cast<uint32>(strlen(buf_xa)), + &my_charset_bin, next_query_id()); + thd->lex->sql_command= SQLCOM_XA_START; + if (trans_xa_start(thd)) + { + DBUG_PRINT("error", ("trans_xa_start() failed")); + thd->is_slave_error= 1; + } + } + else + { + thd->set_query_and_id(gtid_begin_string, sizeof(gtid_begin_string)-1, + &my_charset_bin, next_query_id()); + thd->lex->sql_command= SQLCOM_BEGIN; + if (trans_begin(thd, 0)) + { + DBUG_PRINT("error", ("trans_begin() failed")); + thd->is_slave_error= 1; + } } + status_var_increment(thd->status_var.com_stat[thd->lex->sql_command]); thd->update_stats(); if (likely(!thd->is_slave_error)) @@ -3771,46 +3832,58 @@ bool slave_execute_deferred_events(THD *thd) /************************************************************************** - Xid_log_event methods + Xid_apply_log_event methods **************************************************************************/ #if defined(HAVE_REPLICATION) -void Xid_log_event::pack_info(Protocol *protocol) + +int Xid_apply_log_event::do_record_gtid(THD *thd, rpl_group_info *rgi, + bool in_trans, void **out_hton) { - char buf[128], *pos; - pos= strmov(buf, "COMMIT /* xid="); - pos= longlong10_to_str(xid, pos, 10); - pos= strmov(pos, " */"); - protocol->store(buf, (uint) (pos-buf), &my_charset_bin); -} -#endif + int err= 0; + Relay_log_info const *rli= rgi->rli; + rgi->gtid_pending= false; + err= rpl_global_gtid_slave_state->record_gtid(thd, &rgi->current_gtid, + rgi->gtid_sub_id, + in_trans, false, out_hton); -bool Xid_log_event::write() -{ - DBUG_EXECUTE_IF("do_not_write_xid", return 0;); - return write_header(sizeof(xid)) || - write_data((uchar*)&xid, sizeof(xid)) || - write_footer(); -} + if (unlikely(err)) + { + int ec= thd->get_stmt_da()->sql_errno(); + /* + Do not report an error if this is really a kill due to a deadlock. + In this case, the transaction will be re-tried instead. + */ + if (!is_parallel_retry_error(rgi, ec)) + rli->report(ERROR_LEVEL, ER_CANNOT_UPDATE_GTID_STATE, rgi->gtid_info(), + "Error during XID COMMIT: failed to update GTID state in " + "%s.%s: %d: %s", + "mysql", rpl_gtid_slave_state_table_name.str, ec, + thd->get_stmt_da()->message()); + thd->is_slave_error= 1; + } + return err; +} -#if defined(HAVE_REPLICATION) -int Xid_log_event::do_apply_event(rpl_group_info *rgi) +int Xid_apply_log_event::do_apply_event(rpl_group_info *rgi) { bool res; int err; - rpl_gtid gtid; uint64 sub_id= 0; - Relay_log_info const *rli= rgi->rli; void *hton= NULL; + rpl_gtid gtid; /* - XID_EVENT works like a COMMIT statement. And it also updates the - mysql.gtid_slave_pos table with the GTID of the current transaction. - + An instance of this class such as XID_EVENT works like a COMMIT + statement. It updates mysql.gtid_slave_pos with the GTID of the + current transaction. Therefore, it acts much like a normal SQL statement, so we need to do THD::reset_for_next_command() as if starting a new statement. + + XA_PREPARE_LOG_EVENT also updates the gtid table *but* the update gets + committed as separate "autocommit" transaction. */ thd->reset_for_next_command(); /* @@ -3824,57 +3897,50 @@ int Xid_log_event::do_apply_event(rpl_group_info *rgi) if (rgi->gtid_pending) { sub_id= rgi->gtid_sub_id; - rgi->gtid_pending= false; - gtid= rgi->current_gtid; - err= rpl_global_gtid_slave_state->record_gtid(thd, &gtid, sub_id, true, - false, &hton); - if (unlikely(err)) + + if (!thd->transaction.xid_state.is_explicit_XA()) { - int ec= thd->get_stmt_da()->sql_errno(); - /* - Do not report an error if this is really a kill due to a deadlock. - In this case, the transaction will be re-tried instead. - */ - if (!is_parallel_retry_error(rgi, ec)) - rli->report(ERROR_LEVEL, ER_CANNOT_UPDATE_GTID_STATE, rgi->gtid_info(), - "Error during XID COMMIT: failed to update GTID state in " - "%s.%s: %d: %s", - "mysql", rpl_gtid_slave_state_table_name.str, ec, - thd->get_stmt_da()->message()); - thd->is_slave_error= 1; - return err; + if ((err= do_record_gtid(thd, rgi, true /* in_trans */, &hton))) + return err; + + DBUG_EXECUTE_IF("gtid_fail_after_record_gtid", + { + my_error(ER_ERROR_DURING_COMMIT, MYF(0), + HA_ERR_WRONG_COMMAND); + thd->is_slave_error= 1; + return 1; + }); } - - DBUG_EXECUTE_IF("gtid_fail_after_record_gtid", - { my_error(ER_ERROR_DURING_COMMIT, MYF(0), HA_ERR_WRONG_COMMAND); - thd->is_slave_error= 1; - return 1; - }); } - /* For a slave Xid_log_event is COMMIT */ - general_log_print(thd, COM_QUERY, - "COMMIT /* implicit, from Xid_log_event */"); + general_log_print(thd, COM_QUERY, get_query()); thd->variables.option_bits&= ~OPTION_GTID_BEGIN; - res= trans_commit(thd); /* Automatically rolls back on error. */ - thd->mdl_context.release_transactional_locks(); + res= do_commit(); + if (!res && rgi->gtid_pending) + { + DBUG_ASSERT(!thd->transaction.xid_state.is_explicit_XA()); + if ((err= do_record_gtid(thd, rgi, false, &hton))) + return err; + } if (likely(!res) && sub_id) rpl_global_gtid_slave_state->update_state_hash(sub_id, &gtid, hton, rgi); /* Increment the global status commit count variable */ - status_var_increment(thd->status_var.com_stat[SQLCOM_COMMIT]); + enum enum_sql_command cmd= !thd->transaction.xid_state.is_explicit_XA() ? + SQLCOM_COMMIT : SQLCOM_XA_PREPARE; + status_var_increment(thd->status_var.com_stat[cmd]); return res; } Log_event::enum_skip_reason -Xid_log_event::do_shall_skip(rpl_group_info *rgi) +Xid_apply_log_event::do_shall_skip(rpl_group_info *rgi) { - DBUG_ENTER("Xid_log_event::do_shall_skip"); + DBUG_ENTER("Xid_apply_log_event::do_shall_skip"); if (rgi->rli->slave_skip_counter > 0) { DBUG_ASSERT(!rgi->rli->get_flag(Relay_log_info::IN_TRANSACTION)); @@ -3898,9 +3964,108 @@ Xid_log_event::do_shall_skip(rpl_group_info *rgi) #endif DBUG_RETURN(Log_event::do_shall_skip(rgi)); } +#endif /* HAVE_REPLICATION */ + +/************************************************************************** + Xid_log_event methods +**************************************************************************/ + +#if defined(HAVE_REPLICATION) +void Xid_log_event::pack_info(Protocol *protocol) +{ + char buf[128], *pos; + pos= strmov(buf, "COMMIT /* xid="); + pos= longlong10_to_str(xid, pos, 10); + pos= strmov(pos, " */"); + protocol->store(buf, (uint) (pos-buf), &my_charset_bin); +} + + +int Xid_log_event::do_commit() +{ + bool res; + res= trans_commit(thd); /* Automatically rolls back on error. */ + thd->mdl_context.release_transactional_locks(); + return res; +} +#endif + + +bool Xid_log_event::write() +{ + DBUG_EXECUTE_IF("do_not_write_xid", return 0;); + return write_header(sizeof(xid)) || + write_data((uchar*)&xid, sizeof(xid)) || + write_footer(); +} + +/************************************************************************** + XA_prepare_log_event methods +**************************************************************************/ + +#if defined(HAVE_REPLICATION) +void XA_prepare_log_event::pack_info(Protocol *protocol) +{ + char query[sizeof("XA COMMIT ONE PHASE") + 1 + ser_buf_size]; + + sprintf(query, + (one_phase ? "XA COMMIT %s ONE PHASE" : "XA PREPARE %s"), + m_xid.serialize()); + + protocol->store(query, strlen(query), &my_charset_bin); +} + + +int XA_prepare_log_event::do_commit() +{ + int res; + xid_t xid; + xid.set(m_xid.formatID, + m_xid.data, m_xid.gtrid_length, + m_xid.data + m_xid.gtrid_length, m_xid.bqual_length); + + thd->lex->xid= &xid; + if (!one_phase) + { + if ((res= thd->wait_for_prior_commit())) + return res; + + thd->lex->sql_command= SQLCOM_XA_PREPARE; + res= trans_xa_prepare(thd); + } + else + { + res= trans_xa_commit(thd); + thd->mdl_context.release_transactional_locks(); + } + + return res; +} #endif // HAVE_REPLICATION +bool XA_prepare_log_event::write() +{ + uchar data[1 + 4 + 4 + 4]= {one_phase,}; + uint8 one_phase_byte= one_phase; + + int4store(data+1, static_cast<XID*>(xid)->formatID); + int4store(data+(1+4), static_cast<XID*>(xid)->gtrid_length); + int4store(data+(1+4+4), static_cast<XID*>(xid)->bqual_length); + + DBUG_ASSERT(xid_subheader_no_data == sizeof(data) - 1); + + return write_header(sizeof(one_phase_byte) + xid_subheader_no_data + + static_cast<XID*>(xid)->gtrid_length + + static_cast<XID*>(xid)->bqual_length) || + write_data(data, sizeof(data)) || + write_data((uchar*) static_cast<XID*>(xid)->data, + static_cast<XID*>(xid)->gtrid_length + + static_cast<XID*>(xid)->bqual_length) || + write_footer(); +} + + /************************************************************************** User_var_log_event methods **************************************************************************/ @@ -8303,7 +8468,6 @@ bool event_that_should_be_ignored(const char *buf) event_type == PREVIOUS_GTIDS_LOG_EVENT || event_type == TRANSACTION_CONTEXT_EVENT || event_type == VIEW_CHANGE_EVENT || - event_type == XA_PREPARE_LOG_EVENT || (uint2korr(buf + FLAGS_OFFSET) & LOG_EVENT_IGNORABLE_F)) return 1; return 0; diff --git a/sql/rpl_parallel.cc b/sql/rpl_parallel.cc index 4313840119e..fc6475a170b 100644 --- a/sql/rpl_parallel.cc +++ b/sql/rpl_parallel.cc @@ -27,6 +27,38 @@ struct rpl_parallel_thread_pool global_rpl_thread_pool; static void signal_error_to_sql_driver_thread(THD *thd, rpl_group_info *rgi, int err); +struct XID_cache_insert_element +{ + XID *xid; + uint32 worker_idx; + + XID_cache_insert_element(XID *xid_arg, uint32 idx_arg): + xid(xid_arg), worker_idx(idx_arg) {} +}; + +class XID_cache_element_para +{ +public: + XID xid; + uint32 worker_idx; + Atomic_counter<int32_t> cnt; // of consecutive namesake xa:s queued for exec + static void lf_hash_initializer(LF_HASH *hash __attribute__((unused)), + XID_cache_element_para *element, + XID_cache_insert_element *new_element) + { + element->xid.set(new_element->xid); + element->worker_idx= new_element->worker_idx; + element->cnt= 1; + } + static uchar *key(const XID_cache_element_para *element, size_t *length, + my_bool) + { + *length= element->xid.key_length(); + return element->xid.key(); + } +}; + + static int rpt_handle_event(rpl_parallel_thread::queued_event *qev, struct rpl_parallel_thread *rpt) @@ -271,6 +303,33 @@ finish_event_group(rpl_parallel_thread *rpt, uint64 sub_id, */ thd->get_stmt_da()->reset_diagnostics_area(); wfc->wakeup_subsequent_commits(rgi->worker_error); + + if (!rgi->current_xid.is_null()) + { + Relay_log_info *rli= rgi->rli; + LF_PINS *pins= rli->parallel.rpl_xid_pins[rgi->xid_pins_idx]; + + DBUG_ASSERT(rgi->xid_pins_idx > 0 && + rgi->xid_pins_idx <= opt_slave_parallel_threads); + + XID_cache_element_para* el= + rli->parallel.xid_cache_search(&rgi->current_xid, pins); + + if (el) + { + lf_hash_search_unpin(pins); // it's safe unpin now none but us can delete + if (el->cnt-- == 1) // comparison aganst old value + (void) rli->parallel.xid_cache_delete(el, pins); + + DBUG_ASSERT(el->cnt >= 0); + } + else + { + // no record is begign when replication resumes after XA PREPARE. + sql_print_warning(ER_THD(rli->sql_driver_thd, ER_XAER_NOTA)); + } + } + rgi->current_xid.null(); } @@ -672,12 +731,14 @@ convert_kill_to_deadlock_error(rpl_group_info *rgi) static int is_group_ending(Log_event *ev, Log_event_type event_type) { - if (event_type == XID_EVENT) + if (event_type == XID_EVENT || event_type == XA_PREPARE_LOG_EVENT) return 1; if (event_type == QUERY_EVENT) // COMMIT/ROLLBACK are never compressed { Query_log_event *qev = (Query_log_event *)ev; - if (qev->is_commit()) + if (qev->is_commit() || + !strncmp(qev->query, STRING_WITH_LEN("XA COMMIT")) || + !strncmp(qev->query, STRING_WITH_LEN("XA ROLLBACK"))) return 1; if (qev->is_rollback()) return 2; @@ -1269,6 +1330,12 @@ handle_rpl_parallel_thread(void *arg) slave_output_error_info(rgi, thd); signal_error_to_sql_driver_thread(thd, rgi, 1); } + if (static_cast<Gtid_log_event*>(qev->ev)-> + flags2 & Gtid_log_event::FL_COMPLETED_XA) + { + rgi->current_xid.set(&static_cast<Gtid_log_event*>(qev->ev)->xid); + rgi->xid_pins_idx= static_cast<Gtid_log_event*>(qev->ev)->xid_pins_idx; + } } group_rgi= rgi; @@ -2090,24 +2157,71 @@ rpl_parallel_thread_pool::release_thread(rpl_parallel_thread *rpt) If the flag `reuse' is set, the last worker thread will be returned again, if it is still available. Otherwise a new worker thread is allocated. + + XA flagged as COMPLETED or PREPARED are handled as the following. + + For the PREPARED one, a record consisting of the xid and a choosen worker's + descriptor is inserted into a local rli parallel xid hash. + The record also contains a usage counter field to account for + "duplicate" xid:s which may arise naturally as the result of the + same xid transaction multiple times execution on master. Each emerging + PREPARED xa increments the usage counter of the record keyed by xid. + + For the COMPLETED xa, its xid is searched in the local hash for the + xa prepared worker. A found record's worker is reused, and when not + found (which may be benign) a new worker is allocated by the regular rule. + + While the driver thread is responsible to insert a xid record into + the local hash and possibly increment its usage counter, a Worker assigned + for that xid decrements the counter at the end of xa's completion and deletes + the record when the counter drops to zero. + Pins associated with the Worker are passed to it through Gtid_log_event::pins + of the passed pointer. */ rpl_parallel_thread * rpl_parallel_entry::choose_thread(rpl_group_info *rgi, bool *did_enter_cond, - PSI_stage_info *old_stage, bool reuse) + PSI_stage_info *old_stage, + Gtid_log_event *gtid_ev) { uint32 idx; Relay_log_info *rli= rgi->rli; rpl_parallel_thread *thr; + bool reuse= gtid_ev == NULL; idx= rpl_thread_idx; + if (!reuse) { + if (gtid_ev->flags2 & + (Gtid_log_event::FL_COMPLETED_XA | Gtid_log_event::FL_PREPARED_XA)) + { + LF_PINS *pins= rli->parallel.rpl_xid_pins[0]; + XID_cache_element_para* el= rli->parallel.xid_cache_search(&gtid_ev->xid, + pins); + + if (el) + { + idx= el->worker_idx; + lf_hash_search_unpin(pins); + goto idx_assigned; + } + else + { + // Further execution will clear out whether it's indeed the error case. + // XA completion event may arrive without the prepare one done so. + if (gtid_ev->flags2 & Gtid_log_event::FL_COMPLETED_XA) + sql_print_warning(ER_THD(rli->sql_driver_thd, ER_XAER_NOTA)); + } + } ++idx; if (idx >= rpl_thread_max) idx= 0; + +idx_assigned: rpl_thread_idx= idx; } thr= rpl_threads[idx]; + if (thr) { *did_enter_cond= false; @@ -2177,6 +2291,14 @@ rpl_parallel_entry::choose_thread(rpl_group_info *rgi, bool *did_enter_cond, rpl_threads[idx]= thr= global_rpl_thread_pool.get_thread(&rpl_threads[idx], this); + if (thr && gtid_ev) + { + if (gtid_ev->flags2 & Gtid_log_event::FL_PREPARED_XA) + (void) rli->parallel.xid_cache_replace(&gtid_ev->xid, idx); + else if (gtid_ev->flags2 & Gtid_log_event::FL_COMPLETED_XA) + gtid_ev->xid_pins_idx= idx + 1; // pass pins index to the assigned worker + } + return thr; } @@ -2205,12 +2327,21 @@ rpl_parallel::rpl_parallel() : } -void -rpl_parallel::reset() +bool +rpl_parallel::reset(bool is_parallel) { my_hash_reset(&domain_hash); current= NULL; sql_thread_stopping= false; + if (is_parallel) + { + xid_cache_init(); + rpl_xid_pins= new LF_PINS*[opt_slave_parallel_threads + 1]; + for (ulong i= 0; i <= opt_slave_parallel_threads; i++) + if (!(rpl_xid_pins[i]= lf_hash_get_pins(&xid_cache_para))) + return true; + } + return false; } @@ -2473,6 +2604,76 @@ rpl_parallel::wait_for_workers_idle(THD *thd) } +void rpl_parallel::xid_cache_init() +{ + lf_hash_init(&xid_cache_para, sizeof(XID_cache_element_para), + LF_HASH_UNIQUE, 0, 0, + (my_hash_get_key) XID_cache_element_para::key, &my_charset_bin); + xid_cache_para.alloc.constructor= NULL; + xid_cache_para.alloc.destructor= NULL; + xid_cache_para.initializer= + (lf_hash_initializer) XID_cache_element_para::lf_hash_initializer; +} + + +void rpl_parallel::xid_cache_free() +{ + lf_hash_destroy(&xid_cache_para); +} + + +XID_cache_element_para* rpl_parallel::xid_cache_search(XID *xid, LF_PINS* pins) +{ + return (XID_cache_element_para*) lf_hash_search(&xid_cache_para, pins, + xid->key(), + xid->key_length()); +} + +/** + Insert a first xid-keyed record, or "replace" it with incremented + usage counter. +*/ +void rpl_parallel::xid_cache_replace(XID *xid, uint32 idx) +{ + LF_PINS *pins= rpl_xid_pins[idx + 1]; + XID_cache_element_para* el= xid_cache_search(xid, pins); + + if (el) + { + if (unlikely(el->cnt++ == 0)) + { + lf_hash_search_unpin(pins); + while (xid_cache_search(xid, pins)) // record must be at being deleted + (void) LF_BACKOFF(); + lf_hash_search_unpin(pins); + (void) xid_cache_insert(xid, idx); + } + + DBUG_ASSERT(el->cnt > 0); + } + else + { + (void) xid_cache_insert(xid, idx); + } + lf_hash_search_unpin(pins); +} + +bool rpl_parallel::xid_cache_insert(XID *xid, uint32 idx) +{ + LF_PINS *pins= rpl_xid_pins[idx + 1]; + XID_cache_insert_element new_element(xid, idx); + + return lf_hash_insert(&xid_cache_para, pins, &new_element); +} + + +bool rpl_parallel::xid_cache_delete(XID_cache_element_para* el, LF_PINS *pins) +{ + return lf_hash_delete(&xid_cache_para, pins, + el->xid.key(), el->xid.key_length()); +} + + /* Handle seeing a GTID during slave restart in GTID mode. If we stopped with different replication domains having reached different positions in the relay @@ -2662,7 +2863,7 @@ rpl_parallel::do_event(rpl_group_info *serial_rgi, Log_event *ev, else { DBUG_ASSERT(rli->gtid_skip_flag == GTID_SKIP_TRANSACTION); - if (typ == XID_EVENT || + if (typ == XID_EVENT || typ == XA_PREPARE_LOG_EVENT || (typ == QUERY_EVENT && // COMMIT/ROLLBACK are never compressed (((Query_log_event *)ev)->is_commit() || ((Query_log_event *)ev)->is_rollback()))) @@ -2673,10 +2874,11 @@ rpl_parallel::do_event(rpl_group_info *serial_rgi, Log_event *ev, } } + Gtid_log_event *gtid_ev= NULL; if (typ == GTID_EVENT) { rpl_gtid gtid; - Gtid_log_event *gtid_ev= static_cast<Gtid_log_event *>(ev); + gtid_ev= static_cast<Gtid_log_event *>(ev); uint32 domain_id= (rli->mi->using_gtid == Master_info::USE_GTID_NO || rli->mi->parallel_mode <= SLAVE_PARALLEL_MINIMAL ? 0 : gtid_ev->domain_id); @@ -2715,8 +2917,7 @@ rpl_parallel::do_event(rpl_group_info *serial_rgi, Log_event *ev, instead re-use a thread that we queued for previously. */ cur_thread= - e->choose_thread(serial_rgi, &did_enter_cond, &old_stage, - typ != GTID_EVENT); + e->choose_thread(serial_rgi, &did_enter_cond, &old_stage, gtid_ev); if (!cur_thread) { /* This means we were killed. The error is already signalled. */ @@ -2734,7 +2935,6 @@ rpl_parallel::do_event(rpl_group_info *serial_rgi, Log_event *ev, if (typ == GTID_EVENT) { - Gtid_log_event *gtid_ev= static_cast<Gtid_log_event *>(ev); bool new_gco; enum_slave_parallel_mode mode= rli->mi->parallel_mode; uchar gtid_flags= gtid_ev->flags2; diff --git a/sql/rpl_parallel.h b/sql/rpl_parallel.h index 4579d0da9bc..b27bc63255e 100644 --- a/sql/rpl_parallel.h +++ b/sql/rpl_parallel.h @@ -2,7 +2,7 @@ #define RPL_PARALLEL_H #include "log_event.h" - +#include "lf.h" struct rpl_parallel; struct rpl_parallel_entry; @@ -345,10 +345,14 @@ struct rpl_parallel_entry { group_commit_orderer *current_gco; rpl_parallel_thread * choose_thread(rpl_group_info *rgi, bool *did_enter_cond, - PSI_stage_info *old_stage, bool reuse); + PSI_stage_info *old_stage, + Gtid_log_event *gtid_ev); int queue_master_restart(rpl_group_info *rgi, Format_description_log_event *fdev); }; + +class XID_cache_element_para; + struct rpl_parallel { HASH domain_hash; rpl_parallel_entry *current; @@ -356,13 +360,31 @@ struct rpl_parallel { rpl_parallel(); ~rpl_parallel(); - void reset(); + bool reset(bool is_parallel); rpl_parallel_entry *find(uint32 domain_id); void wait_for_done(THD *thd, Relay_log_info *rli); void stop_during_until(); bool workers_idle(); int wait_for_workers_idle(THD *thd); int do_event(rpl_group_info *serial_rgi, Log_event *ev, ulonglong event_size); + void leave(THD *thd, Relay_log_info *rli) + { + wait_for_done(thd, rli); + for (ulong i= 0; i <= opt_slave_parallel_threads; i++) + lf_hash_put_pins(rpl_xid_pins[i]); + delete[] rpl_xid_pins; + xid_cache_free(); + }; + // XA related. API follows xa.h naming. + LF_HASH xid_cache_para; + LF_PINS **rpl_xid_pins; + + void xid_cache_init(); + void xid_cache_free(); + bool xid_cache_insert(XID *xid, uint32 idx); + void xid_cache_replace(XID *xid, uint32 idx); + bool xid_cache_delete(XID_cache_element_para *el, LF_PINS *pins); + XID_cache_element_para *xid_cache_search(XID *xid, LF_PINS *pins); }; diff --git a/sql/rpl_rli.cc b/sql/rpl_rli.cc index 6d55b06b497..ab035afb276 100644 --- a/sql/rpl_rli.cc +++ b/sql/rpl_rli.cc @@ -35,7 +35,7 @@ #include "sql_table.h" static int count_relay_log_space(Relay_log_info* rli); - +bool xa_trans_force_rollback(THD *thd); /** Current replication state (hash of last GTID executed, per replication domain). @@ -2103,13 +2103,15 @@ rpl_group_info::reinit(Relay_log_info *rli) rpl_group_info::rpl_group_info(Relay_log_info *rli) : thd(0), wait_commit_sub_id(0), wait_commit_group_info(0), parallel_entry(0), - deferred_events(NULL), m_annotate_event(0), is_parallel_exec(false) + deferred_events(NULL), m_annotate_event(0), is_parallel_exec(false), + xid_pins_idx(0) { reinit(rli); bzero(&current_gtid, sizeof(current_gtid)); mysql_mutex_init(key_rpl_group_info_sleep_lock, &sleep_lock, MY_MUTEX_INIT_FAST); mysql_cond_init(key_rpl_group_info_sleep_cond, &sleep_cond, NULL); + current_xid.null(); } @@ -2230,6 +2232,14 @@ void rpl_group_info::cleanup_context(THD *thd, bool error) if (unlikely(error)) { + /*Todo/fixme: does it still not hold? Sort out and optimize if does not. + trans_rollback above does not rollback XA transactions. + It could be done only after necessarily closing tables which dictates + the following placement. + */ + if (thd->transaction.xid_state.is_explicit_XA()) + xa_trans_force_rollback(thd); + thd->mdl_context.release_transactional_locks(); if (thd == rli->sql_driver_thd) diff --git a/sql/rpl_rli.h b/sql/rpl_rli.h index 0e2e42fcb08..0b7d7f6a377 100644 --- a/sql/rpl_rli.h +++ b/sql/rpl_rli.h @@ -808,6 +808,10 @@ struct rpl_group_info }; uchar killed_for_retry; + /* A store to remember xid of being completed XA */ + XID current_xid; + uint32 xid_pins_idx; /* xid pins index to use by XA competing worker */ + rpl_group_info(Relay_log_info *rli_); ~rpl_group_info(); void reinit(Relay_log_info *rli); diff --git a/sql/slave.cc b/sql/slave.cc index 87c1cf6cb77..4b3ad57fe41 100644 --- a/sql/slave.cc +++ b/sql/slave.cc @@ -4230,7 +4230,7 @@ inline void update_state_of_relay_log(Relay_log_info *rli, Log_event *ev) rli->clear_flag(Relay_log_info::IN_TRANSACTION); } } - if (typ == XID_EVENT) + if (typ == XID_EVENT || typ == XA_PREPARE_LOG_EVENT) rli->clear_flag(Relay_log_info::IN_TRANSACTION); if (typ == GTID_EVENT && !(((Gtid_log_event*) ev)->flags2 & Gtid_log_event::FL_STANDALONE)) @@ -5434,7 +5434,6 @@ pthread_handler_t handle_slave_sql(void *arg) But the master timestamp is reset by RESET SLAVE & CHANGE MASTER. */ rli->clear_error(); - rli->parallel.reset(); //tell the I/O thread to take relay_log_space_limit into account from now on rli->ignore_log_space_limit= 0; @@ -5601,7 +5600,9 @@ pthread_handler_t handle_slave_sql(void *arg) } #endif /* WITH_WSREP */ /* Read queries from the IO/THREAD until this thread is killed */ - + if (rli->parallel.reset(mi->using_parallel())) + rli->report(ERROR_LEVEL, ER_SLAVE_FATAL_ERROR, NULL, + "Error initializing parallel mode"); thd->set_command(COM_SLAVE_SQL); while (!sql_slave_killed(serial_rgi)) { @@ -5663,7 +5664,7 @@ pthread_handler_t handle_slave_sql(void *arg) err: if (mi->using_parallel()) - rli->parallel.wait_for_done(thd, rli); + rli->parallel.leave(thd, rli); /* Thread stopped. Print the current replication position to the log */ { @@ -7095,6 +7096,7 @@ static int queue_event(Master_info* mi,const char* buf, ulong event_len) buf[EVENT_TYPE_OFFSET])) || (!mi->last_queued_gtid_standalone && ((uchar)buf[EVENT_TYPE_OFFSET] == XID_EVENT || + (uchar)buf[EVENT_TYPE_OFFSET] == XA_PREPARE_LOG_EVENT || ((uchar)buf[EVENT_TYPE_OFFSET] == QUERY_EVENT && /* QUERY_COMPRESSED_EVENT would never be commmit or rollback */ Query_log_event::peek_is_commit_rollback(buf, event_len, checksum_alg)))))) diff --git a/sql/sql_repl.cc b/sql/sql_repl.cc index 5bfa29b72c4..fd7fa89f227 100644 --- a/sql/sql_repl.cc +++ b/sql/sql_repl.cc @@ -1651,7 +1651,7 @@ is_until_reached(binlog_send_info *info, ulong *ev_offset, return false; break; case GTID_UNTIL_STOP_AFTER_TRANSACTION: - if (event_type != XID_EVENT && + if (event_type != XID_EVENT && event_type != XA_PREPARE_LOG_EVENT && (event_type != QUERY_EVENT || /* QUERY_COMPRESSED_EVENT would never be commmit or rollback */ !Query_log_event::peek_is_commit_rollback (info->packet->ptr()+*ev_offset, @@ -1886,7 +1886,7 @@ send_event_to_slave(binlog_send_info *info, Log_event_type event_type, info->gtid_skip_group= GTID_SKIP_NOT; return NULL; case GTID_SKIP_TRANSACTION: - if (event_type == XID_EVENT || + if (event_type == XID_EVENT || event_type == XA_PREPARE_LOG_EVENT || (event_type == QUERY_EVENT && /* QUERY_COMPRESSED_EVENT would never be commmit or rollback */ Query_log_event::peek_is_commit_rollback(packet->ptr() + ev_offset, len - ev_offset, diff --git a/sql/xa.cc b/sql/xa.cc index e4cad40318e..da6c9c93157 100644 --- a/sql/xa.cc +++ b/sql/xa.cc @@ -20,13 +20,11 @@ #include "sql_class.h" #include "transaction.h" +static bool slave_applier_reset_xa_trans(THD *thd); /*************************************************************************** Handling of XA id cacheing ***************************************************************************/ -enum xa_states { XA_ACTIVE= 0, XA_IDLE, XA_PREPARED, XA_ROLLBACK_ONLY }; - - struct XID_cache_insert_element { enum xa_states xa_state; @@ -78,6 +76,7 @@ class XID_cache_element uint rm_error; enum xa_states xa_state; XID xid; + bool binlogged; bool is_set(int32_t flag) { return m_state.load(std::memory_order_relaxed) & flag; } void set(int32_t flag) @@ -131,6 +130,7 @@ class XID_cache_element { DBUG_ASSERT(!element->is_set(ACQUIRED | RECOVERED)); element->rm_error= 0; + element->binlogged= false; element->xa_state= new_element->xa_state; element->xid.set(new_element->xid); new_element->xid_cache_element= element; @@ -158,6 +158,29 @@ static LF_HASH xid_cache; static bool xid_cache_inited; +bool XID_STATE::is_binlogged() +{ + return is_explicit_XA() && xid_cache_element->binlogged; +} + + +void XID_STATE::set_binlogged() +{ + if (xid_cache_element) + xid_cache_element->binlogged= true; +} + + +void XID_STATE::unset_binlogged() +{ + if (xid_cache_element) + xid_cache_element->binlogged= false; +} + + +enum xa_states XID_STATE::get_state_code() { return xid_cache_element->xa_state; } + + bool THD::fix_xid_hash_pins() { if (!xid_hash_pins) @@ -267,6 +290,7 @@ bool xid_cache_insert(XID *xid) { case 0: new_element.xid_cache_element->set(XID_cache_element::RECOVERED); + new_element.xid_cache_element->binlogged= true; break; case 1: res= 0; @@ -308,7 +332,11 @@ static void xid_cache_delete(THD *thd, XID_cache_element *&element) void xid_cache_delete(THD *thd, XID_STATE *xid_state) { - DBUG_ASSERT(xid_state->is_explicit_XA()); + DBUG_ASSERT(xid_state->is_explicit_XA() || thd->lex->xa_opt == XA_ONE_PHASE); + + if (!xid_state->is_explicit_XA()) + return; + xid_cache_delete(thd, xid_state->xid_cache_element); xid_state->xid_cache_element= 0; } @@ -380,7 +408,7 @@ static bool xa_trans_rolled_back(XID_cache_element *element) @return TRUE if the rollback failed, FALSE otherwise. */ -static bool xa_trans_force_rollback(THD *thd) +bool xa_trans_force_rollback(THD *thd) { bool rc= false; @@ -389,8 +417,8 @@ static bool xa_trans_force_rollback(THD *thd) my_error(ER_XAER_RMERR, MYF(0)); rc= true; } - - thd->variables.option_bits&= ~(OPTION_BEGIN | OPTION_KEEP_LOG); + thd->variables.option_bits&= + ~(OPTION_BEGIN | OPTION_KEEP_LOG | OPTION_GTID_BEGIN); thd->transaction.all.reset(); thd->server_status&= ~(SERVER_STATUS_IN_TRANS | SERVER_STATUS_IN_TRANS_READONLY); @@ -492,6 +520,8 @@ bool trans_xa_end(THD *thd) bool trans_xa_prepare(THD *thd) { + int res= 1; + DBUG_ENTER("trans_xa_prepare"); if (!thd->transaction.xid_state.is_explicit_XA() || @@ -499,16 +529,40 @@ bool trans_xa_prepare(THD *thd) thd->transaction.xid_state.er_xaer_rmfail(); else if (!thd->transaction.xid_state.xid_cache_element->xid.eq(thd->lex->xid)) my_error(ER_XAER_NOTA, MYF(0)); - else if (ha_prepare(thd)) + else { - xid_cache_delete(thd, &thd->transaction.xid_state); - my_error(ER_XA_RBROLLBACK, MYF(0)); + /* + Acquire metadata lock which will ensure that COMMIT is blocked + by active FLUSH TABLES WITH READ LOCK (and vice versa COMMIT in + progress blocks FTWRL). + + We allow FLUSHer to COMMIT; we assume FLUSHer knows what it does. + */ + MDL_request mdl_request; + mdl_request.init(MDL_key::BACKUP, "", "", MDL_BACKUP_COMMIT, + MDL_STATEMENT); + if (thd->mdl_context.acquire_lock(&mdl_request, + thd->variables.lock_wait_timeout) || + ha_prepare(thd)) + { + if (!mdl_request.ticket) + ha_rollback_trans(thd, TRUE); + thd->variables.option_bits&= ~(OPTION_BEGIN | OPTION_KEEP_LOG); + thd->transaction.all.reset(); + thd->server_status&= + ~(SERVER_STATUS_IN_TRANS | SERVER_STATUS_IN_TRANS_READONLY); + xid_cache_delete(thd, &thd->transaction.xid_state); + my_error(ER_XA_RBROLLBACK, MYF(0)); + } + else + { + thd->transaction.xid_state.xid_cache_element->xa_state= XA_PREPARED; + res= thd->variables.pseudo_slave_mode || thd->slave_thread ? + slave_applier_reset_xa_trans(thd) : 0; + } } - else - thd->transaction.xid_state.xid_cache_element->xa_state= XA_PREPARED; - DBUG_RETURN(thd->is_error() || - thd->transaction.xid_state.xid_cache_element->xa_state != XA_PREPARED); + DBUG_RETURN(res); } @@ -523,11 +577,13 @@ bool trans_xa_prepare(THD *thd) bool trans_xa_commit(THD *thd) { - bool res= TRUE; + bool res= true; + XID_STATE &xid_state= thd->transaction.xid_state; + DBUG_ENTER("trans_xa_commit"); - if (!thd->transaction.xid_state.is_explicit_XA() || - !thd->transaction.xid_state.xid_cache_element->xid.eq(thd->lex->xid)) + if (!xid_state.is_explicit_XA() || + !xid_state.xid_cache_element->xid.eq(thd->lex->xid)) { if (thd->in_multi_stmt_transaction_mode() || thd->lex->xa_opt != XA_NONE) { @@ -543,7 +599,45 @@ bool trans_xa_commit(THD *thd) if (auto xs= xid_cache_search(thd, thd->lex->xid)) { res= xa_trans_rolled_back(xs); + /* + Acquire metadata lock which will ensure that COMMIT is blocked + by active FLUSH TABLES WITH READ LOCK (and vice versa COMMIT in + progress blocks FTWRL). + + We allow FLUSHer to COMMIT; we assume FLUSHer knows what it does. + */ + MDL_request mdl_request; + mdl_request.init(MDL_key::BACKUP, "", "", MDL_BACKUP_COMMIT, + MDL_STATEMENT); + if (thd->mdl_context.acquire_lock(&mdl_request, + thd->variables.lock_wait_timeout)) + { + /* + We can't rollback an XA transaction on lock failure due to + Innodb redo log and bin log update is involved in rollback. + Return error to user for a retry. + */ + DBUG_ASSERT(thd->is_error()); + + xs->acquired_to_recovered(); + DBUG_RETURN(true); + } + DBUG_ASSERT(!xid_state.xid_cache_element); + + DEBUG_SYNC(thd, "at_trans_xa_commit"); + if (thd->wait_for_prior_commit()) + { + DBUG_ASSERT(thd->is_error()); + + xs->acquired_to_recovered(); + DBUG_RETURN(true); + } + + xid_state.xid_cache_element= xs; ha_commit_or_rollback_by_xid(thd->lex->xid, !res); + xid_state.xid_cache_element= 0; + + res= res || thd->is_error(); xid_cache_delete(thd, xs); } else @@ -551,19 +645,20 @@ bool trans_xa_commit(THD *thd) DBUG_RETURN(res); } - if (xa_trans_rolled_back(thd->transaction.xid_state.xid_cache_element)) + if (xa_trans_rolled_back(xid_state.xid_cache_element)) { xa_trans_force_rollback(thd); DBUG_RETURN(thd->is_error()); } - else if (thd->transaction.xid_state.xid_cache_element->xa_state == XA_IDLE && + else if (xid_state.xid_cache_element->xa_state == XA_IDLE && thd->lex->xa_opt == XA_ONE_PHASE) { + xid_cache_delete(thd, &xid_state); int r= ha_commit_trans(thd, TRUE); if ((res= MY_TEST(r))) my_error(r == 1 ? ER_XA_RBROLLBACK : ER_XAER_RMERR, MYF(0)); } - else if (thd->transaction.xid_state.xid_cache_element->xa_state == XA_PREPARED && + else if (xid_state.xid_cache_element->xa_state == XA_PREPARED && thd->lex->xa_opt == XA_NONE) { MDL_request mdl_request; @@ -576,26 +671,30 @@ bool trans_xa_commit(THD *thd) We allow FLUSHer to COMMIT; we assume FLUSHer knows what it does. */ mdl_request.init(MDL_key::BACKUP, "", "", MDL_BACKUP_COMMIT, - MDL_TRANSACTION); + MDL_STATEMENT); if (thd->mdl_context.acquire_lock(&mdl_request, thd->variables.lock_wait_timeout)) { - ha_rollback_trans(thd, TRUE); + /* + We can't rollback an XA transaction on lock failure due to + Innodb redo log and bin log update is involved in rollback. + Return error to user for a retry. + */ my_error(ER_XAER_RMERR, MYF(0)); + DBUG_RETURN(true); } else { DEBUG_SYNC(thd, "trans_xa_commit_after_acquire_commit_lock"); - res= MY_TEST(ha_commit_one_phase(thd, 1)); - if (res) + if ((res= MY_TEST(ha_commit_one_phase(thd, 1)))) my_error(ER_XAER_RMERR, MYF(0)); } } else { - thd->transaction.xid_state.er_xaer_rmfail(); + xid_state.er_xaer_rmfail(); DBUG_RETURN(TRUE); } @@ -604,7 +703,7 @@ bool trans_xa_commit(THD *thd) thd->server_status&= ~(SERVER_STATUS_IN_TRANS | SERVER_STATUS_IN_TRANS_READONLY); DBUG_PRINT("info", ("clearing SERVER_STATUS_IN_TRANS")); - xid_cache_delete(thd, &thd->transaction.xid_state); + xid_cache_delete(thd, &xid_state); trans_track_end_trx(thd); @@ -623,10 +722,13 @@ bool trans_xa_commit(THD *thd) bool trans_xa_rollback(THD *thd) { + bool res= false; + XID_STATE &xid_state= thd->transaction.xid_state; + DBUG_ENTER("trans_xa_rollback"); - if (!thd->transaction.xid_state.is_explicit_XA() || - !thd->transaction.xid_state.xid_cache_element->xid.eq(thd->lex->xid)) + if (!xid_state.is_explicit_XA() || + !xid_state.xid_cache_element->xid.eq(thd->lex->xid)) { if (thd->in_multi_stmt_transaction_mode()) { @@ -641,8 +743,36 @@ bool trans_xa_rollback(THD *thd) if (auto xs= xid_cache_search(thd, thd->lex->xid)) { + MDL_request mdl_request; + mdl_request.init(MDL_key::BACKUP, "", "", MDL_BACKUP_COMMIT, + MDL_STATEMENT); + if (thd->mdl_context.acquire_lock(&mdl_request, + thd->variables.lock_wait_timeout)) + { + /* + We can't rollback an XA transaction on lock failure due to + Innodb redo log and bin log update is involved in rollback. + Return error to user for a retry. + */ + DBUG_ASSERT(thd->is_error()); + + xs->acquired_to_recovered(); + DBUG_RETURN(true); + } xa_trans_rolled_back(xs); + DBUG_ASSERT(!xid_state.xid_cache_element); + + DEBUG_SYNC(thd, "at_trans_xa_rollback"); + if (thd->wait_for_prior_commit()) + { + DBUG_ASSERT(thd->is_error()); + xs->acquired_to_recovered(); + DBUG_RETURN(true); + } + + xid_state.xid_cache_element= xs; ha_commit_or_rollback_by_xid(thd->lex->xid, 0); + xid_state.xid_cache_element= 0; xid_cache_delete(thd, xs); } else @@ -650,21 +780,35 @@ bool trans_xa_rollback(THD *thd) DBUG_RETURN(thd->get_stmt_da()->is_error()); } - if (thd->transaction.xid_state.xid_cache_element->xa_state == XA_ACTIVE) + if (xid_state.xid_cache_element->xa_state == XA_ACTIVE) { - thd->transaction.xid_state.er_xaer_rmfail(); + xid_state.er_xaer_rmfail(); DBUG_RETURN(TRUE); } - DBUG_RETURN(xa_trans_force_rollback(thd)); + + MDL_request mdl_request; + mdl_request.init(MDL_key::BACKUP, "", "", MDL_BACKUP_COMMIT, + MDL_STATEMENT); + if (thd->mdl_context.acquire_lock(&mdl_request, + thd->variables.lock_wait_timeout)) + { + /* + We can't rollback an XA transaction on lock failure due to + Innodb redo log and bin log update is involved in rollback. + Return error to user for a retry. + */ + my_error(ER_XAER_RMERR, MYF(0)); + DBUG_RETURN(true); + } + + DBUG_RETURN(res != 0 || xa_trans_force_rollback(thd)); } bool trans_xa_detach(THD *thd) { DBUG_ASSERT(thd->transaction.xid_state.is_explicit_XA()); -#if 1 - return xa_trans_force_rollback(thd); -#else + if (thd->transaction.xid_state.xid_cache_element->xa_state != XA_PREPARED) return xa_trans_force_rollback(thd); thd->transaction.xid_state.xid_cache_element->acquired_to_recovered(); @@ -683,7 +827,6 @@ bool trans_xa_detach(THD *thd) thd->transaction.all.ha_list= 0; thd->transaction.all.no_2pc= 0; return false; -#endif } @@ -877,3 +1020,44 @@ bool mysql_xa_recover(THD *thd) my_eof(thd); DBUG_RETURN(0); } + + +/** + This is a specific to (pseudo-) slave applier collection of standard cleanup + actions to reset XA transaction state sim to @c ha_commit_one_phase. + THD of the slave applier is dissociated from a transaction object in engine + that continues to exist there. + + @param THD current thread + @return the value of is_error() +*/ + +static bool slave_applier_reset_xa_trans(THD *thd) +{ + thd->variables.option_bits&= ~(OPTION_BEGIN | OPTION_KEEP_LOG); + thd->server_status&= + ~(SERVER_STATUS_IN_TRANS | SERVER_STATUS_IN_TRANS_READONLY); + DBUG_PRINT("info", ("clearing SERVER_STATUS_IN_TRANS")); + + thd->transaction.xid_state.xid_cache_element->acquired_to_recovered(); + thd->transaction.xid_state.xid_cache_element= 0; + + for (Ha_trx_info *ha_info= thd->transaction.all.ha_list, *ha_info_next; + ha_info; ha_info= ha_info_next) + { + ha_info_next= ha_info->next(); + ha_info->reset(); + } + thd->transaction.all.ha_list= 0; + + ha_close_connection(thd); + thd->transaction.cleanup(); + thd->transaction.all.reset(); + + DBUG_ASSERT(!thd->transaction.all.ha_list); + DBUG_ASSERT(!thd->transaction.all.no_2pc); + + thd->has_waiter= false; + + return thd->is_error(); +} diff --git a/sql/xa.h b/sql/xa.h index 7cf74efad35..507d07f638f 100644 --- a/sql/xa.h +++ b/sql/xa.h @@ -1,3 +1,5 @@ +#ifndef XA_INCLUDED +#define XA_INCLUDED /* Copyright (c) 2000, 2016, Oracle and/or its affiliates. Copyright (c) 2009, 2019, MariaDB Corporation. @@ -16,17 +18,31 @@ Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA */ - class XID_cache_element; +enum xa_states { XA_ACTIVE= 0, XA_IDLE, XA_PREPARED, XA_ROLLBACK_ONLY }; struct XID_STATE { XID_cache_element *xid_cache_element; - bool check_has_uncommitted_xa() const; bool is_explicit_XA() const { return xid_cache_element != 0; } + /* + Binary logging status of explicit "user" XA. + It is set to TRUE at XA PREPARE if the transaction was written + to the binlog. + It may be FALSE after preparing when the transaction does not modify + transactional tables or binlogging is turned off. + In that case a consequent XA COMMIT/ROLLBACK shouldn't be binlogged. + + The recovered transaction after server restart sets it to TRUE always. + */ + bool is_binlogged(); + bool check_has_uncommitted_xa() const; void set_error(uint error); void er_xaer_rmfail() const; XID *get_xid() const; + void set_binlogged(); + void unset_binlogged(); + enum xa_states get_state_code(); }; void xid_cache_init(void); @@ -42,3 +58,5 @@ bool trans_xa_commit(THD *thd); bool trans_xa_rollback(THD *thd); bool trans_xa_detach(THD *thd); bool mysql_xa_recover(THD *thd); + +#endif /* XA_INCLUDED */ diff --git a/storage/innobase/handler/ha_innodb.cc b/storage/innobase/handler/ha_innodb.cc index a36cc87a201..8a173001ed5 100644 --- a/storage/innobase/handler/ha_innodb.cc +++ b/storage/innobase/handler/ha_innodb.cc @@ -4844,6 +4844,7 @@ innobase_close_connection( if (trx->has_logged_persistent()) { trx_disconnect_prepared(trx); } else { + trx_rollback_for_mysql(trx); trx_deregister_from_2pc(trx); goto rollback_and_free; } diff --git a/storage/innobase/trx/trx0trx.cc b/storage/innobase/trx/trx0trx.cc index c5e48ff65ab..1004198ea0b 100644 --- a/storage/innobase/trx/trx0trx.cc +++ b/storage/innobase/trx/trx0trx.cc @@ -549,8 +549,10 @@ void trx_disconnect_prepared(trx_t *trx) trx->read_view.close(); trx->is_recovered= true; trx->mysql_thd= NULL; + trx->mysql_log_file_name = 0; /* todo/fixme: suggest to do it at innodb prepare */ trx->will_lock= 0; + trx_sys.rw_trx_hash.put_pins(trx); } /****************************************************************//** @@ -1390,8 +1392,21 @@ trx_commit_in_memory( trx->release_locks(); } - DEBUG_SYNC_C("after_trx_committed_in_memory"); - +#ifndef DBUG_OFF + const bool debug_sync = trx->mysql_thd && + trx->has_logged_persistent(); + /* In case of this function is called from a stack executing + THD::free_connection -> ... + innobase_connection_close() -> + trx_rollback_for_mysql... -> . + mysql's thd does not seem to have + thd->debug_sync_control defined any longer. However the stack + is possible only with a prepared trx not updating any data. + */ + if (debug_sync) { + DEBUG_SYNC_C("after_trx_committed_in_memory"); + } +#endif if (trx->read_only || !trx->rsegs.m_redo.rseg) { MONITOR_INC(MONITOR_TRX_RO_COMMIT); } else { diff --git a/storage/rocksdb/ha_rocksdb.cc b/storage/rocksdb/ha_rocksdb.cc index 8488f9ee963..272d2800319 100644 --- a/storage/rocksdb/ha_rocksdb.cc +++ b/storage/rocksdb/ha_rocksdb.cc @@ -3121,6 +3121,8 @@ class Rdb_transaction { s_tx_list.erase(this); RDB_MUTEX_UNLOCK_CHECK(s_tx_list_mutex); } + virtual bool is_prepared() { return false; }; + virtual void detach_prepared_tx() {}; }; /* @@ -3157,7 +3159,16 @@ class Rdb_transaction_impl : public Rdb_transaction { virtual bool is_writebatch_trx() const override { return false; } - private: + bool is_prepared() { + return m_rocksdb_tx && rocksdb::Transaction::PREPARED == m_rocksdb_tx->GetState(); + } + + void detach_prepared_tx() { + DBUG_ASSERT(rocksdb::Transaction::PREPARED == m_rocksdb_tx->GetState()); + m_rocksdb_tx = nullptr; + } + +private: void release_tx(void) { // We are done with the current active transaction object. Preserve it // for later reuse. @@ -3798,7 +3809,8 @@ static int rocksdb_close_connection(handlerton *const hton, THD *const thd) { "disconnecting", rc); } - + if (tx->is_prepared()) + tx->detach_prepared_tx(); delete tx; } return HA_EXIT_SUCCESS; @@ -5301,7 +5313,7 @@ static int rocksdb_init_func(void *const p) { #ifdef MARIAROCKS_NOT_YET rocksdb_hton->update_table_stats = rocksdb_update_table_stats; #endif // MARIAROCKS_NOT_YET - + /* Not needed in MariaDB: rocksdb_hton->flush_logs = rocksdb_flush_wal; diff --git a/storage/rocksdb/mysql-test/rocksdb/r/xa.result b/storage/rocksdb/mysql-test/rocksdb/r/xa.result index 12ae2b474b6..8cb6f39bbac 100644 --- a/storage/rocksdb/mysql-test/rocksdb/r/xa.result +++ b/storage/rocksdb/mysql-test/rocksdb/r/xa.result @@ -1,6 +1,7 @@ -# -# MDEV-13155: XA recovery not supported for RocksDB (Just a testcase) # +# MDEV-742 fixes +# MDEV-13155: XA recovery not supported for RocksDB +# as well. call mtr.add_suppression("Found .* prepared XA transactions"); connect con1,localhost,root,,test; DROP TABLE IF EXISTS t1; @@ -15,19 +16,55 @@ INSERT INTO t1 (a) VALUES (3); INSERT INTO t1 (a) VALUES (4); XA END 'xa2'; XA PREPARE 'xa2'; +connect con3,localhost,root,,test; +XA START 'xa3'; +INSERT INTO t1 (a) VALUES (5); +INSERT INTO t1 (a) VALUES (6); +XA END 'xa3'; +XA PREPARE 'xa3'; +disconnect con3; connection default; SELECT * FROM t1; a +Must be all three XA:s in +XA RECOVER; +formatID gtrid_length bqual_length data +1 3 0 xa3 +1 3 0 xa1 +1 3 0 xa2 # restart connect con3,localhost,root,,test; XA RECOVER; formatID gtrid_length bqual_length data +1 3 0 xa3 1 3 0 xa1 1 3 0 xa2 XA ROLLBACK 'xa1'; XA COMMIT 'xa2'; +XA ROLLBACK 'xa3'; +SELECT a FROM t1; +a +3 +4 +connect con4,localhost,root,,test; +XA START 'xa4'; +INSERT INTO t1 (a) VALUES (7); +INSERT INTO t1 (a) VALUES (8); +XA END 'xa4'; +XA PREPARE 'xa4'; +connection default; +# Now restart through graceful shutdown +# restart +connect con5,localhost,root,,test; +Must have 'xa4' +XA RECOVER; +formatID gtrid_length bqual_length data +1 3 0 xa4 +XA COMMIT 'xa4'; SELECT a FROM t1; a 3 4 +7 +8 DROP TABLE t1; diff --git a/storage/rocksdb/mysql-test/rocksdb/t/xa.test b/storage/rocksdb/mysql-test/rocksdb/t/xa.test index f8f381f0580..0c23e71df8c 100644 --- a/storage/rocksdb/mysql-test/rocksdb/t/xa.test +++ b/storage/rocksdb/mysql-test/rocksdb/t/xa.test @@ -1,6 +1,7 @@ ---echo # ---echo # MDEV-13155: XA recovery not supported for RocksDB (Just a testcase) --echo # +--echo # MDEV-742 fixes +--echo # MDEV-13155: XA recovery not supported for RocksDB +--echo # as well. call mtr.add_suppression("Found .* prepared XA transactions"); @@ -22,17 +23,51 @@ INSERT INTO t1 (a) VALUES (3); INSERT INTO t1 (a) VALUES (4); XA END 'xa2'; XA PREPARE 'xa2'; - + +--connect (con3,localhost,root,,test) +XA START 'xa3'; +INSERT INTO t1 (a) VALUES (5); +INSERT INTO t1 (a) VALUES (6); +XA END 'xa3'; +XA PREPARE 'xa3'; +--disconnect con3 + --connection default SELECT * FROM t1; +--echo Must be all three XA:s in +XA RECOVER; + --let $shutdown_timeout= 0 --source include/restart_mysqld.inc --connect (con3,localhost,root,,test) --disable_abort_on_error -XA RECOVER; +XA RECOVER; # like above XA ROLLBACK 'xa1'; XA COMMIT 'xa2'; +XA ROLLBACK 'xa3'; SELECT a FROM t1; + +--connect (con4,localhost,root,,test) +XA START 'xa4'; +INSERT INTO t1 (a) VALUES (7); +INSERT INTO t1 (a) VALUES (8); +XA END 'xa4'; +XA PREPARE 'xa4'; + +--connection default +--echo # Now restart through graceful shutdown +--source include/restart_mysqld.inc + + +--connect (con5,localhost,root,,test) +--disable_abort_on_error + +--echo Must have 'xa4' +XA RECOVER; +XA COMMIT 'xa4'; + +SELECT a FROM t1; + DROP TABLE t1; diff --git a/storage/rocksdb/mysql-test/rocksdb_rpl/r/rpl_xa.result b/storage/rocksdb/mysql-test/rocksdb_rpl/r/rpl_xa.result new file mode 100644 index 00000000000..b4713c68390 --- /dev/null +++ b/storage/rocksdb/mysql-test/rocksdb_rpl/r/rpl_xa.result @@ -0,0 +1,50 @@ +include/master-slave.inc +[connection master] +connection master; +create table t1 (a int, b int) engine=InnoDB; +insert into t1 values(0, 0); +xa start 't'; +insert into t1 values(1, 2); +xa end 't'; +xa prepare 't'; +xa commit 't'; +connection slave; +include/diff_tables.inc [master:t1, slave:t1] +connection master; +xa start 't'; +insert into t1 values(3, 4); +xa end 't'; +xa prepare 't'; +xa rollback 't'; +connection slave; +include/diff_tables.inc [master:t1, slave:t1] +connection master; +SET pseudo_slave_mode=1; +create table t2 (a int) engine=InnoDB; +xa start 't'; +insert into t1 values (5, 6); +xa end 't'; +xa prepare 't'; +xa start 's'; +insert into t2 values (0); +xa end 's'; +xa prepare 's'; +include/save_master_gtid.inc +connection slave; +include/sync_with_master_gtid.inc +xa recover; +formatID gtrid_length bqual_length data +1 1 0 t +1 1 0 s +connection master; +xa commit 't'; +xa commit 's'; +SET pseudo_slave_mode=0; +Warnings: +Warning 1231 Slave applier execution mode not active, statement ineffective. +connection slave; +include/diff_tables.inc [master:t1, slave:t1] +include/diff_tables.inc [master:t2, slave:t2] +connection master; +drop table t1, t2; +include/rpl_end.inc diff --git a/storage/rocksdb/mysql-test/rocksdb_rpl/t/rpl_xa.inc b/storage/rocksdb/mysql-test/rocksdb_rpl/t/rpl_xa.inc new file mode 100644 index 00000000000..c1300c1e27a --- /dev/null +++ b/storage/rocksdb/mysql-test/rocksdb_rpl/t/rpl_xa.inc @@ -0,0 +1,70 @@ +# +# This "body" file checks general properties of XA transaction replication +# as of MDEV-7974. +# Parameters: +# --let rpl_xa_check= SELECT ... +# +connection master; +create table t1 (a int, b int) engine=InnoDB; +insert into t1 values(0, 0); +xa start 't'; +insert into t1 values(1, 2); +xa end 't'; +xa prepare 't'; +xa commit 't'; + +sync_slave_with_master; +let $diff_tables= master:t1, slave:t1; +source include/diff_tables.inc; + +connection master; + +xa start 't'; +insert into t1 values(3, 4); +xa end 't'; +xa prepare 't'; +xa rollback 't'; + +sync_slave_with_master; +let $diff_tables= master:t1, slave:t1; +source include/diff_tables.inc; + +connection master; +SET pseudo_slave_mode=1; +create table t2 (a int) engine=InnoDB; +xa start 't'; +insert into t1 values (5, 6); +xa end 't'; +xa prepare 't'; +xa start 's'; +insert into t2 values (0); +xa end 's'; +xa prepare 's'; +--source include/save_master_gtid.inc + +connection slave; +source include/sync_with_master_gtid.inc; +if ($rpl_xa_check) +{ + --eval $rpl_xa_check + if ($rpl_xa_verbose) + { + --eval SELECT $rpl_xa_check_lhs + --eval SELECT $rpl_xa_check_rhs + } +} +xa recover; + +connection master; +xa commit 't'; +xa commit 's'; +SET pseudo_slave_mode=0; + +sync_slave_with_master; +let $diff_tables= master:t1, slave:t1; +source include/diff_tables.inc; +let $diff_tables= master:t2, slave:t2; +source include/diff_tables.inc; + +connection master; +drop table t1, t2; diff --git a/storage/rocksdb/mysql-test/rocksdb_rpl/t/rpl_xa.test b/storage/rocksdb/mysql-test/rocksdb_rpl/t/rpl_xa.test new file mode 100644 index 00000000000..7d667aa96d2 --- /dev/null +++ b/storage/rocksdb/mysql-test/rocksdb_rpl/t/rpl_xa.test @@ -0,0 +1,6 @@ +source include/have_rocksdb.inc; +source include/master-slave.inc; +source include/have_binlog_format_row.inc; + +source rpl_xa.inc; +source include/rpl_end.inc; diff --git a/storage/tokudb/mysql-test/tokudb_mariadb/r/xa.result b/storage/tokudb/mysql-test/tokudb_mariadb/r/xa.result index 4724a0af926..34233b6fd8d 100644 --- a/storage/tokudb/mysql-test/tokudb_mariadb/r/xa.result +++ b/storage/tokudb/mysql-test/tokudb_mariadb/r/xa.result @@ -65,4 +65,5 @@ a 20 disconnect con1; connection default; +xa rollback 'testb',0x2030405060,11; drop table t1; diff --git a/storage/tokudb/mysql-test/tokudb_mariadb/t/xa.test b/storage/tokudb/mysql-test/tokudb_mariadb/t/xa.test index dc5520a39b8..a6be07963f5 100644 --- a/storage/tokudb/mysql-test/tokudb_mariadb/t/xa.test +++ b/storage/tokudb/mysql-test/tokudb_mariadb/t/xa.test @@ -68,6 +68,9 @@ xa start 'zzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzzz'; select * from t1; disconnect con1; +xa recover; + connection default; +xa rollback 'testb',0x2030405060,11; drop table t1;

1 0

[Maria-developers] Seeking Guidance to contribute to the organization
by Jeeukrishnan 03 Mar '20

03 Mar '20

2 1

Re: [Maria-developers] 9aace068f0c: MDEV-21743 Split up SUPER privilege to smaller privileges
by Sergei Golubchik 01 Mar '20

01 Mar '20

Hi, Alexander! Just a couple of issues that were left open after our slack chat. Everything else looks fine, nothing new below. On Mar 01, Alexander Barkov wrote: > revision-id: 9aace068f0c (mariadb-10.5.0-277-g9aace068f0c) > parent(s): 607960c7722 > author: Alexander Barkov <bar(a)mariadb.com> > committer: Alexander Barkov <bar(a)mariadb.com> > timestamp: 2020-03-01 11:49:25 +0400 > message: > > MDEV-21743 Split up SUPER privilege to smaller privileges > +constexpr uint PRIVILEGE_T_MAX_BIT= 36; > + > +static_assert((privilege_t)(1ULL << PRIVILEGE_T_MAX_BIT) == LAST_CURRENT_ACL, > + "LAST_CURRENT_ACL and PRIVILEGE_T_MAX_BIT do not match"); I'd still prefer to get rid of static_assert here. E.g. with constexpr uint PRIVILEGE_T_MAX_BIT= my_bit_log2(LAST_CURRENT_ACL); (and fixing my_bit_log2 to work with ulonglong). > @@ -7151,6 +7150,18 @@ bool check_global_access(THD *thd, privilege_t want_access, bool no_errors) > { > #ifndef NO_EMBEDDED_ACCESS_CHECKS > char command[128]; > + /* > + The userstat plugin in > + plugin/userstat/client_stats.cc > + plugin/userstat/user_stats.cc > + calls > + check_global_access(SUPER_ACL | PROCESS_ACL) > + when populating CLIENT_STATISTICS and USER_STATISTICS > + INFORMATION_SCHEMA tables. > + We should eventually fix it to use one privilege only and uncomment > + the DBUG_ASSERT below: > + */ > + //DBUG_ASSERT(my_count_bits(want_access) <= 1); I think you can remove SUPER_ACL from the userstat plugin and uncomment the assert above. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 256753e8ae8: Clean up and speed up interfaces for binary row logging
by Sergei Golubchik 01 Mar '20

01 Mar '20

Hi, Michael! Just a few comments, see below On Feb 28, Michael Widenius wrote: > revision-id: 256753e8ae8 (mariadb-10.5.0-269-g256753e8ae8) > parent(s): 9fd961cf687 > author: Michael Widenius <monty(a)mariadb.com> > committer: Michael Widenius <monty(a)mariadb.com> > timestamp: 2020-02-26 16:05:53 +0200 > message: > > Clean up and speed up interfaces for binary row logging ... > diff --git a/mysql-test/suite/rpl/r/create_or_replace_mix.result b/mysql-test/suite/rpl/r/create_or_replace_mix.result > index 661278aa7ef..6c83d27eef9 100644 > --- a/mysql-test/suite/rpl/r/create_or_replace_mix.result > +++ b/mysql-test/suite/rpl/r/create_or_replace_mix.result > @@ -223,26 +226,12 @@ Log_name Pos Event_type Server_id End_log_pos Info > slave-bin.000001 # Gtid # # GTID #-#-# > slave-bin.000001 # Query # # use `test`; create table t1 (a int) > slave-bin.000001 # Gtid # # BEGIN GTID #-#-# > -slave-bin.000001 # Annotate_rows # # insert into t1 values (0),(1),(2) > -slave-bin.000001 # Table_map # # table_id: # (test.t1) > -slave-bin.000001 # Write_rows_v1 # # table_id: # flags: STMT_END_F > +slave-bin.000001 # Query # # use `test`; insert into t1 values (0),(1),(2) why is it STATEMENT now? > slave-bin.000001 # Query # # COMMIT > -slave-bin.000001 # Gtid # # BEGIN GTID #-#-# > -slave-bin.000001 # Query # # use `test`; CREATE TABLE `t2` ( > - `a` int(11) DEFAULT NULL > -) ENGINE=MyISAM > -slave-bin.000001 # Annotate_rows # # create table t2 engine=myisam select * from t1 > -slave-bin.000001 # Table_map # # table_id: # (test.t2) > -slave-bin.000001 # Write_rows_v1 # # table_id: # flags: STMT_END_F > -slave-bin.000001 # Query # # COMMIT > -slave-bin.000001 # Gtid # # BEGIN GTID #-#-# > -slave-bin.000001 # Query # # use `test`; CREATE OR REPLACE TABLE `t2` ( > - `a` int(11) DEFAULT NULL > -) ENGINE=InnoDB > -slave-bin.000001 # Annotate_rows # # create or replace table t2 engine=innodb select * from t1 > -slave-bin.000001 # Table_map # # table_id: # (test.t2) > -slave-bin.000001 # Write_rows_v1 # # table_id: # flags: STMT_END_F > -slave-bin.000001 # Xid # # COMMIT /* XID */ > +slave-bin.000001 # Gtid # # GTID #-#-# > +slave-bin.000001 # Query # # use `test`; create table t2 engine=myisam select * from t1 > +slave-bin.000001 # Gtid # # GTID #-#-# > +slave-bin.000001 # Query # # use `test`; create or replace table t2 engine=innodb select * from t1 > connection server_1; > drop table t1; > # > diff --git a/mysql-test/suite/rpl/t/rpl_foreign_key.test b/mysql-test/suite/rpl/t/rpl_foreign_key.test > new file mode 100644 > index 00000000000..50be97af24d > --- /dev/null > +++ b/mysql-test/suite/rpl/t/rpl_foreign_key.test > @@ -0,0 +1,18 @@ > +--source include/have_innodb.inc > +--source include/have_binlog_format_row.inc > + > +CREATE TABLE t1 ( > + id INT, > + k INT, > + c CHAR(8), > + KEY (k), > + PRIMARY KEY (id), > + FOREIGN KEY (id) REFERENCES t1 (k) > +) ENGINE=InnoDB; > +LOCK TABLES t1 WRITE; > +SET SESSION FOREIGN_KEY_CHECKS= OFF; > +SET AUTOCOMMIT=OFF; > +INSERT INTO t1 VALUES (1,1,'foo'); > +DROP TABLE t1; > +SET SESSION FOREIGN_KEY_CHECKS= ON; > +SET AUTOCOMMIT=ON; What kind of replication test is it? You don't check binlog events, you don't compare master and slave, you don't even run anything on the slave to check whether your staments were replicated correctly. In fact, you don't have any slave, you have not included master-slave.inc, you only have binlog, so this test should be in the binlog suite, not in the rpl suite - it's a binlog test, not replication test. And even in the binlog suite it would make sense to see what's actually in a binlog. Just as a test that it's ok. > diff --git a/sql/ha_sequence.cc b/sql/ha_sequence.cc > index 6cb9937ebb4..71da208d775 100644 > --- a/sql/ha_sequence.cc > +++ b/sql/ha_sequence.cc > @@ -202,7 +202,11 @@ int ha_sequence::write_row(const uchar *buf) > DBUG_ENTER("ha_sequence::write_row"); > DBUG_ASSERT(table->record[0] == buf); > > - row_already_logged= 0; > + /* > + Log to binary log even if this function has been called before > + (The function ends by setting row_logging to 0) > + */ > + row_logging= row_logging_init; this is a sequence-specific hack, so you should define row_logging_init in ha_sequence class, not in the base handler class > if (unlikely(sequence->initialized == SEQUENCE::SEQ_IN_PREPARE)) > { > /* This calls is from ha_open() as part of create table */ > diff --git a/sql/handler.cc b/sql/handler.cc > index 1e3f987b4e5..4096ae8b90f 100644 > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -6224,32 +6225,37 @@ bool ha_show_status(THD *thd, handlerton *db_type, enum ha_stat_type stat) > 1 Row needs to be logged > */ > > -bool handler::check_table_binlog_row_based(bool binlog_row) > +bool handler::check_table_binlog_row_based() > { > - if (table->versioned(VERS_TRX_ID)) > - return false; > - if (unlikely((table->in_use->variables.sql_log_bin_off))) > - return 0; /* Called by partitioning engine */ > #ifdef WITH_WSREP > - if (!table->in_use->variables.sql_log_bin && > - wsrep_thd_is_applying(table->in_use)) > - return 0; /* wsrep patch sets sql_log_bin to silence binlogging > - from high priority threads */ > #endif /* WITH_WSREP */ That's an empty #ifdef :) > if (unlikely((!check_table_binlog_row_based_done))) > { > check_table_binlog_row_based_done= 1; > check_table_binlog_row_based_result= > - check_table_binlog_row_based_internal(binlog_row); > + check_table_binlog_row_based_internal(); > } > return check_table_binlog_row_based_result; > } > > -bool handler::check_table_binlog_row_based_internal(bool binlog_row) > +bool handler::check_table_binlog_row_based_internal() > { > THD *thd= table->in_use; > > +#ifdef WITH_WSREP > + if (!thd->variables.sql_log_bin && > + wsrep_thd_is_applying(table->in_use)) > + { > + /* > + wsrep patch sets sql_log_bin to silence binlogging from high > + priority threads > + */ > + return 0; > + } > +#endif > return (table->s->can_do_row_logging && > + !table->versioned(VERS_TRX_ID) && > + !(thd->variables.option_bits & OPTION_BIN_TMP_LOG_OFF) && > thd->is_current_stmt_binlog_format_row() && > /* > Wsrep partially enables binary logging if it have not been > @@ -6769,13 +6718,17 @@ int handler::ha_write_row(const uchar *buf) > { error= write_row(buf); }) > > MYSQL_INSERT_ROW_DONE(error); > - if (likely(!error) && !row_already_logged) > + if (likely(!error)) > { > rows_changed++; > - error= binlog_log_row(table, 0, buf, log_func); > + if (row_logging) > + { > + Log_func *log_func= Write_rows_log_event::binlog_row_logging_function; > + error= binlog_log_row(table, 0, buf, log_func); > + } > #ifdef WITH_WSREP > - if (table_share->tmp_table == NO_TMP_TABLE && > - WSREP(ha_thd()) && (error= wsrep_after_row(ha_thd()))) > + if (WSREP_NNULL(ha_thd()) && table_share->tmp_table == NO_TMP_TABLE && why did you swap tests? NO_TMP_TABLE check is cheaper (same in update and delete) > + !error && (error= wsrep_after_row(ha_thd()))) > { > DBUG_RETURN(error); > } > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index 102416ec0a6..9e40c2ae8c8 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -10506,10 +10506,10 @@ do_continue:; > No additional logging of query is needed > */ > binlog_done= 1; > + DBUG_ASSERT(new_table->file->row_logging); > new_table->mark_columns_needed_for_insert(); > thd->binlog_start_trans_and_stmt(); > - binlog_write_table_map(thd, new_table, > - thd->variables.binlog_annotate_row_events); > + thd->binlog_write_table_map(new_table, 1); does it mean you force annotations for ALTER TABLE even if they were configured off? And why would ALTER TABLE generate row events anyway? > } > if (copy_data_between_tables(thd, table, new_table, > alter_info->create_list, ignore, Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

[Maria-developers] GSoC-2020 Query
by Krishna Maheshwari 29 Feb '20

29 Feb '20

Hello, Everyone! I am new here and in open source too. I got to know about MariaDB through GSoC. I have intermediate level of knowledge in sql and have experience working in languages such as c, c++ and java. I did go through the list of tasks mentioned in the mariaDB GSoC 2020, and I was in particularly interested in the tasks listed under MariaDB Server: Optimizer. In that 1) Add FULL OUTER JOIN to MariaDB 2) Recursive CTE support for UPDATE (and DELETE) statements. As I understand the functioning of both of these constructs to some extent, I believe that I would be able to follow the control flow. I would really love to contribute to the code base and be of help but I just don't know where to start from.

1 0

Re: [Maria-developers] 5ae74b4823a: mysqld --help will now load mysqld.options table
by Sergei Golubchik 28 Feb '20

28 Feb '20

Hi, Michael! On Feb 28, Michael Widenius wrote: > revision-id: 5ae74b4823a (mariadb-10.5.0-274-g5ae74b4823a) > parent(s): ee73f2c6e71 > author: Michael Widenius <monty(a)mariadb.com> > committer: Michael Widenius <monty(a)mariadb.com> > timestamp: 2020-02-26 16:05:54 +0200 > message: > > mysqld --help will now load mysqld.options table mysql.plugin > > Changes: > - Initalize Aria early to allow it to load mysql.plugin table with --help > - Don't print 'aborting' when doing --help > - Don't write 'loose' error messages on log_warning < 2 (2 is default) > - Don't write warnings about disabled plugings when doing --help > - Don't write aria_log_control or aria log files when doing --help > - When using --help, open all Aria tables in read only mode (safety) > - If aria_init() fails, do a cleanup(). (Frees used memory) > - If aria_log_control is locked with --help, then don't wait 30 seconds > but instead return at once without initialzing Aria plugin. > > diff --git a/sql/mysqld.cc b/sql/mysqld.cc > index b2f8afca7a6..415a12f4783 100644 > --- a/sql/mysqld.cc > +++ b/sql/mysqld.cc > @@ -8511,8 +8511,8 @@ static void option_error_reporter(enum loglevel level, const char *format, ...) > va_start(args, format); > > /* Don't print warnings for --loose options during bootstrap */ > - if (level == ERROR_LEVEL || !opt_bootstrap || > - global_system_variables.log_warnings) > + if (level == ERROR_LEVEL || > + (!opt_bootstrap && global_system_variables.log_warnings > 1)) You've completely suppressed all --loose warnings during bootstrap. Before your patch they were basically always enabled (because of log_warnings==2). I am not sure it's a good idea to disable warnings completely in bootstrap. If fact, I don't see why bootstrap should be special, so I'd simply remove !opt_bootstrap condition completely here. But if you want to keep it you can do something like global_system_variables > opt_bootstrap it'll make bootstrap a bit quieter than normal startup. > { > vprint_msg_to_log(level, format, args); > } > diff --git a/sql/sql_plugin.cc b/sql/sql_plugin.cc > index d7d7fcca4a2..31de259a218 100644 > --- a/sql/sql_plugin.cc > +++ b/sql/sql_plugin.cc > @@ -1679,7 +1680,22 @@ int plugin_init(int *argc, char **argv, int flags) > global_system_variables.table_plugin = > intern_plugin_lock(NULL, plugin_int_to_ref(plugin_ptr)); > DBUG_SLOW_ASSERT(plugin_ptr->ref_count == 1); > + } > + /* Initialize Aria plugin so that we can load mysql.plugin */ > + plugin_ptr= plugin_find_internal(&Aria, MYSQL_STORAGE_ENGINE_PLUGIN); > + DBUG_ASSERT(plugin_ptr || !mysql_mandatory_plugins[0]); > + if (plugin_ptr) > + { > + DBUG_ASSERT(plugin_ptr->load_option == PLUGIN_FORCE); > > + if (plugin_initialize(&tmp_root, plugin_ptr, argc, argv, false)) > + { > + if (!opt_help) > + goto err_unlock; > + plugin_ptr->state= PLUGIN_IS_DISABLED; > + } > + else > + aria_loaded= 1; > } > mysql_mutex_unlock(&LOCK_plugin); I think this should be done differently. In a completely opposite way. I had unfurtunately hard-coded MyISAM here. A proper fix here could be to remove this special treatment of MyISAM instead of adding another special treatment of Aria. Then plugin_init() could work like: * run dd_frm_type() for the mysql.plugin table - like it's done now * instead of hard-coding MyISAM (and Aria), find this engine name in the plugin_array[] (note, all builtin plugins are already there) * initialize it and (if successful) load mysql.plugin table It only concerns the sql_plugin.cc part of your commit. Your aria part of the commit is still needed, because a good-behaving engine has to be read-only in --help. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 044dfaff670: Replace handler::primary_key_is_clustered() with handler::ha_is_clustered_key()
by Sergei Golubchik 28 Feb '20

28 Feb '20

Hi, Michael! On Feb 28, Michael Widenius wrote: > revision-id: 044dfaff670 (mariadb-10.5.0-275-g044dfaff670) > parent(s): 5ae74b4823a > author: Michael Widenius <monty(a)mariadb.com> > committer: Michael Widenius <monty(a)mariadb.com> > timestamp: 2020-02-26 16:05:55 +0200 > message: > > Replace handler::primary_key_is_clustered() with handler::ha_is_clustered_key() > > This was done to both simplify the code and also to be easier to handle > storage engines that are clustered on some other index than the primary > key. No, I don't get it. Old method meant "Check if the key is a clustered and a reference key". Just "clustered" is marked with HA_CLUSTERED_INDEX. So, 1. You've renamed the method but the name does not match the semantics. It is called is_clustering_key() but it really means "clustered AND reference key" 2. What other engine can be where the reference key isn't a primary key? Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] MDEV-21580: Allow packed sort keys in sort buffer, part #3
by Sergey Petrunia 27 Feb '20

27 Feb '20

Hi Varun, Please more input below: Input on code structure: > SORT_INFO *filesort(THD *thd, TABLE *table, Filesort *filesort... > { > ... > if (!(sort_keys= filesort->make_sortorder(thd, join, first_table_bit))) > DBUG_RETURN(NULL); /* purecov: inspected */ The main effect of this function used to be to create Filesort::sortorder, and so the name 'make_sortorder' used to make sense. Now, it creates a Sort_keys object so the name is counter-intuitive. .. > uint sort_len= sortlength(thd, sort_keys, &multi_byte_charset, > &allow_packing_for_sortkeys); Another counter-intuitive name. Maybe this should be a method of sort_keys object with a different name? And maybe this call and make_sortorder should be moved together since they're logically doing the same thing? ... > Sort_keys* > Filesort::make_sortorder(THD *thd, JOIN *join, table_map first_table_bit) > { ... > if (!sortorder) > sortorder= (SORT_FIELD*) thd->alloc(sizeof(SORT_FIELD) * count); Using "if(!sortorder)" means the sort order can be already present? Is this because we're inside a subquery which we are re-executing? ... > Sort_keys* sort_keys= new Sort_keys(sort_keys_array); But then, we create sort_keys every time, and we do it on a MEM_ROOT which causes a potential O(#rows_examined) memory use. Is my logic correct? I think we should only call make_sortorder() if this hasn't already been done. Any objections to this? > > void Filesort_buffer::sort_buffer(const Sort_param *param, uint count) > { > > ... > qsort2_cmp cmp_func= param->using_packed_sortkeys() ? > get_packed_keys_compare_ptr() : > get_ptr_compare(size); > > my_qsort2(m_sort_keys, count, sizeof(uchar*), cmp_func, > param->using_packed_sortkeys() ? > (void*)param : > (void*) &size); This choose-the-comparison-function logic is duplicated in merge_buffers(). Can we have in one place? I would add appropriate members into class Sort_key (or Sort_param). > if (param->using_packed_addons() || param->using_packed_sortkeys()) > { > /* > The last record read is most likely not complete here. > We need to loop through all the records, reading the length fields, > and then "chop off" the final incomplete record. > */ Why change the identation of this block, from correct to invorrect one? Please move it back two spaces to the left. > static uint make_sortkey(Sort_param *param, uchar *to, uchar *ref_pos) > This function only seems to access Sort_param members that relate to Sort_keys. Did you consider making it accept Sort_keys as an argument? This seems like a sound idea to me: make_sortkey() is only concerned with making sort keys from original records. This is what Sort_keys class should be handling. Sort_param on the other hand covers the entire sorting process: the buffers, total # of rows, etc. Please give it a try. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] 4c263b6d30b: MDEV-20632: Recursive CTE cycle detection using CYCLE clause
by Sergei Golubchik 27 Feb '20

27 Feb '20

Hi, Oleksandr! On Feb 26, Oleksandr Byelkin wrote: > On Wed, Feb 26, 2020 at 1:44 PM Sergei Golubchik <serg(a)mariadb.org> wrote: > > > Hi, Sanja! > > > > _some_ comments are below. > > > > The main thing, I still don't understand your changes in sql_select.cc. > > > > Why did you create separate counters and code paths with if() for CYCLE? > > I'd think it could be just a generalization of DISTINCT > > DISTINCT makes all fields distinct, except hidden, I need also except come > other (or better say by list of not hidden) Exactly. I thought DISTINCT will be just CYCLE with an empty list of non-distinct columns. Or something like that. Not like if (cycle) { cycle code } else { distinct code } > > > --- a/sql/sql_cte.cc > > > +++ b/sql/sql_cte.cc > > > @@ -982,6 +982,38 @@ With_element::rename_columns_of_derived_unit(THD *thd, > > > > are you sure what you're doing below can still be called > > "rename_columns_of_derived_unit" ? > > I can rename it to prepare_olumns_of_derived_unit, is it OK? anything you like :) > > > +opt_cycle: > > > + /* empty */ > > > + { $$= NULL; } > > > + | > > > + CYCLE_SYM > > > + { > > > + if (!Lex->curr_with_clause->with_recursive) > > > + { > > > + thd->parse_error(ER_SYNTAX_ERROR, $1.pos()); > > > + } > > > + } > > > + '(' with_column_list ')' > > > > Where did you see that the standard requires parentheses here? > > Maybe in ORACLE docs... but without it it creates a lot of conflicts. The standard says no parentheses. What kind of conflicts do you get? Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

[Maria-developers] Lag Free ALTER Table for Slave
by Sachin Setiya 27 Feb '20

27 Feb '20

Hi Everyone! This mail will describe usage and implementation of Lag Free Alter. And this will also answer the question raised by Kristian and Simon J Mudd(in 2016) Desc:- This will split Alter into 2 different commits. START ALTER and COMMIT /ROLLBACK ALTER , Start Alter will be written in binlog as soon as we get the locks for the table, alter will proceeds as usual and at the time of writing binlog if alter is successful we will write COMMIT Alter other wise ROLLBACK Alter. Usage:- Using this feature is quite simple. 1. On master you have to turn on `BINLOG_SPLIT_ALTER` dynamic variable. 2. Slave must be using parallel replication. Advance Usage:- So alter is divided like this. 1. START identifier Actual_alter_stmt 2. COMMIT/ROLLBACK identifier Actual_alter_stmt. OR 2. COMMIT/ROLLBACK identifier ALTER identifier is thread_id. Questions by Simon Mudd. >>* this behaviour should be configurable? Yes. >> - new global variable on the master to allow injection of this changed event stream? Right , `BINLOG_SPLIT_ALTER` >> - a new session variable set in the session where the command is triggered ? Right , `BINLOG_SPLIT_ALTER` >> - on slave a setting to indicate how many INPLACE ALTER TABLEs can run at once? No setting so far , but I am thinking of maximum no of CONCURRENT ALTER = slave_parallel_threads >>* how does a DBA monitor what’s going on? >> - the progress and number of active DDL statements So as we there is 2 part, So progress will go like this 1. Executing of start alter (this will take most of time) 2. Waiting for commit/rollback Signal 3. Commit/ Rollback Alter. Number of active ALTER , these will create new threads so DBA can know using this or I am thinking of adding variable in SHOW SLAVE INFO. which will show active DDL. >> - please consider adding counters/metrics for: >> * number of “asynchronous DDLs” in progress / completed successfully / failed or rolled back Okay, We can have counter for these metrics. If i get time to implement this. >> * sizes of the DDL changes made so far Not sure if we need this. >> * number of threads busy in this state (maybe can be implied from SHOW PROCESSLIST or equivalent but nicer to have an easy to query metric) Show processlist will show the busy threads. Sample Output when concurrent alter is running on slave. slave_parallel_threads= 10 Every 1.0s: mysql -uroot -S /home/sachin/alter/build/mysql-test/var/tmp/mysqld.2.sock -e show processlist sachin-sp52: Wed Jan 22 16:56:00 2020 Id User Host db Command Time State Info Progress 1 system user NULL Daemon NULL InnoDB purge coordinator NULL 0.000 3 system user NULL Daemon NULL InnoDB purge worker NULL 0.000 2 system user NULL Daemon NULL InnoDB purge worker NULL 0.000 4 system user NULL Daemon NULL InnoDB purge worker NULL 0.000 5 system user NULL Daemon NULL InnoDB shutdown handler NULL 0.000 10 root localhost:44478 test Sleep 6 NULL 0.000 11 root localhost:44480 test Sleep 7 NULL 0.000 15 root localhost:44492 test Sleep 6 NULL 0.000 16 root localhost:44494 test Sleep 6 NULL 0.000 17 system user NULL Slave_IO 6 Waiting for master to send event NULL 0.000 19 system user NULL Slave_worker 6 Waiting for work from SQL thread NULL 0.000 22 system user NULL Slave_worker 6 Waiting for work from SQL thread NULL 0.000 20 system user NULL Slave_worker 6 Waiting for work from SQL thread NULL 0.000 23 system user NULL Slave_worker 6 Waiting for work from SQL thread NULL 0.000 24 system user NULL Slave_worker 6 Waiting for work from SQL thread NULL 0.000 25 system user NULL Slave_worker 6 Waiting for work from SQL thread NULL 0.000 26 system user NULL Slave_worker 6 Waiting for work from SQL thread NULL 0.000 21 system user NULL Slave_worker 6 Waiting for work from SQL thread NULL 0.000 27 system user NULL Slave_worker 6 Waiting for work from SQL thread NULL 0.000 28 system user NULL Slave_worker 6 Waiting for work from SQL thread NULL 0.000 18 system user NULL Slave_SQL 6 Slave has read all relay log; waiting for the slave I/O thread to update it NULL 0.000 29 system user test Slave_worker 6 NULL /*!100001 START 19 alter table t3 add column c int, force, algorithm=inplace */ 0.000 30 system user test Slave_worker 6 NULL /*!100001 START 17 alter table t1 add column c int, force, algorithm=inplace */ 0.000 31 system user test Slave_worker 6 NULL /*!100001 START 18 alter table t2 add column c int, force, algorithm=inplace */ 0.000 32 system user test Slave_worker 6 NULL /*!100001 START 20 alter table t4 add column c int, force, algorithm=inplace */ 0.000 33 system user test Slave_worker 6 NULL /*!100001 START 21 alter table t5 add column c int, force, algorithm=inplace */ 0.000 34 root localhost NULL Query 0 Init show processlist 0.000 >>* what happens if the commit/rollback ddl event never arrives? (the statement on the master could be skipped for one of several reasons) So here are 2 options If slave is still running DBA can manually run COMMIT/ROLLBACK identifier alter_stmt/ALTER , where identifier is same as START ALTER identifier. so this will either commit or rollback alter. If slave is not running. Since slave is not running all context is lost , so this will take as much as time a normal alter. >> - I assume that users on the slave see the old structure all the time until it completes Right, we work on copy of alter for myisam and for other case innodb will take care of it. >> - would this have a storage or performance penalty on the slave if the commit/rollback DDL event never arrives? Not performance but yes storage and ram penalty. >> - can a “commit” / “rollback” be manually triggered by the DBA in such circumstances? yes. >>* what happens if the server crashes while this process (or these processes) are ongoing when the server starts up again? So if we crash in START ALTER no issue , we can run it again , since it is transactional(we are working on copy so ..) If we crash after start alter and we receive COMMIT alter then it will be treated as normal alter. ROLLBACK ALTER will be instant. If we crash in COMMIT ALTER this will be same as crash in normal ALTER table , so DBA has to deal with it. Architecture:- Lets look at sample alter perf data for innodb and myisam Myisam + 96.72% 0.00% mysqld mysqld [.] mysql_parse + 96.72% 0.00% mysqld mysqld [.] mysql_execute_command + 96.72% 0.00% mysqld mysqld [.] Sql_cmd_alter_table::execute - 96.72% 0.00% mysqld mysqld [.] mysql_alter_table - mysql_alter_table - 96.63% copy_data_between_tables + 79.64% handler::ha_write_row + 6.24% TABLE::update_default_fields + 5.71% READ_RECORD::read_record + 1.87% do_copy_null 1.05% Field::do_field_int + 0.95% _mcount InnoDB + 41.27% 0.00% mysqld mysqld [.] mysql_parse + 41.27% 0.00% mysqld mysqld [.] mysql_execute_command + 41.27% 0.00% mysqld mysqld [.] Sql_cmd_alter_table::execute - 41.27% 0.00% mysqld mysqld [.] mysql_alter_table - mysql_alter_table - 41.27% mysql_inplace_alter_table - 41.15% handler::ha_inplace_alter_table ha_innobase::inplace_alter_table - row_merge_build_indexes - 41.00% row_merge_read_clustered_index + 22.88% row_merge_insert_index_tuples + 5.89% row_build_w_add_vcol + 5.20% BtrBulk::finish + 3.20% row_merge_buf_add + 0.87% page_cur_move_to_next 0.62% rec_get_offsets_func So as we can see most of the work is done by copy_data_between_tables/ mysql_inplace_alter_table. Master Side:- There is not much change into master side. So after locking table and before executing these functions we write START _id_ alter_stmt into binlog. And at the time of write_bin_log we write COMMIT _id_ alter_stmt into binlog. So alter statement will be divided into 2 , hence 2 gtid. START_ALTER will have special flag FL_START_ALTER_E1(4), this will be used on slave side to create new worker thread for start alter processing. Slave Side:- This require parallel replication on slave side. So in do_event when we get a gtid_event with FL_START_ALTER_E1, ::choose_thread will call rpl_parallel_add_extra_worker, which will create a new worker thread and this will do start alter processing. So the gtid_log_event and next Query_log_event will be scheduled to this new thread. And this thread will be exited as soon as alter is finished. On slave side START ALTER will binlog after successfully getting MDL lock and thd lock. Untill this point we are executed as DDL(so new GCO), but after getting locks we will call finish_event_group. So that new events can proceed in parallel. Then it will continue to execute code in mysql_alter_table until we reach a non transactional part(like renaming table/dropping table in myisam) So before executing NON-Transactional part it will wait signal from other worker thread to either abort or proceed forward. We will add a entry into mi-> start_alter_list, with thread_id(of master) as a key. COMMIT/ROLLBACK Alter is treated as a normal query_log_event so it will be assigned normal worker. This command will take SQLCOM_COMMIT_PREVIOUS path in mysql_execute_command. We will simple search for thread_id into start_alter_list and change status from ::WAITING to ::COMMIT_ALTER and signal the wait condition if we dont find the thread_id we will wait on wait condition, So simple it is just consumer producer between start alter and commit/rollback. Questions by Kristian:- >Can you explain what you mean by "true LOCK=NONE"? It is not clear from your >description. I think you mean something that will allow to run an ALTER >TABLE on the slave in parallel with the same statement on the master? Yes >There will be a number of things to consider with this. Previously, all >transactions in the binlog have been atomic; this introduces related events >that can be arbitrarily separated in the binlog. >For example, suppose the slave stops in the middle between BEGIN_DDL_EVENT >and COMMIT_DDL_EVENT/ROLLBACK_DDL_EVENT. What is then the slave's binlog >position / GTID position? Now we are using 2 gtid for ALTER , So it wont be issue. >Hopefully, the exiting pool of slave worker threads (currently used for >parallel replication) can be used for this as well? Doesn't seem right to >introduce a new kind of thread pool... We are using same(global) thread pool but I am creating new threads for START ALTER. Otherwise we will have a deadlock , Suppose if we have 5 concurrent START ALTER and just 5 slave-worker-threads , all worker will be waiting for COMMIT/ROLLBACK, but we cant execute commit rollback because we dont have any free worker. There can be more case when we have concurrent DML too , so creating a new thread was the safest option. >Won't you need some mechanism to prevent following events to access the >table before ALTER TABLE in a worker thread can acquire the metadata lock? Right, So Untill Locking START ALTER is executed as DDL so no following event will be executed before locking. And one more thing regarding locks , So there will we only one thread will be doing whole work of alter, (we can just assume that there is some sleep on slave) so execution on master and slave will be equivalent. >There are some nasty issues to be aware of with potential deadlocks related >to FLUSH TABLES WITH GLOBAL READ LOCK and such, when multiple threads are >trying to coordinate waiting and locking; there were already some quite hard >issues with this for parallel replication. I have to test for it. Code branch bb-10.5-olter Jira (Mdev-11675(https://jira.mariadb.org/browse/MDEV-11675)) Regards Sachin -- Regards Sachin Setiya Software Engineer at MariaDB

3 4

Re: [Maria-developers] ec1a860bd09: MDEV-21743 Split up SUPER privilege to smaller privileges
by Sergei Golubchik 26 Feb '20

26 Feb '20

Hi, Alexander! Thanks! Looks very straightforward. See few comments below. The main one - I am not sure I like the idea of creating numerous aliases for privileges. This approach looks rather confusing to me. On Feb 26, Alexander Barkov wrote: > revision-id: ec1a860bd09 (mariadb-10.5.0-231-gec1a860bd09) > parent(s): b8b5a6a2f9d > author: Alexander Barkov <bar(a)mariadb.com> > committer: Alexander Barkov <bar(a)mariadb.com> > timestamp: 2020-02-23 22:09:55 +0400 > message: > > MDEV-21743 Split up SUPER privilege to smaller privileges > diff --git a/mysql-test/main/alter_user.test b/mysql-test/main/alter_user.test > index 9ea98615272..60a36499a55 100644 > --- a/mysql-test/main/alter_user.test > +++ b/mysql-test/main/alter_user.test > @@ -30,7 +30,7 @@ alter user foo; > > --echo # Grant super privilege to the user. > connection default; > -grant super on *.* to foo; > +grant READ_ONLY ADMIN on *.* to foo; --echo comments are now wrong > --echo # We now have super privilege. We should be able to run alter user. > connect (b, localhost, foo); > diff --git a/mysql-test/main/grant.result b/mysql-test/main/grant.result > index e83083be4ed..1452ada11f5 100644 > --- a/mysql-test/main/grant.result > +++ b/mysql-test/main/grant.result > @@ -631,6 +634,10 @@ Super Server Admin To use KILL thread, SET GLOBAL, CHANGE MASTER, etc. > Trigger Tables To use triggers > Create tablespace Server Admin To create/alter/drop tablespaces > Update Tables To update existing rows > +Set user Server To create views and stored routines with a different definer > +Federated admin Server To execute the CREATE SERVER, ALTER SERVER, DROP SERVER statements > +Connection admin Server To skip connection related limits tests ^^^ allows KILL too > +Read_only admin Server To perform write operations even if @@read_only=ON > Usage Server Admin No privileges - allow connect only > connect root,localhost,root,,test,$MASTER_MYPORT,$MASTER_MYSOCK; > connection root; > diff --git a/sql/lock.cc b/sql/lock.cc > index 6f86c0a38f6..9a4024606f8 100644 > --- a/sql/lock.cc > +++ b/sql/lock.cc > @@ -114,7 +113,7 @@ lock_tables_check(THD *thd, TABLE **tables, uint count, uint flags) > DBUG_ENTER("lock_tables_check"); > > system_count= 0; > - is_superuser= (thd->security_ctx->master_access & SUPER_ACL) != NO_ACL; > + is_superuser= (thd->security_ctx->master_access & IGNORE_READ_ONLY_ACL) != NO_ACL; may be then s/is_superuser/ignores_read_only/ ? > log_table_write_query= (is_log_table_write_query(thd->lex->sql_command) > || ((flags & MYSQL_LOCK_LOG_TABLE) != 0)); > > diff --git a/sql/privilege.h b/sql/privilege.h > index 5dbc0b6dbdf..f80e726d8d0 100644 > --- a/sql/privilege.h > +++ b/sql/privilege.h > @@ -59,24 +59,60 @@ enum privilege_t: unsigned long long > TRIGGER_ACL = (1UL << 27), > CREATE_TABLESPACE_ACL = (1UL << 28), > DELETE_HISTORY_ACL = (1UL << 29), // Added in 10.3.4 > + SET_USER_ACL = (1UL << 30), // Added in 10.5.2 > + FEDERATED_ADMIN_ACL = (1UL << 31), // Added in 10.5.2 > + CONNECTION_ADMIN_ACL = (1ULL << 32), // Added in 10.5.2 > + READ_ONLY_ADMIN_ACL = (1ULL << 33), // Added in 10.5.2 > + REPL_SLAVE_ADMIN_ACL = (1ULL << 34), // Added in 10.5.2 > + REPL_MASTER_ADMIN_ACL = (1ULL << 35), // Added in 10.5.2 > + BINLOG_ADMIN_ACL = (1ULL << 36) // Added in 10.5.2 > /* > - don't forget to update > - 1. static struct show_privileges_st sys_privileges[] > - 2. static const char *command_array[] and static uint command_lengths[] > - 3. mysql_system_tables.sql and mysql_system_tables_fix.sql > - 4. acl_init() or whatever - to define behaviour for old privilege tables > - 5. sql_yacc.yy - for GRANT/REVOKE to work > - 6. Add a new ALL_KNOWN_ACL_VERSION > - 7. Change ALL_KNOWN_ACL to ALL_KNOWN_ACL_VERSION > - 8. Update User_table_json::get_access() > + don't forget to update: > + In this file: > + - Add a new LAST_version_ACL > + - Fix PRIVILEGE_T_MAX_BIT > + - Add a new ALL_KNOWN_ACL_version > + - Change ALL_KNOWN_ACL to ALL_KNOWN_ACL_version > + - Change GLOBAL_ACLS if needed > + - Change SUPER_ADDED_SINCE_USER_TABLE_ACL if needed > + > + In other files: > + - static struct show_privileges_st sys_privileges[] > + - static const char *command_array[] and static uint command_lengths[] > + - mysql_system_tables.sql and mysql_system_tables_fix.sql > + - acl_init() or whatever - to define behaviour for old privilege tables > + - Update User_table_json::get_access() > + - sql_yacc.yy - for GRANT/REVOKE to work > + > + Important: the enum should contain only single-bit values. > + In this case, debuggers print bit combinations in the readable form: > + (gdb) p (privilege_t) (15) > + $8 = (SELECT_ACL | INSERT_ACL | UPDATE_ACL | DELETE_ACL) > + > + Bit-OR combinations of the above values should be declared outside! > */ > - > - // A combination of all bits defined in 10.3.4 (and earlier) > - ALL_KNOWN_ACL_100304 = (1UL << 30) - 1 > }; > > > -constexpr privilege_t ALL_KNOWN_ACL= ALL_KNOWN_ACL_100304; > +// Version markers > +constexpr privilege_t LAST_100304_ACL= DELETE_HISTORY_ACL; > +constexpr privilege_t LAST_100502_ACL= BINLOG_ADMIN_ACL; > +constexpr privilege_t LAST_CURRENT_ACL= LAST_100502_ACL; > +constexpr uint PRIVILEGE_T_MAX_BIT= 36; > + > +static_assert((privilege_t)(1ULL << PRIVILEGE_T_MAX_BIT) == LAST_CURRENT_ACL, > + "LAST_CURRENT_ACL and PRIVILEGE_T_MAX_BIT do not match"); why wouldn't you define PRIVILEGE_T_MAX_BIT via LAST_CURRENT_ACL instead? > + > +// A combination of all bits defined in 10.3.4 (and earlier) > +constexpr privilege_t ALL_KNOWN_ACL_100304 = > + (privilege_t) ((LAST_100304_ACL << 1) - 1); > + > +// A combination of all bits defined in 10.5.2 > +constexpr privilege_t ALL_KNOWN_ACL_100502= > + (privilege_t) ((LAST_100502_ACL << 1) - 1); > + > +// A combination of all bits defined as of the current version > +constexpr privilege_t ALL_KNOWN_ACL= ALL_KNOWN_ACL_100502; > > > // Unary operators > @@ -229,6 +280,104 @@ constexpr privilege_t SHOW_CREATE_TABLE_ACLS= > constexpr privilege_t TMP_TABLE_ACLS= > COL_DML_ACLS | ALL_TABLE_DDL_ACLS; > > + > + > +/* > + If a VIEW has a `definer=invoker@host` clause and > + the specified definer does not exists, then > + - The invoker with REVEAL_MISSING_DEFINER_ACL gets: > + ERROR: The user specified as a definer ('definer1'@'localhost') doesn't exist > + - The invoker without MISSING_DEFINER_ACL gets a generic access error, > + without revealing details that the definer does not exists. > + > + TODO: we should eventually test the same privilege when processing > + other objects that have the DEFINER clause (e.g. routines, triggers). > + Currently the missing definer is revealed for non-privileged invokers > + in case of routines, triggers, etc. > +*/ > +constexpr privilege_t REVEAL_MISSING_DEFINER_ACL= SUPER_ACL; 1. why did you create these aliases? I don't think they make the code simpler, on the contrary, now one can never know whether say, IGNORE_READ_ONLY_ACL is a real privilege like in GRANT IGNORE READ_ONLY ON *.* TO user@host or it's just an alias. 2. REVEAL_MISSING_DEFINER_ACL should be SET_USER_ACL, I think > +constexpr privilege_t DES_DECRYPT_ONE_ARG_ACL= SUPER_ACL; > +constexpr privilege_t LOG_BIN_TRUSTED_SP_CREATOR_ACL= SUPER_ACL; this could be SET_USER_ACL too > +constexpr privilege_t DEBUG_ACL= SUPER_ACL; > +constexpr privilege_t SET_GLOBAL_SYSTEM_VARIABLE_ACL= SUPER_ACL; > +constexpr privilege_t SET_RESTRICTED_SESSION_SYSTEM_VARIABLE_ACL= SUPER_ACL; > + > +/* Privileges related to --read-only */ > +constexpr privilege_t IGNORE_READ_ONLY_ACL= READ_ONLY_ADMIN_ACL; > + > +/* Privileges related to connection handling */ > +constexpr privilege_t IGNORE_INIT_CONNECT_ACL= CONNECTION_ADMIN_ACL; > +constexpr privilege_t IGNORE_MAX_USER_CONNECTIONS_ACL= CONNECTION_ADMIN_ACL; > +constexpr privilege_t IGNORE_MAX_CONNECTIONS_ACL= CONNECTION_ADMIN_ACL; > +constexpr privilege_t IGNORE_MAX_PASSWORD_ERRORS_ACL= CONNECTION_ADMIN_ACL; > +// Was SUPER_ACL prior to 10.5.2 > +constexpr privilege_t KILL_OTHER_USER_PROCESS_ACL= CONNECTION_ADMIN_ACL; > + > + > +/* > + Binary log related privileges that are checked regardless > + of active replication running. > +*/ > + > +/* > + This command was renamed from "SHOW MASTER STATUS" > + to "SHOW BINLOG STATUS" in 10.5.2. > + Was SUPER_ACL | REPL_CLIENT_ACL prior to 10.5.2 > +*/ > +constexpr privilege_t STMT_SHOW_BINLOG_STATUS_ACL= REPL_CLIENT_ACL; > + > +// Was SUPER_ACL | REPL_CLIENT_ACL prior to 10.5.2 > +constexpr privilege_t STMT_SHOW_BINARY_LOGS_ACL= REPL_CLIENT_ACL; > + > +// Was SUPER_ACL prior to 10.5.2 > +constexpr privilege_t STMT_PURGE_BINLOG_ACL= BINLOG_ADMIN_ACL; > + > +// Was REPL_SLAVE_ACL prior to 10.5.2 > +constexpr privilege_t STMT_SHOW_BINLOG_EVENTS_ACL= PROCESS_ACL; > + > + > +/* > + Privileges for replication related statements > + that are executed on the master. > +*/ > +constexpr privilege_t COM_REGISTER_SLAVE_ACL= REPL_SLAVE_ACL; > +constexpr privilege_t COM_BINLOG_DUMP_ACL= REPL_SLAVE_ACL; > +// Was REPL_SLAVE_ACL prior to 10.5.2 > +constexpr privilege_t STMT_SHOW_SLAVE_HOSTS_ACL= REPL_MASTER_ADMIN_ACL; > + > + > +/* Privileges for statements that are executed on the slave */ > +// Was SUPER_ACL prior to 10.5.2 > +constexpr privilege_t STMT_START_SLAVE_ACL= REPL_SLAVE_ADMIN_ACL; > +// Was SUPER_ACL prior to 10.5.2 > +constexpr privilege_t STMT_STOP_SLAVE_ACL= REPL_SLAVE_ADMIN_ACL; > +// Was SUPER_ACL prior to 10.5.2 > +constexpr privilege_t STMT_CHANGE_MASTER_ACL= REPL_SLAVE_ADMIN_ACL; > +// Was (SUPER_ACL | REPL_CLIENT_ACL) prior to 10.5.2 > +constexpr privilege_t STMT_SHOW_SLAVE_STATUS_ACL= REPL_SLAVE_ADMIN_ACL; > +// Was SUPER_ACL prior to 10.5.2 > +constexpr privilege_t STMT_BINLOG_ACL= REPL_SLAVE_ADMIN_ACL; > +// Was REPL_SLAVE_ACL prior to 10.5.2 > +constexpr privilege_t STMT_SHOW_RELAYLOG_EVENTS_ACL= REPL_SLAVE_ADMIN_ACL; > + > + > +/* Privileges for federated database related statements */ > +// Was SUPER_ACL prior to 10.5.2 > +constexpr privilege_t STMT_CREATE_SERVER_ACL= FEDERATED_ADMIN_ACL; > +// Was SUPER_ACL prior to 10.5.2 > +constexpr privilege_t STMT_ALTER_SERVER_ACL= FEDERATED_ADMIN_ACL; > +// Was SUPER_ACL prior to 10.5.2 > +constexpr privilege_t STMT_DROP_SERVER_ACL= FEDERATED_ADMIN_ACL; > + > + > +/* Privileges related to processes */ > +constexpr privilege_t COM_PROCESS_INFO_ACL= PROCESS_ACL; > +constexpr privilege_t STMT_SHOW_EXPLAIN_ACL= PROCESS_ACL; > +constexpr privilege_t STMT_SHOW_ENGINE_STATUS_ACL= PROCESS_ACL; > +constexpr privilege_t STMT_SHOW_ENGINE_MUTEX_ACL= PROCESS_ACL; > +constexpr privilege_t STMT_SHOW_PROCESSLIST_ACL= PROCESS_ACL; > + > + > /* > Defines to change the above bits to how things are stored in tables > This is needed as the 'host' and 'db' table is missing a few privileges > diff --git a/sql/sql_connect.cc b/sql/sql_connect.cc > index e2a3c482ae4..b096bfa7a95 100644 > --- a/sql/sql_connect.cc > +++ b/sql/sql_connect.cc > @@ -1246,7 +1245,8 @@ void prepare_new_connection_state(THD* thd) > thd->set_command(COM_SLEEP); > thd->init_for_queries(); > > - if (opt_init_connect.length && !(sctx->master_access & SUPER_ACL)) > + if (opt_init_connect.length && > + (sctx->master_access & IGNORE_INIT_CONNECT_ACL) == NO_ACL) dunno, I kind of like !(access & SOME_ACL) why not to keep privilege_t -> bool? > { > execute_init_command(thd, &opt_init_connect, &LOCK_sys_init_connect); > if (unlikely(thd->is_error())) > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > index dac5b025821..7f3a436a4c2 100644 > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -7138,8 +7138,7 @@ bool check_some_access(THD *thd, privilege_t want_access, TABLE_LIST *table) > @warning > One gets access right if one has ANY of the rights in want_access. > This is useful as one in most cases only need one global right, > - but in some case we want to check if the user has SUPER or > - REPL_CLIENT_ACL rights. > + but in some case we want to check multiple rights. In what cases? Are there any left? > > @retval > 0 ok Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0