- developers - lists.mariadb.org

[Maria-developers] 4f9c5a8cf54: MDEV-24773: slave_compressed_protocol doesn't work properly with semi-sync replication
by sujatha 08 Feb '21

08 Feb '21

revision-id: 4f9c5a8cf545133c3d29c0de3f7e2e57564607f2 (mariadb-10.3.26-78-g4f9c5a8cf54) parent(s): 59eda73eff1a22ac0373d818bc802c05e82b5449 author: Sujatha committer: Sujatha timestamp: 2021-02-05 16:45:35 +0530 message: MDEV-24773: slave_compressed_protocol doesn't work properly with semi-sync replication Back port upstream fix commit 1800b015a1d487330f7b15f2020b887be348a66b Author: Venkatesh Duggirala <venkatesh.duggirala(a)oracle.com> Date: Fri Sep 8 20:29:22 2017 +0530 Bug#26027024 SLAVE_COMPRESSED_PROTOCOL DOESN'T WORK WITH SEMI-SYNC REPLICATION IN MYSQL-5.7 Analysis: In mysql-5.6, dump thread (the thread that is created on Master after Slave requested for a binlog dump) is also used to receive acknowledgements from the Slave and act on them accordingly. For performance reasons, a special thread called Ack Receiver thread is added in mysql-5.7 Semi synchronous replication plugin. This thread does not have special handling to receive acknowledgements if Slave has enabled compression in the protocol. Hence Master is unable to handle any slave if Slave_compressed_protocol is enabled on it. Fix: Enable compress flag on the communication channels if the Slave has Slave_compressed_protocol ON. --- .../rpl_semi_sync_slave_compressed_protocol.result | 19 ++++++++ ...l_semi_sync_slave_compressed_protocol-slave.opt | 1 + .../t/rpl_semi_sync_slave_compressed_protocol.test | 55 ++++++++++++++++++++++ sql/semisync_master_ack_receiver.cc | 5 ++ 4 files changed, 80 insertions(+) diff --git a/mysql-test/suite/rpl/r/rpl_semi_sync_slave_compressed_protocol.result b/mysql-test/suite/rpl/r/rpl_semi_sync_slave_compressed_protocol.result new file mode 100644 index 00000000000..86816652565 --- /dev/null +++ b/mysql-test/suite/rpl/r/rpl_semi_sync_slave_compressed_protocol.result @@ -0,0 +1,19 @@ +include/master-slave.inc +[connection master] +SET @@GLOBAL.rpl_semi_sync_master_enabled = 1; +connection slave; +include/stop_slave.inc +SET @@GLOBAL.rpl_semi_sync_slave_enabled = 1; +include/start_slave.inc +connection master; +CREATE TABLE t1 (i INT); +DROP TABLE t1; +include/rpl_sync.inc +include/assert_grep.inc [Check that there is no 'Read semi-sync reply magic number error' in error log.] +connection master; +SET @@GLOBAL. rpl_semi_sync_master_enabled = 0; +connection slave; +include/stop_slave.inc +SET @@GLOBAL. rpl_semi_sync_slave_enabled = 0; +include/start_slave.inc +include/rpl_end.inc diff --git a/mysql-test/suite/rpl/t/rpl_semi_sync_slave_compressed_protocol-slave.opt b/mysql-test/suite/rpl/t/rpl_semi_sync_slave_compressed_protocol-slave.opt new file mode 100644 index 00000000000..a1b687d691e --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_semi_sync_slave_compressed_protocol-slave.opt @@ -0,0 +1 @@ +--slave_compressed_protocol diff --git a/mysql-test/suite/rpl/t/rpl_semi_sync_slave_compressed_protocol.test b/mysql-test/suite/rpl/t/rpl_semi_sync_slave_compressed_protocol.test new file mode 100644 index 00000000000..6343fb29ffe --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_semi_sync_slave_compressed_protocol.test @@ -0,0 +1,55 @@ +################################################################################ +# Bug#26027024 SLAVE_COMPRESSED_PROTOCOL DOESN'T WORK WITH SEMI-SYNC +# REPLICATION IN MYSQL-5.7 +# +# Steps to reproduce: +# 1) Set slave_compressed_protocol ON on Slave. +# 2) Do some sample work on Master +# 3) After the work is synced on Slave, check that there is no error +# (Read semi-sync reply magic number error) on Slave. +# 4) Cleanup +################################################################################ +# Test is independent of Binlog format. One of the three formats is enough +# for testing. Choosing 'Row' format. +--source include/have_binlog_format_row.inc +--source include/master-slave.inc + +--let $sav_enabled_master=`SELECT @@GLOBAL.rpl_semi_sync_master_enabled ` +SET @@GLOBAL.rpl_semi_sync_master_enabled = 1; + +--connection slave +source include/stop_slave.inc; +--let $sav_enabled_slave=`SELECT @@GLOBAL.rpl_semi_sync_slave_enabled ` +SET @@GLOBAL.rpl_semi_sync_slave_enabled = 1; +source include/start_slave.inc; + +--connection master +# Do some sample work on Master with slave_compressed_protocol ON. +# (slave_compressed_protocol is set to ON in -slave.opt file of this test.) +CREATE TABLE t1 (i INT); +DROP TABLE t1; + +# Make sure sync is done, so that next 'assert' step can be executed without +# any issues. +--source include/rpl_sync.inc + +# Without the fix, the test would have generated few +# errors in the error log. With the fix, test will +# pass without any errors in the error log. +--let $assert_text= Check that there is no 'Read semi-sync reply magic number error' in error log. +--let $assert_select=Read semi-sync reply magic number error +--let $assert_file= $MYSQLTEST_VARDIR/log/mysqld.1.err +--let $assert_count= 0 +--let $assert_only_after = CURRENT_TEST:rpl.rpl_semi_sync_slave_compressed_protocol.test +--source include/assert_grep.inc + +--connection master +--eval SET @@GLOBAL. rpl_semi_sync_master_enabled = $sav_enabled_master + +--connection slave +source include/stop_slave.inc; +--eval SET @@GLOBAL. rpl_semi_sync_slave_enabled = $sav_enabled_slave +source include/start_slave.inc; + +# Cleanup +--source include/rpl_end.inc diff --git a/sql/semisync_master_ack_receiver.cc b/sql/semisync_master_ack_receiver.cc index 81f494c9d34..cc3a56a56b2 100644 --- a/sql/semisync_master_ack_receiver.cc +++ b/sql/semisync_master_ack_receiver.cc @@ -268,6 +268,11 @@ void Ack_receiver::run() net_clear(&net, 0); net.vio= &slave->vio; + /* + Set compress flag. This is needed to support + Slave_compress_protocol flag enabled Slaves + */ + net.compress= slave->thd->net.compress; len= my_net_read(&net); if (likely(len != packet_error))

1 0

[Maria-developers] RFC: incompatible changes to mtr debugging flags
by Sergei Golubchik 27 Jan '21

27 Jan '21

Hi, all, Here's a feature I want to discuss, as it's rather incompatibly removes a bunch of mtr command-line options. It's a unified handling of all debuggers. I wanted to do it for quite a while, but the actual trigger was me trying to add support for --rr (https://rr-project.org/) in embedded. It was supported only for non-embedded server, and making it work for embedded needed way more copy-pasting than I was comfortable with. So, in my branch all debuggers are not haphazardly added using whatever syntax one thought of at the moment and copy-pasted all over. They've handled by a common module, all support the same set of options (using gdb as an example): --gdb, --boot-gdb, --client-gdb, --manual-gdb. Both --gdb and --client-gdb work for embedded. All four of them accept an optional argument that can be a semicolon separated list of gdb commands, like in ./mtr 1st --gdb='b mysql_parse;r' but can start from command-line options too: ./mtr 1st --boot-gdb='-nh -q;b mysql_parse;r' A "debugger" is anything that wraps mysqld or mysqltest execution, the current list of debuggers is: gdb, ddd, dbx, lldb, valgrind, strace, ktrace, rr, devenv, windbg, vc_express, vc, vsjitdebugger. There's also "valgdb" that does what '--valgrind --gdb' was doing before. This removes the following mtr options: --rr-args --rr-dir --manual-debug --debugger --strace-option --stracer --valgrind-all --valgrind-mysqltest --valgrind-mysqld --valgrind-options --valgrind-option --valgrind-path --callgrind, and --valgrind only enables it for the server, not for everything. If pushed, it'll definitely break backward compatibility. But I don't think mtr backward compatibility is all that important, it might only affect a handful of developers who use scripts to start mtr with specific hard-coded settings, those scripts could be easily adjusted. It's much more important to have same mtr features in all branches, so that when switching between branches, one wouldn't need to guess what command line options mtr supports now. That is, if pushed, this should go into 10.2. Opinions? Missing features? Push / not push? for the reference, it's in bb-10.2-serg branch at the moment. Regards, Sergei

5 7

Re: [Maria-developers] 3e01da668d1: MDEV-21785: sequences used as default by other table not dumped in right order by mysqldump
by Sergei Golubchik 21 Jan '21

21 Jan '21

Hi, Oleksandr! On Jan 21, Oleksandr Byelkin wrote: > revision-id: 3e01da668d1 (mariadb-10.3.26-64-g3e01da668d1) > parent(s): 59998d3480f > author: Oleksandr Byelkin <sanja(a)mariadb.com> > committer: Oleksandr Byelkin <sanja(a)mariadb.com> > timestamp: 2021-01-14 10:45:51 +0100 > message: > > MDEV-21785: sequences used as default by other table not dumped in right order by mysqldump > > Dump sequences first. > > This atch made to keep it small and > to keep number of queries to the server the same. > > Order of tables in a dump can not be changed > (except sequences first) because mysql_list_tables > uses SHOW TABLES and I used SHOW FULL TABLES. it looks like a rather overengineered implementation. I'd suggest to simplify it as: > diff --git a/client/mysqldump.c b/client/mysqldump.c > index a964f96437d..17837713c01 100644 > --- a/client/mysqldump.c > +++ b/client/mysqldump.c > @@ -42,6 +42,11 @@ > /* on merge conflict, bump to a higher version again */ > #define DUMP_VERSION "10.19" > > +/** > + First mysql version supporting sequences. > +*/ > +#define FIRST_SEQUENCE_VERSION 100300 seems unnecessary, you implemented SHOW FULL TABLES in 5.0.2-alpha. Oct 2004. You can safely use SHOW FULL TABLES unconditionally. > + > #include <my_global.h> > #include <my_sys.h> > #include <my_user.h> > @@ -92,6 +97,13 @@ > /* Max length GTID position that we will output. */ > #define MAX_GTID_LENGTH 1024 > > + > +/* Dump sequence/tables control */ > +#define DUMP_TABLE_TABLE 1 > +#define DUMP_TABLE_SEQUENCE 2 > +#define DUMP_TABLE_ALL (DUMP_TABLE_TABLE | DUMP_TABLE_SEQUENCE) > + > + > static my_bool ignore_table_data(const uchar *hash_key, size_t len); > static void add_load_option(DYNAMIC_STRING *str, const char *option, > const char *option_value); > @@ -171,6 +183,7 @@ static DYNAMIC_STRING dynamic_where; > static MYSQL_RES *get_table_name_result= NULL; > static MEM_ROOT glob_root; > static MYSQL_RES *routine_res, *routine_list_res; > +static int get_table_name_result_short= 1; that'll be unnecessary > > > #include <sslopt-vars.h> > @@ -3853,13 +3866,16 @@ static char *alloc_query_str(size_t size) > ARGS > table - table name > db - db name > + type_ctrl - dump table or sequences or both > > RETURNS > void > */ > > > -static void dump_table(const char *table, const char *db, const uchar *hash_key, size_t len) > +static void dump_table(const char *table, const char *db, > + const uchar *hash_key, size_t len, > + const uint type_ctrl) > { > char ignore_flag; > char buf[200], table_buff[NAME_LEN+3]; > @@ -3881,9 +3897,11 @@ static void dump_table(const char *table, const char *db, const uchar *hash_key, > */ > if (check_if_ignore_table(table, table_type) & IGNORE_SEQUENCE_TABLE) > { > - get_sequence_structure(table, db); > + if (type_ctrl & DUMP_TABLE_SEQUENCE) > + get_sequence_structure(table, db); > + DBUG_VOID_RETURN; > + } else if (!(type_ctrl & DUMP_TABLE_TABLE)) > DBUG_VOID_RETURN; > - } here I'd remove the whole if(). Let's say the convention is that dump_table() should never be invoked for sequences, it's the caller's job to ensure it. > /* > Make sure you get the create table info before the following check for > --no-data flag below. Otherwise, the create table info won't be printed. > @@ -4368,18 +4386,49 @@ static void dump_table(const char *table, const char *db, const uchar *hash_key, > } /* dump_table */ > > > -static char *getTableName(int reset) > +static char *getTableName(int reset, uint table_control) > { > MYSQL_ROW row; > > if (!get_table_name_result) > { > - if (!(get_table_name_result= mysql_list_tables(mysql,NullS))) > - return(NULL); > + if (mysql_get_server_version(mysql) >= FIRST_SEQUENCE_VERSION) > + { > + const char *query= > + "SHOW FULL TABLES"; > + > + if (mysql_query_with_error_report(mysql, 0, query)) > + return (NULL); > + > + if (!(get_table_name_result= mysql_store_result(mysql))) > + return (NULL); > + > + get_table_name_result_short= 0; > + } > + else > + { > + if (!(get_table_name_result= mysql_list_tables(mysql,NullS))) > + return(NULL); > + get_table_name_result_short= 1; > + } this will be twice as small, if you won't try to support before-5.0 servers. > } > if ((row= mysql_fetch_row(get_table_name_result))) > - return((char*) row[0]); > - > + { > + int sequence; > + if (get_table_name_result_short || > + table_control == DUMP_TABLE_ALL) > + return((char*) row[0]); > + while (row && > + (((sequence= (strcmp(row[1], "SEQUENCE") == 0)) && > + (table_control & DUMP_TABLE_TABLE)) || > + ((!sequence) && > + (table_control & DUMP_TABLE_SEQUENCE)))) I'd rather just do static char *getTableName(int reset, uint want_sequence) and while (row && strcmp(row[1], "SEQUENCE") == want_sequence) row= mysql_fetch_row(get_table_name_result); > + { > + row= mysql_fetch_row(get_table_name_result); > + } > + if (row) > + return((char*) row[0]); > + } > if (reset) > mysql_data_seek(get_table_name_result,0); /* We want to read again */ > else > @@ -4785,17 +4834,17 @@ static int dump_all_stats() > /* EITS added in 10.0.1 */ > if (mysql_get_server_version(mysql) >= 100001) > { > - dump_table("column_stats", "mysql", NULL, 0); > - dump_table("index_stats", "mysql", NULL, 0); > - dump_table("table_stats", "mysql", NULL, 0); > + dump_table("column_stats", "mysql", NULL, 0, DUMP_TABLE_TABLE); > + dump_table("index_stats", "mysql", NULL, 0, DUMP_TABLE_TABLE); > + dump_table("table_stats", "mysql", NULL, 0, DUMP_TABLE_TABLE); this wouldn't be needed if the caller will only use dump_table for tables > } > /* Innodb may be disabled */ > if (!mysql_query(mysql, "show fields from innodb_index_stats")) > { > MYSQL_RES *tableres= mysql_store_result(mysql); > mysql_free_result(tableres); > - dump_table("innodb_index_stats", "mysql", NULL, 0); > - dump_table("innodb_table_stats", "mysql", NULL, 0); > + dump_table("innodb_index_stats", "mysql", NULL, 0, DUMP_TABLE_TABLE); > + dump_table("innodb_table_stats", "mysql", NULL, 0, DUMP_TABLE_TABLE); > } > opt_no_create_info= prev_no_create_info; > return 0; > @@ -4817,11 +4866,11 @@ static int dump_all_timezones() > opt_prev_no_create_info= opt_no_create_info; > opt_no_create_info= 1; > fprintf(md_result_file,"\nUSE mysql;\n"); > - dump_table("time_zone", "mysql", NULL, 0); > - dump_table("time_zone_name", "mysql", NULL, 0); > - dump_table("time_zone_leap_second", "mysql", NULL, 0); > - dump_table("time_zone_transition", "mysql", NULL, 0); > - dump_table("time_zone_transition_type", "mysql", NULL, 0); > + dump_table("time_zone", "mysql", NULL, 0, DUMP_TABLE_TABLE); > + dump_table("time_zone_name", "mysql", NULL, 0, DUMP_TABLE_TABLE); > + dump_table("time_zone_leap_second", "mysql", NULL, 0, DUMP_TABLE_TABLE); > + dump_table("time_zone_transition", "mysql", NULL, 0, DUMP_TABLE_TABLE); > + dump_table("time_zone_transition_type", "mysql", NULL, 0, DUMP_TABLE_TABLE); > opt_no_create_info= opt_prev_no_create_info; > return 0; > } > @@ -5346,12 +5396,29 @@ static int dump_all_tables_in_db(char *database) > DBUG_RETURN(1); > } > } > - while ((table= getTableName(0))) > + > + if (mysql_get_server_version(mysql) >= FIRST_SEQUENCE_VERSION && > + !opt_no_create_info) > + { > + // First process sequences > + while ((table= getTableName(1, DUMP_TABLE_SEQUENCE))) > + { > + char *end= strmov(afterdot, table); > + if (include_table((uchar*) hash_key, end - hash_key)) > + { > + dump_table(table, database, (uchar*) hash_key, end - hash_key, > + DUMP_TABLE_SEQUENCE); > + } > + } > + table_ctrl= DUMP_TABLE_TABLE; // next pass > + } and instead of all the above, you could do just for (int table_ctrl= 1; table_ctrl >= 0; table_ctrl--) > + while ((table= getTableName(0, table_ctrl))) > { > char *end= strmov(afterdot, table); > if (include_table((uchar*) hash_key, end - hash_key)) > { > - dump_table(table, database, (uchar*) hash_key, end - hash_key); > + dump_table(table, database, (uchar*) hash_key, end - hash_key, > + table_ctrl); > my_free(order_by); > order_by= 0; > if (opt_dump_triggers && mysql_get_server_version(mysql) >= 50009) > @@ -5745,11 +5813,24 @@ static int dump_selected_tables(char *db, char **table_names, int tables) > DBUG_RETURN(1); > } > } > + > + if (mysql_get_server_version(mysql) >= FIRST_SEQUENCE_VERSION && > + !opt_no_create_info) > + { > + /* Dump Sequence first */ > + for (pos= dump_tables; pos < end; pos++) > + { > + DBUG_PRINT("info",("Dumping sequence(?) %s", *pos)); > + dump_table(*pos, db, NULL, 0, DUMP_TABLE_SEQUENCE); > + } > + } > + else > + table_ctrl|= DUMP_TABLE_SEQUENCE; // dump all on next pass and here, in dump_selected_tables(), you'd need to do check_if_ignore_table() == IGNORE_SEQUENCE_TABLE. > /* Dump each selected table */ > for (pos= dump_tables; pos < end; pos++) > { > DBUG_PRINT("info",("Dumping table %s", *pos)); > - dump_table(*pos, db, NULL, 0); > + dump_table(*pos, db, NULL, 0, table_ctrl); > if (opt_dump_triggers && > mysql_get_server_version(mysql) >= 50009) > { Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] eb75e8705d9: MDEV-8134: The relay-log is not flushed after the slave-relay-log.999999 showed
by sujatha 21 Jan '21

21 Jan '21

revision-id: eb75e8705d9a444e10057967eaebf947b1115ff8 (mariadb-10.2.31-695-geb75e8705d9) parent(s): 53acd1c1d88be82190c56af3e4cc11fb2770a169 author: Sujatha committer: Sujatha timestamp: 2021-01-21 13:00:02 +0530 message: MDEV-8134: The relay-log is not flushed after the slave-relay-log.999999 showed Problem: ======== Auto purge of relaylogs stops when relay-log-file is 'slave-relay-log.999999' and slave_parallel_threads is enabled. Analysis: ========= The problem is that in Relay_log_info::inc_group_relay_log_pos() function, when two log names are compared via strcmp() function, it gives correct result, when log name sequence numbers are of same digits(6 digits), But when the number goes to 7 digits, a 999999 compares greater than 1000000, which is wrong, hence the bug. Fix: ==== Extract the numeric extension part of the file name, convert it into unsigned long and compare. Thanks to David Zhao for the contribution. --- .../suite/rpl/r/rpl_relay_max_extension.result | 37 +++++++ .../suite/rpl/t/rpl_relay_max_extension.test | 109 +++++++++++++++++++++ sql/rpl_parallel.cc | 5 +- sql/rpl_rli.cc | 4 +- sql/sql_repl.cc | 17 ++++ sql/sql_repl.h | 1 + 6 files changed, 169 insertions(+), 4 deletions(-) diff --git a/mysql-test/suite/rpl/r/rpl_relay_max_extension.result b/mysql-test/suite/rpl/r/rpl_relay_max_extension.result new file mode 100644 index 00000000000..4444398203e --- /dev/null +++ b/mysql-test/suite/rpl/r/rpl_relay_max_extension.result @@ -0,0 +1,37 @@ +include/rpl_init.inc [topology=1->2] +connection server_2; +include/stop_slave.inc +RESET SLAVE; +include/start_slave.inc +include/stop_slave.inc +# +# Stop slave server +# +# +# Simulate file number get close to 999997 +# by renaming relay logs and modifying index/info files +# +# Restart slave server +# +SET @save_slave_parallel_threads= @@GLOBAL.slave_parallel_threads; +SET @save_max_relay_log_size= @@GLOBAL.max_relay_log_size; +SET GLOBAL slave_parallel_threads=1; +SET GLOBAL max_relay_log_size=100 * 1024; +include/start_slave.inc +connection server_1; +create table t1 (i int, c varchar(1024)); +# +# Insert some data to generate enough amount of binary logs +# +connection server_2; +# +# Assert that 'slave-relay-bin.999999' is purged. +# +NOT FOUND /slave-relay-bin.999999/ in slave-relay-bin.index +include/stop_slave.inc +SET GLOBAL slave_parallel_threads= @save_slave_parallel_threads; +SET GLOBAL max_relay_log_size= @save_max_relay_log_size; +include/start_slave.inc +connection server_1; +DROP TABLE t1; +include/rpl_end.inc diff --git a/mysql-test/suite/rpl/t/rpl_relay_max_extension.test b/mysql-test/suite/rpl/t/rpl_relay_max_extension.test new file mode 100644 index 00000000000..e1e087f2e0e --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_relay_max_extension.test @@ -0,0 +1,109 @@ +# ==== Purpose ==== +# +# Test verifies that auto purging mechanism of relay logs works fine when the +# file extension grows beyond 999999. +# +# ==== Implementation ==== +# +# Steps: +# 0 - In master-slave setup clear all the relay logs on the slave server. +# 1 - Start the slave so that new relay logs starting from +# 'slave-relay-bin.000001' are created. +# 2 - Get the active relay-log file name by using SHOW SLAVE STATUS. +# Shutdown the slave server. +# 3 - Rename active relay log to '999997' in both 'relay-log.info' and +# 'slave-relay-bin.index' files. +# 4 - Restart the slave server by configuring 'slave_parallel_threads=1' +# and 'max_relay_log_size=100K'. +# 5 - Generate load on master such that few relay logs are generated on +# slave. The relay log sequence number will change to 7 digits. +# 6 - Sync slave with master to ensure that relay logs are applied on +# slave. They should have been automatically purged. +# 7 - Assert that there is no 'slave-relay-bin.999999' file in +# 'relay-log.info'. +# +# ==== References ==== +# +# MDEV-8134: The relay-log is not flushed after the slave-relay-log.999999 +# showed +# + +--source include/have_innodb.inc +--source include/have_binlog_format_row.inc +--let $rpl_topology=1->2 +--source include/rpl_init.inc + +--connection server_2 +--source include/stop_slave.inc +RESET SLAVE; +--source include/start_slave.inc +--source include/stop_slave.inc +--let $relay_log=query_get_value(SHOW SLAVE STATUS, Relay_Log_File, 1) + +--echo # +--echo # Stop slave server +--echo # + +--let $datadir = `select @@datadir` +--exec echo "wait" > $MYSQLTEST_VARDIR/tmp/mysqld.2.expect +--shutdown_server 10 +--source include/wait_until_disconnected.inc + +--exec sed -i "s/$relay_log/slave-relay-bin.999997/g" $datadir/relay-log.info +--exec sed -i "s/$relay_log/slave-relay-bin.999997/g" $datadir/slave-relay-bin.index + +--echo # +--echo # Simulate file number get close to 999997 +--echo # by renaming relay logs and modifying index/info files + +--move_file $datadir/$relay_log $datadir/slave-relay-bin.999997 + +--echo # +--echo # Restart slave server +--echo # + +--exec echo "restart" > $MYSQLTEST_VARDIR/tmp/mysqld.2.expect +--enable_reconnect +--source include/wait_until_connected_again.inc +SET @save_slave_parallel_threads= @@GLOBAL.slave_parallel_threads; +SET @save_max_relay_log_size= @@GLOBAL.max_relay_log_size; + +SET GLOBAL slave_parallel_threads=1; +SET GLOBAL max_relay_log_size=100 * 1024; +--source include/start_slave.inc + +--connection server_1 +create table t1 (i int, c varchar(1024)); +--echo # +--echo # Insert some data to generate enough amount of binary logs +--echo # +--let $count = 1000 +--disable_query_log +while ($count) +{ + eval insert into t1 values (1001 - $count, repeat('a',1000)); + dec $count; +} +--enable_query_log +--save_master_pos + +--connection server_2 +--sync_with_master + +--let $relay_log=query_get_value(SHOW SLAVE STATUS, Relay_Log_File, 1) + +--echo # +--echo # Assert that 'slave-relay-bin.999999' is purged. +--echo # +let SEARCH_FILE=$datadir/slave-relay-bin.index; +let SEARCH_PATTERN=slave-relay-bin.999999; +source include/search_pattern_in_file.inc; + +--source include/stop_slave.inc +SET GLOBAL slave_parallel_threads= @save_slave_parallel_threads; +SET GLOBAL max_relay_log_size= @save_max_relay_log_size; +--source include/start_slave.inc + +--connection server_1 +DROP TABLE t1; +--source include/rpl_end.inc diff --git a/sql/rpl_parallel.cc b/sql/rpl_parallel.cc index 4cf87ba73b7..869640fd46f 100644 --- a/sql/rpl_parallel.cc +++ b/sql/rpl_parallel.cc @@ -4,6 +4,7 @@ #include "rpl_mi.h" #include "sql_parse.h" #include "debug_sync.h" +#include "sql_repl.h" /* Code for optional parallel execution of replicated events on the slave. @@ -82,7 +83,7 @@ handle_queued_pos_update(THD *thd, rpl_parallel_thread::queued_event *qev) return; mysql_mutex_lock(&rli->data_lock); - cmp= strcmp(rli->group_relay_log_name, qev->event_relay_log_name); + cmp= compare_log_name(rli->group_relay_log_name, qev->event_relay_log_name); if (cmp < 0) { rli->group_relay_log_pos= qev->future_event_relay_log_pos; @@ -91,7 +92,7 @@ handle_queued_pos_update(THD *thd, rpl_parallel_thread::queued_event *qev) rli->group_relay_log_pos < qev->future_event_relay_log_pos) rli->group_relay_log_pos= qev->future_event_relay_log_pos; - cmp= strcmp(rli->group_master_log_name, qev->future_event_master_log_name); + cmp= compare_log_name(rli->group_master_log_name, qev->future_event_master_log_name); if (cmp < 0) { strcpy(rli->group_master_log_name, qev->future_event_master_log_name); diff --git a/sql/rpl_rli.cc b/sql/rpl_rli.cc index 40ab375571a..5273b33c728 100644 --- a/sql/rpl_rli.cc +++ b/sql/rpl_rli.cc @@ -989,7 +989,7 @@ void Relay_log_info::inc_group_relay_log_pos(ulonglong log_pos, if (rgi->is_parallel_exec) { /* In case of parallel replication, do not update the position backwards. */ - int cmp= strcmp(group_relay_log_name, rgi->event_relay_log_name); + int cmp= compare_log_name(group_relay_log_name, rgi->event_relay_log_name); if (cmp < 0) { group_relay_log_pos= rgi->future_event_relay_log_pos; @@ -1001,7 +1001,7 @@ void Relay_log_info::inc_group_relay_log_pos(ulonglong log_pos, In the parallel case we need to update the master_log_name here, rather than in Rotate_log_event::do_update_pos(). */ - cmp= strcmp(group_master_log_name, rgi->future_event_master_log_name); + cmp= compare_log_name(group_master_log_name, rgi->future_event_master_log_name); if (cmp <= 0) { if (cmp < 0) diff --git a/sql/sql_repl.cc b/sql/sql_repl.cc index 4af8ebc2dd8..59a3f686e45 100644 --- a/sql/sql_repl.cc +++ b/sql/sql_repl.cc @@ -4541,5 +4541,22 @@ rpl_gtid_pos_update(THD *thd, char *str, size_t len) return false; } +int compare_log_name(const char *log_1, const char *log_2) { + int res= 1; + const char *ext1_str= strrchr(log_1, '.'); + const char *ext2_str= strrchr(log_2, '.'); + char file_name_1[255], file_name_2[255]; + strmake(file_name_1, log_1, (ext1_str - log_1)); + strmake(file_name_2, log_2, (ext2_str - log_2)); + char *endptr = NULL; + res= strcmp(file_name_1, file_name_2); + if (!res) + { + ulong ext1= strtoul(++ext1_str, &endptr, 10); + ulong ext2= strtoul(++ext2_str, &endptr, 10); + res= (ext1 > ext2 ? 1 : ((ext1 == ext2) ? 0 : -1)); + } + return res; +} #endif /* HAVE_REPLICATION */ diff --git a/sql/sql_repl.h b/sql/sql_repl.h index 8ddfa9239f6..9129aaeed5e 100644 --- a/sql/sql_repl.h +++ b/sql/sql_repl.h @@ -56,6 +56,7 @@ bool show_binlogs(THD* thd); extern int init_master_info(Master_info* mi); void kill_zombie_dump_threads(uint32 slave_server_id); int check_binlog_magic(IO_CACHE* log, const char** errmsg); +int compare_log_name(const char *log_1, const char *log_2); struct LOAD_FILE_IO_CACHE : public IO_CACHE {

1 0

[Maria-developers] How to change max key length for MyISAM?
by Witold Filipczyk 15 Jan '21

15 Jan '21

Sorry, for private question, but I did not know where to ask. mariadb-10.3.27. I changed in include/myisam.h: #define MI_MAX_KEY_LENGTH 3072 /* Max length in bytes */ but, it did not help, still this ERROR 1071 (42000): Specified key was too long; max key length is 1000 bytes. What more to change to get key length 3072 for MyISAM tables? And what can be negative impact of such a change? Witold Filipczyk

5 8

[Maria-developers] MDEV-13467 review.
by Alexey Botchkov 09 Jan '21

09 Jan '21

Hi, Anel! Sorry for that long delay with this review. Had real difficult weeks of my life :( Now it seems to be over, so getting back to normal. Below is my opinions of the code i see in the branch bb-10.1-anel-MDEV-13467-gis-feature-st_sphere-v2 Hope it's the recent version. + Adapter for functions that takes two or three arguments. +*/ + +class Create_func_arg2_or_3 : public Create_func +{ Do you think we need the separate Create_fund_arg2_or_3 class? If you don't see any other class to inherit from it, i'd recommend to leave the Create_func_distance_sphere alone. No need to have two constructors in the Item_func_sphere_distance too. Just use the list argument as it's done for the Item_func_json_contains (in 10.2). Anyway i belive we should get rid of duplicating code about the 'param1' and 'param2'. They can be popped/checked with no if (arg_count == 2) condition. + double distance= 0.0; + null_value= (args[0]->null_value || args[1]->null_value); + if (null_value) + { + // Returned item must be set to Null + DBUG_ASSERT(maybe_null); don't think it's needed. + return 0; Should be return 0.0 i belive. Or goto handle_errors. Thet goes to all but one 'return' statements in this function. + } > ... > if (!arg1 || !arg2) This last condition should never happen since we have not-null_value. So i'd remove that 'if' section completely. And anyway it should do the 'return' in that branch. + null_value= args[2]->null_value; + if (null_value) + { + return 0; + } seems nicer to me this way if (args[2]->null_value) { null_value= true; goto handle_errors; } + if (!(g1= Geometry::construct(&buffer1, arg1->ptr(), arg1->length())) || + !(g2= Geometry::construct(&buffer2, arg2->ptr(), arg2->length()))) + { + goto handle_errors; we should launch the ER_GIS_INVALID_DATA here. + // Generate error message in case different geometry is used? Yes, i think the explainig message here is good to have. +double Item_func_sphere_distance::spherical_distance(Geometry *g1, + Geometry *g2, + const double sphere_radius) +double Item_func_sphere_distance::spherical_distance_points(Geometry *g1, + Geometry *g2, + const double r) Why these two are member static functions, not simply static? If you plan any 'external' use for them, please add comments about what exacly they do. +double Item_func_sphere_distance::spherical_distance(Geometry *g1, + Geometry *g2, + const double sphere_radius) +{ + double res= 0.0; + switch (g1->get_class_info()->m_type_id) + { + case Geometry::wkb_point: + res= spherical_distance_points(g1, g2, sphere_radius); + break; + + case Geometry::wkb_multipoint: + res= spherical_distance_points(g1, g2, sphere_radius); + break; Why do we need this function? We confirmed already that the type is either wkb_point or wkb_multipolygon, so just can call spherical_distance_points. No? >>>>> +class Item_func_sphere_distance: public Item_real_func +{ + double sphere_radius= 6370986.0; // Default radius equals Earth radius Some compilers don't like this syntax, and we didn't use it yet, so i'd recomment to move this to the constructor or even better to the ::val_real(). +double Gis_point::calculate_haversine(const Geometry *g, + const double sphere_radius) +{ + DBUG_ASSERT(sphere_radius > 0); + double x1r= 0.0; + double x2r= 0.0; + double y1r= 0.0; + double y2r= 0.0; That lot of unnecessary initializations seems excessive. + if (g->get_class_info()->m_type_id == Geometry::wkb_multipoint) + { + const char point_size= 4 + WKB_HEADER_SIZE + POINT_DATA_SIZE+1; //1 for the type + char point_temp[point_size]; + memset(point_temp+4, Geometry::wkb_point, 1); + memcpy(point_temp+5, static_cast<const Gis_multi_point *>(g)->get_data_ptr()+5, 4); + memcpy(point_temp+4+WKB_HEADER_SIZE, g->get_data_ptr()+4+WKB_HEADER_SIZE, + POINT_DATA_SIZE); + point_temp[point_size-1]= '\0'; + Geometry_buffer gbuff; + Geometry *gg= Geometry::construct(&gbuff, point_temp, point_size-1); There is the Gis_multi_point::geometry_n() function, should be used here to get the point. I don't like that lot of static_cast-s. Isn't it nicer to have just the function of double calculate_harvesine_points(Geometry *p1, Geometry *p2, double radius) double calculate_harvesine_multipoint_point(Geometry *mp, Geometry *p, double radius) double calculate_harvesine_multipoint_multipoint(Geometry *mp1, Geometry *mp2, double radius) which would call each other? The ::get_xy_radian() method looks unneeded to me. It's enough to have the to_radian(double d) function. BTW it's possible to do the multipoint/multipoint calculations faster than n^2. Not necessary to do it right now, just not to remember. Best regards. HF

1 0

[Maria-developers] 608b0ee52ef: MDEV-23033: All slaves crash once in ~24 hours and loop restart with signal 11
by sujatha 04 Jan '21

04 Jan '21

revision-id: 608b0ee52ef3e854ce14a407e64e936adbbeba23 (mariadb-10.2.31-648-g608b0ee52ef) parent(s): 25db9ffa8bdab8a2f2af3c7f154343dd6c6d238f author: Sujatha committer: Sujatha timestamp: 2021-01-04 15:06:12 +0530 message: MDEV-23033: All slaves crash once in ~24 hours and loop restart with signal 11 Problem: ======= Upon deleting or updating a row in a parent table (with primary key), if the child table has virtual column and an associated key with ON UPDATE CASCADE/ON DELETE CASCADE, it will result in slave crash. Analysis: ======== Tables which are related through foreign key require prelocking similar to triggers. i.e If a table has triggers/foreign keys we should add all tables and routines used by them to the prelocking set. This prelocking happens during 'open_and_lock_tables' call. Each table being opened is checked for foreign key references. If foreign key reference exists then the child table is opened and it is linked to the table_list. Upon any modification to parent table its corresponding child tables are retried from table_list and they are updated accordingly. This prelocking work fine on master. On slave prelocking works for following cases. - Statement/mixed based replication - In row based replication when trigger execution is enabled through 'slave_run_triggers_for_rbr=YES/LOGGING/ENFORCE' Otherwise it results in an assert/crash, as the parent table will not find the corresponding child table and it will be NULL. Dereferencing NULL pointer leads to slave server exit. Fix: === Introduce a new 'slave_fk_event_map' flag similar to 'trg_event_map'. This flag will ensure that when foreign key is enabled in row based replication all the parent and child tables are prelocked, so that parent is able to locate the child table. Note: This issue is specific to slave, hence only slave needs to be upgraded. --- mysql-test/suite/rpl/r/rpl_row_vcol_crash.result | 380 ++++++++++++++++++++ mysql-test/suite/rpl/t/rpl_row_vcol_crash.test | 425 +++++++++++++++++++++++ sql/log_event.cc | 42 +-- sql/sql_base.cc | 118 ++++--- sql/table.h | 6 +- 5 files changed, 906 insertions(+), 65 deletions(-) diff --git a/mysql-test/suite/rpl/r/rpl_row_vcol_crash.result b/mysql-test/suite/rpl/r/rpl_row_vcol_crash.result new file mode 100644 index 00000000000..f76d8935fa8 --- /dev/null +++ b/mysql-test/suite/rpl/r/rpl_row_vcol_crash.result @@ -0,0 +1,380 @@ +include/master-slave.inc +[connection master] +# +# Test case 1: KEY on a virtual column with ON DELETE CASCADE +# +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +INSERT INTO t1 VALUES (1),(2),(3); +CREATE TABLE t2 (id INT NOT NULL PRIMARY KEY, +t1_id INT NOT NULL, +v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), KEY (t1_id), +CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +INSERT INTO t2 VALUES (90,1,NULL); +INSERT INTO t2 VALUES (91,2,default); +DELETE FROM t1 WHERE id=1; +connection slave; +# +# Verify data consistency on slave +# +include/diff_tables.inc [master:test.t1, slave:test.t1] +include/diff_tables.inc [master:test.t2, slave:test.t2] +connection master; +DROP TABLE t2,t1; +connection slave; +# +# Test case 2: Verify "ON DELETE CASCADE" for parent->child->child scenario +# Parent table: users +# Child tables: matchmaking_groups, matchmaking_group_users +# Parent table: matchmaking_groups +# Child tables: matchmaking_group_users, matchmaking_group_maps +# +# Deleting a row from parent table should be reflected in +# child tables. +# matchmaking_groups->matchmaking_group_users->matchmaking_group_maps +# users->matchmaking_group_users->matchmaking_group_maps +# +connection master; +CREATE TABLE users (id INT UNSIGNED AUTO_INCREMENT PRIMARY KEY, +name VARCHAR(32) NOT NULL DEFAULT '' +) ENGINE=InnoDB DEFAULT CHARSET=utf8; +CREATE TABLE matchmaking_groups ( +id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY, +host_user_id INT UNSIGNED NOT NULL UNIQUE, +v_col INT AS (host_user_id+1) VIRTUAL, KEY (v_col), +CONSTRAINT FOREIGN KEY (host_user_id) REFERENCES users (id) +ON DELETE CASCADE ON UPDATE CASCADE +) ENGINE=InnoDB DEFAULT CHARSET=utf8; +CREATE TABLE matchmaking_group_users ( +matchmaking_group_id BIGINT UNSIGNED NOT NULL, +user_id INT UNSIGNED NOT NULL, +v_col1 int as (user_id+1) virtual, KEY (v_col1), +PRIMARY KEY (matchmaking_group_id,user_id), +UNIQUE KEY user_id (user_id), +CONSTRAINT FOREIGN KEY (matchmaking_group_id) +REFERENCES matchmaking_groups (id) ON DELETE CASCADE ON UPDATE CASCADE, +CONSTRAINT FOREIGN KEY (user_id) +REFERENCES users (id) ON DELETE CASCADE ON UPDATE CASCADE +) ENGINE=InnoDB DEFAULT CHARSET=utf8; +CREATE TABLE matchmaking_group_maps ( +matchmaking_group_id BIGINT UNSIGNED NOT NULL, +map_id TINYINT UNSIGNED NOT NULL, +v_col2 INT AS (map_id+1) VIRTUAL, KEY (v_col2), +PRIMARY KEY (matchmaking_group_id,map_id), +CONSTRAINT FOREIGN KEY (matchmaking_group_id) +REFERENCES matchmaking_groups (id) ON DELETE CASCADE ON UPDATE CASCADE +) ENGINE=InnoDB DEFAULT CHARSET=utf8; +connection slave; +connection master; +INSERT INTO users VALUES (NULL,'foo'),(NULL,'bar'); +INSERT INTO matchmaking_groups VALUES (10,1,default),(11,2,default); +INSERT INTO matchmaking_group_users VALUES (10,1,default),(11,2,default); +INSERT INTO matchmaking_group_maps VALUES (10,55,default),(11,66,default); +DELETE FROM matchmaking_groups WHERE id = 10; +connection slave; +# +# No rows should be returned as ON DELETE CASCASE should have removed +# corresponding rows from child tables. There should not any mismatch +# of 'id' field between parent->child. +# +SELECT * FROM matchmaking_group_users WHERE matchmaking_group_id NOT IN (SELECT id FROM matchmaking_groups); +matchmaking_group_id user_id v_col1 +SELECT * FROM matchmaking_group_maps WHERE matchmaking_group_id NOT IN (SELECT id FROM matchmaking_groups); +matchmaking_group_id map_id v_col2 +# +# Rows with id=11 should be present +# +SELECT * FROM matchmaking_group_users; +matchmaking_group_id user_id v_col1 +11 2 3 +SELECT * FROM matchmaking_group_maps; +matchmaking_group_id map_id v_col2 +11 66 67 +connection master; +DELETE FROM users WHERE id = 2; +connection slave; +# +# No rows should be present in both the child tables +# +SELECT * FROM matchmaking_group_users; +matchmaking_group_id user_id v_col1 +SELECT * FROM matchmaking_group_maps; +matchmaking_group_id map_id v_col2 +connection master; +DROP TABLE matchmaking_group_maps, matchmaking_group_users, matchmaking_groups, users; +connection slave; +# +# Test case 3: KEY on a virtual column with ON UPDATE CASCADE +# +connection master; +CREATE TABLE t1 (a INT NOT NULL PRIMARY KEY, b INT NOT NULL) ENGINE=InnoDB; +INSERT INTO t1 VALUES (1, 80); +CREATE TABLE t2 (a INT KEY, b INT, +v_col int as (b+1) virtual, KEY (v_col), +CONSTRAINT b FOREIGN KEY (b) REFERENCES t1(a) ON UPDATE CASCADE +) ENGINE=InnoDB; +INSERT INTO t2 VALUES (51, 1, default); +connection slave; +connection master; +UPDATE t1 SET a = 50 WHERE a = 1; +# +# Master: Verify that ON UPDATE CASCADE works fine +# old_row: (51, 1, 2) ON UPDATE New_row: (51, 50, 51) +# +SELECT * FROM t2 WHERE b=50; +a b v_col +51 50 51 +connection slave; +# +# Slave: Verify that ON UPDATE CASCADE works fine +# old_row: (51, 1, 2) ON UPDATE New_row: (51, 50, 51) +# +SELECT * FROM t2 WHERE b=50; +a b v_col +51 50 51 +connection master; +DROP TABLE t2, t1; +connection slave; +# +# Test case 4: Define triggers on master, their results should be +# replicated as part of row events and they should be +# applied on slave with the default +# slave_run_triggers_for_rbr=NO +# +connection master; +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +CREATE TABLE t2 (count INT NOT NULL) ENGINE=InnoDB; +CREATE TRIGGER trg AFTER INSERT ON t1 FOR EACH ROW INSERT INTO t2 VALUES (1); +INSERT INTO t1 VALUES (2),(3); +connection slave; +SHOW GLOBAL VARIABLES LIKE 'slave_run_triggers_for_rbr'; +Variable_name Value +slave_run_triggers_for_rbr NO +# +# As two rows are inserted in table 't1', two rows should get inserted +# into table 't2' as part of trigger. +# +include/assert.inc [Table t2 should have two rows.] +connection master; +DROP TABLE t1,t2; +connection slave; +# +# Test case 5: Define triggers + Foreign Keys on master, their results +# should be replicated as part of row events and master +# and slave should be in sync. +# +connection master; +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +CREATE TABLE t2 (t1_id INT NOT NULL, +v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), KEY (t1_id), +CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +CREATE TABLE t3 (count INT NOT NULL) ENGINE=InnoDB; +CREATE TRIGGER trg AFTER INSERT ON t1 FOR EACH ROW INSERT INTO t3 VALUES (1); +INSERT INTO t1 VALUES (2),(3); +INSERT INTO t2 VALUES (2, default), (3, default); +connection slave; +# +# As two rows are inserted in table 't1', two rows should get inserted +# into table 't3' as part of trigger. +# +include/assert.inc [Table t3 should have two rows.] +# +# Verify ON DELETE CASCASE correctness +# +connection master; +DELETE FROM t1 WHERE id=2; +connection slave; +connection master; +include/diff_tables.inc [master:test.t1, slave:test.t1] +include/diff_tables.inc [master:test.t2, slave:test.t2] +include/diff_tables.inc [master:test.t3, slave:test.t3] +DROP TABLE t3,t2,t1; +connection slave; +# +# Test case 6: Triggers are present only on slave and +# 'slave_run_triggers_for_rbr=NO' +# +connection slave; +SET @save_slave_run_triggers_for_rbr= @@GLOBAL.slave_run_triggers_for_rbr; +SET GLOBAL slave_run_triggers_for_rbr= NO;; +SHOW GLOBAL VARIABLES LIKE '%slave_run_triggers_for_rbr%'; +Variable_name Value +slave_run_triggers_for_rbr NO +connection master; +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +CREATE TABLE t2 (t1_id INT NOT NULL, +v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), +KEY (t1_id), CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +CREATE TABLE t3 (count INT NOT NULL) ENGINE=InnoDB; +connection slave; +CREATE TRIGGER trg AFTER INSERT ON t2 FOR EACH ROW INSERT INTO t3 VALUES (1); +connection master; +INSERT INTO t1 VALUES (2),(3); +INSERT INTO t2 VALUES (2, default), (3, default); +connection slave; +# +# Count must be 0 +# +include/assert.inc [Table t3 should have zero rows.] +connection master; +DELETE FROM t1 WHERE id=2; +connection slave; +SET GLOBAL slave_run_triggers_for_rbr= @save_slave_run_triggers_for_rbr; +# +# Verify t1, t2 are consistent on slave. +# +include/diff_tables.inc [master:test.t1, slave:test.t1] +include/diff_tables.inc [master:test.t2, slave:test.t2] +connection master; +DROP TABLE t3,t2,t1; +connection slave; +# +# Test case 7: Triggers are present only on slave and +# 'slave_run_triggers_for_rbr=YES' +# +connection slave; +SET @save_slave_run_triggers_for_rbr= @@GLOBAL.slave_run_triggers_for_rbr; +SET GLOBAL slave_run_triggers_for_rbr= YES;; +SHOW GLOBAL VARIABLES LIKE '%slave_run_triggers_for_rbr%'; +Variable_name Value +slave_run_triggers_for_rbr YES +connection master; +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +CREATE TABLE t2 (t1_id INT NOT NULL, +v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), +KEY (t1_id), CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +CREATE TABLE t3 (count INT NOT NULL) ENGINE=InnoDB; +connection slave; +CREATE TRIGGER trg AFTER INSERT ON t2 FOR EACH ROW INSERT INTO t3 VALUES (1); +connection master; +INSERT INTO t1 VALUES (2),(3); +INSERT INTO t2 VALUES (2, default), (3, default); +connection slave; +# +# Count must be 2 +# +include/assert.inc [Table t3 should have two rows.] +connection master; +DELETE FROM t1 WHERE id=2; +connection slave; +SET GLOBAL slave_run_triggers_for_rbr= @save_slave_run_triggers_for_rbr; +# +# Verify t1, t2 are consistent on slave. +# +include/diff_tables.inc [master:test.t1, slave:test.t1] +include/diff_tables.inc [master:test.t2, slave:test.t2] +connection master; +DROP TABLE t3,t2,t1; +connection slave; +# +# Test case 8: Triggers and Foreign Keys are present only on slave and +# 'slave_run_triggers_for_rbr=NO' +# +connection slave; +SET @save_slave_run_triggers_for_rbr= @@GLOBAL.slave_run_triggers_for_rbr; +SET GLOBAL slave_run_triggers_for_rbr= NO;; +SHOW GLOBAL VARIABLES LIKE '%slave_run_triggers_for_rbr%'; +Variable_name Value +slave_run_triggers_for_rbr NO +connection master; +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +SET sql_log_bin=0; +CREATE TABLE t2 (t1_id INT NOT NULL,v_col INT AS (t1_id+1) VIRTUAL) ENGINE=INNODB; +SET sql_log_bin=1; +CREATE TABLE t3 (count INT NOT NULL) ENGINE=InnoDB; +connection slave; +CREATE TABLE t2 (t1_id INT NOT NULL, +v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), KEY (t1_id), +CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +CREATE TRIGGER trg AFTER INSERT ON t2 FOR EACH ROW INSERT INTO t3 VALUES (1); +connection master; +INSERT INTO t1 VALUES (2),(3); +INSERT INTO t2 VALUES (2, default), (3, default); +connection slave; +# +# Count must be 0 +# +include/assert.inc [Table t3 should have zero rows.] +connection master; +DELETE FROM t1 WHERE id=2; +# t1: Should have one row +SELECT * FROM t1; +id +3 +# t2: Should have two rows +SELECT * FROM t2; +t1_id v_col +2 3 +3 4 +connection slave; +# t1: Should have one row +SELECT * FROM t1; +id +3 +# t2: Should have one row on slave due to ON DELETE CASCASE +SELECT * FROM t2; +t1_id v_col +3 4 +SET GLOBAL slave_run_triggers_for_rbr= @save_slave_run_triggers_for_rbr; +connection master; +DROP TABLE t3,t2,t1; +connection slave; +# +# Test case 9: Triggers are Foreign Keys are present only on slave and +# 'slave_run_triggers_for_rbr=YES' +# +connection slave; +SET @save_slave_run_triggers_for_rbr= @@GLOBAL.slave_run_triggers_for_rbr; +SET GLOBAL slave_run_triggers_for_rbr= YES;; +SHOW GLOBAL VARIABLES LIKE '%slave_run_triggers_for_rbr%'; +Variable_name Value +slave_run_triggers_for_rbr YES +connection master; +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +SET sql_log_bin=0; +CREATE TABLE t2 (t1_id INT NOT NULL,v_col INT AS (t1_id+1) VIRTUAL) ENGINE=INNODB; +SET sql_log_bin=1; +CREATE TABLE t3 (count INT NOT NULL) ENGINE=InnoDB; +connection slave; +CREATE TABLE t2 (t1_id INT NOT NULL, +v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), KEY (t1_id), +CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +CREATE TRIGGER trg AFTER INSERT ON t2 FOR EACH ROW INSERT INTO t3 VALUES (1); +connection master; +INSERT INTO t1 VALUES (2),(3); +INSERT INTO t2 VALUES (2, default), (3, default); +connection slave; +# +# Count must be 2 +# +include/assert.inc [Table t3 should have two rows.] +connection master; +DELETE FROM t1 WHERE id=2; +# t1: Should have one row +SELECT * FROM t1; +id +3 +# t2: Should have two rows +SELECT * FROM t2; +t1_id v_col +2 3 +3 4 +connection slave; +# t1: Should have one row +SELECT * FROM t1; +id +3 +# t2: Should have one row on slave due to ON DELETE CASCASE +SELECT * FROM t2; +t1_id v_col +3 4 +SET GLOBAL slave_run_triggers_for_rbr= @save_slave_run_triggers_for_rbr; +connection master; +DROP TABLE t3,t2,t1; +connection slave; +include/rpl_end.inc diff --git a/mysql-test/suite/rpl/t/rpl_row_vcol_crash.test b/mysql-test/suite/rpl/t/rpl_row_vcol_crash.test new file mode 100644 index 00000000000..84ee14977f3 --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_row_vcol_crash.test @@ -0,0 +1,425 @@ +# ==== Purpose ==== +# +# Test verifies that, slave doesn't report any assert on UPDATE or DELETE of +# row which tries to update the virtual columns with associated KEYs. +# +# Test scenarios are listed below. +# 1) KEY on a virtual column with ON DELETE CASCADE +# 2) Verify "ON DELETE CASCADE" for parent->child->child scenario +# 3) KEY on a virtual column with ON UPDATE CASCADE +# 4) Define triggers on master, their results should be replicated +# as part of row events and they should be applied on slave with +# the default slave_run_triggers_for_rbr=NO +# 5) Define triggers + Foreign Keys on master, their results should be +# replicated as part of row events and master and slave should be in sync. +# 6) Triggers are present only on slave and 'slave_run_triggers_for_rbr=NO' +# 7) Triggers are present only on slave and 'slave_run_triggers_for_rbr=YES' +# 8) Triggers and Foreign Keys are present only on slave and +# 'slave_run_triggers_for_rbr=NO' +# 9) Triggers are Foreign Keys are present only on slave and +# 'slave_run_triggers_for_rbr=YES' +# +# ==== References ==== +# +# MDEV-23033: All slaves crash once in ~24 hours and loop restart with signal 11 +# + +--source include/have_binlog_format_row.inc +--source include/have_innodb.inc +--source include/master-slave.inc + + +--echo # +--echo # Test case 1: KEY on a virtual column with ON DELETE CASCADE +--echo # +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +INSERT INTO t1 VALUES (1),(2),(3); + +CREATE TABLE t2 (id INT NOT NULL PRIMARY KEY, + t1_id INT NOT NULL, + v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), KEY (t1_id), + CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; + +INSERT INTO t2 VALUES (90,1,NULL); +INSERT INTO t2 VALUES (91,2,default); + +# Following query results in an assert on slave +DELETE FROM t1 WHERE id=1; +--sync_slave_with_master + +--echo # +--echo # Verify data consistency on slave +--echo # +--let $diff_tables= master:test.t1, slave:test.t1 +--source include/diff_tables.inc +--let $diff_tables= master:test.t2, slave:test.t2 +--source include/diff_tables.inc + +--connection master +DROP TABLE t2,t1; +--sync_slave_with_master + +--echo # +--echo # Test case 2: Verify "ON DELETE CASCADE" for parent->child->child scenario +--echo # Parent table: users +--echo # Child tables: matchmaking_groups, matchmaking_group_users +--echo # Parent table: matchmaking_groups +--echo # Child tables: matchmaking_group_users, matchmaking_group_maps +--echo # +--echo # Deleting a row from parent table should be reflected in +--echo # child tables. +--echo # matchmaking_groups->matchmaking_group_users->matchmaking_group_maps +--echo # users->matchmaking_group_users->matchmaking_group_maps +--echo # + +--connection master +CREATE TABLE users (id INT UNSIGNED AUTO_INCREMENT PRIMARY KEY, + name VARCHAR(32) NOT NULL DEFAULT '' +) ENGINE=InnoDB DEFAULT CHARSET=utf8; + +CREATE TABLE matchmaking_groups ( + id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY, + host_user_id INT UNSIGNED NOT NULL UNIQUE, + v_col INT AS (host_user_id+1) VIRTUAL, KEY (v_col), + CONSTRAINT FOREIGN KEY (host_user_id) REFERENCES users (id) + ON DELETE CASCADE ON UPDATE CASCADE +) ENGINE=InnoDB DEFAULT CHARSET=utf8; + +CREATE TABLE matchmaking_group_users ( + matchmaking_group_id BIGINT UNSIGNED NOT NULL, + user_id INT UNSIGNED NOT NULL, + v_col1 int as (user_id+1) virtual, KEY (v_col1), + PRIMARY KEY (matchmaking_group_id,user_id), + UNIQUE KEY user_id (user_id), + CONSTRAINT FOREIGN KEY (matchmaking_group_id) + REFERENCES matchmaking_groups (id) ON DELETE CASCADE ON UPDATE CASCADE, + CONSTRAINT FOREIGN KEY (user_id) + REFERENCES users (id) ON DELETE CASCADE ON UPDATE CASCADE +) ENGINE=InnoDB DEFAULT CHARSET=utf8; + +CREATE TABLE matchmaking_group_maps ( + matchmaking_group_id BIGINT UNSIGNED NOT NULL, + map_id TINYINT UNSIGNED NOT NULL, + v_col2 INT AS (map_id+1) VIRTUAL, KEY (v_col2), + PRIMARY KEY (matchmaking_group_id,map_id), + CONSTRAINT FOREIGN KEY (matchmaking_group_id) + REFERENCES matchmaking_groups (id) ON DELETE CASCADE ON UPDATE CASCADE +) ENGINE=InnoDB DEFAULT CHARSET=utf8; +--sync_slave_with_master + +--connection master +INSERT INTO users VALUES (NULL,'foo'),(NULL,'bar'); +INSERT INTO matchmaking_groups VALUES (10,1,default),(11,2,default); +INSERT INTO matchmaking_group_users VALUES (10,1,default),(11,2,default); +INSERT INTO matchmaking_group_maps VALUES (10,55,default),(11,66,default); + +DELETE FROM matchmaking_groups WHERE id = 10; +--sync_slave_with_master + +--echo # +--echo # No rows should be returned as ON DELETE CASCASE should have removed +--echo # corresponding rows from child tables. There should not any mismatch +--echo # of 'id' field between parent->child. +--echo # +SELECT * FROM matchmaking_group_users WHERE matchmaking_group_id NOT IN (SELECT id FROM matchmaking_groups); +SELECT * FROM matchmaking_group_maps WHERE matchmaking_group_id NOT IN (SELECT id FROM matchmaking_groups); + +--echo # +--echo # Rows with id=11 should be present +--echo # +SELECT * FROM matchmaking_group_users; +SELECT * FROM matchmaking_group_maps; + +--connection master +DELETE FROM users WHERE id = 2; +--sync_slave_with_master + +--echo # +--echo # No rows should be present in both the child tables +--echo # +SELECT * FROM matchmaking_group_users; +SELECT * FROM matchmaking_group_maps; + +--connection master +DROP TABLE matchmaking_group_maps, matchmaking_group_users, matchmaking_groups, users; +--sync_slave_with_master + +--echo # +--echo # Test case 3: KEY on a virtual column with ON UPDATE CASCADE +--echo # + +--connection master +CREATE TABLE t1 (a INT NOT NULL PRIMARY KEY, b INT NOT NULL) ENGINE=InnoDB; +INSERT INTO t1 VALUES (1, 80); + +CREATE TABLE t2 (a INT KEY, b INT, + v_col int as (b+1) virtual, KEY (v_col), + CONSTRAINT b FOREIGN KEY (b) REFERENCES t1(a) ON UPDATE CASCADE +) ENGINE=InnoDB; +INSERT INTO t2 VALUES (51, 1, default); +--sync_slave_with_master + +--connection master +UPDATE t1 SET a = 50 WHERE a = 1; + +--echo # +--echo # Master: Verify that ON UPDATE CASCADE works fine +--echo # old_row: (51, 1, 2) ON UPDATE New_row: (51, 50, 51) +--echo # +SELECT * FROM t2 WHERE b=50; +--sync_slave_with_master + +--echo # +--echo # Slave: Verify that ON UPDATE CASCADE works fine +--echo # old_row: (51, 1, 2) ON UPDATE New_row: (51, 50, 51) +--echo # +SELECT * FROM t2 WHERE b=50; + +--connection master +DROP TABLE t2, t1; +--sync_slave_with_master + +--echo # +--echo # Test case 4: Define triggers on master, their results should be +--echo # replicated as part of row events and they should be +--echo # applied on slave with the default +--echo # slave_run_triggers_for_rbr=NO +--echo # + +# In row-based replication, the binary log contains row changes. It will have +# both the changes made by the statement itself, and the changes made by the +# triggers that were invoked by the statement. Slave server(s) do not need to +# run triggers for row changes they are applying. Hence verify that this +# property remains the same and data should be available as if trigger was +# executed. Please note by default slave_run_triggers_for_rbr=NO. + +--connection master +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +CREATE TABLE t2 (count INT NOT NULL) ENGINE=InnoDB; +CREATE TRIGGER trg AFTER INSERT ON t1 FOR EACH ROW INSERT INTO t2 VALUES (1); +INSERT INTO t1 VALUES (2),(3); +--sync_slave_with_master + +SHOW GLOBAL VARIABLES LIKE 'slave_run_triggers_for_rbr'; +--echo # +--echo # As two rows are inserted in table 't1', two rows should get inserted +--echo # into table 't2' as part of trigger. +--echo # +--let $assert_cond= COUNT(*) = 2 FROM t2 +--let $assert_text= Table t2 should have two rows. +--source include/assert.inc + +--connection master +DROP TABLE t1,t2; +--sync_slave_with_master + +--echo # +--echo # Test case 5: Define triggers + Foreign Keys on master, their results +--echo # should be replicated as part of row events and master +--echo # and slave should be in sync. +--echo # +--connection master +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +CREATE TABLE t2 (t1_id INT NOT NULL, + v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), KEY (t1_id), + CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +CREATE TABLE t3 (count INT NOT NULL) ENGINE=InnoDB; +CREATE TRIGGER trg AFTER INSERT ON t1 FOR EACH ROW INSERT INTO t3 VALUES (1); + +INSERT INTO t1 VALUES (2),(3); +INSERT INTO t2 VALUES (2, default), (3, default); +--sync_slave_with_master + +--echo # +--echo # As two rows are inserted in table 't1', two rows should get inserted +--echo # into table 't3' as part of trigger. +--echo # +--let $assert_cond= COUNT(*) = 2 FROM t3 +--let $assert_text= Table t3 should have two rows. +--source include/assert.inc + +--echo # +--echo # Verify ON DELETE CASCASE correctness +--echo # +--connection master +DELETE FROM t1 WHERE id=2; +--sync_slave_with_master + +--connection master +--let $diff_tables= master:test.t1, slave:test.t1 +--source include/diff_tables.inc +--let $diff_tables= master:test.t2, slave:test.t2 +--source include/diff_tables.inc +--let $diff_tables= master:test.t3, slave:test.t3 +--source include/diff_tables.inc + +DROP TABLE t3,t2,t1; +--sync_slave_with_master + +# +# Test case: Triggers only on slave +# +--write_file $MYSQLTEST_VARDIR/tmp/trig_on_slave.inc PROCEDURE + if ($slave_run_triggers_for_rbr == '') { + --die !!!ERROR IN TEST: you must set $slave_run_triggers_for_rbr + } + +--connection slave +SET @save_slave_run_triggers_for_rbr= @@GLOBAL.slave_run_triggers_for_rbr; +--eval SET GLOBAL slave_run_triggers_for_rbr= $slave_run_triggers_for_rbr; +SHOW GLOBAL VARIABLES LIKE '%slave_run_triggers_for_rbr%'; + +--connection master +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +CREATE TABLE t2 (t1_id INT NOT NULL, + v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), + KEY (t1_id), CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +CREATE TABLE t3 (count INT NOT NULL) ENGINE=InnoDB; +--sync_slave_with_master + +CREATE TRIGGER trg AFTER INSERT ON t2 FOR EACH ROW INSERT INTO t3 VALUES (1); + +--connection master +INSERT INTO t1 VALUES (2),(3); +INSERT INTO t2 VALUES (2, default), (3, default); +--sync_slave_with_master + +if ($slave_run_triggers_for_rbr == 'NO') { +--echo # +--echo # Count must be 0 +--echo # +--let $assert_cond= COUNT(*) = 0 FROM t3 +--let $assert_text= Table t3 should have zero rows. +--source include/assert.inc +} +if ($slave_run_triggers_for_rbr == 'YES') { +--echo # +--echo # Count must be 2 +--echo # +--let $assert_cond= COUNT(*) = 2 FROM t3 +--let $assert_text= Table t3 should have two rows. +--source include/assert.inc +} + +--connection master +DELETE FROM t1 WHERE id=2; +--sync_slave_with_master +SET GLOBAL slave_run_triggers_for_rbr= @save_slave_run_triggers_for_rbr; + +--echo # +--echo # Verify t1, t2 are consistent on slave. +--echo # +--let $diff_tables= master:test.t1, slave:test.t1 +--source include/diff_tables.inc +--let $diff_tables= master:test.t2, slave:test.t2 +--source include/diff_tables.inc + +--connection master +DROP TABLE t3,t2,t1; +--sync_slave_with_master +#END OF +PROCEDURE + +--echo # +--echo # Test case 6: Triggers are present only on slave and +--echo # 'slave_run_triggers_for_rbr=NO' +--echo # +--let $slave_run_triggers_for_rbr=NO +--source $MYSQLTEST_VARDIR/tmp/trig_on_slave.inc + +--echo # +--echo # Test case 7: Triggers are present only on slave and +--echo # 'slave_run_triggers_for_rbr=YES' +--echo # +--let $slave_run_triggers_for_rbr=YES +--source $MYSQLTEST_VARDIR/tmp/trig_on_slave.inc +--remove_file $MYSQLTEST_VARDIR/tmp/trig_on_slave.inc + +# +# Test case: Trigger and Foreign Key are present only on slave +# +--write_file $MYSQLTEST_VARDIR/tmp/trig_fk_on_slave.inc PROCEDURE + if ($slave_run_triggers_for_rbr == '') { + --die !!!ERROR IN TEST: you must set $slave_run_triggers_for_rbr + } + +--connection slave +SET @save_slave_run_triggers_for_rbr= @@GLOBAL.slave_run_triggers_for_rbr; +--eval SET GLOBAL slave_run_triggers_for_rbr= $slave_run_triggers_for_rbr; +SHOW GLOBAL VARIABLES LIKE '%slave_run_triggers_for_rbr%'; + +--connection master +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +SET sql_log_bin=0; +CREATE TABLE t2 (t1_id INT NOT NULL,v_col INT AS (t1_id+1) VIRTUAL) ENGINE=INNODB; +SET sql_log_bin=1; +CREATE TABLE t3 (count INT NOT NULL) ENGINE=InnoDB; +--sync_slave_with_master + +# Have foreign key and trigger on slave. +CREATE TABLE t2 (t1_id INT NOT NULL, + v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), KEY (t1_id), + CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +CREATE TRIGGER trg AFTER INSERT ON t2 FOR EACH ROW INSERT INTO t3 VALUES (1); + +--connection master +INSERT INTO t1 VALUES (2),(3); +INSERT INTO t2 VALUES (2, default), (3, default); +--sync_slave_with_master + +if ($slave_run_triggers_for_rbr == 'NO') { +--echo # +--echo # Count must be 0 +--echo # +--let $assert_cond= COUNT(*) = 0 FROM t3 +--let $assert_text= Table t3 should have zero rows. +--source include/assert.inc +} +if ($slave_run_triggers_for_rbr == 'YES') { +--echo # +--echo # Count must be 2 +--echo # +--let $assert_cond= COUNT(*) = 2 FROM t3 +--let $assert_text= Table t3 should have two rows. +--source include/assert.inc +} + +--connection master +DELETE FROM t1 WHERE id=2; +--echo # t1: Should have one row +SELECT * FROM t1; +--echo # t2: Should have two rows +SELECT * FROM t2; +--sync_slave_with_master +--echo # t1: Should have one row +SELECT * FROM t1; +--echo # t2: Should have one row on slave due to ON DELETE CASCASE +SELECT * FROM t2; +SET GLOBAL slave_run_triggers_for_rbr= @save_slave_run_triggers_for_rbr; + +--connection master +DROP TABLE t3,t2,t1; +--sync_slave_with_master +#END OF +PROCEDURE + +--echo # +--echo # Test case 8: Triggers and Foreign Keys are present only on slave and +--echo # 'slave_run_triggers_for_rbr=NO' +--echo # +--let $slave_run_triggers_for_rbr=NO +--source $MYSQLTEST_VARDIR/tmp/trig_fk_on_slave.inc + +--echo # +--echo # Test case 9: Triggers are Foreign Keys are present only on slave and +--echo # 'slave_run_triggers_for_rbr=YES' +--echo # +--let $slave_run_triggers_for_rbr=YES +--source $MYSQLTEST_VARDIR/tmp/trig_fk_on_slave.inc +--remove_file $MYSQLTEST_VARDIR/tmp/trig_fk_on_slave.inc + +--source include/rpl_end.inc diff --git a/sql/log_event.cc b/sql/log_event.cc index c649e1f64fa..c32f31db1f6 100644 --- a/sql/log_event.cc +++ b/sql/log_event.cc @@ -10718,7 +10718,7 @@ int Rows_log_event::do_add_row_data(uchar *row_data, size_t length) There was the same problem with MERGE MYISAM tables and so here we try to go the same way. */ -static void restore_empty_query_table_list(LEX *lex) +inline void restore_empty_query_table_list(LEX *lex) { if (lex->first_not_own_table()) (*lex->first_not_own_table()->prev_global)= NULL; @@ -10733,6 +10733,8 @@ int Rows_log_event::do_apply_event(rpl_group_info *rgi) TABLE* table; DBUG_ENTER("Rows_log_event::do_apply_event(Relay_log_info*)"); int error= 0; + LEX *lex= thd->lex; + uint8 new_trg_event_map= get_trg_event_map(); /* If m_table_id == ~0ULL, then we have a dummy event that does not contain any data. In that case, we just remove all tables in the @@ -10823,27 +10825,29 @@ int Rows_log_event::do_apply_event(rpl_group_info *rgi) DBUG_ASSERT(!debug_sync_set_action(thd, STRING_WITH_LEN(action))); };); - if (slave_run_triggers_for_rbr) - { - LEX *lex= thd->lex; - uint8 new_trg_event_map= get_trg_event_map(); - - /* - Trigger's procedures work with global table list. So we have to add - rgi->tables_to_lock content there to get trigger's in the list. + /* + Trigger's procedures work with global table list. So we have to add + rgi->tables_to_lock content there to get trigger's in the list. - Then restore_empty_query_table_list() restore the list as it was - */ - DBUG_ASSERT(lex->query_tables == NULL); - if ((lex->query_tables= rgi->tables_to_lock)) - rgi->tables_to_lock->prev_global= &lex->query_tables; + Then restore_empty_query_table_list() restore the list as it was + */ + DBUG_ASSERT(lex->query_tables == NULL); + if ((lex->query_tables= rgi->tables_to_lock)) + rgi->tables_to_lock->prev_global= &lex->query_tables; - for (TABLE_LIST *tables= rgi->tables_to_lock; tables; - tables= tables->next_global) + for (TABLE_LIST *tables= rgi->tables_to_lock; tables; + tables= tables->next_global) + { + if (slave_run_triggers_for_rbr) { tables->trg_event_map= new_trg_event_map; lex->query_tables_last= &tables->next_global; } + else if (!WSREP_ON) + { + tables->slave_fk_event_map= new_trg_event_map; + lex->query_tables_last= &tables->next_global; + } } if (open_and_lock_tables(thd, rgi->tables_to_lock, FALSE, 0)) { @@ -11193,8 +11197,7 @@ int Rows_log_event::do_apply_event(rpl_group_info *rgi) } /* remove trigger's tables */ - if (slave_run_triggers_for_rbr) - restore_empty_query_table_list(thd->lex); + restore_empty_query_table_list(thd->lex); #if defined(WITH_WSREP) && defined(HAVE_QUERY_CACHE) if (WSREP(thd) && thd->wsrep_exec_mode == REPL_RECV) @@ -11212,8 +11215,7 @@ int Rows_log_event::do_apply_event(rpl_group_info *rgi) DBUG_RETURN(error); err: - if (slave_run_triggers_for_rbr) - restore_empty_query_table_list(thd->lex); + restore_empty_query_table_list(thd->lex); rgi->slave_close_thread_tables(thd); DBUG_RETURN(error); } diff --git a/sql/sql_base.cc b/sql/sql_base.cc index 8e57ea437b6..b8d18abb50c 100644 --- a/sql/sql_base.cc +++ b/sql/sql_base.cc @@ -4341,6 +4341,70 @@ bool table_already_fk_prelocked(TABLE_LIST *tl, LEX_STRING *db, return false; } +/** + Extend the table_list to include foreign tables for prelocking. + + @param[in] thd Thread context. + @param[in] prelocking_ctx Prelocking context of the statement. + @param[in] table_list Table list element for table. + @param[in] sp Routine body. + @param[out] need_prelocking Set to TRUE if method detects that prelocking + required, not changed otherwise. + + @retval FALSE Success. + @retval TRUE Failure (OOM). +*/ +inline bool +prepare_fk_prelocking_list(THD *thd, Query_tables_list *prelocking_ctx, + TABLE_LIST *table_list, bool *need_prelocking, + uint8 op) +{ + List <FOREIGN_KEY_INFO> fk_list; + List_iterator<FOREIGN_KEY_INFO> fk_list_it(fk_list); + FOREIGN_KEY_INFO *fk; + Query_arena *arena, backup; + + arena= thd->activate_stmt_arena_if_needed(&backup); + + table_list->table->file->get_parent_foreign_key_list(thd, &fk_list); + if (thd->is_error()) + { + if (arena) + thd->restore_active_arena(arena, &backup); + return TRUE; + } + + *need_prelocking= TRUE; + + while ((fk= fk_list_it++)) + { + // FK_OPTION_RESTRICT and FK_OPTION_NO_ACTION only need read access + static bool can_write[]= { true, false, true, true, false, true }; + thr_lock_type lock_type; + + if ((op & (1 << TRG_EVENT_DELETE) && can_write[fk->delete_method]) + || (op & (1 << TRG_EVENT_UPDATE) && can_write[fk->update_method])) + lock_type= TL_WRITE_ALLOW_WRITE; + else + lock_type= TL_READ; + + if (table_already_fk_prelocked(prelocking_ctx->query_tables, + fk->foreign_db, fk->foreign_table, + lock_type)) + continue; + + TABLE_LIST *tl= (TABLE_LIST *) thd->alloc(sizeof(TABLE_LIST)); + tl->init_one_table_for_prelocking(fk->foreign_db->str, fk->foreign_db->length, + fk->foreign_table->str, fk->foreign_table->length, + NULL, lock_type, false, table_list->belong_to_view, + op, &prelocking_ctx->query_tables_last); + } + if (arena) + thd->restore_active_arena(arena, &backup); + + return FALSE; +} + /** Defines how prelocking algorithm for DML statements should handle table list @@ -4381,55 +4445,21 @@ handle_table(THD *thd, Query_tables_list *prelocking_ctx, add_tables_and_routines_for_triggers(thd, prelocking_ctx, table_list)) return TRUE; } - if (table_list->table->file->referenced_by_foreign_key()) { - List <FOREIGN_KEY_INFO> fk_list; - List_iterator<FOREIGN_KEY_INFO> fk_list_it(fk_list); - FOREIGN_KEY_INFO *fk; - Query_arena *arena, backup; - - arena= thd->activate_stmt_arena_if_needed(&backup); - - table_list->table->file->get_parent_foreign_key_list(thd, &fk_list); - if (thd->is_error()) - { - if (arena) - thd->restore_active_arena(arena, &backup); - return TRUE; - } - - *need_prelocking= TRUE; - - while ((fk= fk_list_it++)) - { - // FK_OPTION_RESTRICT and FK_OPTION_NO_ACTION only need read access - static bool can_write[]= { true, false, true, true, false, true }; - uint8 op= table_list->trg_event_map; - thr_lock_type lock_type; - - if ((op & (1 << TRG_EVENT_DELETE) && can_write[fk->delete_method]) - || (op & (1 << TRG_EVENT_UPDATE) && can_write[fk->update_method])) - lock_type= TL_WRITE_ALLOW_WRITE; - else - lock_type= TL_READ; - - if (table_already_fk_prelocked(prelocking_ctx->query_tables, - fk->foreign_db, fk->foreign_table, - lock_type)) - continue; - - TABLE_LIST *tl= (TABLE_LIST *) thd->alloc(sizeof(TABLE_LIST)); - tl->init_one_table_for_prelocking(fk->foreign_db->str, fk->foreign_db->length, - fk->foreign_table->str, fk->foreign_table->length, - NULL, lock_type, false, table_list->belong_to_view, - op, &prelocking_ctx->query_tables_last); - } - if (arena) - thd->restore_active_arena(arena, &backup); + return (prepare_fk_prelocking_list(thd, prelocking_ctx, table_list, + need_prelocking, + table_list->trg_event_map)); } } + else if (table_list->slave_fk_event_map && + table_list->table->file->referenced_by_foreign_key()) + { + return (prepare_fk_prelocking_list(thd, prelocking_ctx, + table_list, need_prelocking, + table_list->slave_fk_event_map)); + } return FALSE; } diff --git a/sql/table.h b/sql/table.h index 9a864f7ce9f..57706655d9b 100644 --- a/sql/table.h +++ b/sql/table.h @@ -2277,8 +2277,12 @@ struct TABLE_LIST Indicates what triggers we need to pre-load for this TABLE_LIST when opening an associated TABLE. This is filled after the parsed tree is created. + + slave_fk_event_map is filled on the slave side with bitmaps value + representing row-based event operation to help find and prelock + possible FK constrain-related child tables. */ - uint8 trg_event_map; + uint8 trg_event_map, slave_fk_event_map; /* TRUE <=> this table is a const one and was optimized away. */ bool optimized_away;

2 2

[Maria-developers] 3f48bc08b92: MDEV-23033: All slaves crash once in ~24 hours and loop restart with signal 11
by sujatha 31 Dec '20

31 Dec '20

revision-id: 3f48bc08b92b2734c2af7408c246b96338675205 (mariadb-10.2.31-646-g3f48bc08b92) parent(s): 78292047a4747ccd9210ba36a185a1dbe825de89 author: Sujatha committer: Sujatha timestamp: 2020-12-31 15:43:53 +0530 message: MDEV-23033: All slaves crash once in ~24 hours and loop restart with signal 11 Problem: ======= Upon deleting or updating a row in a parent table (with primary key), if the child table has virtual column and an associated key with ON UPDATE CASCADE/ON DELETE CASCADE, it will result in slave crash. Analysis: ======== Tables which are related through foreign key require prelocking similar to triggers. i.e If a table has triggers/foreign keys we should add all tables and routines used by them to the prelocking set. This prelocking happens during 'open_and_lock_tables' call. Each table being opened is checked for foreign key references. If foreign key reference exists then the child table is opened and it is linked to the table_list. Upon any modification to parent table its corresponding child tables are retried from table_list and they are updated accordingly. This prelocking work fine on master. On slave prelocking works for following cases. - Statement/mixed based replication - In row based replication when trigger execution is enabled through 'slave_run_triggers_for_rbr=YES/LOGGING/ENFORCE' Otherwise it results in an assert/crash, as the parent table will not find the corresponding child table and it will be NULL. Dereferencing NULL pointer leads to slave server exit. Fix: === Introduce a new 'slave_fk_event_map' flag similar to 'trg_event_map'. This flag will ensure that when foreign key is enabled in row based replication all the parent and child tables are prelocked, so that parent is able to locate the child table. --- mysql-test/suite/rpl/r/rpl_row_vcol_crash.result | 381 +++++++++++++++++++++ mysql-test/suite/rpl/t/rpl_row_vcol_crash.test | 410 +++++++++++++++++++++++ sql/log_event.cc | 44 ++- sql/sql_base.cc | 6 +- sql/table.h | 6 +- 5 files changed, 821 insertions(+), 26 deletions(-) diff --git a/mysql-test/suite/rpl/r/rpl_row_vcol_crash.result b/mysql-test/suite/rpl/r/rpl_row_vcol_crash.result new file mode 100644 index 00000000000..e2a5d9b65e5 --- /dev/null +++ b/mysql-test/suite/rpl/r/rpl_row_vcol_crash.result @@ -0,0 +1,381 @@ +include/master-slave.inc +[connection master] +# +# Test case: KEY on a virtual column with ON DELETE CASCADE +# +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +INSERT INTO t1 VALUES (1),(2),(3); +CREATE TABLE t2 (id INT NOT NULL PRIMARY KEY, +t1_id INT NOT NULL, +v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), KEY (t1_id), +CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +INSERT INTO t2 VALUES (90,1,NULL); +INSERT INTO t2 VALUES (91,2,default); +DELETE FROM t1 WHERE id=1; +connection slave; +# +# Verify data consistency on slave +# +include/diff_tables.inc [master:test.t1, slave:test.t1] +include/diff_tables.inc [master:test.t2, slave:test.t2] +connection master; +DROP TABLE t2,t1; +connection slave; +# +# Test case: Verify cascading on delete with multiple levels of child +# tables works fine. +# Parent table: users +# Child tables: matchmaking_groups, matchmaking_group_users +# Parent table: matchmaking_groups +# Child tables: matchmaking_group_users, matchmaking_group_maps +# +# Verify ON DELETE CASCASE works fine when a row is deleted +# from parent table users +# matchmaking_groups->matchmaking_group_users->matchmaking_group_maps +# users->matchmaking_group_users->matchmaking_group_maps +# +connection master; +CREATE TABLE users (id INT UNSIGNED AUTO_INCREMENT PRIMARY KEY, +name VARCHAR(32) NOT NULL DEFAULT '' +) ENGINE=InnoDB DEFAULT CHARSET=utf8; +CREATE TABLE matchmaking_groups ( +id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY, +host_user_id INT UNSIGNED NOT NULL UNIQUE, +v_col INT AS (host_user_id+1) VIRTUAL, KEY (v_col), +CONSTRAINT FOREIGN KEY (host_user_id) REFERENCES users (id) +ON DELETE CASCADE ON UPDATE CASCADE +) ENGINE=InnoDB DEFAULT CHARSET=utf8; +CREATE TABLE matchmaking_group_users ( +matchmaking_group_id BIGINT UNSIGNED NOT NULL, +user_id INT UNSIGNED NOT NULL, +v_col1 int as (user_id+1) virtual, KEY (v_col1), +PRIMARY KEY (matchmaking_group_id,user_id), +UNIQUE KEY user_id (user_id), +CONSTRAINT FOREIGN KEY (matchmaking_group_id) +REFERENCES matchmaking_groups (id) ON DELETE CASCADE ON UPDATE CASCADE, +CONSTRAINT FOREIGN KEY (user_id) +REFERENCES users (id) ON DELETE CASCADE ON UPDATE CASCADE +) ENGINE=InnoDB DEFAULT CHARSET=utf8; +CREATE TABLE matchmaking_group_maps ( +matchmaking_group_id BIGINT UNSIGNED NOT NULL, +map_id TINYINT UNSIGNED NOT NULL, +v_col2 INT AS (map_id+1) VIRTUAL, KEY (v_col2), +PRIMARY KEY (matchmaking_group_id,map_id), +CONSTRAINT FOREIGN KEY (matchmaking_group_id) +REFERENCES matchmaking_groups (id) ON DELETE CASCADE ON UPDATE CASCADE +) ENGINE=InnoDB DEFAULT CHARSET=utf8; +connection slave; +connection master; +INSERT INTO users VALUES (NULL,'foo'),(NULL,'bar'); +INSERT INTO matchmaking_groups VALUES (10,1,default),(11,2,default); +INSERT INTO matchmaking_group_users VALUES (10,1,default),(11,2,default); +INSERT INTO matchmaking_group_maps VALUES (10,55,default),(11,66,default); +DELETE FROM matchmaking_groups WHERE id = 10; +connection slave; +# +# No rows should be returned as ON DELETE CASCASE should have removed +# corresponding rows from child tables. There should not any mismatch +# of 'id' field between parent->child. +# +SELECT * FROM matchmaking_group_users WHERE matchmaking_group_id NOT IN (SELECT id FROM matchmaking_groups); +matchmaking_group_id user_id v_col1 +SELECT * FROM matchmaking_group_maps WHERE matchmaking_group_id NOT IN (SELECT id FROM matchmaking_groups); +matchmaking_group_id map_id v_col2 +# +# Rows with id=11 should be present +# +SELECT * FROM matchmaking_group_users; +matchmaking_group_id user_id v_col1 +11 2 3 +SELECT * FROM matchmaking_group_maps; +matchmaking_group_id map_id v_col2 +11 66 67 +connection master; +DELETE FROM users WHERE id = 2; +connection slave; +# +# No rows should be present in both the child tables +# +SELECT * FROM matchmaking_group_users; +matchmaking_group_id user_id v_col1 +SELECT * FROM matchmaking_group_maps; +matchmaking_group_id map_id v_col2 +connection master; +DROP TABLE matchmaking_group_maps, matchmaking_group_users, matchmaking_groups, users; +connection slave; +# +# Test case: ON UPDATE CASCADE scenario +# +connection master; +CREATE TABLE t1 (a INT NOT NULL PRIMARY KEY, b INT NOT NULL) ENGINE=InnoDB; +INSERT INTO t1 VALUES (1, 80); +CREATE TABLE t2 (a INT KEY, b INT, +v_col int as (b+1) virtual, KEY (v_col), +CONSTRAINT b FOREIGN KEY (b) REFERENCES t1(a) ON UPDATE CASCADE +) ENGINE=InnoDB; +INSERT INTO t2 VALUES (51, 1, default); +connection slave; +connection master; +UPDATE t1 SET a = 50 WHERE a = 1; +# +# Master: Verify that ON UPDATE CASCADE works fine +# old_row: (51, 1, 2) ON UPDATE New_row: (51, 50, 51) +# +SELECT * FROM t2 WHERE b=50; +a b v_col +51 50 51 +connection slave; +# +# Slave: Verify that ON UPDATE CASCADE works fine +# old_row: (51, 1, 2) ON UPDATE New_row: (51, 50, 51) +# +SELECT * FROM t2 WHERE b=50; +a b v_col +51 50 51 +connection master; +DROP TABLE t2, t1; +connection slave; +# +# Test case: When triggers are defined on master they should be +# replicated as part of row events and they should be +# applied on slave with the default +# slave_run_tiggers_for_rbr=NO +# +connection master; +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +CREATE TABLE t2 (count INT NOT NULL) ENGINE=InnoDB; +CREATE TRIGGER trg AFTER INSERT ON t1 FOR EACH ROW INSERT INTO t2 VALUES (1); +INSERT INTO t1 VALUES (2),(3); +connection slave; +SHOW GLOBAL VARIABLES LIKE 'slave_run_triggers_for_rbr'; +Variable_name Value +slave_run_triggers_for_rbr NO +# +# As two rows are inserted in table 't1', two rows should get inserted +# into table 't2' as part of trigger. +# +include/assert.inc [Table t2 should have two rows.] +connection master; +DROP TABLE t1,t2; +connection slave; +# +# Test case: On master create triggers and tables with foreign key +# relation. Upon replication to slave verify their results +# are fine. Master and Slave should be in sync. +# +connection master; +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +CREATE TABLE t2 (t1_id INT NOT NULL, +v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), KEY (t1_id), +CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +CREATE TABLE t3 (count INT NOT NULL) ENGINE=InnoDB; +CREATE TRIGGER trg AFTER INSERT ON t1 FOR EACH ROW INSERT INTO t3 VALUES (1); +INSERT INTO t1 VALUES (2),(3); +INSERT INTO t2 VALUES (2, default), (3, default); +connection slave; +# +# As two rows are inserted in table 't1', two rows should get inserted +# into table 't3' as part of trigger. +# +include/assert.inc [Table t3 should have two rows.] +# +# Verify ON DELETE CASCASE correctness +# +connection master; +DELETE FROM t1 WHERE id=2; +connection slave; +connection master; +include/diff_tables.inc [master:test.t1, slave:test.t1] +include/diff_tables.inc [master:test.t2, slave:test.t2] +include/diff_tables.inc [master:test.t3, slave:test.t3] +DROP TABLE t3,t2,t1; +connection slave; +# +# Test case: Triggers are present only on slave and +# 'slave_run_triggers_for_rbr=NO' +# +connection slave; +SET @save_slave_run_triggers_for_rbr= @@GLOBAL.slave_run_triggers_for_rbr; +SET GLOBAL slave_run_triggers_for_rbr= NO;; +SHOW GLOBAL VARIABLES LIKE '%slave_run_triggers_for_rbr%'; +Variable_name Value +slave_run_triggers_for_rbr NO +connection master; +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +CREATE TABLE t2 (t1_id INT NOT NULL, +v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), +KEY (t1_id), CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +CREATE TABLE t3 (count INT NOT NULL) ENGINE=InnoDB; +connection slave; +CREATE TRIGGER trg AFTER INSERT ON t2 FOR EACH ROW INSERT INTO t3 VALUES (1); +connection master; +INSERT INTO t1 VALUES (2),(3); +INSERT INTO t2 VALUES (2, default), (3, default); +connection slave; +# +# Count must be 0 +# +include/assert.inc [Table t3 should have zero rows.] +connection master; +DELETE FROM t1 WHERE id=2; +connection slave; +SET GLOBAL slave_run_triggers_for_rbr= @save_slave_run_triggers_for_rbr; +# +# Verify t1,t2 are consistent on slave. +# +include/diff_tables.inc [master:test.t1, slave:test.t1] +include/diff_tables.inc [master:test.t2, slave:test.t2] +connection master; +DROP TABLE t3,t2,t1; +connection slave; +# +# Test case: Triggers are present only on slave and +# 'slave_run_triggers_for_rbr=YES' +# +connection slave; +SET @save_slave_run_triggers_for_rbr= @@GLOBAL.slave_run_triggers_for_rbr; +SET GLOBAL slave_run_triggers_for_rbr= YES;; +SHOW GLOBAL VARIABLES LIKE '%slave_run_triggers_for_rbr%'; +Variable_name Value +slave_run_triggers_for_rbr YES +connection master; +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +CREATE TABLE t2 (t1_id INT NOT NULL, +v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), +KEY (t1_id), CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +CREATE TABLE t3 (count INT NOT NULL) ENGINE=InnoDB; +connection slave; +CREATE TRIGGER trg AFTER INSERT ON t2 FOR EACH ROW INSERT INTO t3 VALUES (1); +connection master; +INSERT INTO t1 VALUES (2),(3); +INSERT INTO t2 VALUES (2, default), (3, default); +connection slave; +# +# Count must be 2 +# +include/assert.inc [Table t3 should have two rows.] +connection master; +DELETE FROM t1 WHERE id=2; +connection slave; +SET GLOBAL slave_run_triggers_for_rbr= @save_slave_run_triggers_for_rbr; +# +# Verify t1,t2 are consistent on slave. +# +include/diff_tables.inc [master:test.t1, slave:test.t1] +include/diff_tables.inc [master:test.t2, slave:test.t2] +connection master; +DROP TABLE t3,t2,t1; +connection slave; +# +# Test case: Triggers and Foreign Keys are present only on slave and +# 'slave_run_triggers_for_rbr=NO' +# +connection slave; +SET @save_slave_run_triggers_for_rbr= @@GLOBAL.slave_run_triggers_for_rbr; +SET GLOBAL slave_run_triggers_for_rbr= NO;; +SHOW GLOBAL VARIABLES LIKE '%slave_run_triggers_for_rbr%'; +Variable_name Value +slave_run_triggers_for_rbr NO +connection master; +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +SET sql_log_bin=0; +CREATE TABLE t2 (t1_id INT NOT NULL,v_col INT AS (t1_id+1) VIRTUAL) ENGINE=INNODB; +SET sql_log_bin=1; +CREATE TABLE t3 (count INT NOT NULL) ENGINE=InnoDB; +connection slave; +CREATE TABLE t2 (t1_id INT NOT NULL, +v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), KEY (t1_id), +CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +CREATE TRIGGER trg AFTER INSERT ON t2 FOR EACH ROW INSERT INTO t3 VALUES (1); +connection master; +INSERT INTO t1 VALUES (2),(3); +INSERT INTO t2 VALUES (2, default), (3, default); +connection slave; +# +# Count must be 0 +# +include/assert.inc [Table t3 should have zero rows.] +connection master; +DELETE FROM t1 WHERE id=2; +# t1: Should have one row +SELECT * FROM t1; +id +3 +# t2: Should have two rows +SELECT * FROM t2; +t1_id v_col +2 3 +3 4 +connection slave; +# t1: Should have one row +SELECT * FROM t1; +id +3 +# t2: Should have one row on slave due to ON DELETE CASCASE +SELECT * FROM t2; +t1_id v_col +3 4 +SET GLOBAL slave_run_triggers_for_rbr= @save_slave_run_triggers_for_rbr; +connection master; +DROP TABLE t3,t2,t1; +connection slave; +# +# Test case: Triggers are Foreign Keys are present only on slave and +# 'slave_run_triggers_for_rbr=YES' +# +connection slave; +SET @save_slave_run_triggers_for_rbr= @@GLOBAL.slave_run_triggers_for_rbr; +SET GLOBAL slave_run_triggers_for_rbr= YES;; +SHOW GLOBAL VARIABLES LIKE '%slave_run_triggers_for_rbr%'; +Variable_name Value +slave_run_triggers_for_rbr YES +connection master; +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +SET sql_log_bin=0; +CREATE TABLE t2 (t1_id INT NOT NULL,v_col INT AS (t1_id+1) VIRTUAL) ENGINE=INNODB; +SET sql_log_bin=1; +CREATE TABLE t3 (count INT NOT NULL) ENGINE=InnoDB; +connection slave; +CREATE TABLE t2 (t1_id INT NOT NULL, +v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), KEY (t1_id), +CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +CREATE TRIGGER trg AFTER INSERT ON t2 FOR EACH ROW INSERT INTO t3 VALUES (1); +connection master; +INSERT INTO t1 VALUES (2),(3); +INSERT INTO t2 VALUES (2, default), (3, default); +connection slave; +# +# Count must be 2 +# +include/assert.inc [Table t3 should have two rows.] +connection master; +DELETE FROM t1 WHERE id=2; +# t1: Should have one row +SELECT * FROM t1; +id +3 +# t2: Should have two rows +SELECT * FROM t2; +t1_id v_col +2 3 +3 4 +connection slave; +# t1: Should have one row +SELECT * FROM t1; +id +3 +# t2: Should have one row on slave due to ON DELETE CASCASE +SELECT * FROM t2; +t1_id v_col +3 4 +SET GLOBAL slave_run_triggers_for_rbr= @save_slave_run_triggers_for_rbr; +connection master; +DROP TABLE t3,t2,t1; +connection slave; +include/rpl_end.inc diff --git a/mysql-test/suite/rpl/t/rpl_row_vcol_crash.test b/mysql-test/suite/rpl/t/rpl_row_vcol_crash.test new file mode 100644 index 00000000000..2ed7c219aeb --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_row_vcol_crash.test @@ -0,0 +1,410 @@ +# ==== Purpose ==== +# +# Test verifies that, slave doesn't report any assert on UPDATE or DELETE of +# row which tries to update the virtual columns with associated KEYs. +# +# ==== References ==== +# +# MDEV-23033: All slaves crash once in ~24 hours and loop restart with signal 11 +# + +--source include/have_binlog_format_row.inc +--source include/have_innodb.inc +--source include/master-slave.inc + + +--echo # +--echo # Test case: KEY on a virtual column with ON DELETE CASCADE +--echo # +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +INSERT INTO t1 VALUES (1),(2),(3); + +CREATE TABLE t2 (id INT NOT NULL PRIMARY KEY, + t1_id INT NOT NULL, + v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), KEY (t1_id), + CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; + +INSERT INTO t2 VALUES (90,1,NULL); +INSERT INTO t2 VALUES (91,2,default); + +# Following query results in an assert on slave +DELETE FROM t1 WHERE id=1; +--sync_slave_with_master + +--echo # +--echo # Verify data consistency on slave +--echo # +--let $diff_tables= master:test.t1, slave:test.t1 +--source include/diff_tables.inc +--let $diff_tables= master:test.t2, slave:test.t2 +--source include/diff_tables.inc + +--connection master +DROP TABLE t2,t1; +--sync_slave_with_master + +--echo # +--echo # Test case: Verify cascading on delete with multiple levels of child +--echo # tables works fine. +--echo # Parent table: users +--echo # Child tables: matchmaking_groups, matchmaking_group_users +--echo # Parent table: matchmaking_groups +--echo # Child tables: matchmaking_group_users, matchmaking_group_maps +--echo # +--echo # Verify ON DELETE CASCASE works fine when a row is deleted +--echo # from parent table users +--echo # matchmaking_groups->matchmaking_group_users->matchmaking_group_maps +--echo # users->matchmaking_group_users->matchmaking_group_maps +--echo # + +--connection master +CREATE TABLE users (id INT UNSIGNED AUTO_INCREMENT PRIMARY KEY, + name VARCHAR(32) NOT NULL DEFAULT '' +) ENGINE=InnoDB DEFAULT CHARSET=utf8; + +CREATE TABLE matchmaking_groups ( + id BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY, + host_user_id INT UNSIGNED NOT NULL UNIQUE, + v_col INT AS (host_user_id+1) VIRTUAL, KEY (v_col), + CONSTRAINT FOREIGN KEY (host_user_id) REFERENCES users (id) + ON DELETE CASCADE ON UPDATE CASCADE +) ENGINE=InnoDB DEFAULT CHARSET=utf8; + +CREATE TABLE matchmaking_group_users ( + matchmaking_group_id BIGINT UNSIGNED NOT NULL, + user_id INT UNSIGNED NOT NULL, + v_col1 int as (user_id+1) virtual, KEY (v_col1), + PRIMARY KEY (matchmaking_group_id,user_id), + UNIQUE KEY user_id (user_id), + CONSTRAINT FOREIGN KEY (matchmaking_group_id) + REFERENCES matchmaking_groups (id) ON DELETE CASCADE ON UPDATE CASCADE, + CONSTRAINT FOREIGN KEY (user_id) + REFERENCES users (id) ON DELETE CASCADE ON UPDATE CASCADE +) ENGINE=InnoDB DEFAULT CHARSET=utf8; + +CREATE TABLE matchmaking_group_maps ( + matchmaking_group_id BIGINT UNSIGNED NOT NULL, + map_id TINYINT UNSIGNED NOT NULL, + v_col2 INT AS (map_id+1) VIRTUAL, KEY (v_col2), + PRIMARY KEY (matchmaking_group_id,map_id), + CONSTRAINT FOREIGN KEY (matchmaking_group_id) + REFERENCES matchmaking_groups (id) ON DELETE CASCADE ON UPDATE CASCADE +) ENGINE=InnoDB DEFAULT CHARSET=utf8; +--sync_slave_with_master + +--connection master +INSERT INTO users VALUES (NULL,'foo'),(NULL,'bar'); +INSERT INTO matchmaking_groups VALUES (10,1,default),(11,2,default); +INSERT INTO matchmaking_group_users VALUES (10,1,default),(11,2,default); +INSERT INTO matchmaking_group_maps VALUES (10,55,default),(11,66,default); + +DELETE FROM matchmaking_groups WHERE id = 10; +--sync_slave_with_master + +--echo # +--echo # No rows should be returned as ON DELETE CASCASE should have removed +--echo # corresponding rows from child tables. There should not any mismatch +--echo # of 'id' field between parent->child. +--echo # +SELECT * FROM matchmaking_group_users WHERE matchmaking_group_id NOT IN (SELECT id FROM matchmaking_groups); +SELECT * FROM matchmaking_group_maps WHERE matchmaking_group_id NOT IN (SELECT id FROM matchmaking_groups); + +--echo # +--echo # Rows with id=11 should be present +--echo # +SELECT * FROM matchmaking_group_users; +SELECT * FROM matchmaking_group_maps; + +--connection master +DELETE FROM users WHERE id = 2; +--sync_slave_with_master + +--echo # +--echo # No rows should be present in both the child tables +--echo # +SELECT * FROM matchmaking_group_users; +SELECT * FROM matchmaking_group_maps; + +--connection master +DROP TABLE matchmaking_group_maps, matchmaking_group_users, matchmaking_groups, users; +--sync_slave_with_master + +--echo # +--echo # Test case: ON UPDATE CASCADE scenario +--echo # + +--connection master +CREATE TABLE t1 (a INT NOT NULL PRIMARY KEY, b INT NOT NULL) ENGINE=InnoDB; +INSERT INTO t1 VALUES (1, 80); + +CREATE TABLE t2 (a INT KEY, b INT, + v_col int as (b+1) virtual, KEY (v_col), + CONSTRAINT b FOREIGN KEY (b) REFERENCES t1(a) ON UPDATE CASCADE +) ENGINE=InnoDB; +INSERT INTO t2 VALUES (51, 1, default); +--sync_slave_with_master + +--connection master +UPDATE t1 SET a = 50 WHERE a = 1; + +--echo # +--echo # Master: Verify that ON UPDATE CASCADE works fine +--echo # old_row: (51, 1, 2) ON UPDATE New_row: (51, 50, 51) +--echo # +SELECT * FROM t2 WHERE b=50; +--sync_slave_with_master + +--echo # +--echo # Slave: Verify that ON UPDATE CASCADE works fine +--echo # old_row: (51, 1, 2) ON UPDATE New_row: (51, 50, 51) +--echo # +SELECT * FROM t2 WHERE b=50; + +--connection master +DROP TABLE t2, t1; +--sync_slave_with_master + +--echo # +--echo # Test case: When triggers are defined on master they should be +--echo # replicated as part of row events and they should be +--echo # applied on slave with the default +--echo # slave_run_tiggers_for_rbr=NO +--echo # + +# In row-based replication, the binary log contains row changes. It will have +# both the changes made by the statement itself, and the changes made by the +# triggers that were invoked by the statement. Slave server(s) do not need to +# run triggers for row changes they are applying. Hence verify that this +# property remains the same and data should be available as if trigger was +# executed. Please note by default slave_run_tiggers_for_rbr=NO. + +--connection master +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +CREATE TABLE t2 (count INT NOT NULL) ENGINE=InnoDB; +CREATE TRIGGER trg AFTER INSERT ON t1 FOR EACH ROW INSERT INTO t2 VALUES (1); +INSERT INTO t1 VALUES (2),(3); +--sync_slave_with_master + +SHOW GLOBAL VARIABLES LIKE 'slave_run_triggers_for_rbr'; +--echo # +--echo # As two rows are inserted in table 't1', two rows should get inserted +--echo # into table 't2' as part of trigger. +--echo # +--let $assert_cond= COUNT(*) = 2 FROM t2 +--let $assert_text= Table t2 should have two rows. +--source include/assert.inc + +--connection master +DROP TABLE t1,t2; +--sync_slave_with_master + +--echo # +--echo # Test case: On master create triggers and tables with foreign key +--echo # relation. Upon replication to slave verify their results +--echo # are fine. Master and Slave should be in sync. +--echo # +--connection master +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +CREATE TABLE t2 (t1_id INT NOT NULL, + v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), KEY (t1_id), + CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +CREATE TABLE t3 (count INT NOT NULL) ENGINE=InnoDB; +CREATE TRIGGER trg AFTER INSERT ON t1 FOR EACH ROW INSERT INTO t3 VALUES (1); + +INSERT INTO t1 VALUES (2),(3); +INSERT INTO t2 VALUES (2, default), (3, default); +--sync_slave_with_master + +--echo # +--echo # As two rows are inserted in table 't1', two rows should get inserted +--echo # into table 't3' as part of trigger. +--echo # +--let $assert_cond= COUNT(*) = 2 FROM t3 +--let $assert_text= Table t3 should have two rows. +--source include/assert.inc + +--echo # +--echo # Verify ON DELETE CASCASE correctness +--echo # +--connection master +DELETE FROM t1 WHERE id=2; +--sync_slave_with_master + +--connection master +--let $diff_tables= master:test.t1, slave:test.t1 +--source include/diff_tables.inc +--let $diff_tables= master:test.t2, slave:test.t2 +--source include/diff_tables.inc +--let $diff_tables= master:test.t3, slave:test.t3 +--source include/diff_tables.inc + +DROP TABLE t3,t2,t1; +--sync_slave_with_master + +# +# Test case: Triggers only on slave +# +--write_file $MYSQLTEST_VARDIR/tmp/trig_on_slave.inc PROCEDURE + if ($slave_run_triggers_for_rbr == '') { + --die !!!ERROR IN TEST: you must set $slave_run_triggers_for_rbr + } + +--connection slave +SET @save_slave_run_triggers_for_rbr= @@GLOBAL.slave_run_triggers_for_rbr; +--eval SET GLOBAL slave_run_triggers_for_rbr= $slave_run_triggers_for_rbr; +SHOW GLOBAL VARIABLES LIKE '%slave_run_triggers_for_rbr%'; + +--connection master +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +CREATE TABLE t2 (t1_id INT NOT NULL, + v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), + KEY (t1_id), CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +CREATE TABLE t3 (count INT NOT NULL) ENGINE=InnoDB; +--sync_slave_with_master + +CREATE TRIGGER trg AFTER INSERT ON t2 FOR EACH ROW INSERT INTO t3 VALUES (1); + +--connection master +INSERT INTO t1 VALUES (2),(3); +INSERT INTO t2 VALUES (2, default), (3, default); +--sync_slave_with_master + +if ($slave_run_triggers_for_rbr == 'NO') { +--echo # +--echo # Count must be 0 +--echo # +--let $assert_cond= COUNT(*) = 0 FROM t3 +--let $assert_text= Table t3 should have zero rows. +--source include/assert.inc +} +if ($slave_run_triggers_for_rbr == 'YES') { +--echo # +--echo # Count must be 2 +--echo # +--let $assert_cond= COUNT(*) = 2 FROM t3 +--let $assert_text= Table t3 should have two rows. +--source include/assert.inc +} + +--connection master +DELETE FROM t1 WHERE id=2; +--sync_slave_with_master +SET GLOBAL slave_run_triggers_for_rbr= @save_slave_run_triggers_for_rbr; + +--echo # +--echo # Verify t1,t2 are consistent on slave. +--echo # +--let $diff_tables= master:test.t1, slave:test.t1 +--source include/diff_tables.inc +--let $diff_tables= master:test.t2, slave:test.t2 +--source include/diff_tables.inc + +--connection master +DROP TABLE t3,t2,t1; +--sync_slave_with_master +#END OF +PROCEDURE + +--echo # +--echo # Test case: Triggers are present only on slave and +--echo # 'slave_run_triggers_for_rbr=NO' +--echo # +--let $slave_run_triggers_for_rbr=NO +--source $MYSQLTEST_VARDIR/tmp/trig_on_slave.inc + +--echo # +--echo # Test case: Triggers are present only on slave and +--echo # 'slave_run_triggers_for_rbr=YES' +--echo # +--let $slave_run_triggers_for_rbr=YES +--source $MYSQLTEST_VARDIR/tmp/trig_on_slave.inc +--remove_file $MYSQLTEST_VARDIR/tmp/trig_on_slave.inc + +# +# Test case: Trigger and Foreign Key are present only on slave +# +--write_file $MYSQLTEST_VARDIR/tmp/trig_fk_on_slave.inc PROCEDURE + if ($slave_run_triggers_for_rbr == '') { + --die !!!ERROR IN TEST: you must set $slave_run_triggers_for_rbr + } + +--connection slave +SET @save_slave_run_triggers_for_rbr= @@GLOBAL.slave_run_triggers_for_rbr; +--eval SET GLOBAL slave_run_triggers_for_rbr= $slave_run_triggers_for_rbr; +SHOW GLOBAL VARIABLES LIKE '%slave_run_triggers_for_rbr%'; + +--connection master +CREATE TABLE t1 (id INT NOT NULL PRIMARY KEY) ENGINE=InnoDB; +SET sql_log_bin=0; +CREATE TABLE t2 (t1_id INT NOT NULL,v_col INT AS (t1_id+1) VIRTUAL) ENGINE=INNODB; +SET sql_log_bin=1; +CREATE TABLE t3 (count INT NOT NULL) ENGINE=InnoDB; +--sync_slave_with_master + +# Have foreign key and trigger on slave. +CREATE TABLE t2 (t1_id INT NOT NULL, + v_col INT AS (t1_id+1) VIRTUAL, KEY (v_col), KEY (t1_id), + CONSTRAINT a FOREIGN KEY (t1_id) REFERENCES t1 (id) ON DELETE CASCADE +) ENGINE=InnoDB; +CREATE TRIGGER trg AFTER INSERT ON t2 FOR EACH ROW INSERT INTO t3 VALUES (1); + +--connection master +INSERT INTO t1 VALUES (2),(3); +INSERT INTO t2 VALUES (2, default), (3, default); +--sync_slave_with_master + +if ($slave_run_triggers_for_rbr == 'NO') { +--echo # +--echo # Count must be 0 +--echo # +--let $assert_cond= COUNT(*) = 0 FROM t3 +--let $assert_text= Table t3 should have zero rows. +--source include/assert.inc +} +if ($slave_run_triggers_for_rbr == 'YES') { +--echo # +--echo # Count must be 2 +--echo # +--let $assert_cond= COUNT(*) = 2 FROM t3 +--let $assert_text= Table t3 should have two rows. +--source include/assert.inc +} + +--connection master +DELETE FROM t1 WHERE id=2; +--echo # t1: Should have one row +SELECT * FROM t1; +--echo # t2: Should have two rows +SELECT * FROM t2; +--sync_slave_with_master +--echo # t1: Should have one row +SELECT * FROM t1; +--echo # t2: Should have one row on slave due to ON DELETE CASCASE +SELECT * FROM t2; +SET GLOBAL slave_run_triggers_for_rbr= @save_slave_run_triggers_for_rbr; + +--connection master +DROP TABLE t3,t2,t1; +--sync_slave_with_master +#END OF +PROCEDURE + +--echo # +--echo # Test case: Triggers and Foreign Keys are present only on slave and +--echo # 'slave_run_triggers_for_rbr=NO' +--echo # +--let $slave_run_triggers_for_rbr=NO +--source $MYSQLTEST_VARDIR/tmp/trig_fk_on_slave.inc + +--echo # +--echo # Test case: Triggers are Foreign Keys are present only on slave and +--echo # 'slave_run_triggers_for_rbr=YES' +--echo # +--let $slave_run_triggers_for_rbr=YES +--source $MYSQLTEST_VARDIR/tmp/trig_fk_on_slave.inc +--remove_file $MYSQLTEST_VARDIR/tmp/trig_fk_on_slave.inc + +--source include/rpl_end.inc diff --git a/sql/log_event.cc b/sql/log_event.cc index c649e1f64fa..10aa0afbad8 100644 --- a/sql/log_event.cc +++ b/sql/log_event.cc @@ -10718,7 +10718,7 @@ int Rows_log_event::do_add_row_data(uchar *row_data, size_t length) There was the same problem with MERGE MYISAM tables and so here we try to go the same way. */ -static void restore_empty_query_table_list(LEX *lex) +inline void restore_empty_query_table_list(LEX *lex) { if (lex->first_not_own_table()) (*lex->first_not_own_table()->prev_global)= NULL; @@ -10733,6 +10733,8 @@ int Rows_log_event::do_apply_event(rpl_group_info *rgi) TABLE* table; DBUG_ENTER("Rows_log_event::do_apply_event(Relay_log_info*)"); int error= 0; + LEX *lex= thd->lex; + uint8 new_trg_event_map= get_trg_event_map(); /* If m_table_id == ~0ULL, then we have a dummy event that does not contain any data. In that case, we just remove all tables in the @@ -10823,27 +10825,25 @@ int Rows_log_event::do_apply_event(rpl_group_info *rgi) DBUG_ASSERT(!debug_sync_set_action(thd, STRING_WITH_LEN(action))); };); - if (slave_run_triggers_for_rbr) - { - LEX *lex= thd->lex; - uint8 new_trg_event_map= get_trg_event_map(); - - /* - Trigger's procedures work with global table list. So we have to add - rgi->tables_to_lock content there to get trigger's in the list. + /* + Trigger's procedures work with global table list. So we have to add + rgi->tables_to_lock content there to get trigger's in the list. - Then restore_empty_query_table_list() restore the list as it was - */ - DBUG_ASSERT(lex->query_tables == NULL); - if ((lex->query_tables= rgi->tables_to_lock)) - rgi->tables_to_lock->prev_global= &lex->query_tables; + Then restore_empty_query_table_list() restore the list as it was + */ + DBUG_ASSERT(lex->query_tables == NULL); + if ((lex->query_tables= rgi->tables_to_lock)) + rgi->tables_to_lock->prev_global= &lex->query_tables; - for (TABLE_LIST *tables= rgi->tables_to_lock; tables; - tables= tables->next_global) - { + for (TABLE_LIST *tables= rgi->tables_to_lock; tables; + tables= tables->next_global) + { + if (slave_run_triggers_for_rbr) tables->trg_event_map= new_trg_event_map; - lex->query_tables_last= &tables->next_global; - } + else + tables->slave_fk_event_map= new_trg_event_map; + + lex->query_tables_last= &tables->next_global; } if (open_and_lock_tables(thd, rgi->tables_to_lock, FALSE, 0)) { @@ -11193,8 +11193,7 @@ int Rows_log_event::do_apply_event(rpl_group_info *rgi) } /* remove trigger's tables */ - if (slave_run_triggers_for_rbr) - restore_empty_query_table_list(thd->lex); + restore_empty_query_table_list(thd->lex); #if defined(WITH_WSREP) && defined(HAVE_QUERY_CACHE) if (WSREP(thd) && thd->wsrep_exec_mode == REPL_RECV) @@ -11212,8 +11211,7 @@ int Rows_log_event::do_apply_event(rpl_group_info *rgi) DBUG_RETURN(error); err: - if (slave_run_triggers_for_rbr) - restore_empty_query_table_list(thd->lex); + restore_empty_query_table_list(thd->lex); rgi->slave_close_thread_tables(thd); DBUG_RETURN(error); } diff --git a/sql/sql_base.cc b/sql/sql_base.cc index 8e57ea437b6..a7cddbca16a 100644 --- a/sql/sql_base.cc +++ b/sql/sql_base.cc @@ -4371,7 +4371,8 @@ handle_table(THD *thd, Query_tables_list *prelocking_ctx, /* We rely on a caller to check that table is going to be changed. */ DBUG_ASSERT(table_list->lock_type >= TL_WRITE_ALLOW_WRITE); - if (table_list->trg_event_map) + if (table_list->trg_event_map || + (thd->rgi_slave && thd->is_current_stmt_binlog_format_row())) { if (table_list->table->triggers) { @@ -4405,7 +4406,8 @@ handle_table(THD *thd, Query_tables_list *prelocking_ctx, { // FK_OPTION_RESTRICT and FK_OPTION_NO_ACTION only need read access static bool can_write[]= { true, false, true, true, false, true }; - uint8 op= table_list->trg_event_map; + uint8 op= ((table_list->trg_event_map) ? table_list->trg_event_map : + table_list->slave_fk_event_map); thr_lock_type lock_type; if ((op & (1 << TRG_EVENT_DELETE) && can_write[fk->delete_method]) diff --git a/sql/table.h b/sql/table.h index 9a864f7ce9f..57706655d9b 100644 --- a/sql/table.h +++ b/sql/table.h @@ -2277,8 +2277,12 @@ struct TABLE_LIST Indicates what triggers we need to pre-load for this TABLE_LIST when opening an associated TABLE. This is filled after the parsed tree is created. + + slave_fk_event_map is filled on the slave side with bitmaps value + representing row-based event operation to help find and prelock + possible FK constrain-related child tables. */ - uint8 trg_event_map; + uint8 trg_event_map, slave_fk_event_map; /* TRUE <=> this table is a const one and was optimized away. */ bool optimized_away;

1 0

Re: [Maria-developers] [Commits] cdc305c8dd8: MDEV-19620: Changing join_buffer_size causes different results
by Sergey Petrunia 29 Dec '20

29 Dec '20

Hi Varun, Please find input below. On Mon, Dec 28, 2020 at 02:34:40PM +0530, varun wrote: > revision-id: cdc305c8dd89a726e09e5fe70ff890d06609cbfb (mariadb-10.3.21-309-gcdc305c8dd8) > parent(s): 043bd85a574a88856ab9c6d497e682ed06fe45e9 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2020-12-28 14:12:14 +0530 > message: > > MDEV-19620: Changing join_buffer_size causes different results > > The scenario here is that query refinement phase decides to use a hash join. > When the join buffers are allocated in the JOIN::init_join_caches, for a table > the size exceeds the value for join_buffer_space_limit (which is the limit of the > space available for all join buffers). When this happens then we disallow join > buffering for the table, this is done in the revise_cache_usage and set_join_cache_denial. > > In this issue the hash key is created on an index for which ref access is possible, so > when we disallow hash join then instead of switching to REF access we switch to a table > scan. This is a problem because the equijoin conditions for which a lookup can be made > are not attached to the table(or are not evaluated for the table). This leads to incorrect > results. > > The fix here would be to switch to using a lookup because it was picked by the join planner > to be more efficient than the table scan. > > --- > mysql-test/main/join_cache.result | 138 ++++++++++++++++++++++++++++++++++++++ > mysql-test/main/join_cache.test | 105 +++++++++++++++++++++++++++++ > sql/sql_select.cc | 69 +++++++++++++++---- > sql/sql_select.h | 6 ++ > 4 files changed, 304 insertions(+), 14 deletions(-) > > diff --git a/mysql-test/main/join_cache.result b/mysql-test/main/join_cache.result > index 3d1d91df997..e58503f422f 100644 > --- a/mysql-test/main/join_cache.result > +++ b/mysql-test/main/join_cache.result > @@ -6128,4 +6128,142 @@ EXPLAIN > } > } > drop table t1,t2,t3; > +# > +# MDEV-19620: Changing join_buffer_size causes different results > +# > +SET @save_join_cache_level= @@join_cache_level; > +SET @save_join_buffer_size= @@join_buffer_size; > +SET @save_join_buffer_space_limit= @@join_buffer_space_limit; > +SET join_cache_level = 3; > +CREATE TABLE t1 (i2 VARCHAR(500), c2 VARCHAR(500)) ENGINE=MyISAM; > +INSERT INTO t1 VALUES ('1','v'),('7','s'); > +CREATE TABLE t2 (pk3 INT PRIMARY KEY, i3 VARCHAR(300), c3 VARCHAR(500)) ENGINE=MyISAM; > +INSERT INTO t2 VALUES (1,'7','g'),(2,'4','p'),(3,'1','q'); > +INSERT INTO t2 VALUES (4,'7','g'),(5,'4','p'),(6,'1','q'); > +INSERT INTO t2 VALUES (16,'7','g'),(17,'4','p'),(28,'1','q'); > +# > +# Hash join + table Scan on t2 > +# > +set join_buffer_size=1024; > +set join_buffer_space_limit=2048; > +EXPLAIN SELECT * FROM t1 LEFT JOIN t2 ON (c3=c2); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 ALL NULL NULL NULL NULL 2 > +1 SIMPLE t2 ALL NULL NULL NULL NULL 9 Using where > +SELECT * FROM t1 LEFT JOIN t2 ON (c3=c2); > +i2 c2 pk3 i3 c3 > +1 v NULL NULL NULL > +7 s NULL NULL NULL > +set join_buffer_space_limit=262144; > +EXPLAIN SELECT * FROM t1 LEFT JOIN t2 ON (c3=c2); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 ALL NULL NULL NULL NULL 2 > +1 SIMPLE t2 hash_ALL NULL #hash#$hj 503 test.t1.c2 9 Using where; Using join buffer (flat, BNLH join) > +SELECT * FROM t1 LEFT JOIN t2 ON (c3=c2); > +i2 c2 pk3 i3 c3 > +1 v NULL NULL NULL > +7 s NULL NULL NULL > +# > +# HASH join + ref access on t2 > +# > +ALTER TABLE t2 ADD KEY k1(c3); > +set join_buffer_size=1024; > +set join_buffer_space_limit=2048; > +EXPLAIN SELECT * FROM t1 LEFT JOIN t2 ON (c3=c2); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 ALL NULL NULL NULL NULL 2 > +1 SIMPLE t2 ref k1 k1 503 test.t1.c2 2 Using where > +SELECT * FROM t1 LEFT JOIN t2 ON (c3=c2); > +i2 c2 pk3 i3 c3 > +1 v NULL NULL NULL > +7 s NULL NULL NULL > +set join_buffer_space_limit=262144; > +EXPLAIN SELECT * FROM t1 LEFT JOIN t2 ON (c3=c2); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 ALL NULL NULL NULL NULL 2 > +1 SIMPLE t2 hash_ALL k1 #hash#k1 503 test.t1.c2 9 Using where; Using join buffer (flat, BNLH join) > +SELECT * FROM t1 LEFT JOIN t2 ON (c3=c2); > +i2 c2 pk3 i3 c3 > +1 v NULL NULL NULL > +7 s NULL NULL NULL > +# > +# Hash join + index scan on t2 > +# > +ALTER TABLE t2 DROP KEY k1; > +ALTER TABLE t2 ADD KEY k1(i3,c3); > +set join_buffer_size=1024; > +set join_buffer_space_limit=2048; > +EXPLAIN SELECT c3,i3, i2,c2 FROM t1 LEFT JOIN t2 ON (c3 = c2); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 ALL NULL NULL NULL NULL 2 > +1 SIMPLE t2 index NULL k1 806 NULL 9 Using where; Using index > +SELECT c3,i3, i2,c2 FROM t1 LEFT JOIN t2 ON (c3 = c2); > +c3 i3 i2 c2 > +NULL NULL 1 v > +NULL NULL 7 s > +set join_buffer_space_limit=262144; > +EXPLAIN SELECT c3,i3, i2,c2 FROM t1 LEFT JOIN t2 ON (c3 = c2); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 ALL NULL NULL NULL NULL 2 > +1 SIMPLE t2 hash_index NULL #hash#$hj:k1 503:806 test.t1.c2 9 Using where; Using index; Using join buffer (flat, BNLH join) > +SELECT c3,i3, i2,c2 FROM t1 LEFT JOIN t2 ON (c3 = c2); > +c3 i3 i2 c2 > +NULL NULL 1 v > +NULL NULL 7 s > +DROP TABLE t1,t2; > +# > +# Hash join + range scan on t2 > +# > +CREATE TABLE t1 (i2 VARCHAR(500), c2 VARCHAR(500)); > +INSERT INTO t1 VALUES ('1','v'),('7','s'); > +CREATE TABLE t2 (pk3 INT PRIMARY KEY, i3 VARCHAR(300), c3 VARCHAR(500), INDEX(i3,c3)); > +INSERT INTO t2 VALUES (1,'7','g'),(2,'4','s'),(3,'1','q'); > +set join_buffer_size=1024; > +set join_buffer_space_limit=2048; > +EXPLAIN SELECT * FROM t1 LEFT JOIN t2 ON (c3 = c2) WHERE i3 >= '4'; > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 ALL NULL NULL NULL NULL 2 Using where > +1 SIMPLE t2 range i3 i3 303 NULL 2 Using index condition; Using where > +SELECT * FROM t1 LEFT JOIN t2 ON (c3 = c2) WHERE i3 >= '4'; > +i2 c2 pk3 i3 c3 > +7 s 2 4 s > +set join_buffer_space_limit=262144; > +EXPLAIN SELECT * FROM t1 LEFT JOIN t2 ON (c3 = c2) WHERE i3 >= '4'; > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 ALL NULL NULL NULL NULL 2 Using where > +1 SIMPLE t2 hash_range i3 #hash#$hj:i3 503:303 test.t1.c2 2 Using where; Using join buffer (flat, BNLH join) > +SELECT * FROM t1 LEFT JOIN t2 ON (c3 = c2) WHERE i3 >= '4'; > +i2 c2 pk3 i3 c3 > +7 s 2 4 s > +DROP TABLE t1,t2; > +# > +# Hash join + eq ref access on t2 > +# > +CREATE TABLE t1 (i2 VARCHAR(500), c2 VARCHAR(500)); > +INSERT INTO t1 VALUES ('1','v'),('7','s'); > +CREATE TABLE t2 (pk3 INT, i3 VARCHAR(300), c3 VARCHAR(500) PRIMARY KEY); > +INSERT INTO t2 VALUES (1,'7','g'),(2,'4','s'),(3,'1','q'); > +set join_buffer_size=1024; > +set join_buffer_space_limit=2048; > +EXPLAIN SELECT c3,i3, i2,c2 FROM t1 LEFT JOIN t2 ON (c3 = c2); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 ALL NULL NULL NULL NULL 2 > +1 SIMPLE t2 eq_ref PRIMARY PRIMARY 502 test.t1.c2 1 Using where > +SELECT c3,i3, i2,c2 FROM t1 LEFT JOIN t2 ON (c3 = c2); > +c3 i3 i2 c2 > +NULL NULL 1 v > +s 4 7 s > +set join_buffer_space_limit=262144; > +EXPLAIN SELECT c3,i3, i2,c2 FROM t1 LEFT JOIN t2 ON (c3 = c2); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 ALL NULL NULL NULL NULL 2 > +1 SIMPLE t2 hash_ALL PRIMARY #hash#PRIMARY 502 test.t1.c2 3 Using where; Using join buffer (flat, BNLH join) > +SELECT c3,i3, i2,c2 FROM t1 LEFT JOIN t2 ON (c3 = c2); > +c3 i3 i2 c2 > +NULL NULL 1 v > +s 4 7 s > +DROP TABLE t1,t2; > +SET @save_join_cache_level= @@join_cache_level; > +SET @save_join_buffer_size= @@join_buffer_size; > +SET @save_join_buffer_space_limit= @@join_buffer_space_limit; > set @@optimizer_switch=@save_optimizer_switch; > diff --git a/mysql-test/main/join_cache.test b/mysql-test/main/join_cache.test > index 91339c2cb21..6670c62516b 100644 > --- a/mysql-test/main/join_cache.test > +++ b/mysql-test/main/join_cache.test > @@ -4054,5 +4054,110 @@ where > > drop table t1,t2,t3; > > +--echo # > +--echo # MDEV-19620: Changing join_buffer_size causes different results > +--echo # > + > +SET @save_join_cache_level= @@join_cache_level; > +SET @save_join_buffer_size= @@join_buffer_size; > +SET @save_join_buffer_space_limit= @@join_buffer_space_limit; > +SET join_cache_level = 3; > + > +CREATE TABLE t1 (i2 VARCHAR(500), c2 VARCHAR(500)) ENGINE=MyISAM; > +INSERT INTO t1 VALUES ('1','v'),('7','s'); > +CREATE TABLE t2 (pk3 INT PRIMARY KEY, i3 VARCHAR(300), c3 VARCHAR(500)) ENGINE=MyISAM; > +INSERT INTO t2 VALUES (1,'7','g'),(2,'4','p'),(3,'1','q'); > +INSERT INTO t2 VALUES (4,'7','g'),(5,'4','p'),(6,'1','q'); > +INSERT INTO t2 VALUES (16,'7','g'),(17,'4','p'),(28,'1','q'); > + > +--echo # > +--echo # Hash join + table Scan on t2 > +--echo # > + > +set join_buffer_size=1024; > +set join_buffer_space_limit=2048; > +EXPLAIN SELECT * FROM t1 LEFT JOIN t2 ON (c3=c2); > +SELECT * FROM t1 LEFT JOIN t2 ON (c3=c2); > + > +set join_buffer_space_limit=262144; > +EXPLAIN SELECT * FROM t1 LEFT JOIN t2 ON (c3=c2); > +SELECT * FROM t1 LEFT JOIN t2 ON (c3=c2); > + > +--echo # > +--echo # HASH join + ref access on t2 > +--echo # > + > +ALTER TABLE t2 ADD KEY k1(c3); > +set join_buffer_size=1024; > +set join_buffer_space_limit=2048; > +EXPLAIN SELECT * FROM t1 LEFT JOIN t2 ON (c3=c2); > +SELECT * FROM t1 LEFT JOIN t2 ON (c3=c2); > + > +set join_buffer_space_limit=262144; > +EXPLAIN SELECT * FROM t1 LEFT JOIN t2 ON (c3=c2); > +SELECT * FROM t1 LEFT JOIN t2 ON (c3=c2); > + > +--echo # > +--echo # Hash join + index scan on t2 > +--echo # > +ALTER TABLE t2 DROP KEY k1; > +ALTER TABLE t2 ADD KEY k1(i3,c3); > + > +set join_buffer_size=1024; > +set join_buffer_space_limit=2048; > + > +EXPLAIN SELECT c3,i3, i2,c2 FROM t1 LEFT JOIN t2 ON (c3 = c2); > +SELECT c3,i3, i2,c2 FROM t1 LEFT JOIN t2 ON (c3 = c2); > + > +set join_buffer_space_limit=262144; > +EXPLAIN SELECT c3,i3, i2,c2 FROM t1 LEFT JOIN t2 ON (c3 = c2); > +SELECT c3,i3, i2,c2 FROM t1 LEFT JOIN t2 ON (c3 = c2); > +DROP TABLE t1,t2; > + > +--echo # > +--echo # Hash join + range scan on t2 > +--echo # > + > +CREATE TABLE t1 (i2 VARCHAR(500), c2 VARCHAR(500)); > +INSERT INTO t1 VALUES ('1','v'),('7','s'); > +CREATE TABLE t2 (pk3 INT PRIMARY KEY, i3 VARCHAR(300), c3 VARCHAR(500), INDEX(i3,c3)); > +INSERT INTO t2 VALUES (1,'7','g'),(2,'4','s'),(3,'1','q'); > + > +set join_buffer_size=1024; > +set join_buffer_space_limit=2048; > + > +EXPLAIN SELECT * FROM t1 LEFT JOIN t2 ON (c3 = c2) WHERE i3 >= '4'; > +SELECT * FROM t1 LEFT JOIN t2 ON (c3 = c2) WHERE i3 >= '4'; > + > +set join_buffer_space_limit=262144; > +EXPLAIN SELECT * FROM t1 LEFT JOIN t2 ON (c3 = c2) WHERE i3 >= '4'; > +SELECT * FROM t1 LEFT JOIN t2 ON (c3 = c2) WHERE i3 >= '4'; > + > +DROP TABLE t1,t2; > + > +--echo # > +--echo # Hash join + eq ref access on t2 > +--echo # > + > +CREATE TABLE t1 (i2 VARCHAR(500), c2 VARCHAR(500)); > +INSERT INTO t1 VALUES ('1','v'),('7','s'); > +CREATE TABLE t2 (pk3 INT, i3 VARCHAR(300), c3 VARCHAR(500) PRIMARY KEY); > +INSERT INTO t2 VALUES (1,'7','g'),(2,'4','s'),(3,'1','q'); > + > +set join_buffer_size=1024; > +set join_buffer_space_limit=2048; > + > +EXPLAIN SELECT c3,i3, i2,c2 FROM t1 LEFT JOIN t2 ON (c3 = c2); > +SELECT c3,i3, i2,c2 FROM t1 LEFT JOIN t2 ON (c3 = c2); > + > +set join_buffer_space_limit=262144; > +EXPLAIN SELECT c3,i3, i2,c2 FROM t1 LEFT JOIN t2 ON (c3 = c2); > +SELECT c3,i3, i2,c2 FROM t1 LEFT JOIN t2 ON (c3 = c2); > +DROP TABLE t1,t2; > + > +SET @save_join_cache_level= @@join_cache_level; > +SET @save_join_buffer_size= @@join_buffer_size; > +SET @save_join_buffer_space_limit= @@join_buffer_space_limit; > + > # The following command must be the last one in the file > set @@optimizer_switch=@save_optimizer_switch; > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index 203422f0b43..8b7deb98dc6 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -111,6 +111,7 @@ static bool best_extension_by_limited_search(JOIN *join, > uint prune_level, > uint use_cond_selectivity); > static uint determine_search_depth(JOIN* join); > +static void pick_table_access_method(JOIN_TAB *tab); > C_MODE_START > static int join_tab_cmp(const void *dummy, const void* ptr1, const void* ptr2); > static int join_tab_cmp_straight(const void *dummy, const void* ptr1, const void* ptr2); > @@ -10081,6 +10082,7 @@ static bool create_ref_for_key(JOIN *join, JOIN_TAB *j, > j->ref.disable_cache= FALSE; > j->ref.null_ref_part= NO_REF_PART; > j->ref.const_ref_part_map= 0; > + j->ref.not_null_keyparts= 0; > keyuse=org_keyuse; > > store_key **ref_key= j->ref.key_copy; > @@ -10173,23 +10175,12 @@ static bool create_ref_for_key(JOIN *join, JOIN_TAB *j, > } > } /* not ftkey */ > *ref_key=0; // end_marker > + j->ref.not_null_keyparts= not_null_keyparts; > if (j->type == JT_FT) > DBUG_RETURN(0); > - ulong key_flags= j->table->actual_key_flags(keyinfo); > if (j->type == JT_CONST) > j->table->const_table= 1; > - else if (!((keyparts == keyinfo->user_defined_key_parts && > - ( > - (key_flags & (HA_NOSAME | HA_NULL_PART_KEY)) == HA_NOSAME || > - /* Unique key and all keyparts are NULL rejecting */ > - ((key_flags & HA_NOSAME) && keyparts == not_null_keyparts) > - )) || > - /* true only for extended keys */ > - (keyparts > keyinfo->user_defined_key_parts && > - MY_TEST(key_flags & HA_EXT_NOSAME) && > - keyparts == keyinfo->ext_key_parts) > - ) || > - null_ref_key) > + else if (!j->is_eq_ref_access()|| null_ref_key) > { > /* Must read with repeat */ > j->type= null_ref_key ? JT_REF_OR_NULL : JT_REF; > @@ -11582,11 +11573,25 @@ void set_join_cache_denial(JOIN_TAB *join_tab) > don't do join buffering for the first table in sjm nest. > */ > join_tab[-1].next_select= sub_select; > - if (join_tab->type == JT_REF && join_tab->is_ref_for_hash_join()) > + if ((join_tab->type == JT_REF || join_tab->type == JT_HASH) && > + join_tab->is_ref_for_hash_join()) > { > join_tab->type= JT_ALL; > join_tab->ref.key_parts= 0; Do we ever get here if join_tab->type == JT_REF? I think we don't. If this is so, please remove it. > } > + > + if (join_tab->type == JT_HASH && !join_tab->is_ref_for_hash_join()) > + { > + join_tab->type= join_tab->is_eq_ref_access() ? JT_EQ_REF : JT_REF; > + pick_table_access_method(join_tab); > + } Also please add a comment explaining that the join optimizer's choice was ref access and we fall back to that. > + > + if (join_tab->type == JT_HASH_NEXT) > + { > + join_tab->type = JT_NEXT; > + DBUG_ASSERT(join_tab->ref.key_parts == 0); > + } > + > join_tab->join->return_tab= join_tab; > } > } > @@ -27954,6 +27959,42 @@ void JOIN_TAB::partial_cleanup() > } > > > +/* > + @brief > + Check if the access method for the table is EQ_REF access or not > + > + @retval > + TRUE EQ_REF access > + FALSE Otherwise > +*/ > +bool JOIN_TAB::is_eq_ref_access() > +{ > + > + KEY *keyinfo; > + if (!is_hash_join_key_no(ref.key)) > + keyinfo= table->key_info + ref.key; > + else > + keyinfo= hj_key; This code now looks as if it was possible that hash join is used and also eq_ref is used. Is this really possible? I've tried catching an example of this with MTR and I wasn't able to do so. Maybe we should just return false if hash join is used and also a comment about this? > + > + uint keyparts= ref.key_parts; > + ulong key_flags= table->actual_key_flags(keyinfo); > + if ( (keyparts == keyinfo->user_defined_key_parts && > + ( > + (key_flags & (HA_NOSAME | HA_NULL_PART_KEY)) == HA_NOSAME || > + /* Unique key and all keyparts are NULL rejecting */ > + ((key_flags & HA_NOSAME) && keyparts == ref.not_null_keyparts) > + ) > + ) || > + /* true only for extended keys */ > + (keyparts > keyinfo->user_defined_key_parts && > + MY_TEST(key_flags & HA_EXT_NOSAME) && > + keyparts == keyinfo->ext_key_parts) > + ) > + return true; > + return false; > +} > + > + > /** > @} (end of group Query_Optimizer) > */ BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

2 1

Re: [Maria-developers] 33787ea47a3: MDEV-6899: Database Dump has Syntax Error When Importing
by Sergei Golubchik 23 Dec '20

23 Dec '20

Hi, Anel! On Dec 23, Anel Husakovic wrote: > revision-id: 33787ea47a3 (mariadb-10.2.31-588-g33787ea47a3) > parent(s): ceef26cf866 > author: Anel Husakovic <anel(a)mariadb.org> > committer: Anel Husakovic <anel(a)mariadb.org> > timestamp: 2020-11-17 15:11:14 +0100 > message: > > MDEV-6899: Database Dump has Syntax Error When Importing > > --- > mysql-test/r/mysqldump.result | 32 ++++++++++++++++++++++++++++++++ > mysql-test/t/mysqldump.test | 26 ++++++++++++++++++++++++++ > sql/events.cc | 10 ++++++++-- > sql/sql_show.cc | 4 +++- > 4 files changed, 69 insertions(+), 3 deletions(-) > > diff --git a/sql/events.cc b/sql/events.cc > index abac2833833..2880113bc75 100644 > --- a/sql/events.cc > +++ b/sql/events.cc > @@ -680,6 +680,12 @@ send_show_create_event(THD *thd, Event_timed *et, Protocol *protocol) > if (et->get_create_event(thd, &show_str)) > DBUG_RETURN(TRUE); > > + LEX_STRING def_show_str= show_str.lex_string(); > + if (!strcmp(&def_show_str.str[def_show_str.length -1], ";")) > + { > + --def_show_str.length; > + def_show_str.str[def_show_str.length]='\0'; > + } No, this is clearly wrong. The bug is that semicolon even gets into the event definition, see other commands in the bug report. You should fix that, not strip it away in SHOW CREATE, which will only hide the bug. Why is the semicolon parsed as a part of the event anyway? > field_list.push_back(new (mem_root) > Item_empty_string(thd, "Event", NAME_CHAR_LEN), > mem_root); Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] ab7092a415f: MDEV-24179: Assertion `m_status == DA_ERROR || m_status == DA_OK ||
by Sergei Golubchik 21 Dec '20

21 Dec '20

Hi, Rucha! On Dec 21, Rucha Deodhar wrote: > revision-id: ab7092a415f (mariadb-10.2.31-584-gab7092a415f) > parent(s): 9b30212f15e > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2020-12-18 10:20:20 +0530 > message: > > MDEV-24179: Assertion `m_status == DA_ERROR || m_status == DA_OK || > m_status == DA_OK_BULK' failed in Diagnostics_area::message() > > Analysis: Assertion failure happens because we reach the maximum limit to > examine rows. > Fix: Return the error state. > > diff --git a/sql/sql_show.cc b/sql/sql_show.cc > index 5544c765775..73b7a84fae1 100644 > --- a/sql/sql_show.cc > +++ b/sql/sql_show.cc > @@ -5046,6 +5046,12 @@ int get_all_tables(THD *thd, TABLE_LIST *tables, COND *cond) > continue; > } > > + if (thd->killed == ABORT_QUERY) > + { > + error= 0; > + goto err; > + } > + you already have exactly the same if() few lines below. may be it shouldn't be duplicated, but moved down to the err: label? > DEBUG_SYNC(thd, "before_open_in_get_all_tables"); > if (fill_schema_table_by_open(thd, FALSE, > table, schema_table, Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 889f90365ef: MDEV-23536 : Race condition between KILL and transaction commit
by Sergei Golubchik 20 Dec '20

20 Dec '20

Hi, Jan! On Dec 19, Jan LindstrÃ¶m wrote: > revision-id: 889f90365ef (mariadb-10.2.31-623-g889f90365ef) > parent(s): 74223c33d1a > author: Jan Lindström <jan.lindstrom(a)mariadb.com> > committer: Jan Lindström <jan.lindstrom(a)mariadb.com> > timestamp: 2020-12-18 16:10:39 +0200 > message: > > MDEV-23536 : Race condition between KILL and transaction commit ... > If you look carefully into the above, you can conclude that > thd->free_connection() can be called concurrently with > KILL/thd->awake(). Which is the bug. And it is partially fixed in > THD::~THD(), that is destructor waits for KILL completion: > > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index 7eaafbd9044..46aeb4f2bb3 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -1437,6 +1437,31 @@ void THD::cleanup(void) > void THD::free_connection() > { > DBUG_ASSERT(free_connection_done == 0); > +#ifdef WITH_WSREP > + /* > + wsrep_rgi should be protected with LOCK_thd_data mutex and > + mutexing ordering rules state that LOCK_thd_data must be taken > + before LOCK_thd_kill. > + */ > + if (wsrep_rgi && WSREP(this)) mysql_mutex_lock(&LOCK_thd_data); > +#endif > + /* > + Other threads may have a lock on LOCK_thd_kill to ensure that this > + THD is not deleted while they access it. The following mutex_lock > + ensures that no one else is using this THD and it's now safe to > + continue. > + */ > + mysql_mutex_lock(&LOCK_thd_kill); > + mysql_mutex_unlock(&LOCK_thd_kill); Note that THD destructor starts from `assert_not_linked()` - it's important, because if the THD is linked in the list of threads, then someone might start using it after you released LOCK_thd_kill. With the new logic you need to move assert_not_linked to free_connection(). otherwise ok to push > + > +#ifdef WITH_WSREP > + if (rpl_group_info *rgi= wsrep_rgi) > + { > + wsrep_rgi= 0; > + if (WSREP(this)) mysql_mutex_unlock(&LOCK_thd_data); > + delete rgi; > + } > +#endif > my_free(db); > db= NULL; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

3 4

Re: [Maria-developers] a36f27e200e: Fixes that enables my_new.cc (new wrapper using my_malloc)
by Sergei Golubchik 11 Dec '20

11 Dec '20

Hi, Michael! On Dec 11, Michael Widenius wrote: > revision-id: a36f27e200e (mariadb-10.5.2-275-ga36f27e200e) > parent(s): 9f2bbf84e8d > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2020-09-24 17:26:40 +0300 > message: > > Fixes that enables my_new.cc (new wrapper using my_malloc) > > This is not enabled by default as there are leaks in the > server that needs to be fixed first. One can compile > with -DREALLY_USE_MYSYS_NEW -DSF_REMEMBER_FRAMES=14 to find the > memory leaks from 'new' The change in my_new.cc is ok. The change in my_global.h and CMakeLists.txt is confusing and redundant. You unconditionaly define USE_MYSYS_NEW then conditionally undefine it again? Everything already worked just fine. By default cxx new is used, if one wants to use mysys version, it's done with -DHAVE_CXX_NEW=0 this works for me in 10.5 and fails to compile my_new.cc, as it should. Please, only keed your my_new.cc changes and say in the commit message "One can compile with -DHAVE_CXX_NEW=0 to find memory leaks from 'new'" As for -DSF_REMEMBER_FRAMES=14, perhaps it's better to do it automatically? Like #ifdef USE_MYSYS_NEW #define SF_REMEMBER_FRAMES 14 #else #define SF_REMEMBER_FRAMES 8 #endif or may be even just always make it 14. It should be easy to enable this feature, right? > diff --git a/CMakeLists.txt b/CMakeLists.txt > index 80fac4b2ccc..f97a6354f19 100644 > --- a/CMakeLists.txt > +++ b/CMakeLists.txt > @@ -362,10 +362,8 @@ ENDIF() > # Run platform tests > INCLUDE(configure.cmake) > > -# force -DUSE_MYSYS_NEW unless already done by HAVE_CXX_NEW > -IF(NOT HAVE_CXX_NEW) > - ADD_DEFINITIONS(-DUSE_MYSYS_NEW) > -ENDIF() > +# Always use USE_MYSYS_NEW > +ADD_DEFINITIONS(-DUSE_MYSYS_NEW) > > # Find header files from the bundled libraries > # (wolfssl, readline, pcre2, etc) > diff --git a/include/my_global.h b/include/my_global.h > index 86ef5f882f6..8862a397f73 100644 > --- a/include/my_global.h > +++ b/include/my_global.h > @@ -527,6 +527,9 @@ typedef unsigned short ushort; > duplicate declaration of __cxa_pure_virtual, solved by declaring it a > weak symbol. > */ > +#ifndef REALLY_USE_MYSYS_NEW > +#undef USE_MYSYS_NEW > +#endif > #if defined(USE_MYSYS_NEW) && ! defined(DONT_DECLARE_CXA_PURE_VIRTUAL) > C_MODE_START > int __cxa_pure_virtual () __attribute__ ((weak)); Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 9f2bbf84e8d1: Fixed my_addr_resolve
by Sergei Golubchik 10 Dec '20

10 Dec '20

Hi, Michael! On Dec 03, Michael Widenius wrote: > revision-id: 9f2bbf84e8d1 (mariadb-10.5.2-274-g9f2bbf84e8d1) > parent(s): 173116c3a573 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2020-09-24 17:26:40 +0300 > message: > > Fixed my_addr_resolve > > At least on openSUSE Leap 15.1 one should not use the dli_fbase offset > provided by dladdr() for addresses in the main program. > Without this patch the stack traces from sf_report_leaked_memory() > are not usable. Sorry, I was not able to repeat it. I built on openSUSE Leap and the stack trace was perfectly usable without your patch. I tried both `kill -ABRT` and an artificially introduced memory leak. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 53a88b2b7032: Optimize usage of c_ptr(), c_ptr_quick() and String::alloc()
by Sergei Golubchik 04 Dec '20

04 Dec '20

Hi, Michael! On Dec 03, Michael Widenius wrote: > revision-id: 53a88b2b7032 (mariadb-10.5.2-272-g53a88b2b7032) > parent(s): 0aab153e05be > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2020-09-24 17:26:40 +0300 > message: > > Optimize usage of c_ptr(), c_ptr_quick() and String::alloc() > > The proble was that hen using String::alloc() to allocate a string, "hen" ? after a minute of staring I realized that you probably meant "when". (and "problem") > the String ensures that there is space for an extra NULL byte in the > buffer and if not, reallocates the string. This is a problem with the > String::set_int() that calls alloc(21), which forces > an extra malloc/free to happen. > > - We do not anymore re-allocate String if alloc() is called with the > Allocated_length. This reduces number of malloc() allocations, > especially one big re-allocation in Protocol::send_result_Set_metadata() > for almost every query that produced a result to the connnected client. > - Avoid extra mallocs when using LONGLONG_BUFFER_SIZE > This can now be done as alloc() doesn't increase buffers if new length is > not bigger than old one. > - c_ptr() is redesigned to be safer (but a bit longer) than before. > - Remove wrong usage of c_ptr_quick() > c_ptr_quick() where used in many cases to get the pointer to the used > buffer, even when it didn't need to be \0 terminated. In this case > ptr() is a better substitute. > Another problem with c_ptr_quick() is that it didn't guarantee that > the string would be \0 terminated. > - item_val_str(), an API function not used currently by the server, > now always returns a null terminated string (before it didn't always > do that). > - Ensure that all String allocations uses STRING_PSI_MEMORY_KEY. The old > mixed usage of performance keys caused assert's when String buffers > where shrunk. > - Binary_string::shrink() is simplifed > > diff --git a/sql/item.cc b/sql/item.cc > index 4a26cb6c140c..328fe96018a5 100644 > --- a/sql/item.cc > +++ b/sql/item.cc > @@ -4574,7 +4574,7 @@ const String *Item_param::value_query_val_str(THD *thd, String *str) const > break; > } > DBUG_ASSERT(str->length() <= typelen); > - buf= str->c_ptr_quick(); > + buf= (char*) str->ptr(); > ptr= buf + str->length(); > *ptr++= '\''; > ptr+= (uint) my_TIME_to_str(&value.time, ptr, decimals); this is just wrong. One cannot simply write data at the String's end pointer. One must alloc() the space first. > diff --git a/sql/partition_info.cc b/sql/partition_info.cc > index 8ad14ca260c4..ce72a7af051d 100644 > --- a/sql/partition_info.cc > +++ b/sql/partition_info.cc > @@ -126,7 +126,7 @@ partition_info *partition_info::get_clone(THD *thd) > /** > Mark named [sub]partition to be used/locked. > > - @param part_name Partition name to match. > + @param part_name Partition name to match. Must be \0 terminated! > @param length Partition name length. > > @return Success if partition found > @@ -172,9 +172,9 @@ bool partition_info::add_named_partition(const char *part_name, size_t length) > else > bitmap_set_bit(&read_partitions, part_def->part_id); > } > - DBUG_PRINT("info", ("Found partition %u is_subpart %d for name %s", > + DBUG_PRINT("info", ("Found partition %u is_subpart %d for name %.*s", > part_def->part_id, part_def->is_subpart, > - part_name)); > + length, part_name)); why did you specify a length if the name "must be \0 terminated!" ? (and I agree that it must be) > DBUG_RETURN(false); > } > > diff --git a/sql/sql_plugin.cc b/sql/sql_plugin.cc > index 633d969b3dec..2b603380aa6c 100644 > --- a/sql/sql_plugin.cc > +++ b/sql/sql_plugin.cc > @@ -404,12 +404,16 @@ static int item_value_type(struct st_mysql_value *value) > static const char *item_val_str(struct st_mysql_value *value, > char *buffer, int *length) > { > + size_t org_length= *length; > - String str(buffer, *length, system_charset_info), *res; > + String str(buffer, org_length, system_charset_info), *res; > if (!(res= ((st_item_value_holder*)value)->item->val_str(&str))) > return NULL; > *length= res->length(); > - if (res->c_ptr_quick() == buffer) > + if (res->ptr() == buffer && res->length() < org_length) > + { > + buffer[res->length()]= 0; > return buffer; > + } Okay. This looked like the *only* correct usage of c_ptr_quick(). Agree, better to remove c_ptr_quick(). > > /* > Lets be nice and create a temporary string since the > diff --git a/sql/sql_string.cc b/sql/sql_string.cc > index e2defba434dd..8ab156d0e556 100644 > --- a/sql/sql_string.cc > +++ b/sql/sql_string.cc > @@ -1251,21 +1259,16 @@ bool String::append_semi_hex(const char *s, uint len, CHARSET_INFO *cs) > // Shrink the buffer, but only if it is allocated on the heap. > void Binary_string::shrink(size_t arg_length) > { > - if (!is_alloced()) > - return; > - if (ALIGN_SIZE(arg_length + 1) < Alloced_length) > + if (is_alloced() && ALIGN_SIZE(arg_length + 1) < Alloced_length) > + { > + char *new_ptr; > + /* my_realloc() can't fail as new buffer is less than the original one */ > + if ((new_ptr= (char*) my_realloc(STRING_PSI_MEMORY_KEY, Ptr, arg_length, > + MYF(thread_specific ? > + MY_THREAD_SPECIFIC : 0)))) why do you need an if() if "my_realloc() can't fail" ? (and it really cannot, you're right) > { > - char* new_ptr; > - if (!(new_ptr = (char*)my_realloc(STRING_PSI_MEMORY_KEY, Ptr, arg_length, > - MYF(thread_specific ? MY_THREAD_SPECIFIC : 0)))) > - { > - Alloced_length = 0; > - real_alloc(arg_length); > - } > - else > - { > - Ptr = new_ptr; > - Alloced_length = (uint32)arg_length; > - } > + Ptr= new_ptr; > + Alloced_length= (uint32) arg_length; > } > + } > } > diff --git a/sql/sql_string.h b/sql/sql_string.h > index 2ef817ea0adc..01d5211fe850 100644 > --- a/sql/sql_string.h > +++ b/sql/sql_string.h > @@ -599,25 +599,34 @@ class Binary_string: public Static_binary_string > > inline char *c_ptr() > { > - DBUG_ASSERT(!alloced || !Ptr || !Alloced_length || > - (Alloced_length >= (str_length + 1))); > - > - if (!Ptr || Ptr[str_length]) // Should be safe > - (void) realloc(str_length); > + if (unlikely(!Ptr)) > + return (char*) ""; > + /* > + Here we assume that any buffer used to initalize String has > + an end \0 or have at least an accessable character at end. > + This is to handle the case of String("Hello",5) efficently. > + */ > + if (unlikely(!alloced && !Ptr[str_length])) No, this is wrong. Note the difference between String a("Hello", 5) and char hello[5]; String a(buf, 5); Your assumption should only apply to the first case, not to the second. In the first case alloced==Alloced_length==0, in the second case only alloced==0 and Alloced_length==5. So in the if() above you need to look at Alloced_length. > + return Ptr; > + if (str_length < Alloced_length) > + { > + Ptr[str_length]=0; > + return Ptr; > + } > + (void) realloc(str_length+1); /* This will add end \0 */ > return Ptr; > } > + /* Use c_ptr() instead. This will be deleted soon, kept for compatiblity */ > inline char *c_ptr_quick() > { > - if (Ptr && str_length < Alloced_length) > - Ptr[str_length]=0; > - return Ptr; > + return c_ptr_safe(); > } > inline char *c_ptr_safe() > { > if (Ptr && str_length < Alloced_length) > Ptr[str_length]=0; > else > - (void) realloc(str_length); > + (void) realloc(str_length + 1); what's the difference between c_ptr_safe and c_ptr, again? they're very similar, so they'd need a good comment explaining when to use what. > return Ptr; > } > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 994ea2af3973: Optimize size of lex structures
by Sergei Golubchik 04 Dec '20

04 Dec '20

Hi, Michael! On Dec 03, Michael Widenius wrote: > revision-id: 994ea2af3973 (mariadb-10.5.2-269-g994ea2af3973) > parent(s): 77500a1a1107 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2020-09-17 12:24:58 +0300 > message: > > Optimize size of lex structures > > LEX, st_select_lex, st_select_unit optimized for space: > - Use bit fields for bool variables > - Ensure that all bit fields are initialized (improves > performance for init functions as all bit fields can be > initalized with one memory access) > - Move members around in above structures to remove alignment > gaps I did not get any reply to my email From: Sergei Golubchik <serg(a)mariadb.org> To: Michael Widenius <michael.widenius(a)gmail.com> Cc: maria-developers(a)lists.launchpad.net Subject: Re: c3ecf0d6243: Optimize size of lex structures Date: Wed, 9 Sep 2020 15:57:23 +0200 and I don't see any related changes in this commit. Please, do at least one of the two. Regards, Sergei

1 0

Re: [Maria-developers] 77500a1a1107: Added typedef decimal_digits_t (uint16) for number of decimals
by Sergei Golubchik 04 Dec '20

04 Dec '20

Hi, Michael! I did not get any reply to my email From: Sergei Golubchik <serg(a)mariadb.org> To: Michael Widenius <michael.widenius(a)gmail.com> Cc: maria-developers(a)lists.launchpad.net Subject: Re: c79c6f78205: Added typedef decimal_digits_t (uint16) for number of decimals Date: Wed, 9 Sep 2020 13:34:57 +0200 and I don't see any related changes in this commit. Please, do at least one of the two. Regards, Sergei On Dec 03, Michael Widenius wrote: > revision-id: 77500a1a1107 (mariadb-10.5.2-268-g77500a1a1107) > parent(s): 32104b969513 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2020-09-17 12:24:58 +0300 > message: > > Added typedef decimal_digits_t (uint16) for number of decimals > > For fields and Item's uint8 should be good enough. After > discussions with Alexander Barkov we choose uint16 (for now) > as some format functions may accept +256 digits. > > The reason for this patch was to make the storage of decimal > digits simlar. Before this patch decimals was stored/used as > uint8, int and uint. > > Changed most decimal variables and functions to use the new > typedef. >

1 0

Re: [Maria-developers] 32104b969513: MDEV-23001 Precreate static Item_bool() to simplify code
by Sergei Golubchik 04 Dec '20

04 Dec '20

Hi, Michael! I did not get any reply to my email From: Sergei Golubchik <serg(a)mariadb.org> To: Michael Widenius <michael.widenius(a)gmail.com> Cc: maria-developers(a)lists.launchpad.net Subject: Re: bf8bd057344: MDEV-23001 Precreate static Item_bool() to simplify code Date: Tue, 8 Sep 2020 20:39:35 +0200 and I don't see any related changes in this commit. Please, do at least one of the two. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org On Dec 03, Michael Widenius wrote: > revision-id: 32104b969513 (mariadb-10.5.2-267-g32104b969513) > parent(s): 32a29afea777 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2020-09-17 12:24:45 +0300 > message: > > MDEV-23001 Precreate static Item_bool() to simplify code > > The following changes where done: > - Create global Item: Item_false and Item_true > - Replace all creation if 'FALSE' and 'TRUE' top level items used for > WHERE/HAVING/ON clauses to use Item_false and Item_true. > > The benefit are: > - Less and faster code > - No test needed if we where able to create the new item. > - Fixed possible errors if 'new' would have failed for the Item_bool's >

1 0

[Maria-developers] MDEV-21287 dev community opinion wanted
by Pierre-Jean Clement 02 Dec '20

02 Dec '20

Hi all, I created the Jira ticket MDEV-21287 10 months ago and didn't get much feedback. I tried to take a shot at it myself, but found out that the resolution MDEV-20403 introduced this behavior by purposefully evaluating 'update functions' before the BEFORE UPDATE trigger. I couldn't really understand why this was necessary, and I'd like to get feedback from the dev community to try and understand what is really supposed to happen on update. I kinda hate getting my trigger called with a new value that's not going to be persisted anyway, what do you think about that? Regards, Pierre-Jean

2 3

Re: [Maria-developers] MDEV-21265: IN predicate conversion to IN subquery should be allowed for a broader set of datatype comparison
by Sergey Petrunia 26 Nov '20

26 Nov '20

Hi Varun, Please find some cosmetic input below. Ok to push after addresed. > diff --git a/mysql-test/main/subselect4.test b/mysql-test/main/subselect4.test > index 6f5eb1f2985f..e9da028618f6 100644 > --- a/mysql-test/main/subselect4.test > +++ b/mysql-test/main/subselect4.test > @@ -2238,3 +2238,38 @@ SET join_cache_level= @save_join_cache_level; > DROP TABLE t1,t2,t3,t4; > > --echo # End of 10.2 tests > + > +--echo # > +--echo # MDEV-21265: IN predicate conversion to IN subquery should be allowed for a broader set of datatype comparison > +--echo # > + > +CREATE TABLE t1(a VARCHAR(50) collate utf8_general_ci, b INT); > +INSERT INTO t1 VALUES ('abc',1), ('def', 2), ('ghi', 3), ('jkl', 4), ('mno', 5); > + > +CREATE TABLE t2(a VARCHAR(50) collate utf8mb4_general_ci, b INT); > +INSERT INTO t2 VALUES ('abc',1), ('def', 2), ('ghi', 3), ('jkl', 4), ('mno', 5); > + > +set in_predicate_conversion_threshold=2; > + ... > +EXPLAIN > +SELECT * FROM t2 WHERE (t2.a,t2.b) IN (('abx',1),('def',2), ('abc', 3)); > + > +set names default; > +DROP TABLE t1,t2; > + > +--echo # End of 10.3 tests Please restore the value of in_predicate_conversion_threshold. > diff --git a/sql/item_subselect.h b/sql/item_subselect.h > index 1cea7291c9ef..d1b2f86088d9 100644 > --- a/sql/item_subselect.h > +++ b/sql/item_subselect.h > @@ -573,6 +573,13 @@ class Item_in_subselect :public Item_exists_subselect > TRUE<=>registered in the list of semijoins in outer select > */ > bool is_registered_semijoin; > + > + Item_func_not_all *upper_item; // point on NOT/NOP before ALL/SOME subquery Is this necessary to touch the above member? I think it's better leave it as is. > + > + /* > + SET to TRUE if IN subququery is converted from an IN rredicate > + */ > + bool converted_from_in_predicate; ... > diff --git a/sql/sql_type.h b/sql/sql_type.h > index 907225b7c505..a1403165fe38 100644 > --- a/sql/sql_type.h > +++ b/sql/sql_type.h > @@ -1291,9 +1291,21 @@ class Type_handler > Item *target_expr, Item *target_value, > Item_bool_func2 *source, > Item *source_expr, Item *source_const) const= 0; > + > + /* > + @brief > + Check if an IN subquery allows materialization or not > + @param > + inner expression on the inner side of the IN subquery > + outer expression on the inner side of the IN subquery Typo. Should it should be 'outer' ? > + is_in_predicate SET to true if IN subquery was converted from an > + IN predicate or we are checking if materialization > + strategy can be used for an IN predicate > + */ > virtual bool > subquery_type_allows_materialization(const Item *inner, > - const Item *outer) const= 0; > + const Item *outer, > + bool is_in_predicate) const= 0; > /** > Make a simple constant replacement item for a constant "src", > so the new item can futher be used for comparison with "cmp", e.g.: BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Question about JSONPath and '**' wildcard
by Sergey Petrunia 20 Nov '20

20 Nov '20

Hi Alexey, I've found this discrepancy in JSONPath evaluation: set @json_doc3 = ' { "root": { "child1" : { "child2" : { "child1" : { "x":124 } } } } } '; select json_extract(@json_doc3, '$**.child1**.x'); MariaDB [test]> select json_extract(@json_doc3, '$**.child1**.x'); +--------------------------------------------+ | json_extract(@json_doc3, '$**.child1**.x') | +--------------------------------------------+ | NULL | +--------------------------------------------+ 1 row in set (0.001 sec) MySQL-8> select json_extract(@json_doc3, '$**.child1**.x'); +--------------------------------------------+ | json_extract(@json_doc3, '$**.child1**.x') | +--------------------------------------------+ | [124] | +--------------------------------------------+ 1 row in set (0.00 sec) Which one is right? My opinion is that MariaDB's answer is definitely incorrect. As for MySQL's answer, it depends on how the result of JSONPath expression is defined. If it is "a set of nodes in the JSON document which match the pattern", MySQL's result is correct. Coming from XPath world, I was expecting a semantics in form find the set of nodes matching the search step#1 ( $**, then apply step #2 ( .child1) then apply step #3 ( any children of those) ... in which case "124" would be in the result twice as it is reachable via two possible paths. Any thoughts about this? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

2 2

Re: [Maria-developers] 255e5f269d80: MDEV-23610: Slave user can't run "SHOW SLAVE STATUS" anymore after upgrade to 10.5, mysql_upgrade should take of that
by Sergei Golubchik 13 Nov '20

13 Nov '20

Hi, Sujatha! Looks good, thanks! One detail to fix - I think you've mistakenly committed an old test from the previous version of the patch. And two minor suggestions, you can change or not, as you like. otherwise ok to push On Nov 13, Sujatha wrote: > revision-id: 255e5f269d80 (mariadb-10.5.4-319-g255e5f269d80) > parent(s): 97569d3c37a6 > author: Sujatha <sujatha.sivakumar(a)mariadb.com> > committer: Sujatha <sujatha.sivakumar(a)mariadb.com> > timestamp: 2020-11-13 19:08:02 +0530 > message: > > MDEV-23610: Slave user can't run "SHOW SLAVE STATUS" anymore after upgrade to 10.5, mysql_upgrade should take of that > > Add a new privilege "SLAVE MONITOR" which will grant user the permission > to execute "SHOW SLAVE STATUS" and "SHOW RELAYLOG EVENTS" commands. > > SHOW SLAVE STATUS requires either SLAVE MONITOR/SUPER > SHOW RELAYLOG EVENTS requires SLAVE MONITOR privilege. > > diff --git a/mysql-test/main/grant_slave_monitor.test b/mysql-test/main/grant_slave_monitor.test > new file mode 100644 > index 000000000000..a4f21085e537 > --- /dev/null > +++ b/mysql-test/main/grant_slave_monitor.test > @@ -0,0 +1,101 @@ > +# ==== Purpose ==== > +# > +# SLAVE MONITOR privilege is required to execute following commands. > +# SHOW SLAVE STATUS > +# SHOW RELAYLOG EVENTS > +# > +# ==== Implementation ==== > +# > +# Step1: GRANT ALL privileges for a new user 'user1' and then REVOKE > +# SLAVE MONITOR and SUPER privileges. > +# Step2: Execute SHOW SLAVE STAUTS/SHOW RELAYLOG EVENTS commands and expect > +# ER_SPECIFIC_ACCESS_DENIED_ERROR. This also verifies that REPLICATION > +# SLAVE ADMIN privilege is not required for these two commands. > +# Step3: GRANT SLAVE MONITOR privilege and observe that both commands are > +# allowd to execute. > +# Step4: GRANT SUPER privilege and observe that only SHOW SLAVE STATUS command > +# is allowed. > +# > +# ==== References ==== > +# > +# MDEV-23610: Slave user can't run "SHOW SLAVE STATUS" anymore after upgrade > +# to 10.5, mysql_upgrade should take of that > +# MDEV-23918: admin privlege required to view contents of relay logs in 10.5 > +# > + > +--source include/not_embedded.inc > + > +CREATE USER user1@localhost IDENTIFIED BY ''; > +GRANT ALL PRIVILEGES ON *.* TO user1@localhost; > +REVOKE SLAVE MONITOR, SUPER ON *.* FROM user1@localhost; > +FLUSH PRIVILEGES; > + > +--connect(con1,localhost,user1,,) > +--connection con1 > +SHOW GRANTS; > + > +--echo # > +--echo # Verify that having REPLICATION SLAVE ADMIN doesn't allow SHOW SLAVE STATUS > +--echo # Expected error: Access denied; you need (at least one of) the SUPER, SLAVE > +--echo # MONITOR privilege(s) for this operation > +--echo # > +--error ER_SPECIFIC_ACCESS_DENIED_ERROR > +SHOW SLAVE STATUS; > + > +--echo # > +--echo # Verify that having REPLICATION SLAVE ADMIN doesn't allow SHOW RELAYLOG EVENTS > +--echo # Expected error: Access denied; you need (at least one of) the REPLICA MONITOR > +--echo # privilege(s) for this operation > +--echo # > +--disable_ps_protocol > +--error ER_SPECIFIC_ACCESS_DENIED_ERROR > +SHOW RELAYLOG EVENTS; > +--enable_ps_protocol > +--disconnect con1 > + > +--echo # > +--echo # SHOW SLAVE STATUS and SHOW RELAYLOG EVENTS are allowed with SLAVE MONITOR privilege > +--echo # > + > +--connection default > +GRANT SLAVE MONITOR ON *.* TO user1@localhost; > +FLUSH PRIVILEGES; > + > +--connect(con1,localhost,user1,,) > +--connection con1 > +SHOW GRANTS; > + > +--disable_result_log > +SHOW SLAVE STATUS; > +#--disable_ps_protocol > +SHOW RELAYLOG EVENTS; > +#--enable_ps_protocol > +--enable_result_log > +--disconnect con1 > + > +--connection default > +DROP USER user1@localhost; > + > +--echo # > +--echo # SHOW SLAVE STATUS command is allowed with SUPER privilege > +--echo # > +CREATE USER user1@localhost IDENTIFIED BY ''; > +GRANT SUPER ON *.* TO user1@localhost; > + > +--connect(con1,localhost,user1,,) > +--disable_result_log > +SHOW SLAVE STATUS; > +--enable_result_log > + > +--echo # > +--echo # SHOW RELAYLOG EVENTS is not allowed with SUPER privilege, it requires SLAVE MONITOR > +--echo # > + > +--disable_ps_protocol > +--error ER_SPECIFIC_ACCESS_DENIED_ERROR > +SHOW RELAYLOG EVENTS; > +--enable_ps_protocol > +--disconnect con1 > + > +--connection default > +DROP USER user1@localhost; > diff --git a/mysql-test/main/rpl_replica_monitor_priv.test b/mysql-test/main/rpl_replica_monitor_priv.test > --- /dev/null > +++ b/mysql-test/main/rpl_replica_monitor_priv.test > @@ -0,0 +1,90 @@ Did you check in this file by mistake? It's a replication test (source include/master-slave.inc) but not in the rpl suite. It tests the feature that does not need master-slave.inc. And it's duplicated by grant_slave_monitor.test > diff --git a/sql/privilege.h b/sql/privilege.h > index 37cdf4da01ad..d0e6e532b4b5 100644 > --- a/sql/privilege.h > +++ b/sql/privilege.h > @@ -93,13 +94,18 @@ enum privilege_t: unsigned long long > */ > }; > > +constexpr static inline privilege_t ALL_KNOWN_BITS(privilege_t x) > +{ > + return (privilege_t)((x << 1) - 1); Generally it's safer to write x | (x - 1) it won't overflow if x is already the highest possible bit. In privileges we're quite far from it at the moment, so it's not a big deal, just a style issue and setting a good example to others. > +} > > // Version markers > constexpr privilege_t LAST_100304_ACL= DELETE_HISTORY_ACL; > constexpr privilege_t LAST_100502_ACL= BINLOG_REPLAY_ACL; > +constexpr privilege_t LAST_100509_ACL= SLAVE_MONITOR_ACL; > > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index ef96e8f24843..b5b1472f004f 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -1015,7 +1015,7 @@ class User_table_tabular: public User_table > access|= LOCK_TABLES_ACL | CREATE_TMP_ACL | SHOW_DB_ACL; > if (access & FILE_ACL) > access|= BINLOG_MONITOR_ACL | REPL_SLAVE_ACL | BINLOG_ADMIN_ACL | > - BINLOG_REPLAY_ACL; > + BINLOG_REPLAY_ACL | SLAVE_MONITOR_ACL; this does not hurt, but it's rather redundant, you set SLAVE_MONITOR_ACL below anyway. > if (access & PROCESS_ACL) > access|= SUPER_ACL | EXECUTE_ACL; > } > @@ -1057,6 +1057,9 @@ class User_table_tabular: public User_table > if (access & REPL_SLAVE_ACL) > access|= REPL_MASTER_ADMIN_ACL; > > + if (access & REPL_SLAVE_ACL) > + access|= SLAVE_MONITOR_ACL; > + > return access & GLOBAL_ACLS; > } > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] d66d5457589d: MDEV-23610: Slave user can't run "SHOW SLAVE STATUS" anymore after upgrade to 10.5, mysql_upgrade should take of that
by Sergei Golubchik 12 Nov '20

12 Nov '20

Hi, Sujatha! On Nov 12, Sujatha wrote: > revision-id: d66d5457589d (mariadb-10.5.4-315-gd66d5457589d) > parent(s): 10b2d5726fa2 > author: Sujatha <sujatha.sivakumar(a)mariadb.com> > committer: Sujatha <sujatha.sivakumar(a)mariadb.com> > timestamp: 2020-11-11 18:41:44 +0530 > message: > > MDEV-23610: Slave user can't run "SHOW SLAVE STATUS" anymore after upgrade to 10.5, mysql_upgrade should take of that > > Add a new privilege "REPLICA MONITOR" which will grant user the permission > to execute "SHOW SLAVE STATUS" and "SHOW RELAYLOG EVENTS" commands. > > SHOW SLAVE STATUS requires either REPLICA MONITOR/SUPER > SHOW RELAYLOG EVENTS requires REPLICA MONITOR privilege. > > diff --git a/mysql-test/main/grant.result b/mysql-test/main/grant.result > index 01daf0271867..a41b38aaa0ba 100644 > --- a/mysql-test/main/grant.result > +++ b/mysql-test/main/grant.result > @@ -1966,10 +1967,11 @@ GRANT REPLICATION SLAVE ON *.* TO mysqltest_u1@localhost; > GRANT SHOW DATABASES ON *.* TO mysqltest_u1@localhost; > GRANT SHUTDOWN ON *.* TO mysqltest_u1@localhost; > GRANT USAGE ON *.* TO mysqltest_u1@localhost; > +GRANT REPLICA MONITOR ON *.* TO mysqltest_u1@localhost; I don't think we should have a mix of REPLICA and SLAVE terms. Call it SLAVE MONITOR and when we flip the switch on grants, we'll do it for all grants at the same time. > SHOW GRANTS FOR mysqltest_u1@localhost; > Grants for mysqltest_u1@localhost > -GRANT RELOAD, SHUTDOWN, PROCESS, FILE, SHOW DATABASES, REPLICATION SLAVE, BINLOG MONITOR, CREATE USER ON *.* TO `mysqltest_u1`@`localhost` > +GRANT RELOAD, SHUTDOWN, PROCESS, FILE, SHOW DATABASES, REPLICATION SLAVE, BINLOG MONITOR, CREATE USER, REPLICA MONITOR ON *.* TO `mysqltest_u1`@`localhost` > GRANT CREATE TEMPORARY TABLES, LOCK TABLES, EXECUTE, CREATE ROUTINE, ALTER ROUTINE, EVENT ON `mysqltest_db1`.* TO `mysqltest_u1`@`localhost` > connect con1,localhost,mysqltest_u1,,mysqltest_db1; > connection con1; > diff --git a/mysql-test/main/grant.test b/mysql-test/main/grant.test > index 38baf6738255..63be18ecb292 100644 > --- a/mysql-test/main/grant.test > +++ b/mysql-test/main/grant.test > @@ -1833,6 +1833,7 @@ GRANT REPLICATION SLAVE ON *.* TO mysqltest_u1@localhost; > GRANT SHOW DATABASES ON *.* TO mysqltest_u1@localhost; > GRANT SHUTDOWN ON *.* TO mysqltest_u1@localhost; > GRANT USAGE ON *.* TO mysqltest_u1@localhost; > +GRANT REPLICA MONITOR ON *.* TO mysqltest_u1@localhost; But the parser should, of course, accept both, as everywhere. > diff --git a/mysql-test/suite/rpl/t/rpl_replica_monitor_priv.test b/mysql-test/suite/rpl/t/rpl_replica_monitor_priv.test > new file mode 100644 > index 000000000000..f7b2af2e0c51 > --- /dev/null > +++ b/mysql-test/suite/rpl/t/rpl_replica_monitor_priv.test > @@ -0,0 +1,90 @@ > +# ==== Purpose ==== > +# > +# REPLICA MONITOR privilege is required to execute following commands. > +# SHOW SLAVE STATUS > +# SHOW RELAYLOG EVENTS > +# > +# ==== Implementation ==== > +# > +# Step1: GRANT ALL privileges for a new user 'user1' and then REVOKE REPLICA > +# MONITOR and SUPER privileges. > +# Step2: Execute SHOW SLAVE STAUTS/SHOW RELAYLOG commands and expect > +# ER_SPECIFIC_ACCESS_DENIED_ERROR. This also verifies that REPLICATION > +# SLAVE ADMIN privilege is not required for these two commands. > +# Step3: GRANT REPLICA MONITOR privilege and observe that both commands are > +# allowd to execute. > +# Step4: GRANT SUPER privilege and observe that only SHOW SLAVE STATUS command > +# is allowed. why do you need a full master-slave replication setup for that? > +# ==== References ==== > +# > +# MDEV-23610: Slave user can't run "SHOW SLAVE STATUS" anymore after upgrade > +# to 10.5, mysql_upgrade should take of that > +# MDEV-23918: admin privlege required to view contents of relay logs in 10.5 > +# > + > +--source include/not_embedded.inc > +--source include/have_binlog_format_mixed.inc > +--source include/master-slave.inc > + > +--connection master > +CREATE USER user1@localhost IDENTIFIED BY ''; > +GRANT ALL PRIVILEGES ON *.* TO user1@localhost; > +REVOKE REPLICA MONITOR, SUPER ON *.* FROM user1@localhost; > +FLUSH PRIVILEGES; > +--sync_slave_with_master > +--connect(con1,127.0.0.1,user1,,test,$SLAVE_MYPORT) > +--connection con1 > +SELECT CURRENT_USER(); > +SHOW GRANTS; > +--echo "Verify that having REPLICATION SLAVE ADMIN doesn't allow SHOW SLAVE STATUS" > +--echo "Expected error:Access denied; you need (at least one of) the SUPER, REPLICA MONITOR privilege(s) for this operation" better use not quotes, but #, like elsewhere. > +--error ER_SPECIFIC_ACCESS_DENIED_ERROR > +SHOW SLAVE STATUS; > +--echo "Verify that having REPLICATION SLAVE ADMIN doesn't allow SHOW RELAYLOG EVENTS" > +--echo "Expected error: Access denied; you need (at least one of) the REPLICA MONITOR privilege(s) for this operation" > +--disable_ps_protocol why? > +--error ER_SPECIFIC_ACCESS_DENIED_ERROR > +SHOW RELAYLOG EVENTS; > +--enable_ps_protocol > +--disconnect con1 > + > diff --git a/sql/privilege.h b/sql/privilege.h > index 37cdf4da01ad..e7686a206b84 100644 > --- a/sql/privilege.h > +++ b/sql/privilege.h > @@ -115,8 +117,12 @@ constexpr privilege_t ALL_KNOWN_ACL_100304 = > constexpr privilege_t ALL_KNOWN_ACL_100502= > (privilege_t) ((LAST_100502_ACL << 1) - 1); > > +// A combination of all bits defined in 10.5.8 > +constexpr privilege_t ALL_KNOWN_ACL_100508= > + (privilege_t) ((LAST_100508_ACL << 1) - 1); > + > // A combination of all bits defined as of the current version > -constexpr privilege_t ALL_KNOWN_ACL= ALL_KNOWN_ACL_100502; > +constexpr privilege_t ALL_KNOWN_ACL= ALL_KNOWN_ACL_100508; may be (privilege_t) ((LAST_CURRENT_ACL << 1) - 1) and then we won't need to change it anymore? Also, I think this deserves a function, like static inline privilege_t ALL_ACLS_TO(privilege_t x) { return (x << 1) - 1; } or, rather, a bit safer return x | (x - 1) > // Unary operators > @@ -505,9 +511,11 @@ constexpr privilege_t PRIV_STMT_STOP_SLAVE= REPL_SLAVE_ADMIN_ACL | SUPER_ACL; > // Was SUPER_ACL prior to 10.5.2 > constexpr privilege_t PRIV_STMT_CHANGE_MASTER= REPL_SLAVE_ADMIN_ACL | SUPER_ACL; > // Was (SUPER_ACL | REPL_CLIENT_ACL) prior to 10.5.2 > -constexpr privilege_t PRIV_STMT_SHOW_SLAVE_STATUS= REPL_SLAVE_ADMIN_ACL | SUPER_ACL; > +// Was REPL_SLAVE_ADMIN_ACL from 10.5.2 to 10.5.7 // Was SUPER_ACL | REPL_SLAVE_ADMIN_ACL from 10.5.2 to 10.5.7 otherwise it's unclear why you have SUPER_ACL below and not just REPLICA_MONITOR_ACL > +constexpr privilege_t PRIV_STMT_SHOW_SLAVE_STATUS= REPLICA_MONITOR_ACL | SUPER_ACL; > // Was REPL_SLAVE_ACL prior to 10.5.2 > -constexpr privilege_t PRIV_STMT_SHOW_RELAYLOG_EVENTS= REPL_SLAVE_ADMIN_ACL; > +// Was REPL_SLAVE_ADMIN_ACL from 10.5.2 to 10.5.7 > +constexpr privilege_t PRIV_STMT_SHOW_RELAYLOG_EVENTS= REPLICA_MONITOR_ACL; > > /* > Privileges related to binlog replying. > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index ef96e8f24843..8a3551c14f53 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -1057,6 +1057,9 @@ class User_table_tabular: public User_table > if (access & REPL_SLAVE_ACL) > access|= REPL_MASTER_ADMIN_ACL; > > + if (access & FILE_ACL) > + access|= REPLICA_MONITOR_ACL; Why is that? > + > return access & GLOBAL_ACLS; > } > > @@ -1541,19 +1548,22 @@ class User_table_json: public User_table > */ > if (access & SUPER_ACL) > { > - if (access & REPL_SLAVE_ACL) > - { > - /* > - The user could do both before the upgrade: > - - set global variables (because of SUPER_ACL) > - - execute "SHOW SLAVE HOSTS" (because of REPL_SLAVE_ACL) > - Grant all new privileges that were splitted from SUPER (in 10.5.2), > - and REPLICATION MASTER ADMIN, so it still can do "SHOW SLAVE HOSTS". > - */ > - access|= REPL_MASTER_ADMIN_ACL; > - } > access|= GLOBAL_SUPER_ADDED_SINCE_USER_TABLE_ACLS; > } > + if (access & REPL_SLAVE_ACL) > + { > + /* > + The user could do both before the upgrade: > + - set global variables (because of SUPER_ACL) this looked fine where it was before, but not anymore after you moved it out of `if (access & SUPER_ACL)` > + - execute "SHOW SLAVE HOSTS" (because of REPL_SLAVE_ACL) > + Grant all new privileges that were splitted from SUPER (in 10.5.2), > + and REPLICATION MASTER ADMIN, so it still can do "SHOW SLAVE HOSTS". > + - execute "SHOW RELAYLOG EVENTS" (because of REPL_SLAVE_ACL) > + */ > + access|= (REPL_MASTER_ADMIN_ACL | REPLICA_MONITOR_ACL); and REPL_MASTER_ADMIN_ACL allows to modify global variables was was earlier possible only with SUPER_ACL. So, indeed, old code was correct, that if() should be inside SUPER_ACL, one should only get REPL_MASTER_ADMIN_ACL if one had both SUPER_ACL and REPL_SLAVE_ACL. > + } > + if (access & BINLOG_MONITOR_ACL) > + access|= REPLICA_MONITOR_ACL; > } > > if (orig_access & ~mask) > @@ -8974,7 +8984,7 @@ static const char *command_array[]= > "CREATE USER", "EVENT", "TRIGGER", "CREATE TABLESPACE", "DELETE HISTORY", > "SET USER", "FEDERATED ADMIN", "CONNECTION ADMIN", "READ_ONLY ADMIN", > "REPLICATION SLAVE ADMIN", "REPLICATION MASTER ADMIN", "BINLOG ADMIN", > - "BINLOG REPLAY" > + "BINLOG REPLAY", "REPLICA MONITOR" SLAVE MONITOR > }; > > static uint command_lengths[]= Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 2

Re: [Maria-developers] e3f45b2f9ea: MDEV-10267 Add ngram fulltext parser plugin
by Sergei Golubchik 11 Nov '20

11 Nov '20

Hi, Rinat! On Nov 02, Rinat Ibragimov wrote: > > > I can't decide. From my point of view, the current approach is > > > fine. Please pick a variant, and I'll try to implement that. > > > > No, I cannot guess which approach will produce more relevant > > searches. Implement something and then we test what works better > > Variable-length n-grams approach is too innovative, and hard to reason > about. I've never heard about such an approach, and it doesn't look > good to me. So I'll stick with a simple slicer. If you mean that variant where it splits "n-grams approach" to "n-gr", "gra", "ram", "ams", "ms a", "s ap", "app", ... then it's just "n letters in every chunk" very easy to explain. But ok, let's start simple and benchmark. > > Of course, it can. Note that fts_get_word() doesn't generate n-grams > > either, it gets the whole word and the n-gram plugin later splits it > > into n-grams. Similarly param->mysql_parse() will extract words for > > you and you'll split them into n-grams. > > Changed to use param->mysql_parse(). > > Turns out that in Aria, MyISAM, and InnoDB, param->mysql_parser() does > call back param->mysql_add_word(). Is it part of the plugin API? Yes, it is. E.g. slide 12 from my old presentation: http://conferences.oreillynet.com/presentations/mysql06/golubchik_sergei.pdf shows that there are three points where a plugin can add functionality. * It can extract the text and then call param->mysql_parser(), this allows to parse, say, gzip-ed texts or EXIF comments in images. * It can replace param->mysql_parser(), to use different rules for spliting the text into words. This is what the n-gram plugin normally does * It can replace param->mysql_add_word() to post-process every word after the built-in parser did the splitting. For example, stemming or soundex plugin can do that. > Comments in include/mysql/plugin_ftparser.h do not mention that at > all. That's why I initially thought that param->mysql_parse() will > parse the string like the default parser do, without any ways to > interact with the process. I've edited the comment to mention this possibility. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 0cd5377e9ff: MDEV-18734 ASAN heap-use-after-free in my_strnxfrm_simple_internal upon update on versioned partitioned table
by Sergei Golubchik 02 Nov '20

02 Nov '20

Hi, Aleksey! On Oct 20, Aleksey Midenkov wrote: > revision-id: 0cd5377e9ff (mariadb-10.2.31-357-g0cd5377e9ff) > parent(s): dc716da4571 > author: Aleksey Midenkov <midenok(a)gmail.com> > committer: Aleksey Midenkov <midenok(a)gmail.com> > timestamp: 2020-08-17 00:52:35 +0300 > message: > > MDEV-18734 ASAN heap-use-after-free in my_strnxfrm_simple_internal upon update on versioned partitioned table > > update_virtual_fields() may reallocate Field_blob::value buffer due to > different reasons (f.ex. different types, see > Field_str::memcpy_field_possible()). But the buffer address may be > already in m_ordered_rec_buffer which is then used in ordered index > scan queue. > > The patch updates blob virtual fields when the ordered index scan > queue is sorted as well as the top record is restored from the queue. > > New VCOL_UPDATE_BLOBS mode is introduced to update only blobs. Swap is > not needed as it was already done if it was needed. > > Related to ea1b2504 No, I think we need to go deeper here. The bug is not only not caused by system versioning, it is also not caused by virtual columns. The problem is that partitioning copies record[0] out and then back to save and restore values. This does not work for blobs, because record[0] may store the pointer to the memory that Field_blob owns and can free or reallocate as needed. Normally it's not the problem because typically after row reads the engine owns the memory, not Field_blob. When a virtual column is set to a calculated value - then Field_blob owns the memory and we have this bug. But there are engines that use field->store() to set values - in this case Field_blob will own the memory too. Here's a test case with FederatedX: ============================================================= source include/have_partition.inc; source include/have_sequence.inc; source have_federatedx.inc; source include/federated.inc; connection slave; use federated; create table t1_1 (x int, b text, key(x)); create table t1_2 (x int, b text, key(x)); connection master; eval create table t1 (x int, b text, key(x)) engine=federated partition by range columns (x) ( partition p1 values less than (40) connection='mysql://root@127.0.0.1:$SLAVE_MYPORT/federated/t1_1', partition pn values less than (maxvalue) connection='mysql://root@127.0.0.1:$SLAVE_MYPORT/federated/t1_2' ); insert t1 select seq, seq from seq_1_to_35; insert t1 select seq, repeat(seq, 100) from seq_50_to_90; flush tables; select x, b from t1 where x > 30 and x < 60 order by x; select x, b from t1 where x > 30 and x < 60 order by b; drop table t1; connection slave; drop table t1_1, t1_2; source include/federated_cleanup.inc; ============================================================= first select doesn't trigger ASAN error, but it produces incorrect results (I'm showing it here, so that you could create a non-ASAN test that could be run on all builders). the second select gets the same ASAN error as your test below. > diff --git a/mysql-test/suite/vcol/r/partition.result b/mysql-test/suite/vcol/r/partition.result > index bd1353fa145..74c3c3394dc 100644 > --- a/mysql-test/suite/vcol/r/partition.result > +++ b/mysql-test/suite/vcol/r/partition.result > @@ -28,3 +28,26 @@ set statement sql_mode= '' for update t1 set i= 1, v= 2; > Warnings: > Warning 1906 The value specified for generated column 'v' in table 't1' has been ignored > drop table t1; > +# > +# MDEV-18734 ASAN heap-use-after-free in my_strnxfrm_simple_internal upon update on versioned partitioned table > +# > +# Cover queue_fix() in ha_partition::handle_ordered_index_scan() > +create table t1 ( > +x int auto_increment primary key, > +b text, v mediumtext as (b) virtual, > +index (v(10)) > +) partition by range columns (x) ( > +partition p1 values less than (3), > +partition p2 values less than (6), > +partition pn values less than (maxvalue)); > +insert into t1 (b) values ('q'), ('a'), ('b'); > +update t1 set b= 'bar' where v > 'a'; > +drop table t1; this test above doesn't cause ASAN failures for me without your fix. the test below fails all right. > +# Cover return_top_record() in ha_partition::handle_ordered_index_scan() > +create table t1 (x int primary key, b tinytext, v text as (b) virtual) > +partition by range columns (x) ( > +partition p1 values less than (4), > +partition pn values less than (maxvalue)); > +insert into t1 (x, b) values (1, ''), (2, ''), (3, 'a'), (4, 'b'); > +update t1 set b= 'bar' where x > 0 order by v limit 2; > +drop table t1; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 3

Re: [Maria-developers] 7807d9b6939: MDEV-22974: mysql_native_password make "invalid" valid
by Sergei Golubchik 30 Oct '20

30 Oct '20

Hi, Daniel! On Oct 22, Daniel Black wrote: > revision-id: 7807d9b6939 (mariadb-10.5.4-62-g7807d9b6939) > parent(s): 054f10365c4 > author: Daniel Black <daniel(a)mariadb.org> > committer: Daniel Black <daniel(a)mariadb.org> > timestamp: 2020-07-16 17:06:36 +1000 > message: > > MDEV-22974: mysql_native_password make "invalid" valid > > Per b9f3f06857ac, mysql_system_tables_data.sql creates > a mysql_native_password with a salted hash of "invalid" so that `set password` > will detect a native password can be pplied:. "applied" > > SHOW CREATE USER; dilignently uses this value in its output > generating the SQL: > > MariaDB [(none)]> show create user; > > +---------------------------------------------------------------------------------------------------+ > | CREATE USER for dan@localhost | > +---------------------------------------------------------------------------------------------------+ > | CREATE USER `dan`@`localhost` IDENTIFIED VIA mysql_native_password USING 'invalid' OR unix_socket | > +---------------------------------------------------------------------------------------------------+ > > Attempting to execute this before this patch resutls in: > > MariaDB [(none)]> CREATE USER `dan2`@`localhost` IDENTIFIED VIA mysql_native_password USING 'invalid' OR unix_socket; > ERROR 1372 (HY000): Password hash should be a 41-digit hexadecimal number > > As such deep the the implementation of mysql_native_password we make > "invalid" valid (pun intended) such that the above create user will > succeed. > > native_password_get_salt is only called in the context of > set_user_salt, so all setting of native passwords to hashed content of > 'invalid', quite literally create an invalid password. > > So other forms of "invalid" are valid SQL in creating invalid passwords: > > MariaDB [(none)]> set password = 'invalid'; > Query OK, 0 rows affected (0.001 sec) > > MariaDB [(none)]> alter user dan@localhost IDENTIFIED BY PASSWORD 'invalid'; > Query OK, 0 rows affected (0.000 sec) > > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index d94016b7815..3cd7a67ae1a 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -14539,6 +14539,12 @@ static int native_password_get_salt(const char *hash, size_t hash_length, > > if (hash_length != SCRAMBLED_PASSWORD_CHAR_LENGTH) > { > + if (hash_length == 7 && strcmp(hash, "invalid") == 0) > + { > + memcpy(out, "invalid", 7); > + *out_length= 7; > + return 0; > + } okay. After you said ASAN, I think I can see why this could be problematic. You can, of course, pad it to SCRAMBLED_PASSWORD_CHAR_LENGTH, but then you'll create a *valid* scramble that would correspond to some actual password. One option would be to allow "invalid" literal in set_user_auth(), before even any plugin checks. But I'm unsure of the implications. > my_error(ER_PASSWD_LENGTH, MYF(0), SCRAMBLED_PASSWORD_CHAR_LENGTH); > return 1; > } > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 3

Re: [Maria-developers] f112c4a07475: MDEV-20787: Script dgcov.pl does not work
by Sergei Golubchik 30 Oct '20

30 Oct '20

Hi, Anel! On Oct 30, Anel Husakovic wrote: > revision-id: f112c4a07475 (mariadb-10.2.31-541-gf112c4a07475) > parent(s): 784473b98662 > author: Anel Husakovic <anel(a)mariadb.org> > committer: Anel Husakovic <anel(a)mariadb.org> > timestamp: 2020-10-26 14:25:41 +0100 > message: > > MDEV-20787: Script dgcov.pl does not work > > - Patch is changing CMakeList with `--coverage` flag as an alias for > `-fprofile-arcs -ftest-coverage -lgcov` in addition. > > - About the usage of `dgcov`: > When the server is compiled with `-DENABLE_GCOV=ON`, from object files are generated > `.gcno` and `.gcda` files. > `./mtr --gcov is_check_constraint` is invoking the following script calls: > - `./dgcov.pl --purge`, `./mtr is_check_constraint` and > - `./dgcov.pl --generate>/var/last_changes.dgcov`. > The `purge` flag is clearing `.gcda` files (and others extensions), > while running the test (which follows after the purge) new `.gcda` files are generated. > With dgcov `generate` flag, `gcov -i` (`intermediate format`) is called > on generated `<source-files-name>.gcda` files (`dbug.c.gcda` e.g.). > The patch is tested on `gcov 6.3` and `gcov 7.4` versions > and can be seen that resulting `.gcov` file for `6.3` creates > `<full path>.gcov` (`dbug.c.gcda.gcov` e.g.) file, > where `gcov 7.4+` is still creating `source-file-names.gcov`(`dbug.c.gcov`) files > as `gcov` in general is doing. > Results are stored in `./mysql-test/var/last_changes.dgcov`. > Note: for the newer versions of `gcov` intermediate format is > deprecated `-i` and should be used `--json-format`. does it mean that eventually we'll have to rewrite dgcov to support json format? > > - This patch doesn't take gcov versions in account since is extracting > filename of a `.gcov` file, which is generated with `gcov -i > <source-file>.gcda` during `dgcov.pl --generate`. > > - There is a check if file exists. > Reason for that is because some generated `gcov` files are outliers from above > rule (that a file generated in intermediate format(gcov) has the same source name). > Example 1: `aestables.cpp.gcda` is not generating `aestables.cpp.gcov`, > but only the files used in #include `modes.hpp.gcov runtime.hpp.gcov` > Example 2: `my_sha256.cc.gcda` is generating `my_sha.ic.gcov` > > - Gcov with out-of-source is not working, this patch will solve that > (probably?) > Make sure to test `MTR_BINDIR` which gets set during out-of-source > build and generate an error when running `./mysql-test/mtr --gcov alias`. > > diff --git a/mysql-test/dgcov.pl b/mysql-test/dgcov.pl > index fbc5540e6973..f2bbdad4d20e 100755 > --- a/mysql-test/dgcov.pl > +++ b/mysql-test/dgcov.pl > @@ -161,8 +161,14 @@ sub gcov_one_file { > system($cmd)==0 or die "system($cmd): $? $!"; > } I don't see any changes since the last review. Is it the correct commit? > - # now, read the generated file > - open FH, '<', "$_.gcov" or die "open(<$_.gcov): $!"; > + (my $filename = $_)=~ s/\.[^.]+$//; # remove extension I think this contradicts your explanation in the commit comment. You wrote that the file name can be either dbug.c.gcda.gcov or dbug.c.gcov. Here $_ is dbug.c.gcda, you unconditionally remove .gcda and then below you open $filename.gcov, that is dbug.c.gcov Which means this won't work with an older gcov. Is this your intention? I think it's fine not to support old gcov. But this should be a conscious decision, e.g. you can write in the commit comment "Now dgcov.pl only supports gcov 7.4+" - if that's indeed what you want. Also, if you do it this way, then there's no need to use s///, it's simpler to do, like sub gcov_one_file { return unless /\.gcda$/; + my $filename= $`; unless ($opt_skip_gcov) { or, may be even sub gcov_one_file { return unless /\.gcda$/; + my $gcov_file_path= "$File::Find::dir$`.gcov"; unless ($opt_skip_gcov) { > + my $gcov_file_path= $File::Find::dir."/$filename.gcov"; still an extra slash > + if (! -f $gcov_file_path) > + { > + return; > + } still not return unless -f $gcov_file_path; > + open FH, '<', "$gcov_file_path" or die "open(<$gcov_file_path): $!"; still extra quotes > + > my $fname; > while (<FH>) { > chomp; > diff --git a/mysql-test/mysql-test-run.pl b/mysql-test/mysql-test-run.pl > index 900ef736a455..2189529d82a7 100755 > --- a/mysql-test/mysql-test-run.pl > +++ b/mysql-test/mysql-test-run.pl > @@ -1791,7 +1791,7 @@ sub command_line_setup { > # -------------------------------------------------------------------------- > # Gcov flag > # -------------------------------------------------------------------------- > - if ( ($opt_gcov or $opt_gprof) and ! $source_dist ) > + if ( ($opt_gcov or $opt_gprof) and (! $source_dist or -d $ENV{MTR_BINDIR})) > { > mtr_error("Coverage test needs the source - please use source dist"); > } > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] MDEV-24015: SQL Error (1038): Out of sort memory when enough memory for the sort buffer is provided
by Sergey Petrunia 27 Oct '20

27 Oct '20

Hi Varun, I have only cosmetic comments, please find them below. Ok to push after they are addressed. > The issue here was with the re-execution of a subquey, after few executions > the subquery execution returned error that not enough space was provided inside the sort > buffer. > This is happening because with the introduction of packed sort keys, during the phase > where sort length is calculated the parameters of the sort keys are not reset and > we keep up adding the sort length to the previous calculated sort length for the > earlier executions of the subquery. This happens because the sort keys structure is > allocated only once and so if their is re-execution of a subquery involving sorting > then we should be resetting the parameters of the sort keys. The comment is hard to read. How about being more specific and using shorter sentences: The error was: filesort was done in a subquery which was executed multiple times. During each execution, sortlength() computed total sort key length in Sort_keys::sort_length, without resetting it first. Eventually Sort_keys::sort_length got larger than @@sort_buffer_size, which caused filesort() to be aborted with error. Fixed by making sortlength() to compute lengths only during the first invocation. Subsequent invocations return pre-computed values. > @@ -2518,7 +2526,7 @@ void Sort_param::try_to_pack_sortkeys() > return; > > const uint sz= Sort_keys::size_of_length_field; > - uint sort_len= sort_keys->get_sort_length(); > + uint sort_len= sort_keys->get_sort_length_with_original_values(); > > /* > Heuristic introduced, skip packing sort keys if saving less than 128 bytes > diff --git a/sql/sql_sort.h b/sql/sql_sort.h > index 40f0c5ede5f..6951d4d190a 100644 > --- a/sql/sql_sort.h > +++ b/sql/sql_sort.h > @@ -258,7 +258,9 @@ class Sort_keys :public Sql_alloc, > Sort_keys_array(arr, count), > m_using_packed_sortkeys(false), > size_of_packable_fields(0), > - sort_length(0) > + sort_length_with_original_values(0), > + sort_length_with_memcmp_values(0), > + first_execution(true) This violates the coding style. Please ident the member initialization (see e.g. item.h for examples) > { > DBUG_ASSERT(!is_null()); > }a ... > @@ -307,9 +319,12 @@ class Sort_keys :public Sql_alloc, > > void increment_original_sort_length(uint len) > { > - sort_length+= len; > + sort_length_with_original_values+= len; > } > > + bool is_first_execution() { return first_execution; } > + void set_first_execution(bool val) { first_execution= val; } A comment just about the naming: I would not drag "execution" into here. "Sort_keys" are not really executed. How about changing the functions to be bool is_parameters_computed(); bool set_parameters_computed(); and the variable to be "bool parameters_computed" ? > + > static const uint size_of_length_field= 4; > > private: BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] 5150dfd6ab3: MDEV-17891 Assertion failures in select_insert::abort_result_set and mysql_load upon attempt to replace into a full table
by Sergei Golubchik 25 Oct '20

25 Oct '20

Hi, Nikita! On Apr 27, Nikita Malyavin wrote: > revision-id: 5150dfd6ab3 (mariadb-10.3.12-230-g5150dfd6ab3) > parent(s): cf78b8c699d > author: Nikita Malyavin <nikitamalyavin(a)gmail.com> > committer: Nikita Malyavin <nikitamalyavin(a)gmail.com> > timestamp: 2019-06-12 22:11:45 +1000 > message: > > MDEV-17891 Assertion failures in select_insert::abort_result_set and mysql_load upon attempt to replace into a full table > > * set modified_non_trans_table in one missed place This commit needs way more text than that. This is just too misterious now. What asserts fails? What has the test case to do with the fix? Why modified_non_trans_table even matters without replication? Why it's specific to versioning and partitioning? Where trans_safe becomes false? > diff --git a/mysql-test/suite/versioning/r/partition.result b/mysql-test/suite/versioning/r/partition.result > index 3fcb59bdb40..99a297417d5 100644 > --- a/mysql-test/suite/versioning/r/partition.result > +++ b/mysql-test/suite/versioning/r/partition.result > @@ -541,6 +541,29 @@ t1 CREATE TABLE `t1` ( > PARTITION BY SYSTEM_TIME INTERVAL 7 SECOND > (PARTITION `ver_p1` HISTORY ENGINE = DEFAULT_ENGINE, > PARTITION `ver_pn` CURRENT ENGINE = DEFAULT_ENGINE) > +set timestamp= default; > +# MDEV-17891 Assertion failures in select_insert::abort_result_set and > +# mysql_load upon attempt to replace into a full table > +set @@max_heap_table_size= 1024*1024; > +create or replace table t1 ( > +pk integer auto_increment, > +primary key (pk), > +f varchar(45000) > +) with system versioning engine=memory > +partition by system_time interval 1 year (partition p1 history, > +partition pn current); > +# fill the table until full > +insert into t1 () values (),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(); > +insert into t1 (f) select f from t1; > +ERROR HY000: The table 't1' is full > +# leave space for exactly one record in current partition > +delete from t1 where pk = 1; > +# copy all data into history partition > +replace into t1 select * from t1; > +replace into t1 select * from t1; > +ERROR HY000: The table 't1' is full > +drop table t1; > +set @@max_heap_table_size= 1048576; > # Test cleanup > drop database test; > create database test; > diff --git a/mysql-test/suite/versioning/t/partition.test b/mysql-test/suite/versioning/t/partition.test > index d5c83b4d3bb..23768836efc 100644 > --- a/mysql-test/suite/versioning/t/partition.test > +++ b/mysql-test/suite/versioning/t/partition.test > @@ -474,6 +474,7 @@ set timestamp=1523466002.799571; > insert into t1 values (11),(12); > set timestamp=1523466004.169435; > delete from t1 where pk in (11, 12); > +set timestamp= default; > > --echo # > --echo # MDEV-18136 Server crashes in Item_func_dyncol_create::prepare_arguments > @@ -489,6 +490,34 @@ partition by system_time interval column_get(column_create(7,7), 7 as int) secon > --replace_result $default_engine DEFAULT_ENGINE > show create table t1; > > +--echo # MDEV-17891 Assertion failures in select_insert::abort_result_set and > +--echo # mysql_load upon attempt to replace into a full table > + > +--let $max_heap_table_size_orig= `select @@max_heap_table_size;` > +set @@max_heap_table_size= 1024*1024; > +create or replace table t1 ( > + pk integer auto_increment, > + primary key (pk), > + f varchar(45000) > +) with system versioning engine=memory > + partition by system_time interval 1 year (partition p1 history, > + partition pn current); > + > +--echo # fill the table until full > +insert into t1 () values (),(),(),(),(),(),(),(),(),(),(),(),(),(),(),(); > +--error ER_RECORD_FILE_FULL > +insert into t1 (f) select f from t1; > +--echo # leave space for exactly one record in current partition > +delete from t1 where pk = 1; > +--echo # copy all data into history partition > +replace into t1 select * from t1; > +--error ER_RECORD_FILE_FULL > +replace into t1 select * from t1; > + > +# cleanup > +drop table t1; > +eval set @@max_heap_table_size= $max_heap_table_size_orig; > + > --echo # Test cleanup > drop database test; > create database test; > diff --git a/sql/sql_insert.cc b/sql/sql_insert.cc > index 1f3a70721fc..44502ea6704 100644 > --- a/sql/sql_insert.cc > +++ b/sql/sql_insert.cc > @@ -1953,6 +1953,8 @@ int write_record(THD *thd, TABLE *table,COPY_INFO *info) > if (likely(!error)) > { > info->deleted++; > + if (!table->file->has_transactions()) > + thd->transaction.stmt.modified_non_trans_table= TRUE; > if (table->versioned(VERS_TIMESTAMP)) > { > store_record(table, record[2]); Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 05968211699: MDEV-16026: Global system_versioning_asof must not be used if client sessions can have non-default time zone
by Sergei Golubchik 25 Oct '20

25 Oct '20

Hi, Nikita! Okay. This reverts my "ok to push" on MDEV-16481 :) If you want to store the value in unix timestamp, you don't need MYSQL_TIME in save_result. You should calculate the value as a unix timestamp in ::check and store that in save_result. Now you still have timezone conversion in ::update and that can fail. Everything that can fail should be done in ::check, that's the contract. May be also I reapplied your patches to 10.3 incorrectly when I was reviewing. Perhaps you could rebase them yourself - to be sure I'm looking at the correct patch? On Oct 22, Nikita Malyavin wrote: > revision-id: 05968211699 (mariadb-10.4.11-291-g05968211699) > parent(s): efb1023b6f4 > author: Nikita Malyavin <nikitamalyavin(a)gmail.com> > committer: Sergei Golubchik <serg(a)mariadb.com> > timestamp: 2020-10-22 17:07:03 +0200 > message: > > MDEV-16026: Global system_versioning_asof must not be used if client sessions can have non-default time zone > > * store `system_versioning_asof` in unix time; > * both session and global vars are processed in session timezone; > * setting `default` does not copy global varibale abymore. Instead, it sets system_time to SYSTEM_TIME_UNSPECIFIED, which means that no 'AS OF' time is applied and `now()` can be assumed > As a regression, we cannot assign values below 1970 anymore > > --- > mysql-test/suite/versioning/r/sysvars.result | 67 +++++++++++++++++++++++ > mysql-test/suite/versioning/t/sysvars.test | 80 ++++++++++++++++++++++++---- > sql/mysqld.h | 3 +- > sql/sql_select.cc | 6 ++- > sql/sys_vars.ic | 27 ++++++---- > 5 files changed, 162 insertions(+), 21 deletions(-) > > diff --git a/sql/mysqld.h b/sql/mysqld.h > index bd45ff7b798..884e5a06066 100644 > --- a/sql/mysqld.h > +++ b/sql/mysqld.h > @@ -194,7 +194,8 @@ enum vers_system_time_t > struct vers_asof_timestamp_t > { > ulong type; > - MYSQL_TIME ltime; > + my_time_t unix_time; > + ulong second_part; > }; > > enum vers_alter_history_enum > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index d0bb0c816ec..4d62a9829eb 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -722,8 +722,12 @@ bool vers_select_conds_t::init_from_sysvar(THD *thd) > if (type != SYSTEM_TIME_UNSPECIFIED && type != SYSTEM_TIME_ALL) > { > DBUG_ASSERT(type == SYSTEM_TIME_AS_OF); > + MYSQL_TIME ltime; > + thd->variables.time_zone->gmt_sec_to_TIME(&ltime, in.unix_time); > + ltime.second_part = in.second_part; > + > start.item= new (thd->mem_root) > - Item_datetime_literal(thd, &in.ltime, TIME_SECOND_PART_DIGITS); > + Item_datetime_literal(thd, &ltime, TIME_SECOND_PART_DIGITS); > if (!start.item) > return true; > } > diff --git a/sql/sys_vars.ic b/sql/sys_vars.ic > index f33f469b160..417fa7842c2 100644 > --- a/sql/sys_vars.ic > +++ b/sql/sys_vars.ic > @@ -2646,9 +2646,11 @@ class Sys_var_vers_asof: public Sys_var_enum > } > > private: > - bool update(set_var *var, vers_asof_timestamp_t &out) > + bool update(set_var *var, vers_asof_timestamp_t &out, system_variables& pool) > { > bool res= false; > + uint error; > + MYSQL_TIME ltime; > out.type= static_cast<enum_var_type>(var->save_result.ulonglong_value); > if (out.type == SYSTEM_TIME_AS_OF) > { > @@ -2658,11 +2660,14 @@ class Sys_var_vers_asof: public Sys_var_enum > Datetime::Options opt(TIME_CONV_NONE | > TIME_NO_ZERO_IN_DATE | > TIME_NO_ZERO_DATE, thd); > - res= var->value->get_date(thd, &out.ltime, opt); > + res= var->value->get_date(thd, &ltime, opt); > + out.unix_time= pool.time_zone->TIME_to_gmt_sec(&ltime, &error); > + out.second_part= ltime.second_part; > + res|= (error != 0); > } > else // set DEFAULT from global var > { > - out= global_var(vers_asof_timestamp_t); > + out.type= SYSTEM_TIME_UNSPECIFIED; > res= false; > } > } > @@ -2672,11 +2677,11 @@ class Sys_var_vers_asof: public Sys_var_enum > public: > virtual bool global_update(THD *thd, set_var *var) > { > - return update(var, global_var(vers_asof_timestamp_t)); > + return update(var, global_var(vers_asof_timestamp_t), thd->variables); > } > virtual bool session_update(THD *thd, set_var *var) > { > - return update(var, session_var(thd, vers_asof_timestamp_t)); > + return update(var, session_var(thd, vers_asof_timestamp_t), thd->variables); > } > > private: > @@ -2690,10 +2695,14 @@ class Sys_var_vers_asof: public Sys_var_enum > case SYSTEM_TIME_AS_OF: > { > uchar *buf= (uchar*) thd->alloc(MAX_DATE_STRING_REP_LENGTH); > - if (buf &&!my_datetime_to_str(&val.ltime, (char*) buf, 6)) > + MYSQL_TIME ltime; > + > + thd->variables.time_zone->gmt_sec_to_TIME(&ltime, val.unix_time); > + ltime.second_part= val.second_part; > + > + if (buf && !my_datetime_to_str(&ltime, (char*) buf, 6)) > { > - // TODO: figure out variable name > - my_error(ER_WRONG_VALUE_FOR_VAR, MYF(0), "system_versioning_asof_timestamp", "NULL (wrong datetime)"); > + my_error(ER_WRONG_VALUE_FOR_VAR, MYF(0), name.str, "NULL (wrong datetime)"); > return (uchar*) thd->strdup("Error: wrong datetime"); > } > return buf; > @@ -2701,7 +2710,7 @@ class Sys_var_vers_asof: public Sys_var_enum > default: > break; > } > - my_error(ER_WRONG_VALUE_FOR_VAR, MYF(0), "system_versioning_asof_timestamp", "NULL (wrong range type)"); > + my_error(ER_WRONG_VALUE_FOR_VAR, MYF(0), name.str, "NULL (wrong range type)"); > return (uchar*) thd->strdup("Error: wrong range type"); > } > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] caec6b4b3d7: MDEV-17573 Assertion in federatedx on multi-update
by Sergei Golubchik 23 Oct '20

23 Oct '20

Hi, Aleksey! Looks reasonable to me. MYSQL_RES is stored in the handler. Of course, MYSQL_ROWS *current, which is a pointer into MYSQL_RES, has to be in the handler, it cannot be shared. Ok to push, thanks! On Oct 23, Aleksey Midenkov wrote: > revision-id: caec6b4b3d7 (mariadb-10.3.21-250-gcaec6b4b3d7) > parent(s): a00c668ddf2 > author: Aleksey Midenkov <midenok(a)gmail.com> > committer: Aleksey Midenkov <midenok(a)gmail.com> > timestamp: 2020-10-21 13:38:37 +0300 > message: > > MDEV-17573 Assertion in federatedx on multi-update > > Cause: shared federatedx_io cannot store table-specific data. > > Fix: move current row reference `federatedx_io_mysql::current` to > ha_federatedx. > > FederatedX connection (represented by federatedx_io) is stored into > federatedx_txn::txn_list of per-server connections (see > federatedx_txn::acquire()). federatedx_txn object is stored into THD > (see ha_federatedx::external_lock()). When multiple handlers acquire > FederatedX connection they get single federatedx_io instance. Multiple > handlers do their operation via federatedx_io_mysql::mark_position() > and federatedx_io_mysql::fetch_row() in arbitrarty manner. They access > the same federatedx_io_mysql instance and same MYSQL_ROWS *current > pointer, so one handler disrupts the work of the other. > > Related to "MDEV-14551 Can't find record in table on multi-table update > with ORDER BY". Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 2e0bac4688a: MDEV-22313: Neither SHOW CREATE USER nor SHOW GRANTS prints a user's default role
by Sergei Golubchik 23 Oct '20

23 Oct '20

Hi, Anel! On Oct 23, Anel Husakovic wrote: > revision-id: 2e0bac4688a (mariadb-10.2.31-517-g2e0bac4688a) > parent(s): 985ede92034 > author: Anel Husakovic <anel(a)mariadb.org> > committer: Anel Husakovic <anel(a)mariadb.org> > timestamp: 2020-10-23 12:38:11 +0200 > message: > > MDEV-22313: Neither SHOW CREATE USER nor SHOW GRANTS prints a user's default role > > diff --git a/mysql-test/r/grant5.result b/mysql-test/r/grant5.result > index 338bbd36e14..a69070e8399 100644 > --- a/mysql-test/r/grant5.result > +++ b/mysql-test/r/grant5.result ... > +SHOW GRANTS; > +Grants for root@localhost > +GRANT test_role TO 'root'@'localhost' WITH ADMIN OPTION > +GRANT ALL PRIVILEGES ON *.* TO 'root'@'localhost' WITH GRANT OPTION > +GRANT PROXY ON ''@'%' TO 'root'@'localhost' WITH GRANT OPTION > +GRANT USAGE ON *.* TO 'test_role' > +SET DEFAULT ROLE test_role FOR 'root'@'localhost' ... Yes, that's what I mean. I don't have a strong opinion on that, it seems that I very slightly prefer to have SET DEFAULT ROLE after GRANT PROXY, that is to have all statements for root, and then statements for roles. But having SET DEFAULT ROLE at the end makes sense too - all GRANTs first, then SET DEFAULT ROLE. Just make it the way you want. But either way it's good to have a test for it, a test that shows we've considered this issue and intentionally implemented that particular order. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 6d881a271a9: MDEV-23178: Qualified asterisk not supported in INSERT .. RETURNING
by Sergei Golubchik 23 Oct '20

23 Oct '20

Hi, Rucha! On Oct 23, Rucha Deodhar wrote: > revision-id: 6d881a271a9 (mariadb-10.5.4-108-g6d881a271a9) > parent(s): bbd70fcc43c > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2020-10-23 12:57:02 +0530 > message: > > MDEV-23178: Qualified asterisk not supported in INSERT .. RETURNING > > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > index 45ce4be3eb5..4ce7b034029 100644 > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -7959,7 +7959,7 @@ insert_fields(THD *thd, Name_resolution_context *context, const char *db_name, > bool any_privileges, uint *hidden_bit_fields) > { > Field_iterator_table_ref field_iterator; > - bool found; > + bool found, is_insert_or_replace_returning= false; > char name_buff[SAFE_NAME_LEN+1]; > DBUG_ENTER("insert_fields"); > DBUG_PRINT("arena", ("stmt arena: %p",thd->stmt_arena)); > @@ -7978,13 +7978,31 @@ insert_fields(THD *thd, Name_resolution_context *context, const char *db_name, > > found= FALSE; > > + if ((thd->lex->sql_command == SQLCOM_INSERT_SELECT || > + thd->lex->sql_command == SQLCOM_REPLACE_SELECT || > + thd->lex->sql_command == SQLCOM_INSERT || > + thd->lex->sql_command == SQLCOM_REPLACE) && > + thd->lex->has_returning()) > + is_insert_or_replace_returning= true; > + > /* > If table names are qualified, then loop over all tables used in the query, > else treat natural joins as leaves and do not iterate over their underlying > tables. > + Also, When we have INSERT/REPLACE...RETURNING and have qualified asterisk, > + table_name is not NULL and context->table_list is either NULL or has > + incorrect reference because context->table_list has tables from the > + FROM clause. > + context->table_list has incorrect reference (has table from the FROM clause > + instead of table we are inserting into) for > + INSERT/REPLACE...SELECT...RETURNING because we have a FROM clause from the > + SELECT statement. For INSERT/REPLACE...RETURNING it is NULL because > + there is no FROM clause. > */ > - for (TABLE_LIST *tables= (table_name ? context->table_list : > - context->first_name_resolution_table); > + for (TABLE_LIST *tables= (table_name ? (is_insert_or_replace_returning ? > + context->first_name_resolution_table : > + context->table_list) : > + (context->first_name_resolution_table)); ok, I've compiled it to run few queries, now I see that this is wrong. you need to understand the old code and the old comment before modifying them. one way to do it is to change the condition from table_name ? context->table_list : context->first_name_resolution_table to simply context->first_name_resolution_table and see what breaks. You'll see that main.type_varchar will break with At line 319: query 'SELECT t1.* FROM t1 LEFT JOIN t2 USING (c1)' failed: 1051: Unknown table 'test.t1' indeed, t1 is only present in context->table_list, but not in context->first_name_resolution_table. So after your fix the following statement will no longer work: INSERT INTO t2 SELECT t1.* FROM t1 JOIN t1 AS t USING (id1) WHERE id1=1 RETURNING *; it's because you set is_insert_or_replace_returning both for the SELECT and for the RETURNING clauses. > tables; > tables= (table_name ? tables->next_local : > tables->next_name_resolution_table) also this looks suspicious. Normally one starts from table_list and follows the next_local pointer or one starts from first_name_resolution_table and follows the next_name_resolution_table pointer. Your change creates a strange mix of both, it looks like it would be wrong. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 581857bd999: MDEV-22313: Neither SHOW CREATE USER nor SHOW GRANTS prints a user's default role
by Sergei Golubchik 23 Oct '20

23 Oct '20

Hi, Anel! Looks good, thanks! A couple of small comments below. On Oct 23, Anel Husakovic wrote: > revision-id: 581857bd999 (mariadb-10.1.43-314-g581857bd999) let's fix it in 10.2 and up, we're doing the very last 10.1 release, so better be extra-super-conservative about it. > parent(s): 43ec9370b32 > author: Anel Husakovic <anel(a)mariadb.org> > committer: Anel Husakovic <anel(a)mariadb.org> > timestamp: 2020-10-23 00:20:53 +0200 > message: > > MDEV-22313: Neither SHOW CREATE USER nor SHOW GRANTS prints a user's default role > > diff --git a/mysql-test/t/grant5.test b/mysql-test/t/grant5.test > index 74a69952124..bf97a930097 100644 > --- a/mysql-test/t/grant5.test > +++ b/mysql-test/t/grant5.test > @@ -52,6 +52,23 @@ disconnect u1; > drop user u1@localhost; > drop database mysqltest1; > > +# > +# MDEV-22313: Neither SHOW CREATE USER nor SHOW GRANTS prints a user's default role > +# > +CREATE ROLE test_role; > +CREATE USER test_user; > +GRANT test_role TO test_user; > +SET DEFAULT ROLE test_role FOR test_user; > +SHOW GRANTS FOR test_user; > +SET DEFAULT ROLE NONE for test_user; > +SHOW GRANTS FOR test_user; > +SET DEFAULT ROLE test_role; oops, sorry, my mistake. Please, add `SET ROLE test_role` before SHOW GRANTS. The goal is to see that SET DEFAULT ROLE is shown *after* all grants to root@localhost, but before any grants to test_role. > +SHOW GRANTS; > +SET DEFAULT ROLE NONE; > +SHOW GRANTS; > +DROP USER test_user; > +DROP ROLE test_role; > + > # > # End of 10.1 tests > # > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index cf0b1d87bd7..6c852976061 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -8001,15 +8007,44 @@ static ROLE_GRANT_PAIR *find_role_grant_pair(const LEX_STRING *u, > my_hash_search(&acl_roles_mappings, (uchar*)pair_key.ptr(), key_length); > } > > +static bool show_default_role(THD *thd, const char *hostname, > + ACL_USER *acl_entry, char *buff, size_t buffsize) > +{ > + Protocol *protocol= thd->protocol; > + LEX_STRING def_rolename= acl_entry->default_rolename; > + > + if (def_rolename.length) > + { > + String def_str(buff, buffsize, system_charset_info); > + def_str.length(0); > + def_str.append(STRING_WITH_LEN("SET DEFAULT ROLE ")); > + def_str.append(&def_rolename); > + def_str.append(" FOR '"); > + def_str.append(&acl_entry->user); > + DBUG_ASSERT(!(acl_entry->flags & IS_ROLE)); > + def_str.append(STRING_WITH_LEN("'@'")); > + def_str.append(acl_entry->host.hostname, acl_entry->hostname_length, > + system_charset_info); > + def_str.append('\''); > + protocol->prepare_for_resend(); > + protocol->store(def_str.ptr(),def_str.length(),def_str.charset()); > + if (protocol->write()) > + { > + return TRUE; > + } > + } > + return FALSE; looks good. The hostname argument seems to be unused now, doesn't it? > +} > + > -static bool show_role_grants(THD *thd, const char *username, > - const char *hostname, ACL_USER_BASE *acl_entry, > +static bool show_role_grants(THD *thd, const char *hostname, > + ACL_USER_BASE *acl_entry, > char *buff, size_t buffsize) > { > uint counter; > Protocol *protocol= thd->protocol; > LEX_STRING host= {const_cast<char*>(hostname), strlen(hostname)}; > > - String grant(buff,sizeof(buff),system_charset_info); > + String grant(buff, buffsize, system_charset_info); ouch, indeed. thanks! > for (counter= 0; counter < acl_entry->role_grants.elements; counter++) > { > grant.length(0); Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 01612f594b3: MDEV-23769: MTR can abort before it prints the test result summary
by Sergei Golubchik 22 Oct '20

22 Oct '20

Hi, Anel! On Oct 22, Anel Husakovic wrote: > Hi Sergei, > thanks for the review. > > Having warning/error is a necessary condition. > Sufficient condition is to call at least 2 tests. > Incorrect behaviour can be seen if: > - Both tests are correct but there is error generated (incorrect option > passed): > So this `./mysql-test/mtr --mysqld=--lock-wait-timeout=-xx 1st` will > generate and write statistic at the end, > but this `./mysql-test/mtr --mysqld=--lock-wait-timeout=-xx 1st 1st > --force` will not generate the statistic. > Note: warning is generated (--mysqld=--lock-wait-timeout=-1) statistic > will be generated Okay, thanks. This would be useful to show in the commit comment, that one can use ./mtr --mysqld=--lock-wait-timeout=-xx 1st 1st --force --parallel 1 to repeat the error. But then I don't understand why this happens. Why one failure is not enough? Why --parallel 2 destroys the effect? Apparently, it's check_warnings_post_shutdown() that causes that, not exit(1). One can change to exit(0) or remove exit(1) completely - the bug will still happen. Why check_warnings_post_shutdown() writing to the master server causes this bug? It looks rather fishy that in your fix it's the worker process that prints the statistics, not the master. Btw, note that check_warnings_post_shutdown() doesn't have any return statement, it very implicitly return "number of bytes sent" - it would be easier to read if it'd return something explicitly. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 99c14476251: MDEV-22313: Neither SHOW CREATE USER nor SHOW GRANTS prints a user's default role
by Sergei Golubchik 22 Oct '20

22 Oct '20

Hi, Anel! On Oct 22, Anel Husakovic wrote: > revision-id: 99c14476251 (mariadb-10.1.43-314-g99c14476251) > parent(s): 43ec9370b32 > author: Anel Husakovic <anel(a)mariadb.org> > committer: Anel Husakovic <anel(a)mariadb.org> > timestamp: 2020-10-22 16:49:28 +0200 > message: > > MDEV-22313: Neither SHOW CREATE USER nor SHOW GRANTS prints a user's default role > > diff --git a/mysql-test/t/grant5.test b/mysql-test/t/grant5.test > index 74a69952124..8756648c3e1 100644 > --- a/mysql-test/t/grant5.test > +++ b/mysql-test/t/grant5.test > @@ -53,5 +53,19 @@ drop user u1@localhost; > drop database mysqltest1; > > # > -# End of 10.1 tests > +# MDEV-22313: Neither SHOW CREATE USER nor SHOW GRANTS prints a user's default role > +# > +CREATE ROLE test_role; > +CREATE USER test_user; > +GRANT test_role TO test_user; > +SET DEFAULT ROLE test_role FOR test_user; > +SHOW GRANTS FOR test_user; > +SET DEFAULT ROLE NONE for test_user; > +SHOW GRANTS FOR test_user; > +SHOW GRANTS; The point is to run `SHOW GRANTS` for a user with a default role. E.g. SET DEFAULT ROLE test_role; SHOW GRANTS; SET DEFAULT ROLE NONE; > +DROP USER test_user; > +DROP ROLE test_role; > + > # > +# End of 10.1 tests > +# > \ No newline at end of file new line at the end of file? > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index cf0b1d87bd7..f42f7bdcfaf 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -8001,6 +8007,36 @@ static ROLE_GRANT_PAIR *find_role_grant_pair(const LEX_STRING *u, > my_hash_search(&acl_roles_mappings, (uchar*)pair_key.ptr(), key_length); > } > > +static bool show_default_role(THD *thd, const char *hostname, > + ACL_USER *acl_entry, char *buff) > +{ > + Protocol *protocol= thd->protocol; > + LEX_STRING host= {const_cast<char*>(hostname), strlen(hostname)}; > + > + LEX_STRING def_rolename= acl_entry->default_rolename; > + if (def_rolename.length) > + { > + String def_str(buff,sizeof(buff),system_charset_info); eh? sizeof(buff) is sizeof(char*) that is usually 8 on 64-bit, 4 on 32-bit architecture. You want the size of the buffer here, not the size of the pointer to the buffer. > + def_str.length(0); > + def_str.append(STRING_WITH_LEN("SET DEFAULT ROLE ")); > + def_str.append(&def_rolename); > + def_str.append(" FOR '"); > + def_str.append(acl_entry->user.str, acl_entry->user.length, > + system_charset_info); again, just as above, you can use def_str.append(&acl_entry->user); > + DBUG_ASSERT(!(acl_entry->flags & IS_ROLE)); > + def_str.append(STRING_WITH_LEN("'@'")); > + def_str.append(&host); why not to use def_str.append(acl_entry->host.hostname, acl_entry->hostname_length, system_charset_info); > + def_str.append('\''); > + protocol->prepare_for_resend(); > + protocol->store(def_str.ptr(),def_str.length(),def_str.charset()); > + if (protocol->write()) > + { > + return TRUE; > + } > + } > + return FALSE; > +} Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 6729bf5c031: MDEV-22313: Neither SHOW CREATE USER nor SHOW GRANTS prints a user's default role
by Sergei Golubchik 22 Oct '20

22 Oct '20

Hi, Anel! On Oct 22, Anel Husakovic wrote: > Hi Sergei, > thank you for your review. > How is easier for you (or to ask a general question: how does the review > process go?) regarding the commits? > Is it better to add a new commit as a review commit and if the patch gets > accepted to squash after, or to rebase commits and send only a single > commit again for the review? One single commit is better, thanks. At least for small patches. > Addressed in the new patch: > https://github.com/MariaDB/server/commit/99c144762514d4dea3b6ec395de96aaf10… Thanks! I'll reply to the commit directly. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 83614913d9a: MDEV-16243 [Warning] "Versioned table ...: partition ... is full, add more HISTORY partitions" floods the error log
by Sergei Golubchik 22 Oct '20

22 Oct '20

Hi, Aleksey! On Oct 22, Aleksey Midenkov wrote: > revision-id: 83614913d9a (mariadb-10.4.11-291-g83614913d9a) > parent(s): efb1023b6f4 > author: Aleksey Midenkov <midenok(a)gmail.com> > committer: Sergei Golubchik <serg(a)mariadb.com> > timestamp: 2020-10-22 16:44:13 +0200 > message: > > MDEV-16243 [Warning] "Versioned table ...: partition ... is full, add more HISTORY partitions" floods the error log > > diff --git a/sql/partition_info.cc b/sql/partition_info.cc > index 8989a918c0c..f287cfddcd2 100644 > --- a/sql/partition_info.cc > +++ b/sql/partition_info.cc > @@ -879,9 +879,12 @@ void partition_info::vers_set_hist_part(THD *thd) > } > return; > warn: > - my_error(WARN_VERS_PART_FULL, MYF(ME_WARNING|ME_ERROR_LOG), > - table->s->db.str, table->s->table_name.str, > - vers_info->hist_part->partition_name); > + if (vers_info->hist_part->warn_vers_part_full) > + { > + my_error(WARN_VERS_PART_FULL, MYF(ME_WARNING|ME_ERROR_LOG), > + table->s->db.str, table->s->table_name.str, > + vers_info->hist_part->partition_name); > + } would it make sense to put warn_vers_part_full into vers_info, instead of storing it per-partition? > } > > > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 2830b9c290e: MDEV-23926: Fix warnings generated during compilation of plugin/auth_pam/mapper/pam_user_map.c on MacOS
by Sergei Golubchik 22 Oct '20

22 Oct '20

Hi, Dmitry! On Oct 22, Dmitry Shulga wrote: > revision-id: 2830b9c290e (mariadb-10.2.31-489-g2830b9c290e) > parent(s): 8984d779103 > author: Dmitry Shulga <dmitry.shulga(a)mariadb.com> > committer: Dmitry Shulga <dmitry.shulga(a)mariadb.com> > timestamp: 2020-10-22 00:42:10 +0700 > message: > > MDEV-23926: Fix warnings generated during compilation of plugin/auth_pam/mapper/pam_user_map.c on MacOS > > Compiler warnings like one listed below are generated during server build on MacOS: > > > --- > plugin/auth_pam/CMakeLists.txt | 26 ++++++++++++++++++++++++++ > plugin/auth_pam/config.h.cmake | 1 + > plugin/auth_pam/mapper/pam_user_map.c | 30 +++++++++++++++++++----------- > 3 files changed, 46 insertions(+), 11 deletions(-) Nice! Could you also - in a second follow-up commit, not to confuse with the pure bug fix - do a small cleanup and move all other checks from plugin/auth_pam/CMakeLists.txt into your new config.h.cmake ? Currently they're added to the command line, like IF(HAVE_PAM_EXT_H) ADD_DEFINITIONS(-DHAVE_PAM_EXT_H) ENDIF() but as you've introduced config.h.cmake, there's no need to put them all on the command line anymore. You'll need to include config_auth_pam.h into more source files. And, I suppose. add config_auth_pam.h to .gitignore too. ok to push these two commits when done. Thanks! Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 65af1d07771: MDEV-23178: Qualified asterisk not supported in INSERT .. RETURNING
by Sergei Golubchik 22 Oct '20

22 Oct '20

Hi, Rucha! On Oct 22, Rucha Deodhar wrote: > revision-id: 65af1d07771 (mariadb-10.5.4-108-g65af1d07771) > parent(s): bbd70fcc43c > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2020-08-06 17:23:49 +0530 > message: > > MDEV-23178: Qualified asterisk not supported in INSERT .. RETURNING > > Anaylsis: When we have INSERT/REPLACE...RETURNING and have qualified asterisk, > table_name is not NULL and context->table_list is either NULL or has > incorrect reference because context->table_list has tables from the FROM clause. > context->table_list has incorrect reference (has table from the FROM clause > instead of table we are inserting into) for INSERT/REPLACE...SELECT...RETURNING > because we have a FROM clause from the SELECT statement. > For INSERT/REPLACE...RETURNING it is NULL because there is no FROM clause. > Fix: If table_name is not null, check if we have INSERT/REPLACE...RETURNING. > If so, the reference should be the table we are inserting into and not > the table in the FROM clause. > > diff --git a/mysql-test/main/insert_returning.result b/mysql-test/main/insert_returning.result > index e664e02bedc..e1bab6f0329 100644 > --- a/mysql-test/main/insert_returning.result > +++ b/mysql-test/main/insert_returning.result > @@ -89,6 +89,9 @@ total val1 id1 && id1 id1 UPPER(val1) f(id1) good tests! > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > index 45ce4be3eb5..4ce7b034029 100644 > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -7959,7 +7959,7 @@ insert_fields(THD *thd, Name_resolution_context *context, const char *db_name, > bool any_privileges, uint *hidden_bit_fields) > { > Field_iterator_table_ref field_iterator; > - bool found; > + bool found, is_insert_or_replace_returning= false; > char name_buff[SAFE_NAME_LEN+1]; > DBUG_ENTER("insert_fields"); > DBUG_PRINT("arena", ("stmt arena: %p",thd->stmt_arena)); > @@ -7978,13 +7978,31 @@ insert_fields(THD *thd, Name_resolution_context *context, const char *db_name, > > found= FALSE; > > + if ((thd->lex->sql_command == SQLCOM_INSERT_SELECT || > + thd->lex->sql_command == SQLCOM_REPLACE_SELECT || > + thd->lex->sql_command == SQLCOM_INSERT || > + thd->lex->sql_command == SQLCOM_REPLACE) && > + thd->lex->has_returning()) > + is_insert_or_replace_returning= true; What about INSERT t1 SELECT t2.* FROM t2 RETURNING t1.*; can your is_insert_or_replace_returning distinguish between the first and the second asterisk? Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 01612f594b3: MDEV-23769: MTR can abort before it prints the test result summary
by Sergei Golubchik 22 Oct '20

22 Oct '20

Hi, Anel! On Oct 22, Anel Husakovic wrote: > revision-id: 01612f594b3 (mariadb-10.1.43-300-g01612f594b3) > parent(s): 82301aea4f2 > author: Anel Husakovic <anel(a)mariadb.org> > committer: Anel Husakovic <anel(a)mariadb.org> > timestamp: 2020-10-03 02:34:03 +0200 > message: > > MDEV-23769: MTR can abort before it prints the test result summary > > - Patch is solving generating report on warning > - Patch is doing cosmetic fix of `current_test` log file which holds the old log value of test `CURRENT TEST:..` in `mark_log()` in case of `unknown option` and as such > the logic which is using it's content doesn't output valid log content and doesn't generate valid `$test->{'comment'}` message. more is not always better. please remove everything in the commit comment below this line - 400 lines that don't explain anything beyond what your *one line* of the comment already did. And now you have to press PageDown many times to get to my main comment about the fix :) > Here are the test cases observed. Note that only in 6. scenario there is `print $child "HELLO\n";` problem in printing (to delete?). > > 1. Valid test case, but warning in server log > ``` > $ ./mtr MDEV-23769 --force > Logging: ./mtr MDEV-23769 --force > vardir: /home/anel/mariadb/10.1-mtr-in-source/mysql-test/var > Checking leftover processes... > Removing old var directory... > Creating var directory '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var'... > Checking supported features... > MariaDB Version 10.1.47-MariaDB-debug > - SSL connections supported > - binaries are debug compiled > Sphinx 'indexer' binary not found, sphinx suite will be skipped > Collecting tests... > Installing system database... > > ============================================================================== > > TEST RESULT TIME (ms) or COMMENT > -------------------------------------------------------------------------- > > worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019 > show databases; > Database > information_schema > mtr > mysql > performance_schema > test > main.MDEV-23769 [ fail ] Found warnings/errors in server log file! > Test ended at 2020-10-03 02:13:00 > line > 2020-10-03 2:13:00 140309996242496 [Warning] option 'lock_wait_timeout': unsigned value 18446744073709551615 adjusted to 31536000 > ^ Found warnings in /home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/mysqld.1.err > ok > > - saving '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.MDEV-23769/' to '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.MDEV-23769/' > -------------------------------------------------------------------------- > The servers were restarted 0 times > Spent 0.000 of 4 seconds executing testcases > mysql-test-run: WARNING: Got errors/warnings while running tests, please examine '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/warnings' for details. > > Completed: Failed 1/1 tests, 0.00% were successful. > > Failing test(s): main.MDEV-23769 > > The log files in var/log may give you some hint of what went wrong. > > If you want to report this error, please read first the documentation > at http://dev.mysql.com/doc/mysql/en/mysql-test-suite.html > > mysql-test-run: *** ERROR: there were failing test cases > ``` > > 2. Valid test cases (same), but warning in server log (Note reported single test case) > ``` > $ ./mtr MDEV-23769 MDEV-23769 --force > Logging: ./mtr MDEV-23769 MDEV-23769 --force > vardir: /home/anel/mariadb/10.1-mtr-in-source/mysql-test/var > Checking leftover processes... > Removing old var directory... > Creating var directory '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var'... > Checking supported features... > MariaDB Version 10.1.47-MariaDB-debug > - SSL connections supported > - binaries are debug compiled > Sphinx 'indexer' binary not found, sphinx suite will be skipped > Collecting tests... > Installing system database... > > ============================================================================== > > TEST RESULT TIME (ms) or COMMENT > -------------------------------------------------------------------------- > > worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019 > show databases; > Database > information_schema > mtr > mysql > performance_schema > test > main.MDEV-23769 [ fail ] Found warnings/errors in server log file! > Test ended at 2020-10-03 02:15:05 > line > 2020-10-03 2:15:04 139868136609344 [Warning] option 'lock_wait_timeout': unsigned value 18446744073709551615 adjusted to 31536000 > ^ Found warnings in /home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/mysqld.1.err > ok > > - saving '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.MDEV-23769/' to '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.MDEV-23769/' > show databases; > Database > information_schema > mtr > mysql > performance_schema > test > main.MDEV-23769 [ fail ] Found warnings/errors in server log file! > Test ended at 2020-10-03 02:15:05 > line > 2020-10-03 2:15:05 140513782085184 [Warning] option 'lock_wait_timeout': unsigned value 18446744073709551615 adjusted to 31536000 > ^ Found warnings in /home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/mysqld.1.err > ok > > - saving '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.MDEV-23769/' to '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.MDEV-23769/' > -------------------------------------------------------------------------- > The servers were restarted 1 times > Spent 0.000 of 5 seconds executing testcases > mysql-test-run: WARNING: Got errors/warnings while running tests, please examine '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/warnings' for details. > > Completed: Failed 2/2 tests, 0.00% were successful. > > Failing test(s): main.MDEV-23769 > > The log files in var/log may give you some hint of what went wrong. > > If you want to report this error, please read first the documentation > at http://dev.mysql.com/doc/mysql/en/mysql-test-suite.html > > mysql-test-run: *** ERROR: there were failing test cases > ``` > > 3. Valid test cases (different test cases), but warning in server log (Note one passed one failed) > > ``` > $ ./mtr MDEV-23769 1st --force > Logging: ./mtr MDEV-23769 1st --force > vardir: /home/anel/mariadb/10.1-mtr-in-source/mysql-test/var > Checking leftover processes... > Removing old var directory... > Creating var directory '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var'... > Checking supported features... > MariaDB Version 10.1.47-MariaDB-debug > - SSL connections supported > - binaries are debug compiled > Sphinx 'indexer' binary not found, sphinx suite will be skipped > Collecting tests... > Installing system database... > > ============================================================================== > > TEST RESULT TIME (ms) or COMMENT > -------------------------------------------------------------------------- > > worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019 > show databases; > Database > information_schema > mtr > mysql > performance_schema > test > main.MDEV-23769 [ fail ] Found warnings/errors in server log file! > Test ended at 2020-10-03 02:17:09 > line > 2020-10-03 2:17:08 140658805515840 [Warning] option 'lock_wait_timeout': unsigned value 18446744073709551615 adjusted to 31536000 > ^ Found warnings in /home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/mysqld.1.err > ok > > - saving '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.MDEV-23769/' to '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.MDEV-23769/' > main.1st [ pass ] 1 > -------------------------------------------------------------------------- > The servers were restarted 1 times > Spent 0.001 of 4 seconds executing testcases > mysql-test-run: WARNING: Got errors/warnings while running tests, please examine '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/warnings' for details. > > Completed: Failed 1/2 tests, 50.00% were successful. > > Failing test(s): main.MDEV-23769 > > The log files in var/log may give you some hint of what went wrong. > > If you want to report this error, please read first the documentation > at http://dev.mysql.com/doc/mysql/en/mysql-test-suite.html > > mysql-test-run: *** ERROR: there were failing test cases > > ``` > > 4. Now for example 3. create artificial failure with warnings in server log (note, both test failed) > > ``` > > $ ./mtr MDEV-23769 1st --force > Logging: ./mtr MDEV-23769 1st --force > vardir: /home/anel/mariadb/10.1-mtr-in-source/mysql-test/var > Checking leftover processes... > Removing old var directory... > Creating var directory '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var'... > Checking supported features... > MariaDB Version 10.1.47-MariaDB-debug > - SSL connections supported > - binaries are debug compiled > Sphinx 'indexer' binary not found, sphinx suite will be skipped > Collecting tests... > Installing system database... > > ============================================================================== > > TEST RESULT TIME (ms) or COMMENT > -------------------------------------------------------------------------- > > worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019 > show databases; > Database > information_schema > mtr > mysql > performance_schema > test > main.MDEV-23769 [ fail ] Found warnings/errors in server log file! > Test ended at 2020-10-03 02:17:52 > line > 2020-10-03 2:17:51 140260493149760 [Warning] option 'lock_wait_timeout': unsigned value 18446744073709551615 adjusted to 31536000 > ^ Found warnings in /home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/mysqld.1.err > ok > > - saving '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.MDEV-23769/' to '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.MDEV-23769/' > main.1st [ fail ] > Test ended at 2020-10-03 02:17:52 > > CURRENT_TEST: main.1st > mysqltest: At line 5: End of line junk detected: "show tables in mysql > " > > The result from queries just before the failure was: > show databases; > Database > information_schema > mtr > mysql > performance_schema > test > > - saving '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.1st/' to '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.1st/' > -------------------------------------------------------------------------- > The servers were restarted 1 times > Spent 0.000 of 4 seconds executing testcases > mysql-test-run: WARNING: Got errors/warnings while running tests, please examine '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/warnings' for details. > > Completed: Failed 2/2 tests, 0.00% were successful. > > Failing test(s): main.MDEV-23769 main.1st > > The log files in var/log may give you some hint of what went wrong. > > If you want to report this error, please read first the documentation > at http://dev.mysql.com/doc/mysql/en/mysql-test-suite.html > > mysql-test-run: *** ERROR: there were failing test cases > > ``` > > 5. Generate unknown option, for the same example as 3. (note: both failed) > ``` > > $ ./mtr MDEV-23769 1st --mysqld=--xx --force > Logging: ./mtr MDEV-23769 1st --mysqld=--xx --force > vardir: /home/anel/mariadb/10.1-mtr-in-source/mysql-test/var > Checking leftover processes... > Removing old var directory... > Creating var directory '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var'... > Checking supported features... > MariaDB Version 10.1.47-MariaDB-debug > - SSL connections supported > - binaries are debug compiled > Sphinx 'indexer' binary not found, sphinx suite will be skipped > Collecting tests... > Installing system database... > > ============================================================================== > > TEST RESULT TIME (ms) or COMMENT > -------------------------------------------------------------------------- > > worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019 > worker[1] mysql-test-run: WARNING: Process [mysqld.1 - pid: 52531, winpid: 52531, exit: 512] died after mysql-test-run waited 0.3 seconds for /home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/run/mysqld.1.pid to be created. > main.MDEV-23769 [ fail ] > Test ended at 2020-10-03 02:19:01 > > CURRENT_TEST: main.MDEV-23769 > > Failed to start mysqld.1 > mysqltest failed but provided no output > > - saving '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.MDEV-23769/' to '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.MDEV-23769/' > ***Warnings generated in error logs during shutdown after running tests: main.MDEV-23769 > > 2020-10-03 2:19:01 140238937720384 [Warning] option 'lock_wait_timeout': unsigned value 18446744073709551615 adjusted to 31536000 > 2020-10-03 2:19:01 140238937720384 [ERROR] /home/anel/mariadb/10.1-mtr-in-source/sql/mysqld: unknown option '--xx' > 2020-10-03 2:19:01 140238937720384 [ERROR] Aborting > > worker[1] mysql-test-run: WARNING: Process [mysqld.1 - pid: 52537, winpid: 52537, exit: 512] died after mysql-test-run waited 0.3 seconds for /home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/run/mysqld.1.pid to be created. > main.1st [ fail ] > Test ended at 2020-10-03 02:19:02 > > CURRENT_TEST: main.1st > > Failed to start mysqld.1 > mysqltest failed but provided no output > > - saving '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.1st/' to '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.1st/' > ***Warnings generated in error logs during shutdown after running tests: main.1st > > 2020-10-03 2:19:02 140048201700928 [ERROR] /home/anel/mariadb/10.1-mtr-in-source/sql/mysqld: unknown option '--xx' > 2020-10-03 2:19:02 140048201700928 [ERROR] Aborting > > Can't use an undefined value as a symbol reference at ./mtr line 657. > -------------------------------------------------------------------------- > The servers were restarted 1 times > Spent 0.000 of 5 seconds executing testcases > > Completed: Failed 2/2 tests, 0.00% were successful. > > Failing test(s): main.MDEV-23769 main.1st > > The log files in var/log may give you some hint of what went wrong. > > If you want to report this error, please read first the documentation > at http://dev.mysql.com/doc/mysql/en/mysql-test-suite.html > > Errors/warnings were found in logfiles during server shutdown after running the > following sequence(s) of tests: > main.MDEV-23769 > main.1st > mysql-test-run: *** ERROR: there were failing test cases > > ``` > > 6. Again the same option, but with 2 same tests -> not perl error: `Can't use an undefined value as a symbol reference at ./mtr line 657. > ` ` print $child "HELLO\n";` (solution delete the line). > > ``` > $ ./mtr MDEV-23769 MDEV-23769 --mysqld=--xx --force > Logging: ./mtr MDEV-23769 MDEV-23769 --mysqld=--xx --force > vardir: /home/anel/mariadb/10.1-mtr-in-source/mysql-test/var > Checking leftover processes... > Removing old var directory... > Creating var directory '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var'... > Checking supported features... > MariaDB Version 10.1.47-MariaDB-debug > - SSL connections supported > - binaries are debug compiled > Sphinx 'indexer' binary not found, sphinx suite will be skipped > Collecting tests... > Installing system database... > > ============================================================================== > > TEST RESULT TIME (ms) or COMMENT > -------------------------------------------------------------------------- > > worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019 > worker[1] mysql-test-run: WARNING: Process [mysqld.1 - pid: 52595, winpid: 52595, exit: 512] died after mysql-test-run waited 0.3 seconds for /home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/run/mysqld.1.pid to be created. > main.MDEV-23769 [ fail ] > Test ended at 2020-10-03 02:20:12 > > CURRENT_TEST: main.MDEV-23769 > > Failed to start mysqld.1 > mysqltest failed but provided no output > > - saving '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.MDEV-23769/' to '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.MDEV-23769/' > ***Warnings generated in error logs during shutdown after running tests: main.MDEV-23769 > > 2020-10-03 2:20:12 139815724283456 [Warning] option 'lock_wait_timeout': unsigned value 18446744073709551615 adjusted to 31536000 > 2020-10-03 2:20:12 139815724283456 [ERROR] /home/anel/mariadb/10.1-mtr-in-source/sql/mysqld: unknown option '--xx' > 2020-10-03 2:20:12 139815724283456 [ERROR] Aborting > > worker[1] mysql-test-run: WARNING: Process [mysqld.1 - pid: 52601, winpid: 52601, exit: 512] died after mysql-test-run waited 0.3 seconds for /home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/run/mysqld.1.pid to be created. > main.MDEV-23769 [ fail ] > Test ended at 2020-10-03 02:20:12 > > CURRENT_TEST: main.MDEV-23769 > > Failed to start mysqld.1 > mysqltest failed but provided no output > > - saving '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.MDEV-23769/' to '/home/anel/mariadb/10.1-mtr-in-source/mysql-test/var/log/main.MDEV-23769/' > ***Warnings generated in error logs during shutdown after running tests: main.MDEV-23769 > > 2020-10-03 2:20:12 140650797745728 [Warning] option 'lock_wait_timeout': unsigned value 18446744073709551615 adjusted to 31536000 > 2020-10-03 2:20:12 140650797745728 [ERROR] /home/anel/mariadb/10.1-mtr-in-source/sql/mysqld: unknown option '--xx' > 2020-10-03 2:20:12 140650797745728 [ERROR] Aborting > > Can't use an undefined value as a symbol reference at ./mtr line 657. > -------------------------------------------------------------------------- > The servers were restarted 1 times > Spent 0.000 of 5 seconds executing testcases > > Completed: Failed 2/2 tests, 0.00% were successful. > > Failing test(s): main.MDEV-23769 > > The log files in var/log may give you some hint of what went wrong. > > If you want to report this error, please read first the documentation > at http://dev.mysql.com/doc/mysql/en/mysql-test-suite.html > > Errors/warnings were found in logfiles during server shutdown after running the > following sequence(s) of tests: > main.MDEV-23769 > main.MDEV-23769 > mysql-test-run: *** ERROR: there were failing test cases > > ``` > > diff --git a/mysql-test/disabled.def b/mysql-test/disabled.def > index ee30c4f4d3c..fdb62a7e5f5 100644 > --- a/mysql-test/disabled.def > +++ b/mysql-test/disabled.def > @@ -20,3 +20,4 @@ ssl_crl_clrpath : broken upstream > file_contents : MDEV-6526 these files are not installed anymore > max_statement_time : cannot possibly work, depends on timing > partition_open_files_limit : open_files_limit check broken by MDEV-18360 > +MDEV-23769 : Intentionally disabled, for validating purposes. I don't know if it makes any sense. A dead test that nobody will ever run. May be it's better not to add it at all? In the past I've added tests like that too, the "mtr" and "mtr2" suites. I regret it now :) Tests only help if they're run. > diff --git a/mysql-test/mysql-test-run.pl b/mysql-test/mysql-test-run.pl > index 52920530132..8d761fd70cf 100755 > --- a/mysql-test/mysql-test-run.pl > +++ b/mysql-test/mysql-test-run.pl > @@ -1005,6 +1005,13 @@ sub run_worker ($) { > stop_servers(reverse all_servers()); > if(check_warnings_post_shutdown($server)) { > # Warnings appeared in log file(s) during final server shutdown. > + # Make sure first to run statistics > + my $tests= collect_test_cases($opt_reorder, $opt_suites, \@opt_cases, \@opt_skip_test_list); > + my ($prefix, $fail, $completed, $extra_warnings)= > + run_test_server($server, $tests, $opt_parallel); > + mtr_print_line(); > + print_total_times($opt_parallel) if $opt_report_times; > + mtr_report_stats($prefix, $fail, $completed, $extra_warnings); I don't understand the fix. You're saying that if a warning happens on shutdown then mtr won't print the result summary. This doesn't seem to be the case, I've tried few times to repeat it - modified mtr to always pretend there's a warning on shutdown, and modified mysqld to always issue a warning on shutdown - and in all cases I saw the result summary all right. > exit(1); > } > } > @@ -3971,6 +3978,7 @@ sub run_testcase ($$) { > if (start_servers($tinfo)) > { > report_failure_and_restart($tinfo); > + unlink $path_current_testlog; > return 1; > } > } > diff --git a/mysql-test/t/MDEV-23769.opt b/mysql-test/t/MDEV-23769.opt > new file mode 100644 > index 00000000000..fd8dbcab53f > --- /dev/null > +++ b/mysql-test/t/MDEV-23769.opt > @@ -0,0 +1 @@ > +--lock-wait-timeout=-1 > diff --git a/mysql-test/t/MDEV-23769.test b/mysql-test/t/MDEV-23769.test > new file mode 100644 > index 00000000000..9938c7cf644 > --- /dev/null > +++ b/mysql-test/t/MDEV-23769.test > @@ -0,0 +1,4 @@ > +--echo # > +--echo # MDEV-23769: MTR can abort before it prints the test result summary > +--echo # > +show databases; > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 8c745467560: MDEV-23836: Assertion `! is_set() || m_can_overwrite_status' in
by Sergei Golubchik 21 Oct '20

21 Oct '20

Hi, Rucha! what happens in this bug? Why does the server need to open_table() after my_ok()? On Oct 21, Rucha Deodhar wrote: > revision-id: 8c745467560 (mariadb-10.5.4-226-g8c745467560) > parent(s): 308f8350c7b > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2020-10-20 11:53:22 +0530 > message: > > MDEV-23836: Assertion `! is_set() || m_can_overwrite_status' in > Diagnostics_area::set_error_status (interrupted ALTER TABLE under LOCK) > > Analysis: Assertion failure happens because we don't ignore kill query > even if 'OK' is already sent. > Fix: Check if we already sent 'OK'. If sent, ignore kill query. > > diff --git a/mysql-test/main/alter_table.result b/mysql-test/main/alter_table.result > index 197e52b35ca..204bfa5249c 100644 > --- a/mysql-test/main/alter_table.result > +++ b/mysql-test/main/alter_table.result > @@ -3322,5 +3322,23 @@ Note 1176 Key 'x' doesn't exist in table 't1' > unlock tables; > drop table t1; > # > +# MDEV-23836: Assertion `! is_set() || m_can_overwrite_status' in > +# Diagnostics_area::set_error_status (interrupted ALTER TABLE under LOCK) > +# > +set @max_session_mem_used_save= @@max_session_mem_used; > +CREATE TABLE t1 (a INT); > +SELECT * FROM t1; > +a > +ALTER TABLE x MODIFY xx INT; > +ERROR 42S02: Table 'test.x' doesn't exist > +SET SESSION max_session_mem_used= 8192; > +LOCK TABLE t1 WRITE; > +ALTER TABLE t1 CHANGE COLUMN IF EXISTS b c INT; > +Warnings: > +Note 1054 Unknown column 'b' in 't1' > +set SESSION max_session_mem_used = @max_session_mem_used_save; > +UNLOCK TABLES; > +DROP TABLE t1; > +# > # End of 10.5 tests > # > diff --git a/mysql-test/main/alter_table.test b/mysql-test/main/alter_table.test > index 6fe61a3222a..d09c4378339 100644 > --- a/mysql-test/main/alter_table.test > +++ b/mysql-test/main/alter_table.test > @@ -2529,6 +2529,27 @@ alter table t1 rename key if exists x to xx; > unlock tables; > drop table t1; > > +--echo # > +--echo # MDEV-23836: Assertion `! is_set() || m_can_overwrite_status' in > +--echo # Diagnostics_area::set_error_status (interrupted ALTER TABLE under LOCK) > +--echo # > + > +set @max_session_mem_used_save= @@max_session_mem_used; > + > +CREATE TABLE t1 (a INT); > +SELECT * FROM t1; > + > +--error ER_NO_SUCH_TABLE > +ALTER TABLE x MODIFY xx INT; > + > +SET SESSION max_session_mem_used= 8192; > +LOCK TABLE t1 WRITE; > +ALTER TABLE t1 CHANGE COLUMN IF EXISTS b c INT; > + > +set SESSION max_session_mem_used = @max_session_mem_used_save; > +UNLOCK TABLES; > +DROP TABLE t1; > + > --echo # > --echo # End of 10.5 tests > --echo # > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > index e6d0ab13e75..e8d39917ba6 100644 > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -1682,7 +1682,9 @@ bool open_table(THD *thd, TABLE_LIST *table_list, Open_table_context *ot_ctx) > > if (!(flags & MYSQL_OPEN_IGNORE_KILLED) && thd->killed) > { > - thd->send_kill_message(); > + /* If we already sent 'ok', we can ignore any kill query statements */ > + if (! thd->get_stmt_da()->is_set()) > + thd->send_kill_message(); > DBUG_RETURN(TRUE); > } > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 1060cc4235a: MDEV-23500: Assertion `!is_set() || (m_status == DA_OK_BULK && is_bulk_op())'
by Sergei Golubchik 21 Oct '20

21 Oct '20

Hi, Rucha! On Oct 21, Rucha Deodhar wrote: > revision-id: 1060cc4235a (mariadb-10.2.31-438-g1060cc4235a) > parent(s): 80075ba011f > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2020-09-22 17:12:57 +0530 > message: > > MDEV-23500: Assertion `!is_set() || (m_status == DA_OK_BULK && is_bulk_op())' > failed in Diagnostics_area::set_ok_status upon multi-table update > > Analysis: When NULL field is copied to NOT-NULL so there is error but > this error is not reported so we get the failure. > Fix: Return the error state. > > diff --git a/mysql-test/r/update.result b/mysql-test/r/update.result > index 9e19abc4e9c..c1d37fc36e7 100644 > --- a/mysql-test/r/update.result > +++ b/mysql-test/r/update.result > @@ -731,4 +731,14 @@ INSERT INTO t2 VALUES > UPDATE t1,t2 SET t1.i1 = -39 WHERE t2.d1 <> t1.i1 AND t2.d1 = t1.d2; > ERROR 22007: Incorrect datetime value: '19' for column `test`.`t1`.`i1` at row 1 > DROP TABLE t1,t2; > +# > +# MDEV-23500: Assertion `!is_set() || (m_status == DA_OK_BULK && > +# is_bulk_op())' failed in Diagnostics_area::set_ok_status upon > +# multi-table update > +# > +CREATE TABLE t (f INT NOT NULL); > +INSERT INTO t VALUES (0),(0); > +UPDATE t AS t1 LEFT JOIN t AS t2 ON t1.f <=> t2.f SET t2.f = NULL; > +ERROR 01000: Data truncated for column 'f' at row 4 with a single-table update I get MariaDB [test]> update t set f=NULL; ERROR 1048 (23000): Column 'f' cannot be null should multi-table update produce the same error? > +DROP TABLE t; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 9c2570e6c42: MDEV-22422: Assertion `! is_set()' failed in Diagnostics_area::set_eof_status
by Sergei Golubchik 21 Oct '20

21 Oct '20

Hi, Rucha! On Oct 21, Rucha Deodhar wrote: > revision-id: 9c2570e6c42 (mariadb-10.1.43-279-g9c2570e6c42) > parent(s): 5768f57d243 > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2020-09-22 17:18:37 +0530 > message: > > MDEV-22422: Assertion `! is_set()' failed in Diagnostics_area::set_eof_status > > Analysis: The error is not returned when the statement can't be used. > And so we still go on to search for keywords. > Fix: Return the error state. > > diff --git a/mysql-test/t/errors.test b/mysql-test/t/errors.test > index 55461002fd4..e2b10559299 100644 > --- a/mysql-test/t/errors.test > +++ b/mysql-test/t/errors.test > @@ -213,3 +213,22 @@ select * from seq_1_to_1000; > --enable_result_log > # We may not be able to execute any more queries with this connection > # because of too little memory# > + > +--echo # > +--echo # 10.1 Test > +--echo # > +--echo # MDEV-22422: Assertion `! is_set()' failed in > +--echo # Diagnostics_area::set_eof_status > +--echo # > + > +SET @saved_max_session_mem_used = @@max_session_mem_used; > + > +SET max_session_mem_used = 50000; > +--error ER_OPTION_PREVENTS_STATEMENT > +help 'it is going to crash'; > + > +SET @@max_session_mem_used=@saved_max_session_mem_used; > + > +--echo # > +--echo # End of 10.1 Test > +--echo # > diff --git a/sql/sql_help.cc b/sql/sql_help.cc > index cd9da566e51..654a3915b29 100644 > --- a/sql/sql_help.cc > +++ b/sql/sql_help.cc > @@ -736,6 +736,9 @@ static bool mysqld_help_internal(THD *thd, const char *mask) > &name, &description, &example); > delete select; > > + if (thd->is_error()) > + goto error; does the error happen during the first count_topics= search_topics() ? There are lots of places below that look kind of fragile too. May be they're safe. Or may be by increasing max_session_mem_used somewhat you can get mysqld_help_internal to crash below again and again? Could you look into that please? > if (count_topics == 0) > { > int UNINIT_VAR(key_id); > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 6729bf5c031: MDEV-22313: Neither SHOW CREATE USER nor SHOW GRANTS prints a user's default role
by Sergei Golubchik 21 Oct '20

21 Oct '20

Hi, Anel! On Oct 21, Anel Husakovic wrote: > revision-id: 6729bf5c031 (mariadb-10.1.43-112-g6729bf5c031) > parent(s): ad4b70562bb > author: Anel Husakovic <anel(a)mariadb.org> > committer: Anel Husakovic <anel(a)mariadb.org> > timestamp: 2020-04-22 20:48:55 +0200 > message: > > MDEV-22313: Neither SHOW CREATE USER nor SHOW GRANTS prints a user's default role > > diff --git a/mysql-test/t/grant5.test b/mysql-test/t/grant5.test > index 649bba7d1ca..5eefd9b9185 100644 > --- a/mysql-test/t/grant5.test > +++ b/mysql-test/t/grant5.test > @@ -33,3 +33,16 @@ REVOKE EXECUTE ON PROCEDURE sp FROM u; > --error ER_TABLE_NOT_LOCKED > REVOKE PROCESS ON *.* FROM u; > DROP TABLE t1; > + > +# > +# MDEV-22313: Neither SHOW CREATE USER nor SHOW GRANTS prints a user's default role > +# > +CREATE ROLE test_role; > +CREATE USER test_user; > +GRANT test_role TO test_user; > +SET DEFAULT ROLE test_role FOR test_user; > +SHOW GRANTS FOR test_user; > +SET DEFAULT ROLE NONE for test_user; > +SHOW GRANTS FOR test_user; add a test for SHOW GRANTS without FOR username > +DROP USER test_user; > +DROP ROLE test_role; > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index b6a6f806e50..90dd552ec43 100644 > --- a/sql/sql_acl.cc > @@ -7891,6 +7893,10 @@ bool mysql_show_grants(THD *thd, LEX_USER *lex_user) > if (show_role_grants(thd, username, hostname, acl_user, buff, sizeof(buff))) > goto end; > > + /* Show default role to acl_user */ > + if (show_default_role(thd, username, hostname, acl_user, buff, sizeof(buff))) > + goto end; Please, print SET DEFAULT ROLE after all grants for this user. > + > /* Add first global access grants */ > if (show_global_privileges(thd, acl_user, FALSE, buff, sizeof(buff))) > goto end; > @@ -7963,6 +7969,42 @@ static ROLE_GRANT_PAIR *find_role_grant_pair(const LEX_STRING *u, > my_hash_search(&acl_roles_mappings, (uchar*)pair_key.ptr(), key_length); > } > > +static bool show_default_role(THD *thd, const char *username, > + const char *hostname, ACL_USER_BASE *acl_entry, why did you cast acl_entry (which you know is ACL_USER) to the base class ACL_USER_BASE? why did you pass the username if you don't use it? > + char *buff, size_t buffsize) > +{ > + Protocol *protocol= thd->protocol; > + LEX_STRING host= {const_cast<char*>(hostname), strlen(hostname)}; This doesn't make much sense. You only create this LEX_STRING to use it for String::append(LEX_STRING *). But you can directly use String::append(const char*) and you won't need any LEX_STRING here. Furthermore, you can print acl_entry->host.hostname just as you print acl_entry->user, so you don't need the above hostname argument at all. > + String def_role(buff,sizeof(buff),system_charset_info); > + def_role.length(0); you don't use def_role anywhere. > + LEX_STRING def_rolename= ((ACL_USER *)acl_entry)->default_rolename; > + if (def_rolename.length) > + { > + String def_str(buff,sizeof(buff),system_charset_info); > + def_str.length(0); > + def_str.append("SET DEFAULT ROLE "); prefer to use STRING_WITH_LEN() for string literals. E.g. here it should be def_str.append(STRING_WITH_LEN("SET DEFAULT ROLE ")); > + def_str.append(def_rolename.str, def_rolename.length, > + system_charset_info); here you can do def_str.append(&def_rolename); > + def_str.append(" FOR '"); > + def_str.append(acl_entry->user.str, acl_entry->user.length, > + system_charset_info); > + if (!(acl_entry->flags & IS_ROLE)) better put an assert here not an if > + { > + def_str.append(STRING_WITH_LEN("'@'")); > + def_str.append(&host); > + } > + def_str.append('\''); > + protocol->prepare_for_resend(); > + protocol->store(def_str.ptr(),def_str.length(),def_str.charset()); > + if (protocol->write()) > + { > + return TRUE; > + } > + } > + return FALSE; > +} > + > static bool show_role_grants(THD *thd, const char *username, > const char *hostname, ACL_USER_BASE *acl_entry, > char *buff, size_t buffsize) > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 8c34a242aa2: MDEV-20787: Script dgcov.pl does not work
by Sergei Golubchik 21 Oct '20

21 Oct '20

Hi, Anel! On Oct 21, Anel Husakovic wrote: > revision-id: 8c34a242aa2 (mariadb-10.2.31-478-g8c34a242aa2) > parent(s): 01ffccd6a4e > author: Anel Husakovic <anel(a)mariadb.org> > committer: Anel Husakovic <anel(a)mariadb.org> > timestamp: 2020-10-06 00:00:58 +0200 > message: > > MDEV-20787: Script dgcov.pl does not work > > - This patch is done after doing test on 00f9df29224ada9cdbb796c99f4 > where difference was found between versions of gcov. don't refer to commits that are not pushed and are not going to be pushed. Because if you push this one you'll have a commit forever in the history that refers to an non-existing commit. It isn't helpful. > - This patch doesn't take gcov versions in account since is extracting > filename of a `.gcov` file, which is generated with `gcov -i > <source-file>.gcda` during `dgcov.pl --generate`. > - There is a check if file exists. > Reason for that is because some generated `gcov` files are outliers from above > rule (that a file generated in intermediate format(gcov) has the same source name). > Example 1: `aestables.cpp.gcda` is not generating `aestables.cpp.gcov`, > but only the files used in #include `modes.hpp.gcov runtime.hpp.gcov` > Example 2: `my_sha256.cc.gcda` is generating `my_sha.ic.gcov` > - Make sure to test `MTR_BINDIR` which gets set during out-of-source > build and generate error when running `./mysql-test/mtr --gcov alias` What is a problem with out of source builds? If they don't work, perhaps it would be better to fix dgcov to work with out of source builds? > - During the testing: > 10.1 results are stored in > - `<basedir>//mysql-test-gcov.msg` and errors in > - `<basedir>//mysql-test-gcov.err` > 10.2 results should be stored in > `./mysql-test/var/last_changes.dgcov`, but only empty results are > obtained (tests `1st` and `alias`). I don't understand this ^^^ paragraph. And also because your patch is 10.2+ so who cares what it does on 10.1? > diff --git a/mysql-test/dgcov.pl b/mysql-test/dgcov.pl > index fbc5540e697..f2bbdad4d20 100755 > --- a/mysql-test/dgcov.pl > +++ b/mysql-test/dgcov.pl > @@ -161,8 +161,14 @@ sub gcov_one_file { > system($cmd)==0 or die "system($cmd): $? $!"; > } > > - # now, read the generated file > - open FH, '<', "$_.gcov" or die "open(<$_.gcov): $!"; > + (my $filename = $_)=~ s/\.[^.]+$//; # remove extension In your previous patch you either removed the extension or not based on gcov version. Why did you abandon this logic? > + my $gcov_file_path= $File::Find::dir."/$filename.gcov"; File::Find documentation seems to be saying that $File::Find::dir ends with a slash (/) so you don't need to prefix $filename.gcov with it. > + if (! -f $gcov_file_path) > + { > + return; > + } may be return unless -f $gcov_file_path it's perl after all :) > + open FH, '<', "$gcov_file_path" or die "open(<$gcov_file_path): $!"; you don't need double quotes around "$gcov_file_path" - they don't serve any purpose. > + > my $fname; > while (<FH>) { > chomp; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] f83c1615d97: MDEV-20207: Assertion `! is_set()' failed in Diagnostics_area::set_eof_status
by Sergei Golubchik 21 Oct '20

21 Oct '20

Hi, Rucha! On Oct 21, Rucha Deodhar wrote: > revision-id: f83c1615d97 (mariadb-10.1.43-240-gf83c1615d97) > parent(s): 7e9ffc69ecd > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2020-08-05 00:33:03 +0530 > message: > > MDEV-20207: Assertion `! is_set()' failed in Diagnostics_area::set_eof_status > upon HANDLER READ > > Analysis: The error state is not stored while checking condition and key name. > Fix: Return true while checking condition and key name if error is reported > because geometry object can't be created from the data in the index value for > HANDLER READ. > > diff --git a/mysql-test/suite/handler/aria.test b/mysql-test/suite/handler/aria.test > index 1913d2b791c..50ef43ce5b2 100644 > --- a/mysql-test/suite/handler/aria.test > +++ b/mysql-test/suite/handler/aria.test > @@ -80,3 +80,23 @@ HANDLER t1 CLOSE; > DROP TABLE t1; > > --echo End of 5.1 tests > + > +--echo # > +--echo # 10.1 Test > +--echo # > +--echo # MDEV-20207: Assertion `! is_set()' failed in > +--echo # Diagnostics_area::set_eof_status upon HANDLER READ > +--echo # > + > +DROP TABLE IF EXISTS t1; there's no need to start the test from DROP TABLE IF EXISTS t1; historically people customarily did that. before mtr had after-test checks. but now all tests are supposed to clean up after itself, you do not need to worry about tests leaving tables around. > + > +CREATE TABLE t1 (a POINT, KEY(a)); > +HANDLER t1 OPEN h; > + > +--error ER_CANT_CREATE_GEOMETRY_OBJECT > +HANDLER h READ a = (0); I don't think it should be an error. SELECT * FROM t1 WHERE a=(0) -- this does not fail, it just doesn't find anything. I think HANDLER should do the same. > + > +HANDLER h CLOSE; > +DROP TABLE t1; note that all handler test files source handler.inc. that is, if you want a test to run for all engines, you do not need to add it to all individual test files, add it to handler.inc Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] e3f45b2f9ea: MDEV-10267 Add ngram fulltext parser plugin
by Sergei Golubchik 21 Oct '20

21 Oct '20

Hi, Rinat! On Oct 21, Rinat Ibragimov wrote: > > > + > > > + // Avoid creating partial n-grams by stopping at word boundaries. > > > + // Underscore is treated as a word character too, since identifiers in > > > + // programming languages often contain them. > > > + if (!is_alnum(char_type) && !is_underscore(cs, next, end)) { > > > + start= next + char_len; > > > + next= start; > > > + n_chars= 0; > > > + continue; > > > > I don't think it's a good idea. On the opposite, I would rather create > > ngrams over word boundaries, so that, say, I'd split "n-gram plugin" to > > > > "n-g", "-gr", "gra", "ram", "am ", "m p", " pl", "plu", "lug", "ugi", "gin". > > I see how this can increase index size: more different strings are > included. Implementation will become a tiny bit simpler. But can it > make searching better somehow? Yes, the idea is that it'll rank "One can use n-gram plugin in MariaDB fulltext search" higher than "Plugins are useful for extending functionality of a program" because some ngrams encode that "n-gram" and "plugin" words must be together as a phrase. > I can't decide. From my point of view, the current approach is fine. > Please pick a variant, and I'll try to implement that. No, I cannot guess which approach will produce more relevant searches. Implement something and then we test what works better > > > + } > > > + > > > + next += char_len; > > > + n_chars++; > > > + > > > + if (n_chars == ngram_token_size) { > > > + bool_info->position= static_cast<uint>(start - doc); > > > > nope, we don't have MYSQL_FTPARSER_FULL_BOOLEAN_INFO::position and this > > plugin is definitely not the reason to break the API and add it. > > There is a patch in the patchset that adds the "position" field. Without > that field, > it's possible to trigger an assertion in InnoDB's FTS code by executing: > > INSTALL PLUGIN simple_parser SONAME 'mypluglib'; > CREATE TABLE articles( > a TEXT DEFAULT NULL, > b TEXT DEFAULT NULL, > FULLTEXT (a, b) WITH PARSER simple_parser > ) ENGINE=InnoDB; > INSERT INTO articles VALUES ('111', '1234 1234 1234'); > DROP TABLE articles; > UNINSTALL PLUGIN simple_parser; > > Same assertion may be triggered by any fts plugin. Are you sure you > want that patch to be removed? I don't think InnoDB assert is a reason to extend the fulltext plugin API. What about this fix instead: diff --git a/storage/innobase/fts/fts0fts.cc b/storage/innobase/fts/fts0fts.cc --- a/storage/innobase/fts/fts0fts.cc +++ b/storage/innobase/fts/fts0fts.cc @@ -4669,7 +4669,7 @@ fts_tokenize_add_word_for_parser( ut_ad(boolean_info->position >= 0); position = boolean_info->position + fts_param->add_pos; */ - position = fts_param->add_pos; + position = fts_param->add_pos++; fts_add_token(result_doc, str, position); this fixed the crash for me. > > > + n_chars= ngram_token_size - 1; > > > + is_first= false; > > > + } > > > + } > > > + > > > + // If nothing was emitted yet, emit the remainder. > > > > what reminder, what is it for? > > It's "remainder", with "a" before "i". Remainder of the processed > string. That way strings that are too short for having even a single > n-gram, will still generate some index entries and will be > discoverable by exact queries. Perhaps the comment should say it? That if the string too short, it'll be emitted here? > > > + case MYSQL_FTPARSER_FULL_BOOLEAN_INFO: > > > + // Reusing InnoDB's boolean mode parser to handle all special characters. > > > + while (fts_get_word(cs, &start, end, &word, &bool_info)) { > > > > nope. InnoDB might not even be compiled it. Use param->mysql_parse() > > Copied fts_get_word() implementation into the plugin code. > > As far as I understand, param->mysql_parse() cannot be used here as it > splits a string into words like default fulltext search does. This > plugin however must generate n-grams. Of course, it can. Note that fts_get_word() doesn't generate n-grams either, it gets the whole word and the n-gram plugin later splits it into n-grams. Similarly param->mysql_parse() will extract words for you and you'll split them into n-grams. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] Please assign PRs that modify debian/ contents for Otto to review
by Otto Kekäläinen 20 Oct '20

20 Oct '20

Hello! While working on putting MariaDB 10.5.5 into Debian I noticed that there were a couple of commits that landed more or less on the same day as the release was made and "broke" it just before release. Now there are surprises in a stable release I need to fix and the situation puts extra strain on me. It is good that many people participate in maintaining the contents of debian/ in the MariaDB Server, because I don't have time to do everything. I would however appreciate that when changes are made, I would be given a chance to review them. I will review them post-release anyway and it would save a lot of trouble, if we can fix stuff during the development cycle and not having to patch a post-release in order to be able to put it forward to Debian/Ubuntu. So, please make PRs of any debian/ changes you make, and feel free to add me as the reviewer (or one of the reviewers). This policy can also be automated using a .github/CODEOWNERS file, so I made a PR about it at https://github.com/MariaDB/server/pull/1650 Thanks! PS. I will deliver a MariaDB 10.5 and MariaDB in Debian status talk at https://debconf20.debconf.org/talks/98-whats-new-in-mariadb-server-105-and-… in case anybody is interested to listen in. -- Follow me at @ottokekalainen

2 3

Re: [Maria-developers] db1fbafbead: MDEV-23935: Fix warnings generated during compilation of plugin/auth_pam/testing/pam_mariadb_mtr.c on MacOS
by Sergei Golubchik 19 Oct '20

19 Oct '20

Hi, Dmitry! ok to push On Oct 19, Dmitry Shulga wrote: > revision-id: db1fbafbead (mariadb-10.4.11-420-gdb1fbafbead) > parent(s): c2ac0ce1f02 > author: Dmitry Shulga <dmitry.shulga(a)mariadb.com> > committer: Dmitry Shulga <dmitry.shulga(a)mariadb.com> > timestamp: 2020-10-11 21:51:45 +0700 > message: > > MDEV-23935: Fix warnings generated during compilation of plugin/auth_pam/testing/pam_mariadb_mtr.c on MacOS > > During build server 10.4 on MacOS warnings like the following one > are generated on compiling the file plugin/auth_pam/testing/pam_mariadb_mtr.c > > server-10.4/plugin/auth_pam/testing/pam_mariadb_mtr.c:25:22: error: > initializing 'char *' with an expression of type 'const char [23]' > discards qualifiers [-Werror,-Wincompatible-pointer-types-discards-qualifiers] > { PAM_TEXT_INFO, "Challenge input first." } > > The reason of the warnings is that the data member pam_message::msg is declared > as 'char *' on MacOS but initializer of the data member is 'const char *'. > > To eliminate warnings the compiler option > -Wno-incompatible-pointer-types-discards-qualifiers > has been added to the compiler flags used for compiling the file > pam_mariadb_mtr.c. > > --- > plugin/auth_pam/testing/CMakeLists.txt | 6 ++++++ > 1 file changed, 6 insertions(+) > > diff --git a/plugin/auth_pam/testing/CMakeLists.txt b/plugin/auth_pam/testing/CMakeLists.txt > index 889cb524402..c8d2e3cbb97 100644 > --- a/plugin/auth_pam/testing/CMakeLists.txt > +++ b/plugin/auth_pam/testing/CMakeLists.txt > @@ -4,6 +4,12 @@ ADD_LIBRARY(pam_mariadb_mtr MODULE pam_mariadb_mtr.c) > SET_TARGET_PROPERTIES (pam_mariadb_mtr PROPERTIES PREFIX "") > TARGET_LINK_LIBRARIES(pam_mariadb_mtr pam) > > +IF(APPLE) > + SET_SOURCE_FILES_PROPERTIES( > + pam_mariadb_mtr.c > + PROPERTY COMPILE_FLAGS "-Wno-incompatible-pointer-types-discards-qualifiers") > +ENDIF() > + > SET(dest DESTINATION "${INSTALL_MYSQLTESTDIR}/suite/plugins/pam" COMPONENT Test) > INSTALL(TARGETS pam_mariadb_mtr ${dest}) > INSTALL(FILES mariadb_mtr.conf RENAME mariadb_mtr ${dest}) > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 727d97feb99: MDEV-23926: Fix warnings generated during compilation of plugin/auth_pam/mapper/pam_user_map.c on MacOS
by Sergei Golubchik 19 Oct '20

19 Oct '20

Hi, Dmitry! plugin code should stay in the plugin directory. please move the check to plugin/auth_pam/CMakeLists.txt Also, what about an alternative fix: #ifdef HAVE_POSIX_GETGROUPLIST typedef gid_t my_gid_t; #else typedef int my_gid_t; #end would that help but with less ifdefs? On Oct 19, Dmitry Shulga wrote: > revision-id: 727d97feb99 (mariadb-10.2.31-489-g727d97feb99) > parent(s): 8984d779103 > author: Dmitry Shulga <dmitry.shulga(a)mariadb.com> > committer: Dmitry Shulga <dmitry.shulga(a)mariadb.com> > timestamp: 2020-10-09 20:35:48 +0700 > message: > > MDEV-23926: Fix warnings generated during compilation of plugin/auth_pam/mapper/pam_user_map.c on MacOS > > Compiler warnings listed below are generated during server build on MacOS: > > 88%] Building C object plugin/auth_pam/CMakeFiles/pam_user_map.dir/mapper/pam_user_map.c.o > /Users/shulga/projects/mariadb/server-10.2/plugin/auth_pam/mapper/pam_user_map.c:87:41: error: passing > 'gid_t *' (aka 'unsigned int *') to parameter of type 'int *' converts between pointers to integer types > with different sign [-Werror,-Wpointer-sign] > if (getgrouplist(user, user_group_id, loc_groups, &ng) < 0) > ^~~~~~~~~~ > /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/include/unistd.h:650:43: note: > passing argument to parameter here > int getgrouplist(const char *, int, int *, int *); > ^ > /Users/shulga/projects/mariadb/server-10.2/plugin/auth_pam/mapper/pam_user_map.c:95:46: error: passing > 'gid_t *' (aka 'unsigned int *') to parameter of type 'int *' converts between pointers to integer types > with different sign [-Werror,-Wpointer-sign] > (void) getgrouplist(user, user_group_id, loc_groups, &ng); > ^~~~~~~~~~ > /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.15.sdk/usr/includ > e/unistd.h:650:43: note: > passing argument to parameter here > int getgrouplist(const char *, int, int *, int *); > ^ > 2 errors generated. > > In case MariaDB serer is build with -DCMAKE_BUILD_TYPE=Debug it results in > uild error. > > The reason of compiler warnings is that declaration of the Posix C API function > getgrouplist() on MacOS differs from declration of getgrouplist() proposed > by Posix. > > To suppress this compiler warning cmake configure was adapted to detect what > kind of getgrouplist() function is declared on the building platform and > set the macros HAVE_POSIX_GETGROUPLIST in case the building platform suppors > Posix compatible interface of the getgrouplist() function. Depending on > whether this macros is set the compatible type of arguments is used to pass > parameter values to the function. > > --- > config.h.cmake | 1 + > configure.cmake | 22 ++++++++++++++++++++++ > plugin/auth_pam/mapper/pam_user_map.c | 13 ++++++++++++- > 3 files changed, 35 insertions(+), 1 deletion(-) > > diff --git a/config.h.cmake b/config.h.cmake > index 0e19dd44694..07e4647f0bf 100644 > --- a/config.h.cmake > +++ b/config.h.cmake > @@ -188,6 +188,7 @@ > #cmakedefine HAVE_POSIX_FALLOCATE 1 > #cmakedefine HAVE_LINUX_FALLOC_H 1 > #cmakedefine HAVE_FALLOC_PUNCH_HOLE_AND_KEEP_SIZE 1 > +#cmakedefine HAVE_POSIX_GETGROUPLIST 1 > #cmakedefine HAVE_PREAD 1 > #cmakedefine HAVE_PAUSE_INSTRUCTION 1 > #cmakedefine HAVE_FAKE_PAUSE_INSTRUCTION 1 > diff --git a/configure.cmake b/configure.cmake > index 58e1111bfc7..c6db3ecb912 100644 > --- a/configure.cmake > +++ b/configure.cmake > @@ -1031,3 +1031,25 @@ IF(NOT MSVC) > HAVE_FALLOC_PUNCH_HOLE_AND_KEEP_SIZE > ) > ENDIF() > + > +# Check whether getgrouplist uses git_t for second and third arguments. > +SET(CMAKE_REQUIRED_FLAGS -Werror) > +CHECK_C_SOURCE_COMPILES( > +" > +#ifdef HAVE_GRP_H > +#include <grp.h> > +#endif > +#ifdef HAVE_UNISTD_H > +#include <unistd.h> > +#endif > +int main() { > + char *arg_1; > + gid_t arg_2, arg_3; > + int arg_4; > + (void)getgrouplist(arg_1,arg_2,&arg_3,arg_4); > + return 0; > +} > +" > +HAVE_POSIX_GETGROUPLIST > +) > +SET(CMAKE_REQUIRED_FLAGS) > diff --git a/plugin/auth_pam/mapper/pam_user_map.c b/plugin/auth_pam/mapper/pam_user_map.c > index 9d7ed53f8b1..b0d2affac5c 100644 > --- a/plugin/auth_pam/mapper/pam_user_map.c > +++ b/plugin/auth_pam/mapper/pam_user_map.c > @@ -31,6 +31,8 @@ These comments are written to the syslog as 'authpriv.debug' > and usually end up in /var/log/secure file. > */ > > +#include <my_config.h> > + > #include <stdlib.h> > #include <stdio.h> > #include <ctype.h> > @@ -72,8 +74,13 @@ static const char debug_keyword[]= "debug"; > > static int populate_user_groups(const char *user, gid_t **groups) > { > +#ifdef HAVE_POSIX_GETGROUPLIST > gid_t user_group_id; > gid_t *loc_groups= *groups; > +#else > + int user_group_id; > + int *loc_groups= (int*)*groups; > +#endif > int ng; > > { > @@ -88,12 +95,16 @@ static int populate_user_groups(const char *user, gid_t **groups) > { > /* The rare case when the user is present in more than */ > /* GROUP_BUFFER_SIZE groups. */ > +#ifdef HAVE_POSIX_GETGROUPLIST > loc_groups= (gid_t *) malloc(ng * sizeof (gid_t)); > +#else > + loc_groups= (int *) malloc(ng * sizeof (gid_t)); > +#endif > if (!loc_groups) > return 0; > > (void) getgrouplist(user, user_group_id, loc_groups, &ng); > - *groups= loc_groups; > + *groups= (gid_t*)loc_groups; > } > > return ng; > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 5e71a3e0df7: MDEV-23925: Fix warnings generated during compilation of mysys_ssl/openssl.c on MacOS
by Sergei Golubchik 19 Oct '20

19 Oct '20

Hi, Dmitry! ok to push (again, comments below) On Oct 19, Dmitry Shulga wrote: > revision-id: 5e71a3e0df7 (mariadb-10.2.31-489-g5e71a3e0df7) > parent(s): 8984d779103 > author: Dmitry Shulga <dmitry.shulga(a)mariadb.com> > committer: Dmitry Shulga <dmitry.shulga(a)mariadb.com> > timestamp: 2020-10-09 20:55:14 +0700 > message: > > MDEV-23925: Fix warnings generated during compilation of mysys_ssl/openssl.c on MacOS > > Compiler warnings listed below are generated during server build on MacOS: > In file included from server-10.2-MDEV-23564/mysys_ssl/openssl.c:33: > In file included from /usr/local/include/openssl/evp.h:16: > In file included from /usr/local/include/openssl/bio.h:20: > /usr/local/include/openssl/crypto.h:206:10: warning: 'CRYPTO_cleanup_all_ex_data' macro redefined [-Wmacro-redefined] > ^ > /mariadb/server-10.2-MDEV-23564/include/ssl_compat.h:46:9: note: previous definition is here > ^ > In file included from /mariadb/server-10.2-MDEV-23564/mysys_ssl/openssl.c:33: > /usr/local/include/openssl/evp.h:501:10: warning: 'EVP_MD_CTX_init' macro redefined [-Wmacro-redefined] > ^ > /mariadb/server-10.2-MDEV-23564/include/ssl_compat.h:33:9: note: previous definition is here > ^ > In file included from /mariadb/server-10.2-MDEV-23564/mysys_ssl/openssl.c:33: > /usr/local/include/openssl/evp.h:627:11: warning: 'EVP_CIPHER_CTX_init' macro redefined [-Wmacro-redefined] > ^ > /mariadb/server-10.2-MDEV-23564/include/ssl_compat.h:35:9: note: previous definition is here > ^ > In file included from /mariadb/server-10.2-MDEV-23564/mysys_ssl/openssl.c:33: > /usr/local/include/openssl/evp.h:866:11: warning: 'EVP_cleanup' macro redefined [-Wmacro-redefined] > ^ > /mariadb/server-10.2-MDEV-23564/include/ssl_compat.h:44:9: note: previous definition is here > ^ > 4 warnings generated. generally these long commit comments are not any more clear than if you'd only listed one compilation failure. > > In case MariaDB serer is build with -DCMAKE_BUILD_TYPE=Debug it results in > build error. > > The reason of compiler warnings is that header file <ssl_compat.h> > included before the openssl system header files. File ssl_compat.h > contains some macros with the name as SSL API functions declared > in the openssl system header files. It resulted in duplicate > symbols thats produces compiler warnings. > > To fix the issue the header file ssl_compat.h should be included > after a line where openssl system header is included. > > --- > mysys_ssl/openssl.c | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) > > diff --git a/mysys_ssl/openssl.c b/mysys_ssl/openssl.c > index f3dc8f4277c..a11db113b14 100644 > --- a/mysys_ssl/openssl.c > +++ b/mysys_ssl/openssl.c > @@ -15,7 +15,6 @@ > Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA */ > > #include <my_global.h> > -#include <ssl_compat.h> > > /* > The check is only done for OpenSSL 1.1.x. > @@ -25,13 +24,14 @@ > */ > > #ifndef HAVE_OPENSSL11 > +#include <ssl_compat.h> > int check_openssl_compatibility() > { > return 0; > } > #else > #include <openssl/evp.h> > - > +#include <ssl_compat.h> > static uint testing, alloc_size, alloc_count; > > static void *coc_malloc(size_t size, const char *f __attribute__((unused)), > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] b92597c220f: MDEV-23564: CMAKE failing due to deprecated Apple GSS method
by Sergei Golubchik 19 Oct '20

19 Oct '20

Hi, Dmitry! ok to push. but see comments below On Oct 19, Dmitry Shulga wrote: > revision-id: b92597c220f (mariadb-10.2.31-479-gb92597c220f) > parent(s): 577c61e8be0 > author: Dmitry Shulga <dmitry.shulga(a)mariadb.com> > committer: Dmitry Shulga <dmitry.shulga(a)mariadb.com> > timestamp: 2020-10-08 00:48:47 +0700 > message: > > MDEV-23564: CMAKE failing due to deprecated Apple GSS method > > Some GSS-API functions like gss_import_name(), gss_release_buffer() > used in plugin/auth_gssapi and libmariadb/plugins/auth are marked > as deprecated in MacOS starting from version 10.14+. It results in > extra warnings output on server building. > > To eliminate extra warnings the flag '-Wno-deprecated-declarations' > has been added to compiler invocation string for those source > files that invoke deprecated GSS-API functions. > > diff --git a/libmariadb b/libmariadb > index a746c3af449..fd0dc672309 160000 > --- a/libmariadb > +++ b/libmariadb > @@ -1 +1 @@ > -Subproject commit a746c3af449a8754e78ad7971e59e79af7957cdb > +Subproject commit fd0dc6723095f066648a7c3b2a48523b3002e392 be careful when pushing this commit. you'd need something like 1. rebase your libmariadb fix on top of 3.1, push into 3.1 2. rebase your server fix on top of 10.2 2. update submodule reference in the server repo 3. amend your commit fix and push > diff --git a/plugin/auth_gssapi/CMakeLists.txt b/plugin/auth_gssapi/CMakeLists.txt > index bca4f5af3a1..7a3c32a00b1 100644 > --- a/plugin/auth_gssapi/CMakeLists.txt > +++ b/plugin/auth_gssapi/CMakeLists.txt > @@ -18,6 +18,13 @@ ELSE() > SET(GSSAPI_SERVER gssapi_server.cc) > SET(GSSAPI_ERRMSG gssapi_errmsg.cc) > > + IF(APPLE) > + SET_SOURCE_FILES_PROPERTIES( > + ${GSSAPI_CLIENT} ${GSSAPI_SERVER} ${GSSAPI_ERRMSG} > + PROPERTY COMPILE_FLAGS "-Wno-deprecated-declarations") > +# SET_SOURCE_FILES_PROPERTIES(${GSSAPI_SERVER} PROPERTY COMPILE_FLAGS "-Wno-deprecated-declarations") > +# SET_SOURCE_FILES_PROPERTIES(${GSSAPI_ERRMSG} PROPERTY COMPILE_FLAGS "-Wno-deprecated-declarations") you don't need these commented lines anymore, do you? Is it possible to use a compiler on APPLE that doesn't understand -Wno-deprecated-declarations? If not - your fix it ok. If it is - better use MY_CHECK_AND_SET_COMPILER_FLAG > + ENDIF() > SET(CMAKE_REQUIRED_INCLUDES ${GSSAPI_INCS}) > SET(CMAKE_REQUIRED_LIBRARIES ${GSSAPI_LIBS}) > INCLUDE(CheckCXXSymbolExists) Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 2d73406dad8: MDEV-16481: set global system_versioning_asof=sf() crashes in specific case
by Sergei Golubchik 19 Oct '20

19 Oct '20

Hi, Nikita! ok to push (better late than never :) On Oct 19, Nikita Malyavin wrote: > revision-id: 2d73406dad8 (mariadb-10.3.16-39-g2d73406dad8) > parent(s): 609f358a9bd > author: Nikita Malyavin <nikitamalyavin(a)gmail.com> > committer: Sergei Golubchik <serg(a)mariadb.com> > timestamp: 2020-10-19 17:37:04 +0200 > message: > > MDEV-16481: set global system_versioning_asof=sf() crashes in specific case > > * sys_vars.h: add `MYSQL_TIME` field to `set_var::save_result` > * sys_vars.ic: get rid of calling `var->value->get_date()` from `Sys_var_vers_asof::update()` > * versioning.sysvars: add test; remove double warning > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] e3f45b2f9ea: MDEV-10267 Add ngram fulltext parser plugin
by Sergei Golubchik 19 Oct '20

19 Oct '20

Hi, Rinat! On Oct 19, Rinat Ibragimov wrote: > revision-id: e3f45b2f9ea (mariadb-10.2.31-368-ge3f45b2f9ea) > parent(s): 44c48cefdbe > author: Rinat Ibragimov <ri(a)tempesta-tech.com> > committer: Rinat Ibragimov <ri(a)tempesta-tech.com> > timestamp: 2020-08-25 01:59:06 +0300 > message: > > MDEV-10267 Add ngram fulltext parser plugin > > Ngram is a fulltext parser plugin that splits words into overlapping > segments of fixed lengths. For example, if 3-grams are in use, string > "abcdef" is split into "abc", "bcd", "cde", "def", thus allowing an > efficient substring search for CJK texts where splitting by words in not > feasible. Only word characters are going into n-grams. Spaces and > punctuation characters are treated as separators. > > diff --git a/plugin/fulltext/CMakeLists.txt b/plugin/fulltext/CMakeLists.txt > index 3c37666cb3e..9704733a9aa 100644 > --- a/plugin/fulltext/CMakeLists.txt > +++ b/plugin/fulltext/CMakeLists.txt > @@ -18,3 +18,7 @@ set(CMAKE_CXX_FLAGS_RELWITHDEBINFO "${CMAKE_CXX_FLAGS_RELWITHDEBINFO} -DNDEBUG") > > MYSQL_ADD_PLUGIN(ftexample plugin_example.c > MODULE_ONLY MODULE_OUTPUT_NAME "mypluglib" COMPONENT Test) > + > +# n-gram parser > +MYSQL_ADD_PLUGIN(ftngram ngram_parser/plugin_ngram.cc > + MODULE_ONLY MODULE_OUTPUT_NAME "fulltext_ngram") why to put the plugin under plugin/fulltext/ngram_parser/ ? it's only one file, just put it in plugin/fulltext/ > diff --git a/plugin/fulltext/mysql-test/ngram/disabled.def b/plugin/fulltext/mysql-test/ngram/disabled.def > new file mode 100644 > index 00000000000..0ca9dd33ef7 > --- /dev/null > +++ b/plugin/fulltext/mysql-test/ngram/disabled.def > @@ -0,0 +1 @@ > +ngram_gb18030 : GB18030 character set is not supported yet. I don't think there's any need to add tests for unsupported charsets. The plugin is trivial, no need to test it for every possible charset that we have and don't have. > diff --git a/plugin/fulltext/mysql-test/ngram/ngram_basic.inc b/plugin/fulltext/mysql-test/ngram/ngram_basic.inc > --- /dev/null > +++ b/plugin/fulltext/mysql-test/ngram/ngram_basic.inc > @@ -0,0 +1,25 @@ > +--echo > +--eval CREATE TABLE t (lang TEXT, str TEXT, FULLTEXT(str) WITH PARSER ngram) ENGINE=$engine CHARSET $charset > + > +# Some character are not possible to represent in some encodings. Ignore them. > +--disable_warnings > +--disable_query_log > +# http://www.gerolf.org/doc/polyglot/polyglot.htm. > +INSERT IGNORE INTO t VALUES > + ('Chinese', '他们为什么不说中文？'), > + ('English', 'Why can''t they just speak English?'), > + ('Japanese', 'なぜ、みんな日本語を話してくれないのか？'), > + ('Korean', '세계의 모든 사람들이 한국어를 이해한다면 얼마나 좋을까?'); > +--enable_query_log > +--enable_warnings > + > +SELECT * FROM t ORDER BY lang; > + > +SELECT * FROM t WHERE MATCH(str) AGAINST('之一' IN BOOLEAN MODE) ORDER BY lang; > +SELECT * FROM t WHERE MATCH(str) AGAINST('数据' IN BOOLEAN MODE) ORDER BY lang; > +SELECT * FROM t WHERE MATCH(str) AGAINST('zzz' IN BOOLEAN MODE) ORDER BY lang; > +SELECT * FROM t WHERE MATCH(str) AGAINST('gli' IN BOOLEAN MODE) ORDER BY lang; > +SELECT * FROM t WHERE MATCH(str) AGAINST('ない' IN BOOLEAN MODE) ORDER BY lang; > +SELECT * FROM t WHERE MATCH(str) AGAINST('람들' IN BOOLEAN MODE) ORDER BY lang; > + > +DROP TABLE t; > diff --git a/plugin/fulltext/mysql-test/ngram/ngram_basic.test b/plugin/fulltext/mysql-test/ngram/ngram_basic.test > new file mode 100644 > index 00000000000..eaa8829c3ee > --- /dev/null > +++ b/plugin/fulltext/mysql-test/ngram/ngram_basic.test > @@ -0,0 +1,25 @@ > +# Ngram fulltext parser basic functionality. > + > +SET NAMES utf8mb4; > +SELECT @@ngram_token_size; > + > +let $engine_index= 3; > +while ($engine_index) { > + if ($engine_index == 3) { let $engine= InnoDB; } > + if ($engine_index == 2) { let $engine= MyISAM; } > + if ($engine_index == 1) { let $engine= Aria; } > + > + let $charset_index= 6; > + while ($charset_index) { > + if ($charset_index == 6) { let $charset= utf8mb4; } > + if ($charset_index == 5) { let $charset= big5; } > + if ($charset_index == 4) { let $charset= gbk; } > + if ($charset_index == 3) { let $charset= gb2312; } > + if ($charset_index == 2) { let $charset= sjis; } > + if ($charset_index == 1) { let $charset= ujis; } > + > + source ngram_basic.inc; > + dec $charset_index; > + } eh. unroll the loop please, write let $engine= InnoDB; let $charset= utf8mb4; source ngram_basic.inc; let $charset= big5; source ngram_basic.inc; ... Or, better: CREATE TABLE t (lang TEXT, str TEXT, FULLTEXT(str) WITH PARSER ngram) ENGINE=InnoDB CHARSET utf8mb4; source ngram_basic.inc; CREATE TABLE t (lang TEXT, str TEXT, FULLTEXT(str) WITH PARSER ngram) ENGINE=InnoDB CHARSET big5; source ngram_basic.inc; ... > + dec $engine_index; > +} > diff --git a/plugin/fulltext/mysql-test/ngram/ngram_missing_plugin.result b/plugin/fulltext/mysql-test/ngram/ngram_missing_plugin.result > new file mode 100644 > index 00000000000..aa17b1d6053 > --- /dev/null > +++ b/plugin/fulltext/mysql-test/ngram/ngram_missing_plugin.result > @@ -0,0 +1,23 @@ there are tests for the missing fulltest parser plugin. they don't depend on the individial plugin (that is, btw, *not installed*). No need to repeat them for every plugin, remove this test. > +SET NAMES utf8mb4; > +CREATE TABLE t (str TEXT, FULLTEXT(str) WITH PARSER ngram) > +ENGINE=InnoDB; > +INSERT INTO t VALUES ('Hello World'), ('abcdef ghijklm'); > +# Restart with ngram plugin disabled. > +SET NAMES utf8mb4; > +SELECT * FROM t WHERE MATCH(str) AGAINST('bcd') ORDER BY str; > +ERROR HY000: Plugin 'ngram' is not loaded > +SELECT * FROM t ORDER BY str; > +ERROR HY000: Plugin 'ngram' is not loaded > +INSERT INTO t VALUES ('something'); > +ERROR HY000: Plugin 'ngram' is not loaded > +# Restart with ngram plugin enabled again. > +SET NAMES utf8mb4; > +SELECT * FROM t WHERE MATCH(str) AGAINST('bcd') ORDER BY str; > +str > +abcdef ghijklm > +SELECT * FROM t ORDER BY str; > +str > +abcdef ghijklm > +Hello World > +INSERT INTO t VALUES ('something'); > +DROP TABLE t; > diff --git a/plugin/fulltext/mysql-test/ngram/ngram_token_size.result b/plugin/fulltext/mysql-test/ngram/ngram_token_size.result > new file mode 100644 > index 00000000000..7043e147d5b > --- /dev/null > +++ b/plugin/fulltext/mysql-test/ngram/ngram_token_size.result > @@ -0,0 +1,109 @@ > +# Check 'ngram_token_size' variable. > +# > +# It's a global variable. > +SELECT @@global.ngram_token_size; > +@@global.ngram_token_size > +2 > +SHOW GLOBAL VARIABLES LIKE 'ngram_token_size'; > +Variable_name Value > +ngram_token_size 2 > +SELECT * FROM information_schema.global_variables WHERE variable_name='ngram_token_size'; > +VARIABLE_NAME VARIABLE_VALUE > +NGRAM_TOKEN_SIZE 2 > +# > +# Not a session variable. > +SELECT @@session.ngram_token_size; > +ERROR HY000: Variable 'ngram_token_size' is a GLOBAL variable > +SHOW SESSION VARIABLES LIKE 'ngram_token_size'; > +Variable_name Value > +ngram_token_size 2 > +SELECT * FROM information_schema.session_variables WHERE variable_name='ngram_token_size'; > +VARIABLE_NAME VARIABLE_VALUE > +NGRAM_TOKEN_SIZE 2 > +# > +# 0 is outside allowed range [1, 10], so it's changed to 1. > +SET GLOBAL ngram_token_size= 0; > +Warnings: > +Warning 1292 Truncated incorrect ngram_token_size value: '0' > +SELECT * FROM information_schema.global_variables WHERE variable_name='ngram_token_size'; > +VARIABLE_NAME VARIABLE_VALUE > +NGRAM_TOKEN_SIZE 1 > +# > +# 10 is an acceptable value. > +SET GLOBAL ngram_token_size= 10; > +SELECT * FROM information_schema.global_variables WHERE variable_name='ngram_token_size'; > +VARIABLE_NAME VARIABLE_VALUE > +NGRAM_TOKEN_SIZE 10 > +# > +# 1 is an acceptable value. > +SET GLOBAL ngram_token_size= 1; > +SELECT * FROM information_schema.global_variables WHERE variable_name='ngram_token_size'; > +VARIABLE_NAME VARIABLE_VALUE > +NGRAM_TOKEN_SIZE 1 > +# > +# 11 is outside allowed range [1, 10], so it's changed to 10. > +SET GLOBAL ngram_token_size= 11; > +Warnings: > +Warning 1292 Truncated incorrect ngram_token_size value: '11' > +SELECT * FROM information_schema.global_variables WHERE variable_name='ngram_token_size'; > +VARIABLE_NAME VARIABLE_VALUE > +NGRAM_TOKEN_SIZE 10 ^^^^ all tests above aren't needed, it's enough to do query_vertical select * from information_schema.system_variables where variable_name='ngram_token_size'; all the behavior above is implemented outside of the plugin, no need to test it. > +# > +# Try 3-grams. > +SET GLOBAL ngram_token_size= 3; > +CREATE TABLE t (str TEXT, FULLTEXT INDEX (str) WITH PARSER ngram) Engine=InnoDB; > +INSERT INTO t VALUES > +('Lorem ipsum dolor sit amet, consectetur adipiscing elit.'), > +('Maecenas non pellentesque odio.'), > +('Sed fermentum egestas sapien vel consequat.'); > +# Only one row contains substring 'api'. > +SELECT str FROM t WHERE MATCH(str) AGAINST('api'); > +str > +Sed fermentum egestas sapien vel consequat. > +# 'lent' is parsed into ('len', 'ent') list. Two rows contain at least one of the tokens. > +SELECT str FROM t WHERE MATCH(str) AGAINST('lent*'); > +str > +Maecenas non pellentesque odio. > +Sed fermentum egestas sapien vel consequat. > +# 'zzz' is not present anywhere, so result is empty. > +SELECT str FROM t WHERE MATCH(str) AGAINST('zzzz'); > +str > +# 'piz' cannot be found anywhere, but 'api' is in one of the rows. > +SELECT str FROM t WHERE MATCH(str) AGAINST('apiz'); > +str > +Sed fermentum egestas sapien vel consequat. > +# > +# Try switching to 2-grams while keeping the index. > +SET GLOBAL ngram_token_size= 2; > +# 'api' is now parsed into ('ap', 'pi'). Since index only contains 3-grams, nothing is found. > +SELECT str FROM t WHERE MATCH(str) AGAINST('api'); > +str > +# > +# Add some data while ngram size is set to 2. > +INSERT INTO t VALUES('1234 56789'); > +# > +# Now index contains 3-grams from the rows added before and a number of > +# 2-grams from the just added row. > +# Querying for 'api' still returns nothing, but line with '678' is found. > +SELECT str from t WHERE MATCH(str) AGAINST('api'); > +str > +SELECT str FROM t WHERE MATCH(str) AGAINST('678'); > +str > +1234 56789 > +# > +# Switching back to 3-grams make intitially added rows searchable. > +SET GLOBAL ngram_token_size= 3; > +SELECT str FROM t WHERE MATCH(str) AGAINST('api'); > +str > +Sed fermentum egestas sapien vel consequat. > +# > +# But row with '678' in it cannot be found, as it was added with 2-grams. > +SELECT str FROM t WHERE MATCH(str) AGAINST('678'); > +str > +# > +# Try matching in boolean mode. > +SELECT str FROM t WHERE MATCH(str) AGAINST('+ent -api' IN BOOLEAN MODE); > +str > +Maecenas non pellentesque odio. > +# > +DROP TABLE t; > diff --git a/plugin/fulltext/mysql-test/ngram/ngram_ucs2.result b/plugin/fulltext/mysql-test/ngram/ngram_ucs2.result > new file mode 100644 > index 00000000000..9d82743bd24 > --- /dev/null > +++ b/plugin/fulltext/mysql-test/ngram/ngram_ucs2.result > @@ -0,0 +1,8 @@ > +CREATE TABLE t (str TEXT, FULLTEXT INDEX (str) WITH PARSER ngram) Engine=InnoDB CHARSET=UCS2; > +ERROR HY000: Column 'str' cannot be part of FULLTEXT index > +CREATE TABLE t (str TEXT, FULLTEXT INDEX (str) WITH PARSER ngram) Engine=InnoDB CHARSET=UTF16; > +ERROR HY000: Column 'str' cannot be part of FULLTEXT index > +CREATE TABLE t (str TEXT, FULLTEXT INDEX (str) WITH PARSER ngram) Engine=InnoDB CHARSET=UTF16LE; > +ERROR HY000: Column 'str' cannot be part of FULLTEXT index > +CREATE TABLE t (str TEXT, FULLTEXT INDEX (str) WITH PARSER ngram) Engine=InnoDB CHARSET=UTF32; > +ERROR HY000: Column 'str' cannot be part of FULLTEXT index same as above. This is fulltext index limitation, it does not depend on the parser plugin. The plugin doesn't need to test for it. > diff --git a/plugin/fulltext/mysql-test/ngram/suite.opt b/plugin/fulltext/mysql-test/ngram/suite.opt > new file mode 100644 > index 00000000000..ab37cea996e > --- /dev/null > +++ b/plugin/fulltext/mysql-test/ngram/suite.opt > @@ -0,0 +1,4 @@ > +--plugin-load-add=$FULLTEXT_NGRAM_SO > +--loose-innodb no need to enable innodb explicitly, `source have_innodb.inc` does that. > +--loose-enable-innodb_ft_index_table > +--loose-innodb_ft_index_cache why did you enable these two plugins if you don't use them in your tests? > diff --git a/plugin/fulltext/ngram_parser/plugin_ngram.cc b/plugin/fulltext/ngram_parser/plugin_ngram.cc > new file mode 100644 > index 00000000000..80498c3299b > --- /dev/null > +++ b/plugin/fulltext/ngram_parser/plugin_ngram.cc > @@ -0,0 +1,256 @@ > +/* > + Copyright (c) 2014, 2017, Oracle and/or its affiliates. All rights reserved. > + Copyright (c) 2020, MariaDB. > + > + This program is free software; you can redistribute it and/or modify > + it under the terms of the GNU General Public License, version 2.0, > + as published by the Free Software Foundation. > + > + This program is also distributed with certain software (including > + but not limited to OpenSSL) that is licensed under separate terms, > + as designated in a particular file or component or in included license > + documentation. The authors of MySQL hereby grant you an additional > + permission to link the program and your derivative works with the > + separately licensed software that they have included with MySQL. > + > + This program is distributed in the hope that it will be useful, > + but WITHOUT ANY WARRANTY; without even the implied warranty of > + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + GNU General Public License, version 2.0, for more details. > + > + You should have received a copy of the GNU General Public License > + along with this program; if not, write to the Free Software > + Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA */ > + > +#include <stddef.h> > +#include <limits.h> > +#include <my_attribute.h> > +#include <my_global.h> > +#include "../storage/innobase/include/fts0tokenize.h" No, the plugin should be self-contained and not include anything from other plugins. At least if a plugin is as trivial as that one. > + > +#define U_LOW_LINE 0x5f // Code point for underscore '_'. > +#define is_alnum(ct) ((ct) & (_MY_U | _MY_L | _MY_NMR)) > + > +static int ngram_token_size; > + > +static MYSQL_SYSVAR_INT(token_size, ngram_token_size, > + 0, > + "Ngram full text plugin parser token size in characters", > + NULL, NULL, 2, 1, 10, 0); > + > +static int my_ci_charlen(CHARSET_INFO *cs, const uchar *str, const uchar *end) > +{ > + return my_charlen(cs, reinterpret_cast<const char *>(str), > + reinterpret_cast<const char *>(end)); > +} > + > +static int my_ci_ctype(CHARSET_INFO *cs, int *char_type, const uchar *str, > + const uchar *end) > +{ > + return cs->cset->ctype(cs, char_type, str, end); > +} > + > +static bool is_underscore(CHARSET_INFO *cs, const uchar *str, const uchar *end) > +{ > + my_wc_t wc; > + cs->cset->mb_wc(cs, &wc, str, end); > + return wc == U_LOW_LINE; > +} > + > +// Splits a string into ngrams and emits them. > +static int split_into_ngrams(MYSQL_FTPARSER_PARAM *param, const uchar *doc, > + int len, MYSQL_FTPARSER_BOOLEAN_INFO *bool_info) > +{ > + const CHARSET_INFO *cs= param->cs; > + const uchar *start= doc; > + const uchar *end= doc + len; > + const uchar *next= start; > + int n_chars= 0; > + int ret= 0; > + bool is_first= true; > + > + while (next < end) { > + int char_type; > + int char_len= my_ci_ctype(cs, &char_type, next, end); > + > + // Broken data? > + if (next + char_len > end || char_len == 0) > + break; > + > + // Avoid creating partial n-grams by stopping at word boundaries. > + // Underscore is treated as a word character too, since identifiers in > + // programming languages often contain them. > + if (!is_alnum(char_type) && !is_underscore(cs, next, end)) { > + start= next + char_len; > + next= start; > + n_chars= 0; > + continue; I don't think it's a good idea. On the opposite, I would rather create ngrams over word boundaries, so that, say, I'd split "n-gram plugin" to "n-g", "-gr", "gra", "ram", "am ", "m p", " pl", "plu", "lug", "ugi", "gin". May be better would be to split it into "n-gr", "gra", "ram", "am p", "m pl", "plu", "lug", "ugi", "gin". (that is, always three letters in an n-gram, not three characters). This needs testing to find out what approach works better. > + } > + > + next += char_len; > + n_chars++; > + > + if (n_chars == ngram_token_size) { > + bool_info->position= static_cast<uint>(start - doc); nope, we don't have MYSQL_FTPARSER_FULL_BOOLEAN_INFO::position and this plugin is definitely not the reason to break the API and add it. > + ret= param->mysql_add_word(param, reinterpret_cast<const char *>(start), > + static_cast<int>(next - start), bool_info); > + if (ret != 0) > + return ret; > + > + start += my_ci_charlen(cs, start, end); why do you scan the string again here, you've already counted character by character above? > + n_chars= ngram_token_size - 1; > + is_first= false; > + } > + } > + > + // If nothing was emitted yet, emit the remainder. what reminder, what is it for? > + switch (param->mode) { > + case MYSQL_FTPARSER_FULL_BOOLEAN_INFO: > + case MYSQL_FTPARSER_WITH_STOPWORDS: > + if (n_chars > 0 && is_first) { > + assert(next > start); > + assert(n_chars < ngram_token_size); > + > + bool_info->position= static_cast<uint>(start - doc); > + ret= param->mysql_add_word(param, reinterpret_cast<const char *>(start), > + static_cast<int>(next - start), bool_info); > + } > + break; > + > + default: > + break; > + } > + > + return ret; > +} > + > +static int number_of_chars(const CHARSET_INFO *cs, const uchar *buf, int len) > +{ > + const uchar *ptr= buf; > + const uchar *end= buf + len; > + int size= 0; > + > + while (ptr < end) { > + ptr += my_ci_charlen(cs, ptr, end); > + size += 1; > + } > + > + return size; > +} > + > +// Convert term into phrase and handle wildcard. > +static int ngram_term_convert(MYSQL_FTPARSER_PARAM *param, const uchar *token, > + int len, MYSQL_FTPARSER_BOOLEAN_INFO *bool_info) > +{ > + MYSQL_FTPARSER_BOOLEAN_INFO token_info= > + { FT_TOKEN_WORD, 0, 0, 0, 0, 0, ' ', 0 }; > + const CHARSET_INFO *cs= param->cs; > + int ret= 0; > + > + assert(bool_info->type == FT_TOKEN_WORD); > + assert(bool_info->quot == NULL); > + > + /* Convert rules: > + * 1. if term with wildcard and term length is less than ngram_token_size, > + * we keep it as normal term search. > + * 2. otherwise, term is converted to phrase and wildcard is ignored. > + * e.g. 'abc' and 'abc*' are both equivalent to '"ab bc"'. > + */ > + > + if (bool_info->trunc && number_of_chars(cs, token, len) < ngram_token_size) { > + ret= param->mysql_add_word(param, reinterpret_cast<const char *>(token), > + len, bool_info); > + } else { > + bool_info->type= FT_TOKEN_LEFT_PAREN; > + bool_info->quot= reinterpret_cast<char*>(1); > + > + ret= param->mysql_add_word(param, NULL, 0, bool_info); > + if (ret != 0) > + return ret; > + > + ret= split_into_ngrams(param, token, len, &token_info); > + if (ret != 0) > + return ret; > + > + bool_info->type= FT_TOKEN_RIGHT_PAREN; > + ret= param->mysql_add_word(param, NULL, 0, bool_info); > + > + assert(bool_info->quot == NULL); > + bool_info->type= FT_TOKEN_WORD; > + } > + > + return ret; > +} > + > +static int ngram_parser_parse(MYSQL_FTPARSER_PARAM *param) > +{ > + MYSQL_FTPARSER_BOOLEAN_INFO bool_info= > + { FT_TOKEN_WORD, 0, 0, 0, 0, 0, ' ', 0 }; > + const CHARSET_INFO *cs= param->cs; > + uchar *start= reinterpret_cast<uchar *>(const_cast<char *>(param->doc)); > + uchar *end= start + param->length; > + FT_WORD word= {NULL, 0, 0}; > + int ret= 0; > + > + switch (param->mode) { > + case MYSQL_FTPARSER_SIMPLE_MODE: > + case MYSQL_FTPARSER_WITH_STOPWORDS: > + ret= split_into_ngrams(param, start, static_cast<int>(end - start), > + &bool_info); > + break; > + > + case MYSQL_FTPARSER_FULL_BOOLEAN_INFO: > + // Reusing InnoDB's boolean mode parser to handle all special characters. > + while (fts_get_word(cs, &start, end, &word, &bool_info)) { nope. InnoDB might not even be compiled it. Use param->mysql_parse() > + if (bool_info.type == FT_TOKEN_WORD) { > + if (bool_info.quot != NULL) { > + ret= split_into_ngrams(param, word.pos, word.len, &bool_info); > + } else { > + ret= ngram_term_convert(param, word.pos, word.len, &bool_info); > + } > + } else { > + ret= param->mysql_add_word(param, reinterpret_cast<char*>(word.pos), > + word.len, &bool_info); > + } > + > + if (ret != 0) > + return ret; > + } > + > + break; > + } > + > + return ret; > +} > + > +static struct st_mysql_ftparser ngram_parser_descriptor= > +{ > + MYSQL_FTPARSER_INTERFACE_VERSION, > + ngram_parser_parse, > + NULL, > + NULL, > +}; > + > +static struct st_mysql_sys_var *ngram_system_variables[]= > +{ > + MYSQL_SYSVAR(token_size), > + NULL > +}; > + > +maria_declare_plugin(ngram_parser) > +{ > + MYSQL_FTPARSER_PLUGIN, // Type. > + &ngram_parser_descriptor, // Descriptor. > + "ngram", // Name. > + "Oracle Corp", // Author. > + "Ngram Full-Text Parser", // Description. > + PLUGIN_LICENSE_GPL, // License. > + NULL, // Initialization function. > + NULL, // Deinitialization function. > + 0x0100, // Numeric version. > + NULL, // Status variables. > + ngram_system_variables, // System variables. > + "1.0", // String version representation. > + MariaDB_PLUGIN_MATURITY_EXPERIMENTAL, // Maturity. > +} > +maria_declare_plugin_end; > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 12d5c4f2813: Fixed problems with S3 after "DROP TABLE FORCE" changes
by Sergei Golubchik 19 Oct '20

19 Oct '20

Hi, Monty! On Oct 16, Michael Widenius wrote: > revision-id: 12d5c4f2813 (mariadb-10.5.4-190-g12d5c4f2813) > parent(s): afa5f735f30 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2020-09-13 17:48:15 +0300 > message: > > Fixed problems with S3 after "DROP TABLE FORCE" changes > > MDEV-23691 S3 storage engine: delayed slave can drop the table please put the MDEV on the first line of the commit comment > > This commit also fixes all failing replication S3 tests. > > A slave is delayed if it is trying to execute replicated queries on a > table that is already converted to S3 by the master. What do you mean, a slave is delayed? > > Fixes for replication events on S3 tables for delayed slaves: > - INSERT and INSERT ... SELECT and CREATE TABLE are ignored but written > to the binary log. UPDATE & DELETE will be fixed in a future commit. > > Other things: > - On slaves with --s3-slave-ignore-updates set, allow S3 tables to be > opened in read-write mode. This was done to be able to > ignore-but-replicate queries like insert. Without this change any > open of an S3 table failed with 'Table is read only' which is too > early to be able to replicate the original query. > - Errors are now printed if handler::extra() call fails in > wait_while_tables_are_used() > - Error message for row changes are changed from HA_ERR_WRONG_COMMAND > to HA_ERR_TABLE_READONLY > - Disable some maria_extra() calls for S3 tables. This could cause > S3 tables to fail in some cases. > - Added missing thr_lock_delete() to ma_open() in case of failure > > diff --git a/sql/log.cc b/sql/log.cc > index 2a887a68606..9c656387e90 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -2151,6 +2151,12 @@ static int binlog_commit(handlerton *hton, THD *thd, bool all) > > DBUG_RETURN(0); > } > + /* > + This is true if we are doing an alter table that is replicated as > + CREATE TABLE ... SELECT > + */ > + if (thd->variables.option_bits & OPTION_BIN_COMMIT_OFF) > + DBUG_RETURN(0); why binlog_commit is even called here? > > DBUG_PRINT("debug", > ("all: %d, in_transaction: %s, all.modified_non_trans_table: %s, stmt.modified_non_trans_table: %s", > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index f4cbda5490c..7c103756566 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -7476,6 +7476,34 @@ int THD::binlog_query(THD::enum_binlog_query_type qtype, char const *query_arg, > DBUG_RETURN(0); > } > > + > +/** > + Binlog current query as a statement if binary log is open > + > + @retval false ok > + @retval true error > +*/ > + > + > +bool THD::binlog_current_query() > +{ > + if (!mysql_bin_log.is_open()) > + return 0; > + > + /* > + The following is needed if the query was "ignored" by a shared distributed > + engine, in which case decide_logging_format() will set the binlog filter > + */ > + reset_binlog_local_stmt_filter(); > + clear_binlog_local_stmt_filter(); > + return binlog_query(THD::STMT_QUERY_TYPE, query(), query_length(), > + /* is_trans */ FALSE, > + /* direct */ FALSE, > + /* suppress_use */ FALSE, > + /* Error */ 0) > 0; > +} you created a generally looking function with a rather special behavior. Please rename it to make sure it's not a general purpose helper. For example, call it binlog_current_query_unfiltered() or force_binlog_current_query(). And the function comment should say Force-binlog current query as a statement ignoring the binlog filter setting > + > + > void > THD::wait_for_wakeup_ready() > { > diff --git a/sql/sql_insert.cc b/sql/sql_insert.cc > index 4ad4c478937..5f59e064a35 100644 > --- a/sql/sql_insert.cc > +++ b/sql/sql_insert.cc > @@ -724,6 +724,7 @@ bool mysql_insert(THD *thd, TABLE_LIST *table_list, > Item *unused_conds= 0; > DBUG_ENTER("mysql_insert"); > > + bzero((char*) &info,sizeof(info)); why did you move it up? > create_explain_query(thd->lex, thd->mem_root); > /* > Upgrade lock type if the requested lock is incompatible with > @@ -764,16 +765,27 @@ bool mysql_insert(THD *thd, TABLE_LIST *table_list, > DBUG_RETURN(TRUE); > value_count= values->elements; > > - if (mysql_prepare_insert(thd, table_list, table, fields, values, > - update_fields, update_values, duplic, > - &unused_conds, FALSE)) > + if ((res= mysql_prepare_insert(thd, table_list, table, fields, values, > + update_fields, update_values, duplic, > + &unused_conds, FALSE))) > + { > + retval= thd->is_error(); > + if (res < 0) > + { > + /* > + Insert should be ignored but we have to log the query in statement > + format in the binary log > + */ > + res= thd->binlog_current_query(); abort label doesn't look at `res` it returns `retval`. So I don't think you need `retval= thd->is_error()` at all, and instead you need if (res < 0) retval= thd->binlog_current_query(); > + } > goto abort; > + } > + /* mysql_prepare_insert sets table_list->table if it was not set */ > + table= table_list->table; > > /* Prepares LEX::returing_list if it is not empty */ > if (returning) > result->prepare(returning->item_list, NULL); > - /* mysql_prepare_insert sets table_list->table if it was not set */ > - table= table_list->table; why did you move this up? > > context= &thd->lex->first_select_lex()->context; > /* > @@ -1575,21 +1587,29 @@ bool mysql_prepare_insert(THD *thd, TABLE_LIST *table_list, > DBUG_ASSERT (!select_insert || !values); > > if (mysql_handle_derived(thd->lex, DT_INIT)) > DBUG_RETURN(1); > if (table_list->handle_derived(thd->lex, DT_MERGE_FOR_INSERT)) > DBUG_RETURN(1); > if (thd->lex->handle_list_of_derived(table_list, DT_PREPARE)) > DBUG_RETURN(1); > > if (duplic == DUP_UPDATE) > { > /* it should be allocated before Item::fix_fields() */ > if (table_list->set_insert_values(thd->mem_root)) > DBUG_RETURN(1); > + } > + > + if (!table) > + table= table_list->table; As far as I can see, table in mysql_prepare_insert is *always* either table_list->table or NULL. At least in 10.5. So I'd rather remove the table argument completely and let mysql_prepare_insert always use table_list->table. > + > + if (table->file->check_if_updates_are_ignored("INSERT")) > + { > + DBUG_RETURN(-1); > } > > if (mysql_prepare_insert_check_table(thd, table_list, fields, select_insert)) > DBUG_RETURN(1); > > /* Prepare the fields in the statement. */ > if (values) > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index 15d190c3139..2843496f783 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -5206,7 +5215,19 @@ int create_table_impl(THD *thd, const LEX_CSTRING &orig_db, > goto err; > } > else if (options.if_not_exists()) > + { > + /* > + We never come here as part of normal create table as table existance > + is checked in open_and_lock_tables(). We may come here as part of > + ALTER TABLE > + */ How so? There is no ALTER IF NOT EXISTS. And this is the old code, so it seems to be for CREATE TABLE IF NOT EXISTS. > + > + /* Log CREATE IF NOT EXISTS on slave for distributed engines */ > + if (thd->slave_thread && (exists_hton && exists_hton->flags & > + HTON_IGNORE_UPDATES)) > + thd->log_current_statement= 1; > goto warn; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] MDEV-21265: IN predicate conversion to IN subquery should be allowed for a broader set of datatype comparison
by Sergey Petrunia 16 Oct '20

16 Oct '20

Hi Varun, > Commit 3e29c5a1620deef2fa9b25a0d42b4c07fe59e96f > > MDEV-21265: IN predicate conversion to IN subquery should be allowed for a broader set of datatype comparison > > Allow materialization strategy when the collation of the columns of the > left side of the IN subquery are not same as the one on the right side > of the IN subquery but are compatible with each other. > This allows conversion from utf8mb3 to utf8mb4 as the former is a subset of the later. Please fix the comment to say what the patch does: When collations on the inner and outer sides are the same, and the character set of the inner side is a proper subset of the charset on the outer. > > diff --git a/mysql-test/main/subselect4.result b/mysql-test/main/subselect4.result > index 020a70642913..f3ae2398d45e 100644 > --- a/mysql-test/main/subselect4.result > +++ b/mysql-test/main/subselect4.result > @@ -2718,3 +2718,172 @@ Warning 1931 Query execution was interrupted. The query examined at least 3020 r > SET join_cache_level= @save_join_cache_level; > DROP TABLE t1,t2,t3,t4; > # End of 10.2 tests > +# > +# MDEV-21265: IN predicate conversion to IN subquery should be allowed for a broader set of datatype comparison > +# > +CREATE TABLE t1 ( col1 varchar (32) collate utf8_general_ci); > +CREATE TABLE t2 ( col1 varchar (32) collate utf8mb4_general_ci, key(col1)); > +INSERT INTO t1 select seq from seq_1_to_100; > +INSERT INTO t2 values (_utf8mb4 X'F09F9883'), (1),('?'); ... > EXPLAIN FORMAT=JSON SELECT HEX(col1), col1 IN (SELECT col1 FROM t1) FROM t2; Please insert the '?' into the table t1, not t2. The idea of the check is to check for the following: - We get smiley symbol from table t2. - We make the lookup key from it. This results in an error and the '?' as output. - Then, a lookup is made in t1. And we are trying to provoke a wrong result by having the '?' in t1. OK to push into the custom-build tree when the above is addressed. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] 0d05c9f8699: Add mytop to Client component for rpm package
by Sergei Golubchik 13 Oct '20

13 Oct '20

Hi, Anel! About the diff to 1.9, I tend to agree that our mytop is a superset. I did have some comments when looking at the diff, couldn't help it :) Here they are: > diff --git a/scripts/mytop.sh b/../mytop-1.9.1-4 > index 321b79537fc..056b7a9a806 100644 > --- b/../mytop-1.9.1-4 > +++ a/scripts/mytop.sh > @@ -1,13 +1,13 @@ > -#!/usr/bin/perl -w > +#!/usr/bin/env perl I thought we're getting rid of "env perl" ? > # > -# $Id: mytop,v 1.91 2012/01/18 16:49:12 mgrennan Exp $ > +# $Id: mytop,v 1.99-maria6 2019/10/22 14:53:51 jweisbuch Exp $ > > =pod > > =head1 NAME > > -mytop - display MySQL server performance info like `top' > +mytop - display MariaDB server performance info like `top' may be "MariaDB/MySQL" here and below? > > =cut > > @@ -242,11 +253,15 @@ my $dsn; > > ## Socket takes precedence. > > -$dsn ="DBI:mysql:database=$config{db};mysql_read_default_group=mytop;"; > +if (eval {DBI->install_driver("MariaDB")}) { > + $dsn = "DBI:MariaDB:database=$config{db};mariadb_read_default_group=mytop;"; > +} else { > + $dsn = "DBI:mysql:database=$config{db};mysql_read_default_group=mytop;"; > +} > > if ($config{socket} and -S $config{socket}) > { > - $dsn .= "mysql_socket=$config{socket}"; > + $dsn .= "mariadb_socket=$config{socket}"; if you support both drivers, you need to use the correct option here > } > else > { > @@ -1027,10 +918,15 @@ sub GetData() > } > } > > - open L, "</proc/loadavg"; > - my $l = <L>; > - close L; > - chomp $l; > + my $l; > + if (-e "/proc/loadavg") > + { > + ## To avoid warnings if the OS is not Linux > + open (my $fh, "<", "/proc/loadavg"); > + ## Only the first 3 values are interresting rr? > + $l = join(" ", (split /\s+/, <$fh>)[0..2]); > + close $fh; > + } > > $last_time = $now_time; > > @@ -1960,11 +1694,15 @@ sub PrintHelp() > S - change slow query hightlighting > t - switch to thread view (default) > u - show only a specific user > + V - show variables > : - enter a command (not yet implemented) > ! - Skip an error that has stopped replications (at your own risk) > + L - show full queries (do not strip to terminal width) > + w - adjust the User and DB columns width > + a - toggle the progress column > > -${GREEN}http://jeremy.zawodny.com/mysql/mytop/${RESET} > -${GREEN}http://www.mysqlfanboy.com/mytop-3/${RESET} > +Base version from ${GREEN}http://www.mysqlfanboy.com/mytop-3${RESET} if it's mytop-3, perhaps it should have the version 3.x and not 1.99-maria6 ? > +This version comes as part of the ${GREEN}MariaDB${RESET} distribution. > ]; > > print $help; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

3 2

Re: [Maria-developers] d19f380d62c: MDEV-21445 Strange/inconsistent behavior of IN condition when mixing numbers and strings
by Sergei Golubchik 12 Oct '20

12 Oct '20

Hi, Alexander! On Oct 12, Alexander Barkov wrote: > revision-id: d19f380d62c (mariadb-10.3.21-213-gd19f380d62c) > parent(s): 90c8d773ed0 > author: Alexander Barkov <bar(a)mariadb.com> > committer: Alexander Barkov <bar(a)mariadb.com> > timestamp: 2020-08-18 13:19:21 +0400 > message: > > MDEV-21445 Strange/inconsistent behavior of IN condition when mixing numbers and strings > > Comparison is now performed as DECIMAL for pairs (int,string) and (string,int). Are there tests with strings like '1e100', outside of bigint range ? > Before the patch the comparison was performed in DOUBLE format, > so precision loss happened on huge values, e.g.: > > '97716021308405775' was converted to > 97716021308405770 (notice, the last digit was lost) > > diff --git a/sql/field.h b/sql/field.h > index be3a648617b..c9f5a34a3ae 100644 > --- a/sql/field.h > +++ b/sql/field.h > @@ -311,7 +311,7 @@ class Value_source > const char *end) > { > Converter_str2my_decimal_with_warn(NULL, Warn_filter_all(), > - E_DEC_FATAL_ERROR & ~E_DEC_BAD_NUM, > + 0, why did you remove warnings (here and elsewhere)? would be good to have it in the commit comment too, not just in email. > cs, cptr, end - cptr, decimal_value); > return decimal_value; > } > diff --git a/sql/sql_type.cc b/sql/sql_type.cc > index 9d3a47adfa5..7a847c90b8f 100644 > --- a/sql/sql_type.cc > +++ b/sql/sql_type.cc > @@ -764,6 +764,11 @@ Type_handler_hybrid_field_type::aggregate_for_comparison(const Type_handler *h) > m_type_handler= &type_handler_datetime; > } > } > + else if ((a == INT_RESULT && b == STRING_RESULT) || > + (b == INT_RESULT && a == INT_RESULT)) looks like a typo, the second line should be b == INT_RESULT && a == STRING_RESULT please add some tests to cover it. > + { > + m_type_handler= &type_handler_newdecimal; > + } > else if ((a == INT_RESULT || a == DECIMAL_RESULT) && > (b == INT_RESULT || b == DECIMAL_RESULT)) > { Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 2792940391a: MDEV-23327 Can't uninstall UDF if the implementation library file doesn't exist
by Sergei Golubchik 11 Oct '20

11 Oct '20

Hi, Oleksandr! Mostly ok, one problem below: On Oct 10, Oleksandr Byelkin wrote: > revision-id: 2792940391a (mariadb-10.2.31-343-g2792940391a) > parent(s): 2107e3bb9c0 > author: Oleksandr Byelkin <sanja(a)mariadb.com> > committer: Oleksandr Byelkin <sanja(a)mariadb.com> > timestamp: 2020-07-29 21:54:24 +0200 > message: > > MDEV-23327 Can't uninstall UDF if the implementation library file doesn't exist > > Made cleanup of DROP (udf) FUNCTION procedure and also check of mysql.func (not only loaded udf). > > @@ -606,43 +648,54 @@ int mysql_create_function(THD *thd,udf_func *udf) > } > > > -int mysql_drop_function(THD *thd,const LEX_STRING *udf_name) > +enum drop_udf_result mysql_drop_function(THD *thd, const LEX_STRING *udf_name) > { > TABLE *table; > - TABLE_LIST tables; > udf_func *udf; > DBUG_ENTER("mysql_drop_function"); > > + if (!(table= open_udf_func_table(thd))) > + DBUG_RETURN(UDF_DEL_RESULT_ERROR); > + > + // Pre-check without protection > + if (!find_udf_everywhere(thd, udf_name->str, udf_name->length, table)) you cannot check the udf_hash without a lock > + { > + close_mysql_tables(thd); Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 2

[Maria-developers] Unsubscribe
by TINKU BANSAL 11 Oct '20

11 Oct '20

Hi there, Please Unsubscribe me to this mailing list. Thanks

2 1

Re: [Maria-developers] 62c35fe93e1: MDEV-15703 Crash in EXECUTE IMMEDIATE 'CREATE OR REPLACE TABLE t1 (a INT DEFAULT ?)' USING DEFAULT
by Sergei Golubchik 09 Oct '20

09 Oct '20

Hi, Oleksandr! On Oct 09, Oleksandr Byelkin wrote: > revision-id: 62c35fe93e1 (mariadb-10.2.31-487-g62c35fe93e1) > parent(s): f584d567dbb > author: Oleksandr Byelkin <sanja(a)mariadb.com> > committer: Oleksandr Byelkin <sanja(a)mariadb.com> > timestamp: 2020-10-07 13:43:39 +0200 > message: > > MDEV-15703 Crash in EXECUTE IMMEDIATE 'CREATE OR REPLACE TABLE t1 (a INT DEFAULT ?)' USING DEFAULT > > part 2: > > - check that expressions are evaluable for > making empty row and assigning PS variable > > - correctly handling writing to a temporary tabe during multi-update > by setting associated field > > - Item::raise_error_not_evaluable() bakported from 10.4 and made vitrual > > - Item::is_evaluable_expression() bakported from 10.4 > > - Item::check_is_evaluable_expression_or_error() bakported from 10.4 I still don't quite understand all details of your patch. But here're more questions and comments below: > diff --git a/sql/field.cc b/sql/field.cc > index bdaaecc2026..8ce412a207d 100644 > --- a/sql/field.cc > +++ b/sql/field.cc > @@ -11183,6 +11205,16 @@ bool Field::save_in_field_default_value(bool view_error_processing) > { > THD *thd= table->in_use; > > + if ( // table is not opened properly > + table == NULL || table->pos_in_table_list == NULL || > + table->pos_in_table_list->select_lex == NULL || > + // we are in subquery/derived/CTE > + !table->pos_in_table_list->top_table()->select_lex->is_top_level_select()) > + { > + DBUG_ASSERT(0); // shoud not happened > + my_error(ER_INVALID_DEFAULT_PARAM, MYF(0)); > + return false; > + } I'd still think it'd be better to do a series of asserts. a complex condition to check for something that cannot ever happen? > if (flags & NO_DEFAULT_VALUE_FLAG && > real_type() != MYSQL_TYPE_ENUM) > { > @@ -11221,13 +11253,30 @@ bool Field::save_in_field_default_value(bool view_error_processing) > > bool Field::save_in_field_ignore_value(bool view_error_processing) > { > - enum_sql_command com= table->in_use->lex->sql_command; > - // All insert-like commands > - if (com == SQLCOM_INSERT || com == SQLCOM_REPLACE || > - com == SQLCOM_INSERT_SELECT || com == SQLCOM_REPLACE_SELECT || > - com == SQLCOM_LOAD) > - return save_in_field_default_value(view_error_processing); > - return 0; // ignore > + if ( // table is not opened properly > + table == NULL || table->pos_in_table_list == NULL || > + table->pos_in_table_list->select_lex == NULL || > + // we are in subquery/derived/CTE > + !table->pos_in_table_list->top_table()->select_lex->is_top_level_select()) > + { > + DBUG_ASSERT(0); // shoud not happened > + my_error(ER_INVALID_DEFAULT_PARAM, MYF(0)); > + return false; > + } ditto > + switch (find_ignore_reaction(table->in_use)) > + { > + case IGNORE_MEANS_DEFAULT: > + return save_in_field_default_value(view_error_processing); > + case IGNORE_MEANS_FIELD_VALUE: > + return 0; // ignore > + default: > + ; // fall through to error > + } > + > + // unexpected command > + DBUG_ASSERT(0); // shoud not happened > + my_error(ER_INVALID_DEFAULT_PARAM, MYF(0)); > + return false; > } > > > diff --git a/sql/item.cc b/sql/item.cc > index 994d45a9dc3..401ee5d6be2 100644 > --- a/sql/item.cc > +++ b/sql/item.cc > @@ -84,6 +85,17 @@ void item_init(void) > } > > > +void Item::raise_error_not_evaluable() > +{ > + String tmp; > + this->print(&tmp, QT_ORDINARY); > + // TODO-10.5: add an error message to errmsg-utf8.txt > + my_printf_error(ER_UNKNOWN_ERROR, > + "'%s' is not allowed in this context", MYF(0), tmp.ptr()); > + DBUG_ASSERT(0); // cannot be happen before 10.5 > +} > + > + better remove it, this dead code won't simplify merges. In 10.3 it'll be added with a merge normally. having check_is_evaluable_expression_or_error() makes sense though, it is used and copying it from 10.3 will help to merge indeed or, may be, just backport the patch from 10.3 to 10.2? > void Item::push_note_converted_to_negative_complement(THD *thd) > { > push_warning(thd, Sql_condition::WARN_LEVEL_NOTE, ER_UNKNOWN_ERROR, > @@ -8968,18 +9084,26 @@ bool Item_default_value::fix_fields(THD *thd, Item **items) > def_field->move_field_offset((my_ptrdiff_t) > (def_field->table->s->default_values - > def_field->table->record[0])); > - set_field(def_field); > - return FALSE; > + return def_field; > +} > > -error: > - context->process_error(thd); > - return TRUE; > +bool Item_default_value::associate_with_target_field(THD *thd, > + Item_field *field) > +{ > + associated= true; > + arg= field; iirc, `arg` used to tell the difference between DEFAULT and DEFAULT(x). how are you doing it now? > + return tie_field(thd); > } > > void Item_default_value::cleanup() > { > delete cached_field; // Free cached blob data > cached_field= 0; > + if (associated) > + { > + arg= NULL; > + associated= false; > + } > Item_field::cleanup(); > } > > @@ -9102,8 +9226,54 @@ void Item_ignore_value::print(String *str, enum_query_type query_type) > str->append(STRING_WITH_LEN("ignore")); > } > > + > +bool Item_ignore_value::associate_with_target_field(THD *thd, > + Item_field *field) > +{ > + bitmap_set_bit(field->field->table->read_set, field->field->field_index); > + return Item_default_value::associate_with_target_field(thd, field); > +} > + > + > int Item_ignore_value::save_in_field(Field *field_arg, bool no_conversions) > { > + if (arg) how is IGNORE with arg!=0 differs from IGNORE with arg==0? when happens what? > + { > + switch (find_ignore_reaction(field->table->in_use)) > + { > + case IGNORE_MEANS_DEFAULT: > + DBUG_ASSERT(0); // impossible now, but fully working code if needed > + /* > + save default value > + Note: (((Field*)(arg()->real_item()) is Item_field* (checked in > + tie_field(). > + */ > + if ((((Item_field*)(arg->real_item()))->field->flags & > + NO_DEFAULT_VALUE_FLAG)) > + { > + my_error(ER_NO_DEFAULT_FOR_FIELD, MYF(0), > + ((Item_field*)(arg->real_item()))->field->field_name); > + return true; > + } > + calculate(); > + return Item_field::save_in_field(field_arg, no_conversions); > + case IGNORE_MEANS_FIELD_VALUE: > + // checked on tie_field > + DBUG_ASSERT(arg->real_item()->type() == FIELD_ITEM); > + /* > + save original field (Item::save_in_field is not applicable because > + result_field for the referenced field set to this temporary table). > + */ > + arg->save_val(field_arg); > + return false; > + default: > + ; // fall through to error > + } > + DBUG_ASSERT(0); //impossible > + my_error(ER_INVALID_DEFAULT_PARAM, MYF(0)); > + return true; > + } > + > return field_arg->save_in_field_ignore_value(context->error_processor == > &view_error_processor); > } > diff --git a/sql/sp_head.cc b/sql/sp_head.cc > index 6a650183fb8..41a27a3aaaf 100644 > --- a/sql/sp_head.cc > +++ b/sql/sp_head.cc > @@ -418,6 +418,9 @@ sp_eval_expr(THD *thd, Field *result_field, Item **expr_item_ptr) > if (!(expr_item= sp_prepare_func_item(thd, expr_item_ptr))) > goto error; > > + if (expr_item->check_is_evaluable_expression_or_error()) > + goto error; for CALL(?) ? > + > /* > Set THD flags to emit warnings/errors in case of overflow/type errors > during saving the item into the field. > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > index cc77b58cb3e..b52d6f96a84 100644 > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -8345,6 +8346,9 @@ fill_record(THD *thd, TABLE *table, Field **ptr, List<Item> &values, > field->field_name, table->s->table_name.str); > } > > + if(evaluable && value->check_is_evaluable_expression_or_error()) > + goto err; Is it a good idea to do this check constantly for every user and for every row in selects, insert/update, etc? Could it be done inside IGNORE/DEFAULT? For example, by checking if field->table->tmp_table == INTERNAL_TMP_TABLE. > + > if (use_value) > value->save_val(field); > else > diff --git a/sql/unireg.cc b/sql/unireg.cc > index 083960523c1..91f9073c7ef 100644 > --- a/sql/unireg.cc > +++ b/sql/unireg.cc > @@ -1002,6 +1002,12 @@ static bool make_empty_rec(THD *thd, uchar *buff, uint table_options, > > int res= !expr->fixed && // may be already fixed if ALTER TABLE > expr->fix_fields(thd, &expr); > + if (!res && expr->check_is_evaluable_expression_or_error()) is this for CREATE TABLE case? > + { > + error= 1; > + delete regfield; //To avoid memory leak > + goto err; > + } > if (!res) > res= expr->save_in_field(regfield, 1); > if (!res && (field->flags & BLOB_FLAG)) Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 56f2b3d12b6: MDEV-23264 Unique blobs allow duplicate values upon UPDATE
by Sergei Golubchik 08 Oct '20

08 Oct '20

Hi, Sachin! I suspect is_null() could be wrong here. Please, add a test case where a table has a BEFORE UPDATE trigger and a long UNIQUE over a NOT NULL field. And see how cmp_binary_offset() is used in compare_record(), for example. On Oct 08, Sachin Setiya wrote: > revision-id: 56f2b3d12b6 (mariadb-10.4.11-385-g56f2b3d12b6) > parent(s): af834c218a5 > author: Sachin Setiya <sachin.setiya(a)mariadb.com> > committer: Sachin Setiya <sachin.setiya(a)mariadb.com> > timestamp: 2020-09-14 20:08:18 +0100 > message: > > MDEV-23264 Unique blobs allow duplicate values upon UPDATE > > Problem:- > We are able to insert duplicate value in table because cmp_binary_offset > is not able to differentiate between NULL and empty string. So > check_duplicate_long_entry_key is never called and we don't check for > duplicate. > Solution > Added a if condition with is_null() on field which can differentiate > between NULL and empty string. > > diff --git a/sql/handler.cc b/sql/handler.cc > index 40dea349272..71e6a904f23 100644 > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -6699,8 +6699,13 @@ static int check_duplicate_long_entries_update(TABLE *table, handler *h, uchar * > for (uint j= 0; j < key_parts; j++, keypart++) > { > field= keypart->field; > - /* Compare fields if they are different then check for duplicates*/ > - if(field->cmp_binary_offset(reclength)) > + /* > + Compare fields if they are different then check for duplicates > + cmp_binary_offset cannot differentiate between null and empty string > + So also check for that too > + */ > + if((field->is_null(0) != field->is_null(reclength)) || > + field->cmp_binary_offset(reclength)) > { > if((error= check_duplicate_long_entry_key(table, table->update_handler, > new_rec, i))) > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] Review of MDEV-4851
by Vicențiu Ciorbaru 07 Oct '20

07 Oct '20

Hi Daniel! The patch looks good, some comments about the test cases. About the commit message, we have --write-binlog not just --binlog. Also, check for typos in the last commit message and also, I would squash the two test cases commits into just one. > diff --git a/mysql-test/suite/rpl/t/rpl_mysql_upgrade.test b/mysql-test/suite/rpl/t/rpl_mysql_upgrade.test > index 1931e2eab2a..b836e889429 100644 > --- a/mysql-test/suite/rpl/t/rpl_mysql_upgrade.test > +++ b/mysql-test/suite/rpl/t/rpl_mysql_upgrade.test > @@ -14,8 +14,8 @@ call mtr.add_suppression("table or database name 'mysqltest-1'"); > > connection master; > --disable_warnings > -DROP DATABASE IF EXISTS `#mysql50#mysqltest-1`; > -CREATE DATABASE `#mysql50#mysqltest-1`; > +DROP DATABASE IF EXISTS `mysqltest-1`; > +CREATE DATABASE `mysqltest-1`; Since you are doing cleanup here anyway, let's get rid of the DROP statements a test case should not leave any dirty context behind and if we drop things just to be sure, we are masking potential problems. > --enable_warnings > sync_slave_with_master; > > @@ -34,13 +34,20 @@ if ($before_position == $after_position) > echo Master position is not changed; > } > > -#Some log events of the mysql_upgrade's will cause errors on slave. > +# Some log events of the mysql_upgrade previously caused errors on slave, > +# however with MDEV-4851 this should be ok, so we test it: > connection slave; > -STOP SLAVE SQL_THREAD; > -source include/wait_for_slave_sql_to_stop.inc; > +SET @old_general_log_state = @@global.general_log; > +SET @old_slow_log_state = @@global.slow_query_log; > +SET @old_log_output = @@global.log_output; > +SET GLOBAL general_log = 'ON'; > +SET GLOBAL slow_query_log = 'ON'; > +SET GLOBAL log_output = 'FILE'; > > connection master; > #With '--force' option, mysql_upgrade always executes all sql statements for upgrading. > +ALTER TABLE mysql.slow_log DROP COLUMN thread_id, DROP COLUMN rows_affected; > +DROP DATABASE IF EXISTS `mysqltest-1`; Why IF EXISTS? It *should* be there already. Can you add some syncs between master and slave and then show from the slave connection that the log table table is missing those columns and also that they get applied *after* mysql_upgrade is run? > --exec $MYSQL_UPGRADE --skip-verbose --write-binlog --force --user=root > $MYSQLTEST_VARDIR/log/mysql_upgrade.log 2>&1 > > let $datadir= `select @@datadir`; > @@ -55,8 +62,14 @@ if ($before_position != $after_position) > echo Master position has been changed; > } > > -DROP DATABASE `mysqltest-1`; Notice that the original code didn't use "IF EXISTS". > +sync_slave_with_master; > connection slave; > -DROP DATABASE `#mysql50#mysqltest-1`; > ---let $rpl_only_running_threads= 1 > +SET GLOBAL general_log = 'OFF'; > +SET GLOBAL slow_query_log = 'OFF'; > +truncate mysql.slow_log; > +truncate mysql.general_log; Why do you need to clear the tables? I tested without clearing and it still needs to pass. > +SET GLOBAL general_log = @old_general_log_state; > +SET GLOBAL slow_query_log = @old_slow_log_state; > +SET GLOBAL log_output = @old_log_output; > + > --source include/rpl_end.inc > diff --git a/mysql-test/t/log_tables.test b/mysql-test/t/log_tables.test > index f822ec8d758..0196e62f12f 100644 > --- a/mysql-test/t/log_tables.test > +++ b/mysql-test/t/log_tables.test

1 0

Re: [Maria-developers] Fwd: [Commits] 244f5bd: MDEV-23811: With large number of indexes optimizer chooses an inefficient plan
by Sergey Petrunia 06 Oct '20

06 Oct '20

Hello Igor, On Mon, Oct 05, 2020 at 10:13:17PM -0700, Igor Babaev wrote: > > I've accepted all your changes for my original fix. > This is a new patch for 10.2. > ... > diff --git a/sql/opt_range.cc b/sql/opt_range.cc > index e933d2a..9f1fb9a 100644 > --- a/sql/opt_range.cc > +++ b/sql/opt_range.cc > @@ -8899,11 +8902,10 @@ tree_or(RANGE_OPT_PARAM *param,SEL_TREE > *tree1,SEL_TREE *tree2) > { > SEL_ARG *key1= tree1->keys[key_no]; > SEL_ARG *key2= tree2->keys[key_no]; > - if (!must_be_ored) > - { > + if (!must_be_ored || no_merges1) > key1->incr_refs(); > + if (!must_be_ored || no_merges2) > key2->incr_refs(); > - } > if ((result->keys[key_no]= key_or(param, key1, key2))) > result->keys_map.set_bit(key_no); > } What is the purpose of this change? As far as I understand, the bug is fixed without it, too? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Test failure in main.flush_logs_not_windows on x32
by Otto Kekäläinen 06 Oct '20

06 Oct '20

Hello! After uploading MariaDB 10.5.5 to Debian I came across that the test main.mysqld--help is failing on arch x32: ****************** main.flush_logs_not_windows w2 [ fail ] Test ended at 2020-09-28 19:25:55 CURRENT_TEST: main.flush_logs_not_windows mysqltest: At line 10: query 'flush logs' succeeded - should have failed with errno 1004... ****************** Full log: https://people.debian.org/~jrtc27/mariadb-10.5_10.5.5-1+jrtc27.1_x32-2020-0… Any tips on what to do about this one?

1 0

[Maria-developers] Test failure in main.mysqld--help on x32
by Otto Kekäläinen 05 Oct '20

05 Oct '20

Hello! After uploading MariaDB 10.5.5 to Debian I came across that the test main.mysqld--help is failing on arch x32: ****************** main.mysqld--help 'unix' w1 [ fail ] Test ended at 2020-09-28 19:24:57 CURRENT_TEST: main.mysqld--help --- /<<PKGBUILDDIR>>/mysql-test/main/mysqld--help.result 2020-08-07 12:57:35.000000000 +0000 +++ /<<PKGBUILDDIR>>/mysql-test/main/mysqld--help.reject 2020-09-28 19:24:57.628353544 +0000 @@ -1797,10 +1797,10 @@ sync-relay-log-info 10000 sysdate-is-now FALSE system-versioning-alter-history ERROR -table-cache 421 +table-cache 2000 table-definition-cache 400 -table-open-cache 421 -table-open-cache-instances 1 +table-open-cache 2000 +table-open-cache-instances 8 tc-heuristic-recover OFF tcp-keepalive-interval 0 tcp-keepalive-probes 0 main.flush_logs_not_windows w2 [ fail ] Test ended at 2020-09-28 19:25:55 CURRENT_TEST: main.flush_logs_not_windows mysqltest: At line 10: query 'flush logs' succeeded - should have failed with errno 1004... ****************** Any tips on what to do about this? On Zulip I was tipped about https://github.com/MariaDB/server/commit/deb365581b96527c5c96b3f5ed1235e105… but it does not apply on 10.5.

2 2

[Maria-developers] Test failure in main.index_intersect on hppa
by Otto Kekäläinen 04 Oct '20

04 Oct '20

Hello! I already asked this on Zulip but did not get anywhere, so re-posting here on the mailing list. The basic test suite run directly after the build fails with the following test on hppa on Debian: https://buildd.debian.org/status/fetch.php?pkg=mariadb-10.5&arch=hppa&ver=1… ****************** main.index_intersect w3 [ fail ] Test ended at 2020-09-27 16:41:38 CURRENT_TEST: main.index_intersect --- /<<PKGBUILDDIR>>/mysql-test/main/index_intersect.result 2020-08-07 12:57:35.000000000 +0000 +++ /<<PKGBUILDDIR>>/mysql-test/main/index_intersect.reject 2020-09-27 16:41:38.035115473 +0000 @@ -715,7 +715,7 @@ SELECT * FROM City WHERE Name BETWEEN 'G' AND 'J' AND Population > 500000 AND Country LIKE 'C%'; id select_type table type possible_keys key key_len ref rows Extra -1 SIMPLE City range Population,Country,Name Name 35 NULL # Using index condition; Using where +1 SIMPLE City index_merge Population,Country,Name Name,Country 35,3 NULL # Using sort_intersect(Name,Country); Using where EXPLAIN SELECT * FROM City WHERE ID BETWEEN 1 AND 500 AND Population > 700000 AND Country LIKE 'C%'; mysqltest: Result l ****************** Any suggestions what this 'range' -> 'index_merge' word change in the result might step from?

1 0

Re: [Maria-developers] 19bdb03e97d: MDEV-21211 LOCK_plugin optimization.
by Sergei Golubchik 02 Oct '20

02 Oct '20

Hi, Alexey! Thanks, this is great! Comments are below, there aren't many. The main concern - you only do RCU when installingplugins, but not when uninstalling. I'm not sure how it's supposed to work. On Oct 02, Alexey Botchkov wrote: > revision-id: 19bdb03e97d (mariadb-10.4.11-337-g19bdb03e97d) > parent(s): 0041dacc1b8 > author: Alexey Botchkov <holyfoot(a)mariadb.com> > committer: Alexey Botchkov <holyfoot(a)mariadb.com> > timestamp: 2020-08-11 00:01:53 +0400 > message: > > MDEV-21211 LOCK_plugin optimization. > > Use Apc_target calls to 'notify' threads about changes in plugins. > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > index bf33148811e..1f38c55b922 100644 > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -1348,10 +1348,10 @@ bool do_command(THD *thd) > my_net_set_read_timeout(net, thd->variables.net_read_timeout); > > DBUG_ASSERT(packet_length); > - DBUG_ASSERT(!thd->apc_target.is_enabled()); > + thd->apc_target.enable(); > return_value= dispatch_command(command, thd, packet+1, > (uint) (packet_length-1), FALSE, FALSE); > - DBUG_ASSERT(!thd->apc_target.is_enabled()); > + thd->apc_target.disable(); is this because you don't want to post apc requests to threads that wait for the client? > > out: > thd->lex->restore_set_statement_var(); > diff --git a/sql/sql_plugin.cc b/sql/sql_plugin.cc > index 981420f03ae..84438e3ff70 100644 > --- a/sql/sql_plugin.cc > +++ b/sql/sql_plugin.cc > @@ -217,17 +217,28 @@ static struct > #include "sql_plugin_services.ic" > > /* > - A mutex LOCK_plugin must be acquired before accessing the > - following variables/structures. > - We are always manipulating ref count, so a rwlock here is unneccessary. > + A mutex LOCK_plugin must be acquired when we change the > + new_plugins_state content and when we swap the global_plugins_state > + and the new_plugins_state. > */ > mysql_mutex_t LOCK_plugin; > +struct st_plugins_state > +{ > + DYNAMIC_ARRAY m_array; > + HASH m_hash[MYSQL_MAX_PLUGIN_TYPE_NUM]; > +#ifndef DBUG_OOF DBUG_OFF > + uint m_ref_counter; > +#endif > +}; > + > static DYNAMIC_ARRAY plugin_dl_array; > -static DYNAMIC_ARRAY plugin_array; > -static HASH plugin_hash[MYSQL_MAX_PLUGIN_TYPE_NUM]; > static MEM_ROOT plugin_mem_root; > static bool reap_needed= false; > volatile int global_plugin_version= 1; > +static st_plugins_state plugins_states[2]; > +st_plugins_state *global_plugins_state= plugins_states; it seems this can be static too, you don't use it outside of sql_plugin.cc (and making it global_plugins_state= &plugins_states[0], new_plugins_state= &plugins_state[1]; would be a bit more explicit) > +static st_plugins_state *new_plugins_state= plugins_states + 1; > + > > static bool initialized= 0; > ulong dlopen_count; > @@ -871,7 +882,8 @@ static void plugin_dl_del(struct st_plugin_dl *plugin_dl) > } > > > -static struct st_plugin_int *plugin_find_internal(const LEX_CSTRING *name, > +static struct st_plugin_int *plugin_find_internal(st_plugins_state *st, shouldn't `st` be const here too? > + const LEX_CSTRING *name, > int type) > { > uint i; > @@ -879,21 +891,19 @@ static struct st_plugin_int *plugin_find_internal(const LEX_CSTRING *name, > if (! initialized) > DBUG_RETURN(0); > > - mysql_mutex_assert_owner(&LOCK_plugin); may be you'd want here something like if (st == new_plugins_state) mysql_mutex_assert_owner(&LOCK_plugin); > - > if (type == MYSQL_ANY_PLUGIN) > { > for (i= 0; i < MYSQL_MAX_PLUGIN_TYPE_NUM; i++) > { > struct st_plugin_int *plugin= (st_plugin_int *) > - my_hash_search(&plugin_hash[i], (const uchar *)name->str, name->length); > + my_hash_search(&st->m_hash[i], (const uchar *)name->str, name->length); > if (plugin) > DBUG_RETURN(plugin); > } > } > else > DBUG_RETURN((st_plugin_int *) > - my_hash_search(&plugin_hash[type], (const uchar *)name->str, > + my_hash_search(&st->m_hash[type], (const uchar *)name->str, > name->length)); > DBUG_RETURN(0); > } > @@ -1210,13 +1209,6 @@ static void plugin_variables_deinit(struct st_plugin_int *plugin) > > static void plugin_deinitialize(struct st_plugin_int *plugin, bool ref_check) > { > - /* > - we don't want to hold the LOCK_plugin mutex as it may cause > - deinitialization to deadlock if plugins have worker threads > - with plugin locks > - */ > - mysql_mutex_assert_not_owner(&LOCK_plugin); does it mean you *do* keep LOCK_plugin here now? > - > if (plugin->plugin->status_vars) > { > /* > @@ -1280,24 +1273,25 @@ static void plugin_del(struct st_plugin_int *plugin) > DBUG_VOID_RETURN; > } > > + > static void reap_plugins(void) > { > uint count; > struct st_plugin_int *plugin, **reap, **list; > > - mysql_mutex_assert_owner(&LOCK_plugin); > - > if (!reap_needed) > return; > > + mysql_mutex_lock(&LOCK_plugin); > reap_needed= false; > - count= plugin_array.elements; > + count= global_plugins_state->m_array.elements; > reap= (struct st_plugin_int **)my_alloca(sizeof(plugin)*(count+1)); > *(reap++)= NULL; > > for (uint i=0; i < MYSQL_MAX_PLUGIN_TYPE_NUM; i++) > { > - HASH *hash= plugin_hash + plugin_type_initialization_order[i]; > + HASH *hash= global_plugins_state->m_hash + > + plugin_type_initialization_order[i]; do you need LOCK_plugin when changing plugin states? your comment at the LOCK_plugin declaration doesn't mention it. > for (uint j= 0; j < hash->records; j++) > { > plugin= (struct st_plugin_int *) my_hash_element(hash, j); > @@ -1542,6 +1529,38 @@ static void init_plugin_psi_keys(void) > } > #endif /* HAVE_PSI_INTERFACE */ > > + > +static bool copy_plugins_state(st_plugins_state *dest, st_plugins_state *src) > +{ > + uint i; > + mysql_mutex_assert_owner ? for documentation purposes. > + reset_dynamic(&dest->m_array); > + for (i= 0; i < MYSQL_MAX_PLUGIN_TYPE_NUM; i++) > + my_hash_reset(&dest->m_hash[i]); > + > + for (i= 0; i < src->m_array.elements; i++) > + { > + struct st_plugin_int *ptr; > + ptr= *dynamic_element(&src->m_array, i, struct st_plugin_int **); > + if (insert_dynamic(&dest->m_array, (uchar *) &ptr) || > + (ptr->state != PLUGIN_IS_FREED && > + my_hash_insert(&dest->m_hash[ptr->plugin->type], > + (uchar*) ptr))) > + goto err; > + } > + > + return false; > +err: > + return true; > +} > + > + > +static void set_new_global_plugins_state() > +{ mysql_mutex_assert_owner ? for documentation purposes. > + swap_variables(st_plugins_state *, global_plugins_state, new_plugins_state); > +} > + > + > /* > The logic is that we first load and initialize all compiled in plugins. > From there we load up the dynamic types (assuming we have not been told to > @@ -1815,7 +1834,9 @@ static void plugin_load(MEM_ROOT *tmp_root) > tables.init_one_table(&MYSQL_SCHEMA_NAME, &MYSQL_PLUGIN_NAME, 0, TL_READ); > tables.open_strategy= TABLE_LIST::OPEN_NORMAL; > > + set_new_global_plugins_state(); > result= open_and_lock_tables(new_thd, &tables, FALSE, MYSQL_LOCK_IGNORE_TIMEOUT); > + set_new_global_plugins_state(); why here, why twice? > > table= tables.table; > if (result) > @@ -1851,52 +1872,31 @@ static void plugin_load(MEM_ROOT *tmp_root) > if (!name.length || !dl.length) > continue; > > - /* > - Pre-acquire audit plugins for events that may potentially occur > - during [UN]INSTALL PLUGIN. > - > - When audit event is triggered, audit subsystem acquires interested > - plugins by walking through plugin list. Evidently plugin list > - iterator protects plugin list by acquiring LOCK_plugin, see > - plugin_foreach_with_mask(). > - > - On the other hand plugin_load is acquiring LOCK_plugin > - rather for a long time. > - > - When audit event is triggered during plugin_load plugin > - list iterator acquires the same lock (within the same thread) > - second time. > - > - This hack should be removed when LOCK_plugin is fixed so it > - protects only what it supposed to protect. > - > - See also mysql_install_plugin(), mysql_uninstall_plugin() and > - initialize_audit_plugin() > - */ > - if (mysql_audit_general_enabled()) > - mysql_audit_acquire_plugins(new_thd, event_class_mask); > - \o/ > /* > there're no other threads running yet, so we don't need a mutex. > but plugin_add() before is designed to work in multi-threaded > environment, and it uses mysql_mutex_assert_owner(), so we lock > the mutex here to satisfy the assert > */ > - mysql_mutex_lock(&LOCK_plugin); > plugin_add(tmp_root, false, &name, &dl, MYF(ME_ERROR_LOG)); > free_root(tmp_root, MYF(MY_MARK_BLOCKS_FREE)); > - mysql_mutex_unlock(&LOCK_plugin); > } > if (unlikely(error > 0)) > sql_print_error(ER_THD(new_thd, ER_GET_ERRNO), my_errno, > table->file->table_type()); > + set_new_global_plugins_state(); > end_read_record(&read_record_info); > table->mark_table_for_reopen(); > close_mysql_tables(new_thd); > -end: > +err_ret: > new_thd->db= null_clex_str; // Avoid free on thd->db > delete new_thd; > + set_new_global_plugins_state(); > DBUG_VOID_RETURN; > + > +end: > + set_new_global_plugins_state(); > + goto err_ret; eh, really? you swap twice again? > } > > > @@ -2058,13 +2055,8 @@ void plugin_shutdown(void) > plugin_deinitialize(plugins[i], false); > } > > - /* > - It's perfectly safe not to lock LOCK_plugin, as there're no > - concurrent threads anymore. But some functions called from here > - use mysql_mutex_assert_owner(), so we lock the mutex to satisfy it > - */ > - mysql_mutex_lock(&LOCK_plugin); > > + mysql_mutex_lock(&LOCK_plugin); the comment was helpful, I think > /* > We defer checking ref_counts until after all plugins are deinitialized > as some may have worker threads holding on to plugin references. > @@ -2176,6 +2173,44 @@ static bool finalize_install(THD *thd, TABLE *table, const LEX_CSTRING *name, > return 0; > } > > + > +/* > + Request object for ending the new_plugins_state change. > +*/ > + > +class Ping_all_threads_request : public Apc_target::Apc_call > +{ > +public: > + /* Overloaded virtual functions. */ > + void call_in_target_thread(); > +}; > + > + > +void Ping_all_threads_request::call_in_target_thread() > +{ > + /* Do nothing. We just make sure the thread got to this point. */ > +} > + > + > +static my_bool fix_plugins_state_callback(THD *thd, THD *caller_thd) > +{ > + bool timed_out; > + int timeout_sec= 30; > + > + Ping_all_threads_request ping_req; > + > + mysql_mutex_lock(&thd->LOCK_thd_kill); > + if (thd->get_command() == COM_SLEEP) here you also skip threads waiting on the client? > + { > + mysql_mutex_unlock(&thd->LOCK_thd_kill); > + return FALSE; > + } > + thd->apc_target.make_apc_call(caller_thd, &ping_req, > + timeout_sec, &timed_out); > + return FALSE; > +} > + > + > bool mysql_install_plugin(THD *thd, const LEX_CSTRING *name, > const LEX_CSTRING *dl_arg) > { > @@ -2252,12 +2272,19 @@ bool mysql_install_plugin(THD *thd, const LEX_CSTRING *name, > > if (unlikely(error != INSTALL_GOOD)) > { > + mysql_mutex_unlock(&LOCK_plugin); > reap_needed= true; > reap_plugins(); > + goto err; > } > -err: > global_plugin_version++; > + set_new_global_plugins_state(); > + thd->apc_target.disable(); > mysql_mutex_unlock(&LOCK_plugin); > + server_threads.iterate(fix_plugins_state_callback, thd); > + thd->apc_target.enable(); a comment be good here. to explain why you disable apc and why disable/unlock/iterate must happen in this particlar order. > + > +err: > if (argv) > free_defaults(argv); > DBUG_RETURN(error == INSTALL_FAIL_NOT_OK); > @@ -2414,10 +2416,12 @@ bool mysql_uninstall_plugin(THD *thd, const LEX_CSTRING *name, > error|= !MyFlags; > } > } > - reap_plugins(); > - > global_plugin_version++; > + thd->apc_target.disable(); > mysql_mutex_unlock(&LOCK_plugin); > + server_threads.iterate(fix_plugins_state_callback, thd); > + reap_plugins(); > + thd->apc_target.enable(); I don't quite understand this. You don't do copy-update here. I presume because plugins are only marked PLUGIN_IS_FREED but not actually removed from the array or hash? but you modify plugin state under a mutex, and read it without a mutex. how is that supposed to work? > DBUG_RETURN(error); > #ifdef WITH_WSREP > wsrep_error_label: Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 6f211b1f01b: MDEV-23518 Syntax error in ond SP results in misleading message on SHOW CREATE PROCEDURE
by Sergei Golubchik 02 Oct '20

02 Oct '20

Hi, Oleksandr! On Oct 01, Oleksandr Byelkin wrote: > revision-id: 6f211b1f01b (mariadb-10.4.11-369-g6f211b1f01b) > parent(s): 571764c04fc > author: Oleksandr Byelkin <sanja(a)mariadb.com> > committer: Oleksandr Byelkin <sanja(a)mariadb.com> > timestamp: 2020-08-28 16:05:38 +0200 > message: > > MDEV-23518 Syntax error in ond SP results in misleading message on SHOW CREATE PROCEDURE > > Add info to the error message how to get details about error which happened. > > diff --git a/mysql-test/main/errors.test b/mysql-test/main/errors.test > index f3aedc8e491..a4b95d48e5d 100644 > --- a/mysql-test/main/errors.test > +++ b/mysql-test/main/errors.test > @@ -246,3 +246,39 @@ SELECT UPDATEXML(-73 * -2465717823867977728,@@global.auto_increment_increment,nu > SELECT UPDATEXML(-73 * -2465717823867977728,@@global.long_query_time,null); > > -- echo # End of 10.3 tests > + > +--echo # > +--echo # MDEV-23518: Syntax error in ond SP results in misleading > +--echo # message on SHOW CREATE PROCEDURE > +--echo # > + > +DELIMITER $$; > + > +CREATE PROCEDURE P1 () > +BEGIN NOT ATOMIC > + IF (SELECT 2) THEN > + SELECT 4; > + END IF ; > +END; > +$$ > + > +DELIMITER ;$$ you could've had just CREATE PROCEDURE P1 () SELECT 1; one line, no need to change a delimiter. You overwrite the body below anyway :) > + > +select name,db,body from mysql.proc where name = "P1"; > +update mysql.proc set body_utf8="BEGIN NOT ATOMIC > +IF (SELECT 2) OR foo = 3 THEN > +SELECT 4; > +END IF ; > +END", body="BEGIN NOT ATOMIC > +IF (SELECT 2) OR foo = 3 THEN > +SELECT 4; > +END IF ; > +END"where name = "P1"; > + > +--error ER_SP_PROC_TABLE_CORRUPT > +show create procedure P1; > +show warnings; > + > +drop procedure P1; > + > diff --git a/sql/share/errmsg-utf8.txt b/sql/share/errmsg-utf8.txt > index ddc52702e42..582c80e2ad8 100644 > --- a/sql/share/errmsg-utf8.txt > +++ b/sql/share/errmsg-utf8.txt > @@ -5714,8 +5714,7 @@ ER_SP_RECURSION_LIMIT > eng "Recursive limit %d (as set by the max_sp_recursion_depth variable) was exceeded for routine %.192s" > ger "Rekursionsgrenze %d (durch Variable max_sp_recursion_depth gegeben) wurde für Routine %.192s überschritten" > ER_SP_PROC_TABLE_CORRUPT > - eng "Failed to load routine %-.192s. The table mysql.proc is missing, corrupt, or contains bad data (internal code %d)" > - ger "Routine %-.192s konnte nicht geladen werden. Die Tabelle mysql.proc fehlt, ist beschädigt, oder enthält fehlerhaften Daten (interner Code: %d)" > + eng "Failed to load routine %-.192s. The table mysql.proc is missing, corrupt, etc. (internal code %d). For more details, run SHOW WARNINGS" Okay, so you just extended the error message. Perhaps "The table mysql.proc is missing, corrupt, etc." is now redundant and shoud go away too? Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] New patch on the JSON_TABLE.
by Sergey Petrunia 27 Sep '20

27 Sep '20

Hi Alexey, On Mon, Sep 14, 2020 at 03:38:38PM +0400, Alexey Botchkov wrote: > Hi, Sergei!. > > so the branch name is 'bb-mdef17399-hf'. > It has slightly changed since last patch so please pay attention. > > There i present the patch that resolves issues that you mentioned > in your last comments. > One important is not fixed though. > That is: > CREATE TABLE t1(id INT, f1 JSON); > INSERT INTO t1 VALUES (1, '{\"1\": 1}'), (2, '{\"1\": 2}'), (3, '{\"1\": > 3}'), > (4, '{\"1\": 4}'), (5, '{\"1\": 5}'), (6, '{\"1\": 6}'); > > SELECT * FROM t1 WHERE id IN > (SELECT id FROM t1 as tt2, > JSON_TABLE(f1, "$" COLUMNS (jf FOR ORDINALITY)) AS tbl); > > That SELECT crashes in the optimizer, as the greedy_search() can't find any > satisfying plan. > I got rid of the crash with this line: > @@ -9520,7 +9528,7 @@ best_extension_by_limited_search(JOIN *join, > table_map real_table_bit= s->table->map; > if ((remaining_tables & real_table_bit) && > (allowed_tables & real_table_bit) && > - !(remaining_tables & s->dependent) && > +// !(remaining_tables & s->dependent) && > > But i don't mean it's an acceptable fix. > > It seems to me the problem is in the optimizer, so i'd like to ask your > opinion. > Maybe you just know the answer at once. After some debugging I see that the following happens: 1. Execution reaches Table_function_json_table::setup It computes m_dep_tables= m_json->used_tables(); this is =1 (In the subquery, table tt2 has map=1) 2. Then, subquery is converted into a semi-join. That is, it is merged into its parent subquery. There, we have: table t1: map=1 table tt2: map=2 table JSON_TABLE(...) AS tbl: map=3 3. The subquery was uncorrelated, so SJ-Materialization is a possible option and the join optimizer attempts to construct a join order for the subquery tables. However it can't succeed, as the table "JSON_TABLE(..) AS tbl" is set to be dependent on table with map=1, which is not put into the join order because it is not a part of subquery. As a result, we get assertion failure when the optimizer fails to produce any join orders. The probem here is on step #2. When the subquery's tables are moved to the parent query, they get new values of TABLE::map, and also all their attributes that contain table maps need to be updated. This done here in convert_subq_to_sj() function: ``` /* n. Walk through child's tables and adjust table->map */ List_iterator_fast<TABLE_LIST> si(subq_lex->leaf_tables); while ((tl= si++)) { tl->set_tablenr(table_no); if (tl->is_jtbm()) { tl->jtbm_table_no= table_no; Item *dummy= tl->jtbm_subselect; tl->jtbm_subselect->fix_after_pullout(parent_lex, &dummy, true); ``` In that code one can see: if the subquery has table that has 'is_jtbm()==true', then ... some processing is done to re-compute its attributes after the table has been pulled up into the parent select. The same should be done for TABLE_LIST elements that are JSON_TABLE(..). If you'll introduce a new function, fix_after_pullout() is a good name. It is already used for Item-derived classes. > If not, i'll dig into it. > And you're welcome to observe the new patch. > > Best regards. > HF -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

2 1

[Maria-developers] MDEV-21829: Unique class: discussion followup part #2
by Sergey Petrunia 25 Sep '20

25 Sep '20

Hello Igor, Trying to capture the input from the last call. Currently class Unique is using RB-Tree for its in-memory storage. RB-Tree is implemented by struct TREE and tree_XXX() functions (further I'll refer to them both as just "TREE"). It is possible to get Unique to use some other data structure for its in-memory storage. The most obvious candidate is a hash table. (There are currently no plans to do this, though). So, your request was that - An interface is developed for in-memory storage (let's refer to it as Unique::MemStorage) - An implementation of this interface is developed that uses TREE. (We can't change the TREE itself as it is used in multple places). Is this really the intent? Please clarify. My reaction is that the above described change does not seem to have any relevance to MDEV-21829. MDEV-21829 is about supporting variable-length keys, not about changing, or preparing to change Unique::MemStorage. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 1

Re: [Maria-developers] 2564334b4c1: Backported setting of transcation.on=1 in THD::reset_for_reuse()
by Sergei Golubchik 25 Sep '20

25 Sep '20

Hi, Michael! > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index 7327f270c33..8f6356b15c7 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -1640,6 +1640,7 @@ void THD::reset_for_reuse() > abort_on_warning= 0; > free_connection_done= 0; > m_command= COM_CONNECT; > + transaction.on= 1; > #if defined(ENABLED_PROFILING) > profiling.reset(); > #endif This looks risky, to change transaction.on at some random point in time. I understand that it's not a random point and it should be actually safe here. But perhaps you can add an assert to document that it is safe? Like, transaction.on should be already 1 or there should be no active transaction. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] fc07306120f: MDEV-23586 Mariabackup: GTID saved for replication in 10.4.14 is wrong
by Sergei Golubchik 25 Sep '20

25 Sep '20

Hi, Monty! Just a couple of minor comments Note, the below is from the combined diff of this commit and your earlier MDEV-21953 commit, so you can see some lines here that are not part of the fc07306120f. > diff --git a/sql/handler.cc b/sql/handler.cc > index cc7eedd18f2..402db310ed6 100644 > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -133,6 +133,23 @@ static plugin_ref ha_default_tmp_plugin(THD *thd) > return ha_default_plugin(thd); > } > > +#if defined(WITH_ARIA_STORAGE_ENGINE) && MYSQL_VERSION_ID < 100500 > +void ha_maria_implicit_commit(THD *thd, bool new_trn) > +{ > + if (ha_maria::has_active_transaction(thd)) > + { > + int error; > + MDL_request mdl_request; > + mdl_request.init(MDL_key::BACKUP, "", "", MDL_BACKUP_COMMIT, MDL_EXPLICIT); > + error= thd->mdl_context.acquire_lock(&mdl_request, > + thd->variables.lock_wait_timeout); > + ha_maria::implicit_commit(thd, new_trn); > + if (!error) > + thd->mdl_context.release_lock(mdl_request.ticket); This looks rather fishy. You commit in aria unconditionally? whether you got the lock or not? > + } > +} > +#endif > + > > /** @brief > Return the default storage engine handlerton for thread > @@ -1753,13 +1777,14 @@ int ha_commit_one_phase(THD *thd, bool all) > if ((res= thd->wait_for_prior_commit())) > DBUG_RETURN(res); > } > - res= commit_one_phase_2(thd, all, trans, is_real_trans); > + res= commit_one_phase_2(thd, all, trans, is_real_trans, rw_trans); > DBUG_RETURN(res); > } > > > static int > -commit_one_phase_2(THD *thd, bool all, THD_TRANS *trans, bool is_real_trans) > +commit_one_phase_2(THD *thd, bool all, THD_TRANS *trans, bool is_real_trans, > + bool rw_trans) rw_trans seems to be unused here (also in ha_commit_one_phase()) > { > int error= 0; > uint count= 0; > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index 40e606425c5..71b447fb920 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -1383,7 +1384,11 @@ void THD::update_all_stats() > void THD::init_for_queries() > { > set_time(); > - ha_enable_transaction(this,TRUE); > + /* > + We don't need to call ha_enable_transaction() as we can't have > + any active transactions that has to be commited > + */ > + transaction.on= TRUE; I just commented on the similar code in another commit of yours. don't just say "can't have any active transactions", add an assert instead, please. > > reset_root_defaults(mem_root, variables.query_alloc_block_size, > variables.query_prealloc_size); Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] New patch on the JSON_TABLE.
by Sergey Petrunia 23 Sep '20

23 Sep '20

One more thing: On Mon, Sep 14, 2020 at 03:38:38PM +0400, Alexey Botchkov wrote: > CREATE TABLE t1(id INT, f1 JSON); > INSERT INTO t1 VALUES (1, '{\"1\": 1}'), (2, '{\"1\": 2}'), (3, '{\"1\": > 3}'), > (4, '{\"1\": 4}'), (5, '{\"1\": 5}'), (6, '{\"1\": 6}'); > > SELECT * FROM t1 WHERE id IN > (SELECT id FROM t1 as tt2, > JSON_TABLE(f1, "$" COLUMNS (jf FOR ORDINALITY)) AS tbl); I've noticed that Optimizer Trace is not saved for this query. I get MariaDB [test]> select * from information_schema.optimizer_trace\G *************************** 1. row *************************** QUERY: TRACE: MISSING_BYTES_BEYOND_MAX_MEM_SIZE: 0 INSUFFICIENT_PRIVILEGES: 1 1 row in set (0.001 sec) INSUFFICIENT_PRIVILEGES: 1! This is only supposed to happen when the query uses VIEWs which definition cannot be viewed by the user executing the query. Could you investigate and fix this please? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] On bb-10.6-monty work
by Nikita Malyavin 16 Sep '20

16 Sep '20

Hello, Monty! I've been asked to review your recent work regarding memory footprint optimizations. I went through all the work done, and mostly liked the changes done. So here are only two commits i'd like to highlight > commit bf8bd0573440dc39490eb795b5455735dd63132b > Author: Monty <monty(a)mariadb.org> > Date: Tue Jul 14 18:36:05 2020 +0300 > > MDEV-23001 Precreate static Item_bool() to simplify code > > The following changes where done: > - Create global Item: Item_false and Item_true > - Replace all creation if 'FALSE' and 'TRUE' top level items used for > WHERE/HAVING/ON clauses to use Item_false and Item_true. > > The benefit are: > - Less and faster code > - No test needed if we where able to create the new item. > - Fixed possible errors if 'new' would have failed for the Item_bool's > > diff --git a/sql/item.h b/sql/item.h > index c4fbf8f9c0a..4cdee637415 100644 > --- a/sql/item.h > +++ b/sql/item.h > @@ -4236,6 +4239,23 @@ class Item_bool :public Item_int > }; > > > +class Item_bool_static :public Item_bool > +{ > +public: > + Item_bool_static(const char *str_arg, longlong i): > + Item_bool(NULL, str_arg, i) {}; > + > + /* Create dummy functions for things that may change the static item */ > + void fix_length_and_charset(uint32 max_char_length_arg, CHARSET_INFO *cs) > + {} > + void fix_char_length(size_t max_char_length_arg) > + {} > + void set_join_tab_idx(uint join_tab_idx_arg) > + { DBUG_ASSERT(0); } > +}; > + > +extern const Item_bool_static Item_false, Item_true; > + > class Item_uint :public Item_int > { > public: Do we really need an additional class for static Item_bool's? I know that we have constructors hidden for Item, so maybe just unhide it in Item_bool, and that's it. Besides, I see fix_char_length is not virtual at all, so reimplementing it has no efect. And why can't we do all that in Item_bool itself? Note that adding new class increases a virtual method table, and it's huge already, for Item hierarchy. We could also add some other frequently used Items, like 0, 1, '', NULL, to a static memory. > commit 6ca3ab6c2c18cdbf614984cb9d5312d58e718372 > Author: Monty <monty(a)mariadb.org> > Date: Wed Sep 2 03:13:32 2020 +0300 > > Split item->flags into base_flags and with_flags > > This was done to simplify copying of with_* flags > > Other things: > - Changed Flags to C++ enums, which enables gdb to print > out bit values for the flags. This also enables compiler > errors if one tries to manipulate a non existing bit in > a variable. > - Added set_maybe_null() as a shortcut as setting the > MAYBE_NULL flags was used in a LOT of places. > - Renamed PARAM flag to SP_VAR to ensure it's not confused with persistent > statement parameters. > > diff --git a/sql/item.h b/sql/item.h > index 09618bbb52c..9a794d96982 100644 > --- a/sql/item.h > +++ b/sql/item.h > @@ -717,28 +717,94 @@ class Item_const > > #define STOP_PTR ((void *) 1) > > -/* Bits used in Item.flags */ > -/* If item may be null */ > -#define ITEM_FLAG_MAYBE_NULL_SHIFT 0 > -#define ITEM_FLAG_MAYBE_NULL (1<<ITEM_FLAG_MAYBE_NULL_SHIFT) > -/* If used in GROUP BY list of a query with ROLLUP */ > -#define ITEM_FLAG_IN_ROLLUP (1<<1) > -/* If Item contains an SP parameter */ > -#define ITEM_FLAG_WITH_PARAM (1<<2) > -/* If item contains a window func */ > -#define ITEM_FLAG_WITH_WINDOW_FUNC (1<<3) > -/* True if any item except Item_sum contains a field. Set during parsing. */ > -#define ITEM_FLAG_WITH_FIELD (1<<4) > -/* If item was fixed with fix_fields */ > -#define ITEM_FLAG_FIXED (1<<5) > -/* Indicates that name of this Item autogenerated or set by user */ > -#define ITEM_FLAG_IS_AUTOGENERATED_NAME (1 << 6) > -/* Indicates that this item is in CYCLE clause of WITH */ > -#define ITEM_FLAG_IS_IN_WITH_CYCLE (1<<7) > -/* True if item contains a sum func */ > -#define ITEM_FLAG_WITH_SUM_FUNC (1<< 8) > -/* True if item containts a sub query */ > -#define ITEM_FLAG_WITH_SUBQUERY (1<< 9) > + > +/* Base flags (including IN) for an item */ > + > + > +typedef uint8 item_flags_t; > + > +enum class item_base_t : item_flags_t > +{ > + NONE= 0, > +#define ITEM_FLAGS_MAYBE_NULL_SHIFT 0 // Must match MAYBE_NULL > + MAYBE_NULL= (1<<0), // May be NULL. > + IN_ROLLUP= (1<<1), // Appears in GROUP BY list > + // of a query with ROLLUP. > + FIXED= (1<<2), // Was fixed with fix_fields(). > + IS_AUTOGENERATED_NAME= (1<<3), // The name if this Item was > + // generated or set by the user. > + IS_IN_WITH_CYCLE= (1<<4) // This item is in CYCLE clause > + // of WITH. > +}; > + > + > +/* Flags that tells us what kind of items the item contains */ > + > +enum class item_with_t : item_flags_t > +{ > + NONE= 0, > + SP_VAR= (1<<0), // If Item contains a stored procedure variable > + WINDOW_FUNC= (1<<1), // If item contains a window func > + FIELD= (1<<2), // If any item except Item_sum contains a field. > + SUM_FUNC= (1<<3), // If item contains a sum func > + SUBQUERY= (1<<4) // If item containts a sub query > +}; > + > + > +/* Make operations in item_base_t and item_with_t work like 'int' */ > +static inline item_base_t operator&(const item_base_t a, const item_base_t b) > +{ > + return (item_base_t) (((item_flags_t) a) & ((item_flags_t) b)); > +} > + > +static inline item_base_t & operator&=(item_base_t &a, item_base_t b) > +{ > + a= (item_base_t) (((item_flags_t) a) & (item_flags_t) b); > + return a; > +} > + > +static inline item_base_t operator|(const item_base_t a, const item_base_t b) > +{ > + return (item_base_t) (((item_flags_t) a) | ((item_flags_t) b)); > +} > + > +static inline item_base_t & operator|=(item_base_t &a, item_base_t b) > +{ > + a= (item_base_t) (((item_flags_t) a) | (item_flags_t) b); > + return a; > +} > + > +static inline item_base_t operator~(const item_base_t a) > +{ > + return (item_base_t) ~(item_flags_t) a; > +} > + > +static inline item_with_t operator&(const item_with_t a, const item_with_t b) > +{ > + return (item_with_t) (((item_flags_t) a) & ((item_flags_t) b)); > +} > + > +static inline item_with_t & operator&=(item_with_t &a, item_with_t b) > +{ > + a= (item_with_t) (((item_flags_t) a) & (item_flags_t) b); > + return a; > +} > + > +static inline item_with_t operator|(const item_with_t a, const item_with_t b) > +{ > + return (item_with_t) (((item_flags_t) a) | ((item_flags_t) b)); > +} > + > +static inline item_with_t & operator|=(item_with_t &a, item_with_t b) > +{ > + a= (item_with_t) (((item_flags_t) a) | (item_flags_t) b); > + return a; > +} > + > +static inline item_with_t operator~(const item_with_t a) > +{ > + return (item_with_t) ~(item_flags_t) a; > +} > > > class Item :public Value_source, > @@ -932,8 +998,8 @@ class Item :public Value_source, > const char *orig_name; > > /* All common bool variables for an Item is stored here */ > - typedef uint16 item_flags_t; > - item_flags_t flags; > + item_base_t base_flags; > + item_with_t with_flags; > > /* Marker is used in some functions to temporary mark an item */ > int16 marker; Nice experimentation! I wanted to suggest using constexprs inside Item, but this is even better, because you cant freely add a bit that not in the enum. Though I think it is too bloaty to specify all the operators each time, so I'd prefer to see duplications metaprogrammed out somehow. There is no sane way in C++ (even in modern one) to specify traits, which extend the class functionality (see Golang's traits), so I'd suggest to use good old preprocessor way, like following: #define T item_base_t #include "setup_bitwise_operators.inc" #define T item_with_t #include "setup_bitwise_operators.inc" and then in setup_bitwise_operators.inc: static inline T operator&(const T a, const T b) { return (T) (((item_flags_t) a) & ((item_flags_t) b)); } And so on. Another variant I've just googled is to make a template operator for all types, but restrict its applicatory type set with std::enable_if<T>: https://www.justsoftwaresolutions.co.uk/cplusplus/using-enum-classes-as-bit… So we would only need to add a structure: template<> struct enable_bitmask_operators<item_base_t>{ static constexpr bool enable=true; }; To work. Thereby, the first approach is: + Good old way + Works fast - not a C++ way, though we work in a C++-only file The second approach is: + modern + C++ way - may compile slow - may produce uncomfortable errors when compiler tries to deduce a function/operator to apply Up to you whic way to use, but I think it should be done, since we have many other flag sets, which are likely to be prettified with the same style. -- Yours truly, Nikita Malyavin

2 3

[Maria-developers] MDEV-21829:Followup: On Unique object and variable-size keys
by Sergey Petrunia 15 Sep '20

15 Sep '20

Hi Varun, I was looking at the code for MDEV-21829, trying to understand whether there are any issues with how Unique object was extended to handle variable-size keys. == TREE object (the RB tree) == TREE (the structure and functions that operate on it) support variable-sized keys, however the keys "are expected to know" their size. tree_insert() has "uint key_size" argument. This way, tree_insert() knows how many bytes to allocate and copy when putting the key into the tree. The rest of the TREE code doesn't seem to care about the size of the keys. When TREE code needs to compare a couple of keys, it will call TREE::compare callack. The callback only receives key pointers as arguments, that is, it is expected to infer key size from the contents of the key. The same happens when when walking the tree. Only "uchar *key" is passed to tree_walk_action callback. The walk function has to figure out they key size on its own. == Unique == Unique class cannot do the same what TREE functions function do. It needs to be aware of the sizes of the values it is storing: - It will need to pass value size to tree_insert() - It will need to know value size to write it into tmp file. - It will need to read it back when merging. How does it know it? Unique::merge() and Unique::walk() with merge_walk() functions do make certain assumptions about the data format being used. == The review points == Is this bad? I think tight coupling between the Unique object and the specifics of the data format it is storing is bad. What is worse is that currently this coupling is implicit, it is not immediately apparent to somebody just looking at the code. Possible ways out: 1. Explicitly declare in Unique class that variable-sized records must use certain data format. 2. Isolate the Unique from the specifics of the data format used. 2A. Let Unique get the size of the key as argument of unique_add() and then keep it together with the key. This creates some RAM/storage overhead. 2B. Pass a read_size() callback function which would get the size from the data? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

2 1

Re: [Maria-developers] 099596296ce: Avoid mallocs when using LONGLONG_BUFFER_SIZE
by Sergei Golubchik 13 Sep '20

13 Sep '20

Hi, Michael! This is a good catch! But note that LONGLONG_BUFFER_SIZE already reserves +1 for '\0', it's defined as /* Max length needed for a buffer to hold a longlong or ulonglong + end \0 */ #define LONGLONG_BUFFER_SIZE 21 The problem is that String::set_int doesn't use LONGLONG_BUFFER_SIZE, it uses 20*cs->mbmaxlen+1 This +1 is supposed to reserve one byte for '\0'. Which is incorrect, because alloc() does it too, so a better fix would be to remove +1 from String::set_int(). On Sep 08, Michael Widenius wrote: > revision-id: 099596296ce (mariadb-10.5.2-272-g099596296ce) > parent(s): 2de0c74f94c > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2020-09-03 15:17:56 +0300 > message: > > Avoid mallocs when using LONGLONG_BUFFER_SIZE > > When using String::alloc() to allocate a string, the String ensures that > there is space for an extra NULL byte in the buffer and if not, reallocates > the string. This is a problem with the String::set_int() that calls > alloc(LONGLONG_BUFFER_SIZE), which forces an extra malloc/free to > happen. > > Fixed by using LONGLONG_BUFFER_SIZE+1 for StringBuffer's. > > diff --git a/sql/field.cc b/sql/field.cc > index ab7254e483d..2be4da8d4a8 100644 > --- a/sql/field.cc > +++ b/sql/field.cc > @@ -1085,7 +1085,7 @@ Field_longstr::make_packed_sort_key_part(uchar *buff, > uchar* > Field_longstr::pack_sort_string(uchar *to, const SORT_FIELD_ATTR *sort_field) > { > - StringBuffer<LONGLONG_BUFFER_SIZE> buf; > + StringBuffer<LONGLONG_BUFFER_SIZE+1> buf; > val_str(&buf, &buf); > return to + sort_field->pack_sort_string(to, &buf, field_charset()); > } > diff --git a/sql/item.cc b/sql/item.cc > index 4a26cb6c140..0d947749e01 100644 > --- a/sql/item.cc > +++ b/sql/item.cc > @@ -3612,7 +3612,7 @@ String *Item_int::val_str(String *str) > > void Item_int::print(String *str, enum_query_type query_type) > { > - StringBuffer<LONGLONG_BUFFER_SIZE> buf; > + StringBuffer<LONGLONG_BUFFER_SIZE+1> buf; > // my_charset_bin is good enough for numbers > buf.set_int(value, unsigned_flag, &my_charset_bin); > str->append(buf); > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 23814b75241: Report memory leaks from mariadbd if -T or --debug is used
by Sergei Golubchik 11 Sep '20

11 Sep '20

Hi, Michael! ok to push, of course On Sep 08, Michael Widenius wrote: > revision-id: 23814b75241 (mariadb-10.5.2-273-g23814b75241) > parent(s): 099596296ce > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2020-09-03 15:17:56 +0300 > message: > > Report memory leaks from mariadbd if -T or --debug is used > > Before memory leaks was only reported if server stopped normally. > This made it harder to find out where the leaks happened when > debugging test cases. > > diff --git a/sql/mysqld.cc b/sql/mysqld.cc > index 331684ff728..7468f436a03 100644 > --- a/sql/mysqld.cc > +++ b/sql/mysqld.cc > @@ -1939,7 +1939,7 @@ static void mysqld_exit(int exit_code) > { > fprintf(stderr, "Warning: Memory not freed: %lld\n", > (longlong) global_status_var.global_memory_used); > - if (exit_code == 0) > + if (exit_code == 0 || opt_endinfo) > SAFEMALLOC_REPORT_MEMORY(0); > } > DBUG_LEAVE; > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 2de0c74f94c: Optimize Sql_alloc
by Sergei Golubchik 11 Sep '20

11 Sep '20

Hi, Michael! ok to push On Sep 08, Michael Widenius wrote: > revision-id: 2de0c74f94c (mariadb-10.5.2-271-g2de0c74f94c) > parent(s): 21eb8969ce9 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2020-09-03 15:17:56 +0300 > message: > > Optimize Sql_alloc > > - Remove 'dummy_for_valgrind' overrun marker as this doesn't help much. > The element also distorts the sizes of objects a bit, which makes it > harder to calculate gain in object sizes when doing size optimizations. > - Avoid one extra call indirection when using thd_get_current_thd(), which > is used by Sql_alloc. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] c3ecf0d6243: Optimize size of lex structures
by Sergei Golubchik 09 Sep '20

09 Sep '20

Hi, Michael! On Sep 08, Michael Widenius wrote: > revision-id: c3ecf0d6243 (mariadb-10.5.2-269-gc3ecf0d6243) > parent(s): c79c6f78205 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2020-09-02 20:58:33 +0300 > message: > > Optimize size of lex structures > > LEX, st_select_lex, st_select_unit optimized for space: > - Use bit fields for bool variables > - Ensure that all bit fields are initialized (improves > performance for init functions as all bit fields can be > initalized with one memory access) > - Move members around in above structures to remove alignment > gaps > > diff --git a/sql/sql_lex.h b/sql/sql_lex.h > index f516219c01a..1f6e50d8b60 100644 > --- a/sql/sql_lex.h > +++ b/sql/sql_lex.h > @@ -725,12 +725,15 @@ class st_select_lex_node { > st_select_lex_node *next, **prev, /* neighbor list */ > *master, *slave, /* vertical links */ > *link_next, **link_prev; /* list of whole SELECT_LEX */ > + enum sub_select_type linkage; > > void init_query_common(); > -public: > > +public: > ulonglong options; > - > + uint8 uncacheable; > + bool distinct:1; > + bool no_table_names_allowed:1; /* used for global order by */ I thought you didn't like this approach and preferred flags & DISTINCT instead. Either you forgot to change LEX to flags or you forgot to squash your commits. > /* > result of this query can't be cached, bit field, can be : Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] c79c6f78205: Added typedef decimal_digits_t (uint16) for number of decimals
by Sergei Golubchik 09 Sep '20

09 Sep '20

Hi, Michael! On Sep 08, Michael Widenius wrote: > revision-id: c79c6f78205 (mariadb-10.5.2-268-gc79c6f78205) > parent(s): bf8bd057344 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2020-09-02 20:58:33 +0300 > message: > > Added typedef decimal_digits_t (uint16) for number of decimals > > For fields and Item's uint8 should be good enough. After > discussions with Alexander Barkov we choose uint16 (for now) > as some format functions may accept +256 digits. > > The reason for this patch was to make the storage of decimal > digits simlar. Before this patch decimals was stored/used as > uint8, int and uint. > > Changed most decimal variables and functions to use the new > typedef. > > diff --git a/include/decimal.h b/include/decimal.h > index cab18f99348..f713409ede1 100644 > --- a/include/decimal.h > +++ b/include/decimal.h > @@ -53,10 +53,11 @@ int decimal2double(const decimal_t *from, double *to); > int double2decimal(double from, decimal_t *to); > int decimal_actual_fraction(const decimal_t *from); > int decimal2bin(const decimal_t *from, uchar *to, int precision, int scale); > -int bin2decimal(const uchar *from, decimal_t *to, int precision, int scale); > +int bin2decimal(const uchar *from, decimal_t *to, int precision, > + decimal_digits_t scale); I don't understand. Is it the type for precision (max number of digits in a decimal number) or for scale (mux number of digits after the dot)? If the former - why did you not change the type of the precision argument? If the latter - why is it called not decimal_scale_t? I personally think the former, using decimal_digits_t for both precision and scale, should be fine. > -int decimal_size(int precision, int scale); > -int decimal_bin_size(int precision, int scale); > +int decimal_size(int precision, decimal_digits_t scale); > +int decimal_bin_size(int precision, decimal_digits_t scale); > int decimal_result_size(decimal_t *from1, decimal_t *from2, char op, > int param); > > diff --git a/include/m_ctype.h b/include/m_ctype.h > index 59ac7814aee..34821f76474 100644 > --- a/include/m_ctype.h > +++ b/include/m_ctype.h > @@ -79,6 +79,7 @@ typedef const struct my_collation_handler_st MY_COLLATION_HANDLER; > typedef const struct unicase_info_st MY_UNICASE_INFO; > typedef const struct uni_ctype_st MY_UNI_CTYPE; > typedef const struct my_uni_idx_st MY_UNI_IDX; > +typedef uint16 decimal_digits_t; A bit strange to see it here and not in decimal.h above. > > typedef struct unicase_info_char_st > { > diff --git a/plugin/type_inet/sql_type_inet.h b/plugin/type_inet/sql_type_inet.h > index 8c42431ccaa..df74b91172a 100644 > --- a/plugin/type_inet/sql_type_inet.h > +++ b/plugin/type_inet/sql_type_inet.h > @@ -402,19 +402,19 @@ class Type_handler_inet6: public Type_handler > bool can_return_time() const override { return false; } > bool convert_to_binary_using_val_native() const override { return true; } > > - uint Item_time_precision(THD *thd, Item *item) const override > + decimal_digits_t Item_time_precision(THD *thd, Item *item) const override here you return the precision using the decimal_digits_t type. So it seems that it should apply to both precision and scale. > { > return 0; > } > - uint Item_datetime_precision(THD *thd, Item *item) const override > +decimal_digits_t Item_datetime_precision(THD *thd, Item *item) const override > { > return 0; > } > - uint Item_decimal_scale(const Item *item) const override > + decimal_digits_t Item_decimal_scale(const Item *item) const override > { > return 0; > } > - uint Item_decimal_precision(const Item *item) const override > + decimal_digits_t Item_decimal_precision(const Item *item) const override > { > /* > This will be needed if we ever allow cast from INET6 to DECIMAL. > diff --git a/sql/field.cc b/sql/field.cc > index f50ddec1c80..a5d00dd6fd5 100644 > --- a/sql/field.cc > +++ b/sql/field.cc > @@ -3242,10 +3242,11 @@ Field *Field_decimal::make_new_field(MEM_ROOT *root, TABLE *new_table, > ** Field_new_decimal > ****************************************************************************/ > > -static uint get_decimal_precision(uint len, uint8 dec, bool unsigned_val) > +static decimal_digits_t get_decimal_precision(uint len, decimal_digits_t dec, > + bool unsigned_val) > { > uint precision= my_decimal_length_to_precision(len, dec, unsigned_val); shouldn't my_decimal_length_to_precision() return decimal_digits_t result? > - return MY_MIN(precision, DECIMAL_MAX_PRECISION); > + return (decimal_digits_t) MY_MIN(precision, DECIMAL_MAX_PRECISION); > } > > Field_new_decimal::Field_new_decimal(uchar *ptr_arg, > diff --git a/sql/item.h b/sql/item.h > index 4cdee637415..948bb85f253 100644 > --- a/sql/item.h > +++ b/sql/item.h > @@ -1644,14 +1644,14 @@ class Item: public Value_source, > return type_handler()->Item_decimal_precision(this); > } > /* Returns the number of integer part digits only */ > - inline int decimal_int_part() const > - { return my_decimal_int_part(decimal_precision(), decimals); } > + inline decimal_digits_t decimal_int_part() const > + { return (decimal_digits_t) my_decimal_int_part(decimal_precision(), decimals); } again, shouldn't my_decimal_int_part() return decimal_digits_t? > /* > Returns the number of fractional digits only. > NOT_FIXED_DEC is replaced to the maximum possible number > of fractional digits, taking into account the data type. > */ > - uint decimal_scale() const > + decimal_digits_t decimal_scale() const > { > return type_handler()->Item_decimal_scale(this); > } > @@ -4856,7 +4856,7 @@ class Item_temporal_literal :public Item_literal > Item_literal(thd) > { > collation= DTCollation_numeric(); > - decimals= dec_arg; > + decimals= (decimal_digits_t) dec_arg; shouldnt dec_arg be decimal_digits_t? > cached_time= *ltime; > } > > diff --git a/sql/item_func.cc b/sql/item_func.cc > index 8a75d2c9946..805647d2a04 100644 > --- a/sql/item_func.cc > +++ b/sql/item_func.cc > @@ -2713,7 +2713,7 @@ my_decimal *Item_func_round::decimal_op(my_decimal *decimal_value) > dec= INT_MIN; > > if (!(null_value= (value.is_null() || args[1]->null_value || > - value.round_to(decimal_value, (uint) dec, > + value.round_to(decimal_value, (int) dec, You don't need to cast uint16 to int explicitly > truncate ? TRUNCATE : HALF_UP) > 1))) > return decimal_value; > return 0; > diff --git a/sql/item_func.h b/sql/item_func.h > index 5b4acdce1c6..d5aa1477ff3 100644 > --- a/sql/item_func.h > +++ b/sql/item_func.h > @@ -1369,10 +1369,10 @@ class Item_decimal_typecast :public Item_func > { > my_decimal decimal_value; > public: > - Item_decimal_typecast(THD *thd, Item *a, uint len, uint dec) > + Item_decimal_typecast(THD *thd, Item *a, uint len, decimal_digits_t dec) > :Item_func(thd, a) > { > - decimals= (uint8) dec; > + decimals= (decimal_digits_t) dec; you don't need a cast here > collation= DTCollation_numeric(); > fix_char_length(my_decimal_precision_to_length_no_truncation(len, dec, > unsigned_flag)); > diff --git a/sql/my_decimal.cc b/sql/my_decimal.cc > index edf3e82de40..ac86ff71b64 100644 > --- a/sql/my_decimal.cc > +++ b/sql/my_decimal.cc > @@ -198,7 +198,8 @@ str_set_decimal(uint mask, const my_decimal *val, > E_DEC_OVERFLOW > */ > > -int my_decimal::to_binary(uchar *bin, int prec, int scale, uint mask) const > +int my_decimal::to_binary(uchar *bin, int prec, decimal_digits_t scale, again, shouldn't prec be decimal_digits_t too? > + uint mask) const > { > int err1= E_DEC_OK, err2; > my_decimal rounded; > @@ -329,7 +330,7 @@ my_decimal *date2my_decimal(const MYSQL_TIME *ltime, my_decimal *dec) > } > > > -void my_decimal_trim(ulonglong *precision, uint *scale) > +void my_decimal_trim(ulonglong *precision, decimal_digits_t *scale) and here? > { > if (!(*precision) && !(*scale)) > { > diff --git a/sql/my_decimal.h b/sql/my_decimal.h > index 2db08bf01e3..9f7232f94b4 100644 > --- a/sql/my_decimal.h > +++ b/sql/my_decimal.h > @@ -50,7 +50,7 @@ typedef struct st_mysql_time MYSQL_TIME; > #define DECIMAL_MAX_FIELD_SIZE DECIMAL_MAX_PRECISION > > > -inline uint my_decimal_size(uint precision, uint scale) > +inline uint my_decimal_size(uint precision, decimal_digits_t scale) and here? (and in many places below, that I'll stop commenting on) > { > /* > Always allocate more space to allow library to put decimal point > diff --git a/sql/sql_type.cc b/sql/sql_type.cc > index 9e0f9b013c0..6569ea1a415 100644 > --- a/sql/sql_type.cc > +++ b/sql/sql_type.cc > @@ -1197,11 +1198,12 @@ Type_numeric_attributes::aggregate_numeric_attributes_decimal(Item **item, > uint nitems, > bool unsigned_arg) > { > - int max_int_part= find_max_decimal_int_part(item, nitems); > + decimal_digits_t max_int_part= find_max_decimal_int_part(item, nitems); > decimals= find_max_decimals(item, nitems); > - int precision= MY_MIN(max_int_part + decimals, DECIMAL_MAX_PRECISION); > + decimal_digits_t precision= (decimal_digits_t) why do you need a cast here? > + MY_MIN(max_int_part + decimals, DECIMAL_MAX_PRECISION); > max_length= my_decimal_precision_to_length_no_truncation(precision, > - (uint8) decimals, > + decimals, > unsigned_flag); > } > > @@ -6910,20 +6912,20 @@ bool Type_handler_string_result:: > > /***************************************************************************/ > > -uint Type_handler::Item_time_precision(THD *thd, Item *item) const > +decimal_digits_t Type_handler::Item_time_precision(THD *thd, Item *item) const > { > return MY_MIN(item->decimals, TIME_SECOND_PART_DIGITS); note, no cast here (good) > } > > > -uint Type_handler::Item_datetime_precision(THD *thd, Item *item) const > +decimal_digits_t Type_handler::Item_datetime_precision(THD *thd, Item *item) const > { > return MY_MIN(item->decimals, TIME_SECOND_PART_DIGITS); and not here > } > > > -uint Type_handler_string_result::Item_temporal_precision(THD *thd, Item *item, > - bool is_time) const > +decimal_digits_t Type_handler_string_result:: > +Item_temporal_precision(THD *thd, Item *item, bool is_time) const > { > StringBuffer<64> buf; > String *tmp; > @@ -6939,34 +6941,34 @@ uint Type_handler_string_result::Item_temporal_precision(THD *thd, Item *item, > Datetime(thd, &status, tmp->ptr(), tmp->length(), tmp->charset(), > Datetime::Options(TIME_FUZZY_DATES, TIME_FRAC_TRUNCATE)). > is_valid_datetime())) > - return MY_MIN(status.precision, TIME_SECOND_PART_DIGITS); > - return MY_MIN(item->decimals, TIME_SECOND_PART_DIGITS); > + return (decimal_digits_t) MY_MIN(status.precision, TIME_SECOND_PART_DIGITS); > + return (decimal_digits_t) MY_MIN(item->decimals, TIME_SECOND_PART_DIGITS); and here again. why? (and below too, I'll stop commenting on casts now) > } > > diff --git a/sql/sql_type.h b/sql/sql_type.h > index a6d85b5bb47..7b1a4f730db 100644 > --- a/sql/sql_type.h > +++ b/sql/sql_type.h > @@ -2955,9 +2955,10 @@ class Type_numeric_attributes > class Type_temporal_attributes: public Type_numeric_attributes > { > public: > - Type_temporal_attributes(uint int_part_length, uint dec, bool unsigned_arg) > + Type_temporal_attributes(uint32 int_part_length, decimal_digits_t dec, bool unsigned_arg) > :Type_numeric_attributes(int_part_length + (dec ? 1 : 0), > - MY_MIN(dec, TIME_SECOND_PART_DIGITS), > + MY_MIN(dec, > + (decimal_digits_t) TIME_SECOND_PART_DIGITS), this should not be necessary > unsigned_arg) > { > max_length+= decimals; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] bf8bd057344: MDEV-23001 Precreate static Item_bool() to simplify code
by Sergei Golubchik 08 Sep '20

08 Sep '20

Hi, Michael! On Sep 08, Michael Widenius wrote: > revision-id: bf8bd057344 (mariadb-10.5.2-267-gbf8bd057344) > parent(s): 32a29afea77 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2020-09-02 20:58:32 +0300 > message: > > MDEV-23001 Precreate static Item_bool() to simplify code > > The following changes where done: > - Create global Item: Item_false and Item_true > - Replace all creation if 'FALSE' and 'TRUE' top level items used for > WHERE/HAVING/ON clauses to use Item_false and Item_true. > > The benefit are: > - Less and faster code > - No test needed if we where able to create the new item. > - Fixed possible errors if 'new' would have failed for the Item_bool's > > diff --git a/sql/item.cc b/sql/item.cc > index dbf20f31d7a..b24103f2330 100644 > --- a/sql/item.cc > +++ b/sql/item.cc > @@ -55,7 +55,8 @@ const char *item_empty_name=""; > const char *item_used_name= "\0"; > > static int save_field_in_field(Field *, bool *, Field *, bool); > - > +const Item_bool_static Item_false("FALSE", 0); > +const Item_bool_static Item_true("TRUE", 1); The maain question, that Bar asked in his email - we have tons of modifiable Item members. How do you know none of them will modified in your Item_bool_static objects? > /** > Compare two Items for List<Item>::add_unique() > @@ -412,7 +413,6 @@ Item::Item(THD *thd): > is_expensive_cache(-1), rsize(0), name(null_clex_str), orig_name(0), > common_flags(IS_AUTO_GENERATED_NAME) > { > - DBUG_ASSERT(thd); > marker= 0; > maybe_null= null_value= with_window_func= with_field= false; > in_rollup= 0; > @@ -421,6 +421,9 @@ Item::Item(THD *thd): > /* Initially this item is not attached to any JOIN_TAB. */ > join_tab_idx= MAX_TABLES; > > + /* thd is NULL in case of static items like Item_true */ > + if (!thd) > + return; I don't like that. thd can be NULL only twice, for statically initialized Item_true and Item_false. For that you removed a useful assert and added an if() that is completely unnecessary for billions of items created runtime. Better to have special contructor just for these two very special items. Or to create a THD temporarily when constructing them. > /* Put item in free list so that we can free all items at end */ > next= thd->free_list; > thd->free_list= this; > @@ -3614,6 +3618,19 @@ void Item_int::print(String *str, enum_query_type query_type) > } > > > +/* > + This function is needed to ensure that Item_bool_static doesn't change > + the value of the member str_value. > +*/ > + > +void Item_bool::print(String *str, enum_query_type query_type) > +{ > + // my_charset_bin is good enough for numbers > + String tmp(value ? (char*) "1" : (char*) "0" , 1, &my_charset_bin); > + str->append(tmp); > +} This is not needed anymore. Item_bool inherits from Item_int, and Item_int::print is void Item_int::print(String *str, enum_query_type query_type) { StringBuffer<LONGLONG_BUFFER_SIZE> buf; // my_charset_bin is good enough for numbers buf.set_int(value, unsigned_flag, &my_charset_bin); str->append(buf); } You removed str_value from Item_int::print in July. > + > + > Item *Item_bool::neg_transformer(THD *thd) > { > value= !value; > diff --git a/sql/item.h b/sql/item.h > index c4fbf8f9c0a..4cdee637415 100644 > --- a/sql/item.h > +++ b/sql/item.h > @@ -1969,7 +1969,7 @@ class Item: public Value_source, > virtual bool limit_index_condition_pushdown_processor(void *arg) { return 0; } > virtual bool exists2in_processor(void *arg) { return 0; } > virtual bool find_selective_predicates_list_processor(void *arg) { return 0; } > - bool cleanup_is_expensive_cache_processor(void *arg) > + virtual bool cleanup_is_expensive_cache_processor(void *arg) You've added a new virtual method? You tried so hard to remove virtual methods that it's a bit strange to see a new virtual method added specifically for these two static items. Why does it matter what cleanup_is_expensive_cache_processor will do with them? You overwrite is_expensive() anyway. > { > is_expensive_cache= (int8)(-1); > return 0; > @@ -3225,6 +3225,8 @@ class Item_literal: public Item_basic_constant > bool check_partition_func_processor(void *int_arg) { return false;} > bool const_item() const { return true; } > bool basic_const_item() const { return true; } > + bool is_expensive() { return false; } > + bool cleanup_is_expensive_cache_processor(void *arg) { return 0; } > }; > > diff --git a/sql/opt_index_cond_pushdown.cc b/sql/opt_index_cond_pushdown.cc > index 360ae028f36..d66f2f2e14e 100644 > --- a/sql/opt_index_cond_pushdown.cc > +++ b/sql/opt_index_cond_pushdown.cc > @@ -185,8 +185,8 @@ bool uses_index_fields_only(Item *item, TABLE *tbl, uint keyno, > static Item *make_cond_for_index(THD *thd, Item *cond, TABLE *table, uint keyno, > bool other_tbls_ok) > { > - if (!cond) > - return NULL; > + if (!cond || cond->basic_const_item()) > + return cond; The effect of this change is that you don't set cond->marker= ICP_COND_USES_INDEX_ONLY for basic_const_item's anymore. This looked suspicious at first. But then I noticed (and Sergey Petrunia confirmed) that ICP_COND_USES_INDEX_ONLY is not used at all. Since 5.3. So here I'd suggest to revert this your change and completely remove any ICP_COND_USES_INDEX_ONLY manipulations together with the ICP_COND_USES_INDEX_ONLY definition. > if (cond->type() == Item::COND_ITEM) > { > uint n_marked= 0; > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > index 144b86e8fc9..1140f32c9c9 100644 > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -2313,6 +2313,7 @@ bool dispatch_command(enum enum_server_command command, THD *thd, > thd->update_server_status(); > thd->protocol->end_statement(); > query_cache_end_of_result(thd); > + /* Check that the query didn't change const items */ what does that mean? Is it a TODO comment? Or it documents what happens below? > } > if (drop_more_results) > thd->server_status&= ~SERVER_MORE_RESULTS_EXISTS; > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index 22bf2eb37b5..fdd609a96ce 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -22560,6 +22560,8 @@ make_cond_for_table_from_pred(THD *thd, Item *root_cond, Item *cond, > return new_cond; > } > } > + else if (cond->basic_const_item()) > + return cond; this is rather fragile, isn't it? You skip item->set_join_tab_idx(join_tab_idx_arg), quite understandably. And currently, as far as I can see, item->get_join_tab_idx() is always used under if (!item->const_item()), so what you did is safe. But can it change in the future? Or is join_tab_idx something that conceptually doesn't make sense for const items? > > if (is_top_and_level && used_table == rand_table_bit && > (cond->used_tables() & ~OUTER_REF_TABLE_BIT) != rand_table_bit) Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] 5d9386892b5: MDEV-4633: multi_source.simple test fails sporadically
by sujatha 08 Sep '20

08 Sep '20

revision-id: 5d9386892b58477c4e28de3e40d930b44f5b42d4 (mariadb-10.1.43-274-g5d9386892b5) parent(s): 3ee2422624ffb3d7ffefff8db7ef9398816299bc author: Sujatha committer: Sujatha timestamp: 2020-09-08 13:10:36 +0530 message: MDEV-4633: multi_source.simple test fails sporadically Analysis: ======== Writes to 'rli->log_space_total' needs to be synchronized, otherwise both SQL_THREAD and IO_THREAD can try to modify the variable simultaneously resulting in incorrect rli->log_space_total. In the current test scenario SQL_THREAD is trying to decrement 'rli->log_space_total' in 'purge_first_log' and IO_THREAD is trying to increment the 'rli->log_space_total' in 'queue_event' simultaneously. Hence test occasionally fails with result mismatch. Fix: === Convert 'rli->log_space_total' variable to atomic type. --- sql/log.cc | 4 +++- sql/log.h | 4 +++- sql/rpl_rli.cc | 14 ++++++++------ sql/slave.cc | 19 +++++++++++++++---- 4 files changed, 29 insertions(+), 12 deletions(-) diff --git a/sql/log.cc b/sql/log.cc index 8049b94bab1..90d5c46a15f 100644 --- a/sql/log.cc +++ b/sql/log.cc @@ -4400,7 +4400,9 @@ int MYSQL_BIN_LOG::purge_first_log(Relay_log_info* rli, bool included) 0, 0, &log_space_reclaimed); mysql_mutex_lock(&rli->log_space_lock); - rli->log_space_total-= log_space_reclaimed; + my_atomic_add64_explicit((volatile int64*)(&rli->log_space_total), + (-(ulonglong)log_space_reclaimed), + MY_MEMORY_ORDER_RELAXED); mysql_cond_broadcast(&rli->log_space_cond); mysql_mutex_unlock(&rli->log_space_lock); diff --git a/sql/log.h b/sql/log.h index 277e5c6f69c..9e3807e9939 100644 --- a/sql/log.h +++ b/sql/log.h @@ -712,7 +712,9 @@ class MYSQL_BIN_LOG: public TC_LOG, private MYSQL_LOG char buf1[22],buf2[22]; #endif DBUG_ENTER("harvest_bytes_written"); - (*counter)+=bytes_written; + + my_atomic_add64_explicit((volatile int64*)(counter), bytes_written, + MY_MEMORY_ORDER_RELAXED); DBUG_PRINT("info",("counter: %s bytes_written: %s", llstr(*counter,buf1), llstr(bytes_written,buf2))); bytes_written=0; diff --git a/sql/rpl_rli.cc b/sql/rpl_rli.cc index c196a65809a..8815af4ea4a 100644 --- a/sql/rpl_rli.cc +++ b/sql/rpl_rli.cc @@ -140,8 +140,8 @@ int init_relay_log_info(Relay_log_info* rli, rli->slave_skip_counter=0; rli->abort_pos_wait=0; rli->log_space_limit= relay_log_space_limit; - rli->log_space_total= 0; - + my_atomic_store64_explicit((volatile int64*)(&rli->log_space_total), 0, + MY_MEMORY_ORDER_RELAXED); char pattern[FN_REFLEN]; (void) my_realpath(pattern, slave_load_tmpdir, 0); if (fn_format(pattern, PREFIX_SQL_LOAD, pattern, "", @@ -467,7 +467,8 @@ static inline int add_relay_log(Relay_log_info* rli,LOG_INFO* linfo) linfo->log_file_name); DBUG_RETURN(1); } - rli->log_space_total += s.st_size; + my_atomic_add64_explicit((volatile int64*)(&rli->log_space_total), + s.st_size, MY_MEMORY_ORDER_RELAXED); DBUG_PRINT("info",("log_space_total: %llu", rli->log_space_total)); DBUG_RETURN(0); } @@ -477,7 +478,8 @@ static int count_relay_log_space(Relay_log_info* rli) { LOG_INFO linfo; DBUG_ENTER("count_relay_log_space"); - rli->log_space_total= 0; + my_atomic_store64_explicit((volatile int64*)(&rli->log_space_total), 0, + MY_MEMORY_ORDER_RELAXED); if (rli->relay_log.find_log_pos(&linfo, NullS, 1)) { sql_print_error("Could not find first log while counting relay log space"); @@ -1202,8 +1204,8 @@ int purge_relay_logs(Relay_log_info* rli, THD *thd, bool just_reset, strmake_buf(rli->group_relay_log_name, rli->relay_log.get_log_fname()); strmake_buf(rli->event_relay_log_name, rli->relay_log.get_log_fname()); rli->group_relay_log_pos= rli->event_relay_log_pos= BIN_LOG_HEADER_SIZE; - rli->log_space_total= 0; - + my_atomic_store64_explicit((volatile int64*)(&rli->log_space_total), 0, + MY_MEMORY_ORDER_RELAXED); if (count_relay_log_space(rli)) { *errmsg= "Error counting relay log space"; diff --git a/sql/slave.cc b/sql/slave.cc index 06f2b0d955a..9f4b0f53fb4 100644 --- a/sql/slave.cc +++ b/sql/slave.cc @@ -2371,7 +2371,10 @@ static bool wait_for_relay_log_space(Relay_log_info* rli) &rli->log_space_lock, &stage_waiting_for_relay_log_space, &old_stage); - while (rli->log_space_limit < rli->log_space_total && + while (rli->log_space_limit < + (ulonglong)my_atomic_load64_explicit((volatile int64*) + (&rli->log_space_total), + MY_MEMORY_ORDER_RELAXED) && !(slave_killed=io_slave_killed(mi)) && !rli->ignore_log_space_limit) mysql_cond_wait(&rli->log_space_cond, &rli->log_space_lock); @@ -2912,7 +2915,10 @@ static bool send_show_master_info_data(THD *thd, Master_info *mi, bool full, protocol->store(mi->rli.last_error().message, &my_charset_bin); protocol->store((uint32) mi->rli.slave_skip_counter); protocol->store((ulonglong) mi->rli.group_master_log_pos); - protocol->store((ulonglong) mi->rli.log_space_total); + protocol->store((ulonglong) + my_atomic_load64_explicit((volatile int64*) + (&mi->rli.log_space_total), + MY_MEMORY_ORDER_RELAXED)); protocol->store( mi->rli.until_condition==Relay_log_info::UNTIL_NONE ? "None": @@ -4428,7 +4434,9 @@ Stopping slave I/O thread due to out-of-memory error from master"); #endif if (rli->log_space_limit && rli->log_space_limit < - rli->log_space_total && + (ulonglong) my_atomic_load64_explicit((volatile int64*) + (&rli->log_space_total), + MY_MEMORY_ORDER_RELAXED) && !rli->ignore_log_space_limit) if (wait_for_relay_log_space(rli)) { @@ -7031,7 +7039,10 @@ static Log_event* next_event(rpl_group_info *rgi, ulonglong *event_size) is are able to rotate and purge sometime soon. */ if (rli->log_space_limit && - rli->log_space_limit < rli->log_space_total) + rli->log_space_limit < + (ulonglong) my_atomic_load64_explicit((volatile int64*) + (&rli->log_space_total), + MY_MEMORY_ORDER_RELAXED)) { /* force rotation if not in an unfinished group */ rli->sql_force_rotate_relay= !rli->is_in_group();

1 0

[Maria-developers] 41d807dfc8c: MENT-731:Setting of SHUTDOWN_WAIT_FOR_SLAVES should require special privilege, not generic SUPER
by sujatha 31 Aug '20

31 Aug '20

revision-id: 41d807dfc8c5f640ccbbac5838cf572b2b1712c1 (mariadb-10.4.4-1804-g41d807dfc8c) parent(s): 38cbb0915eb4a0c43f1a83afa37c7de5e481a718 author: Sujatha committer: Sujatha timestamp: 2020-08-31 22:30:38 +0530 message: MENT-731:Setting of SHUTDOWN_WAIT_FOR_SLAVES should require special privilege, not generic SUPER Fix: === 'shutdown_wait_for_slaves' variable can be set either by having SUPER or SHUTDOWN privilege. --- mysql-test/suite/binlog/r/binlog_grant.result | 34 +++++++++++++++++++++++ mysql-test/suite/binlog/t/binlog_grant.test | 40 +++++++++++++++++++++++++++ sql/privilege.h | 3 ++ sql/sys_vars.cc | 5 +++- 4 files changed, 81 insertions(+), 1 deletion(-) diff --git a/mysql-test/suite/binlog/r/binlog_grant.result b/mysql-test/suite/binlog/r/binlog_grant.result index af4c02411d5..aee0730c0ab 100644 --- a/mysql-test/suite/binlog/r/binlog_grant.result +++ b/mysql-test/suite/binlog/r/binlog_grant.result @@ -124,6 +124,40 @@ disconnect user1; connection default; DROP USER user1@localhost; # +# MENT-731: Setting of SHUTDOWN_WAIT_FOR_SLAVES should require special +# privilege, not generic SUPER +# +# Test that 'shutdown_wait_for_slaves' variable cannot be set without +# SHUTDOWN or SUPER +CREATE USER user1@localhost; +GRANT ALL PRIVILEGES ON *.* TO user1@localhost; +REVOKE SHUTDOWN, SUPER ON *.* FROM user1@localhost; +connect user1,localhost,user1,,; +connection user1; +SET GLOBAL SHUTDOWN_WAIT_FOR_SLAVES=ON; +ERROR 42000: Access denied; you need (at least one of) the SHUTDOWN, SUPER privilege(s) for this operation +disconnect user1; +connection default; +DROP USER user1@localhost; +# Test that 'shutdown_wait_for_slaves' variable can be set with SHUTDOWN +CREATE USER user1@localhost; +GRANT SHUTDOWN ON *.* TO user1@localhost; +connect user1,localhost,user1,,; +connection user1; +SET GLOBAL SHUTDOWN_WAIT_FOR_SLAVES=ON; +disconnect user1; +connection default; +DROP USER user1@localhost; +# Test that 'shutdown_wait_for_slaves' variable can be set with SUPER +CREATE USER user1@localhost; +GRANT SUPER ON *.* TO user1@localhost; +connect user1,localhost,user1,,; +connection user1; +SET GLOBAL SHUTDOWN_WAIT_FOR_SLAVES=OFF; +disconnect user1; +connection default; +DROP USER user1@localhost; +# # MDEV-21975 Add BINLOG REPLAY privilege and bind new privileges to # gtid_seq_no, preudo_thread_id, server_id, gtid_domain_id # diff --git a/mysql-test/suite/binlog/t/binlog_grant.test b/mysql-test/suite/binlog/t/binlog_grant.test index ec8d2aeccd2..ca0e40299a9 100644 --- a/mysql-test/suite/binlog/t/binlog_grant.test +++ b/mysql-test/suite/binlog/t/binlog_grant.test @@ -172,6 +172,46 @@ SHOW BINLOG EVENTS; connection default; DROP USER user1@localhost; +--echo # +--echo # MENT-731: Setting of SHUTDOWN_WAIT_FOR_SLAVES should require special +--echo # privilege, not generic SUPER +--echo # + +--echo # Test that 'shutdown_wait_for_slaves' variable cannot be set without +--echo # SHUTDOWN or SUPER +CREATE USER user1@localhost; +GRANT ALL PRIVILEGES ON *.* TO user1@localhost; +REVOKE SHUTDOWN, SUPER ON *.* FROM user1@localhost; +--connect(user1,localhost,user1,,) +--connection user1 +--error ER_SPECIFIC_ACCESS_DENIED_ERROR +SET GLOBAL SHUTDOWN_WAIT_FOR_SLAVES=ON; +--disconnect user1 +--connection default +DROP USER user1@localhost; + + +--echo # Test that 'shutdown_wait_for_slaves' variable can be set with SHUTDOWN +CREATE USER user1@localhost; +GRANT SHUTDOWN ON *.* TO user1@localhost; +--connect(user1,localhost,user1,,) +--connection user1 +SET GLOBAL SHUTDOWN_WAIT_FOR_SLAVES=ON; +--disconnect user1 +connection default; +DROP USER user1@localhost; + + +--echo # Test that 'shutdown_wait_for_slaves' variable can be set with SUPER +CREATE USER user1@localhost; +GRANT SUPER ON *.* TO user1@localhost; +--connect(user1,localhost,user1,,) +--connection user1 +SET GLOBAL SHUTDOWN_WAIT_FOR_SLAVES=OFF; +--disconnect user1 +connection default; +DROP USER user1@localhost; + --echo # --echo # MDEV-21975 Add BINLOG REPLAY privilege and bind new privileges to --echo # gtid_seq_no, preudo_thread_id, server_id, gtid_domain_id diff --git a/sql/privilege.h b/sql/privilege.h index 37cdf4da01a..f4bfd835013 100644 --- a/sql/privilege.h +++ b/sql/privilege.h @@ -433,6 +433,9 @@ constexpr privilege_t PRIV_SET_SYSTEM_GLOBAL_VAR_SLOW_LAUNCH_TIME= constexpr privilege_t PRIV_SET_SYSTEM_GLOBAL_VAR_THREAD_POOL= CONNECTION_ADMIN_ACL | SUPER_ACL; +// Was SUPER_ACL prior to 10.5.5 +constexpr privilege_t PRIV_SET_SYSTEM_VAR_SHUTDOWN_WAIT_FOR_SLAVES= + SHUTDOWN_ACL | SUPER_ACL; /* Binary log related privileges that are checked regardless diff --git a/sql/sys_vars.cc b/sql/sys_vars.cc index 1577457a5d7..396c0570215 100644 --- a/sql/sys_vars.cc +++ b/sql/sys_vars.cc @@ -4573,7 +4573,10 @@ static Sys_var_ulong Sys_profiling_history_size( VALID_RANGE(0, 100), DEFAULT(15), BLOCK_SIZE(1)); #endif -static Sys_var_mybool Sys_shutdown_wait_for_slaves( +static Sys_var_on_access_global<Sys_var_mybool, + PRIV_SET_SYSTEM_VAR_SHUTDOWN_WAIT_FOR_SLAVES> + +Sys_shutdown_wait_for_slaves( "shutdown_wait_for_slaves", "when ON, SHUTDOWN command runs with implicit WAIT FOR ALL SLAVES option.", GLOBAL_VAR(opt_shutdown_wait_for_slaves), CMD_LINE(OPT_ARG),

1 0

[Maria-developers] e4fa0c805ad: MDEV-4633: multi_source.simple test fails sporadically
by sujatha 28 Aug '20

28 Aug '20

revision-id: e4fa0c805adf06c637cc3867427ca529b40e19ce (mariadb-10.1.43-267-ge4fa0c805ad) parent(s): de49e603770463e64cf4a79a7414bac93c02254f author: Sujatha committer: Sujatha timestamp: 2020-08-28 12:03:09 +0530 message: MDEV-4633: multi_source.simple test fails sporadically Analysis: ======== Writes to 'rli->log_space_total' needs to be synchronized. All modifications to 'rli->log_space_total' should be performed by holding 'rli->log_space_lock'. But IO thread doesn't acquire this mutex while writing events to relay log. Hence there is a race condition between SQL THREAD and IO THREAD, when both try to modify the same variable. SQL_THREAD is trying to decrement 'rli->log_space_total' in 'purge_first_log' and IO_THREAD is trying to increment the 'rli->log_space_total' in 'queue_event' simultaneously . Hence test occasionally fails with result mismatch. Fix: === Acquire 'rli->log_space_lock' while modifying 'rli->log_space_total'. --- sql/log.cc | 15 +++++++++++++++ sql/log.h | 13 +------------ sql/rpl_rli.cc | 4 ++++ sql/slave.cc | 28 ++++++++++++++-------------- 4 files changed, 34 insertions(+), 26 deletions(-) diff --git a/sql/log.cc b/sql/log.cc index 8049b94bab1..28a3d58d9e4 100644 --- a/sql/log.cc +++ b/sql/log.cc @@ -10265,6 +10265,21 @@ MYSQL_BIN_LOG::do_binlog_recovery(const char *opt_name, bool do_xa_recovery) return error; } +void MYSQL_BIN_LOG::harvest_bytes_written(Relay_log_info *rli) +{ +#ifndef DBUG_OFF + char buf1[22],buf2[22]; +#endif + DBUG_ENTER("harvest_bytes_written"); + mysql_mutex_lock(&rli->log_space_lock); + rli->log_space_total+=bytes_written; + DBUG_PRINT("info",("relay_log_space: %s bytes_written: %s", + llstr(rli->log_space_total,buf1), + llstr(bytes_written,buf2))); + bytes_written=0; + mysql_mutex_unlock(&rli->log_space_lock); + DBUG_VOID_RETURN; +} #ifdef INNODB_COMPATIBILITY_HOOKS /** diff --git a/sql/log.h b/sql/log.h index 277e5c6f69c..3b136c47bcf 100644 --- a/sql/log.h +++ b/sql/log.h @@ -706,18 +706,7 @@ class MYSQL_BIN_LOG: public TC_LOG, private MYSQL_LOG { bytes_written = 0; } - void harvest_bytes_written(ulonglong* counter) - { -#ifndef DBUG_OFF - char buf1[22],buf2[22]; -#endif - DBUG_ENTER("harvest_bytes_written"); - (*counter)+=bytes_written; - DBUG_PRINT("info",("counter: %s bytes_written: %s", llstr(*counter,buf1), - llstr(bytes_written,buf2))); - bytes_written=0; - DBUG_VOID_RETURN; - } + void harvest_bytes_written(Relay_log_info *rli); void set_max_size(ulong max_size_arg); void signal_update(); void wait_for_sufficient_commits(); diff --git a/sql/rpl_rli.cc b/sql/rpl_rli.cc index c196a65809a..18d7db234fe 100644 --- a/sql/rpl_rli.cc +++ b/sql/rpl_rli.cc @@ -467,7 +467,9 @@ static inline int add_relay_log(Relay_log_info* rli,LOG_INFO* linfo) linfo->log_file_name); DBUG_RETURN(1); } + mysql_mutex_lock(&rli->log_space_lock); rli->log_space_total += s.st_size; + mysql_mutex_unlock(&rli->log_space_lock); DBUG_PRINT("info",("log_space_total: %llu", rli->log_space_total)); DBUG_RETURN(0); } @@ -477,7 +479,9 @@ static int count_relay_log_space(Relay_log_info* rli) { LOG_INFO linfo; DBUG_ENTER("count_relay_log_space"); + mysql_mutex_lock(&rli->log_space_lock); rli->log_space_total= 0; + mysql_mutex_unlock(&rli->log_space_lock); if (rli->relay_log.find_log_pos(&linfo, NullS, 1)) { sql_print_error("Could not find first log while counting relay log space"); diff --git a/sql/slave.cc b/sql/slave.cc index 06f2b0d955a..3d9e23441bc 100644 --- a/sql/slave.cc +++ b/sql/slave.cc @@ -2508,7 +2508,7 @@ static void write_ignored_events_info_to_relay_log(THD *thd, Master_info *mi) } if (likely (rev || glev)) { - rli->relay_log.harvest_bytes_written(&rli->log_space_total); + rli->relay_log.harvest_bytes_written(rli); if (flush_master_info(mi, TRUE, TRUE)) sql_print_error("Failed to flush master info file"); } @@ -5270,7 +5270,7 @@ static int process_io_create_file(Master_info* mi, Create_file_log_event* cev) "error writing Exec_load event to relay log"); goto err; } - mi->rli.relay_log.harvest_bytes_written(&mi->rli.log_space_total); + mi->rli.relay_log.harvest_bytes_written(&mi->rli); break; } if (unlikely(cev_not_written)) @@ -5285,7 +5285,7 @@ static int process_io_create_file(Master_info* mi, Create_file_log_event* cev) goto err; } cev_not_written=0; - mi->rli.relay_log.harvest_bytes_written(&mi->rli.log_space_total); + mi->rli.relay_log.harvest_bytes_written(&mi->rli); } else { @@ -5299,7 +5299,7 @@ static int process_io_create_file(Master_info* mi, Create_file_log_event* cev) "error writing Append_block event to relay log"); goto err; } - mi->rli.relay_log.harvest_bytes_written(&mi->rli.log_space_total) ; + mi->rli.relay_log.harvest_bytes_written(&mi->rli) ; } } } @@ -5489,7 +5489,7 @@ static int queue_binlog_ver_1_event(Master_info *mi, const char *buf, mysql_mutex_unlock(&mi->data_lock); DBUG_RETURN(1); } - rli->relay_log.harvest_bytes_written(&rli->log_space_total); + rli->relay_log.harvest_bytes_written(rli); } delete ev; mi->master_log_pos+= inc_pos; @@ -5547,7 +5547,7 @@ static int queue_binlog_ver_3_event(Master_info *mi, const char *buf, mysql_mutex_unlock(&mi->data_lock); DBUG_RETURN(1); } - rli->relay_log.harvest_bytes_written(&rli->log_space_total); + rli->relay_log.harvest_bytes_written(rli); delete ev; mi->master_log_pos+= inc_pos; err: @@ -5784,9 +5784,7 @@ static int queue_event(Master_info* mi,const char* buf, ulong event_len) mysql_mutex_lock(log_lock); if (likely(!rli->relay_log.write_event(&fdle) && !rli->relay_log.flush_and_sync(NULL))) - { - rli->relay_log.harvest_bytes_written(&rli->log_space_total); - } + rli->relay_log.harvest_bytes_written(rli); else { error= ER_SLAVE_RELAY_LOG_WRITE_FAILURE; @@ -6221,7 +6219,7 @@ static int queue_event(Master_info* mi,const char* buf, ulong event_len) (rli->relay_log.description_event_for_queue)) error= ER_SLAVE_RELAY_LOG_WRITE_FAILURE; else - rli->relay_log.harvest_bytes_written(&rli->log_space_total); + rli->relay_log.harvest_bytes_written(rli); } else if (mi->gtid_reconnect_event_skip_count == 0) { @@ -6234,7 +6232,7 @@ static int queue_event(Master_info* mi,const char* buf, ulong event_len) if (rli->relay_log.append_no_lock(&fake_rev)) error= ER_SLAVE_RELAY_LOG_WRITE_FAILURE; else - rli->relay_log.harvest_bytes_written(&rli->log_space_total); + rli->relay_log.harvest_bytes_written(rli); } } else @@ -6302,7 +6300,7 @@ static int queue_event(Master_info* mi,const char* buf, ulong event_len) { mi->master_log_pos+= inc_pos; DBUG_PRINT("info", ("master_log_pos: %lu", (ulong) mi->master_log_pos)); - rli->relay_log.harvest_bytes_written(&rli->log_space_total); + rli->relay_log.harvest_bytes_written(rli); } else { @@ -6382,7 +6380,7 @@ void end_relay_log_info(Relay_log_info* rli) log_lock= rli->relay_log.get_log_lock(); mysql_mutex_lock(log_lock); rli->relay_log.close(LOG_CLOSE_INDEX | LOG_CLOSE_STOP_EVENT); - rli->relay_log.harvest_bytes_written(&rli->log_space_total); + rli->relay_log.harvest_bytes_written(rli); mysql_mutex_unlock(log_lock); /* Delete the slave's temporary tables from memory. @@ -7318,7 +7316,9 @@ int rotate_relay_log(Master_info* mi) Note that it needs to be protected by mi->data_lock. */ mysql_mutex_assert_owner(&mi->data_lock); - rli->relay_log.harvest_bytes_written(&rli->log_space_total); + + rli->relay_log.harvest_bytes_written(rli); + end: DBUG_RETURN(error); }

1 0

[Maria-developers] 2f859962b03: MDEV-23534: SIGSEGV in sf_malloc_usable_size/my_free on SET GLOBAL REPLICATE_DO_TABLE
by sujatha 27 Aug '20

27 Aug '20

revision-id: 2f859962b032cc75176e05df8d704eec413cdb17 (mariadb-10.1.43-266-g2f859962b03) parent(s): 62d1e3bf67a12eb6f48ac615bda119e2ed65cf16 author: Sujatha committer: Sujatha timestamp: 2020-08-26 16:25:28 +0530 message: MDEV-23534: SIGSEGV in sf_malloc_usable_size/my_free on SET GLOBAL REPLICATE_DO_TABLE Backporting fixes for: MDEV-22317: SIGSEGV in my_free/delete_dynamic in optimized builds (ARIA) MDEV-22059: MSAN report at replicate_ignore_table_grant --- .../suite/rpl/r/rpl_filter_tables_dynamic.result | 2 + .../rpl/r/rpl_filter_wild_tables_dynamic.result | 2 + .../suite/rpl/t/rpl_filter_tables_dynamic.test | 2 + .../rpl/t/rpl_filter_wild_tables_dynamic.test | 2 + sql/rpl_filter.cc | 52 ++++++++++++++++------ 5 files changed, 46 insertions(+), 14 deletions(-) diff --git a/mysql-test/suite/rpl/r/rpl_filter_tables_dynamic.result b/mysql-test/suite/rpl/r/rpl_filter_tables_dynamic.result index 5a746c88458..9709e24fbde 100644 --- a/mysql-test/suite/rpl/r/rpl_filter_tables_dynamic.result +++ b/mysql-test/suite/rpl/r/rpl_filter_tables_dynamic.result @@ -5,6 +5,8 @@ ERROR HY000: This operation cannot be performed as you have a running slave ''; SET @@GLOBAL.replicate_ignore_table="test.t4,test.t5,test.t6"; ERROR HY000: This operation cannot be performed as you have a running slave ''; run STOP SLAVE '' first include/stop_slave.inc +SET @@GLOBAL.replicate_do_table=""; +SET @@GLOBAL.replicate_ignore_table=""; SET @@GLOBAL.replicate_do_table="test.t1,test.t2,test.t3"; SET @@GLOBAL.replicate_ignore_table="test.t4,test.t5,test.t6"; include/start_slave.inc diff --git a/mysql-test/suite/rpl/r/rpl_filter_wild_tables_dynamic.result b/mysql-test/suite/rpl/r/rpl_filter_wild_tables_dynamic.result index 19d8e513e6f..338f4b3bbcf 100644 --- a/mysql-test/suite/rpl/r/rpl_filter_wild_tables_dynamic.result +++ b/mysql-test/suite/rpl/r/rpl_filter_wild_tables_dynamic.result @@ -5,6 +5,8 @@ ERROR HY000: This operation cannot be performed as you have a running slave ''; SET @@GLOBAL.replicate_wild_ignore_table="test.b%"; ERROR HY000: This operation cannot be performed as you have a running slave ''; run STOP SLAVE '' first include/stop_slave.inc +SET @@GLOBAL.replicate_wild_do_table=""; +SET @@GLOBAL.replicate_wild_ignore_table=""; SET @@GLOBAL.replicate_wild_do_table="test.a%"; SET @@GLOBAL.replicate_wild_ignore_table="test.b%"; include/start_slave.inc diff --git a/mysql-test/suite/rpl/t/rpl_filter_tables_dynamic.test b/mysql-test/suite/rpl/t/rpl_filter_tables_dynamic.test index 97ecc167356..ebededc36b3 100644 --- a/mysql-test/suite/rpl/t/rpl_filter_tables_dynamic.test +++ b/mysql-test/suite/rpl/t/rpl_filter_tables_dynamic.test @@ -51,6 +51,8 @@ SET @@GLOBAL.replicate_ignore_table="test.t4,test.t5,test.t6"; connection slave; source include/stop_slave.inc; +SET @@GLOBAL.replicate_do_table=""; +SET @@GLOBAL.replicate_ignore_table=""; SET @@GLOBAL.replicate_do_table="test.t1,test.t2,test.t3"; SET @@GLOBAL.replicate_ignore_table="test.t4,test.t5,test.t6"; source include/start_slave.inc; diff --git a/mysql-test/suite/rpl/t/rpl_filter_wild_tables_dynamic.test b/mysql-test/suite/rpl/t/rpl_filter_wild_tables_dynamic.test index c822c81f270..09db91aa4d3 100644 --- a/mysql-test/suite/rpl/t/rpl_filter_wild_tables_dynamic.test +++ b/mysql-test/suite/rpl/t/rpl_filter_wild_tables_dynamic.test @@ -13,6 +13,8 @@ SET @@GLOBAL.replicate_wild_ignore_table="test.b%"; connection slave; source include/stop_slave.inc; +SET @@GLOBAL.replicate_wild_do_table=""; +SET @@GLOBAL.replicate_wild_ignore_table=""; SET @@GLOBAL.replicate_wild_do_table="test.a%"; SET @@GLOBAL.replicate_wild_ignore_table="test.b%"; source include/start_slave.inc; diff --git a/sql/rpl_filter.cc b/sql/rpl_filter.cc index 366902c1f26..0d5d9ffeea8 100644 --- a/sql/rpl_filter.cc +++ b/sql/rpl_filter.cc @@ -349,14 +349,20 @@ Rpl_filter::set_do_table(const char* table_spec) int status; if (do_table_inited) - my_hash_reset(&do_table); + { + my_hash_free(&do_table); + do_table_inited= 0; + } status= parse_filter_rule(table_spec, &Rpl_filter::add_do_table); - if (!do_table.records) + if (do_table_inited && status) { - my_hash_free(&do_table); - do_table_inited= 0; + if (!do_table.records) + { + my_hash_free(&do_table); + do_table_inited= 0; + } } return status; @@ -369,14 +375,20 @@ Rpl_filter::set_ignore_table(const char* table_spec) int status; if (ignore_table_inited) - my_hash_reset(&ignore_table); + { + my_hash_free(&ignore_table); + ignore_table_inited= 0; + } status= parse_filter_rule(table_spec, &Rpl_filter::add_ignore_table); - if (!ignore_table.records) + if (ignore_table_inited && status) { - my_hash_free(&ignore_table); - ignore_table_inited= 0; + if (!ignore_table.records) + { + my_hash_free(&ignore_table); + ignore_table_inited= 0; + } } return status; @@ -411,14 +423,20 @@ Rpl_filter::set_wild_do_table(const char* table_spec) int status; if (wild_do_table_inited) + { free_string_array(&wild_do_table); + wild_do_table_inited= 0; + } status= parse_filter_rule(table_spec, &Rpl_filter::add_wild_do_table); - if (!wild_do_table.elements) + if (wild_do_table_inited && status) { - delete_dynamic(&wild_do_table); - wild_do_table_inited= 0; + if (!wild_do_table.elements) + { + delete_dynamic(&wild_do_table); + wild_do_table_inited= 0; + } } return status; @@ -431,14 +449,20 @@ Rpl_filter::set_wild_ignore_table(const char* table_spec) int status; if (wild_ignore_table_inited) + { free_string_array(&wild_ignore_table); + wild_ignore_table_inited= 0; + } status= parse_filter_rule(table_spec, &Rpl_filter::add_wild_ignore_table); - if (!wild_ignore_table.elements) + if (wild_ignore_table_inited && status) { - delete_dynamic(&wild_ignore_table); - wild_ignore_table_inited= 0; + if (!wild_ignore_table.elements) + { + delete_dynamic(&wild_ignore_table); + wild_ignore_table_inited= 0; + } } return status;

1 0

[Maria-developers] Weekly Report
by Kartik Soneji 24 Aug '20

24 Aug '20

Hi, This week - Implemented the BZip2, LZ4 and LZ4HC services along with tests, as per original project schedule. This means that all services have been implemented (🎉), although some of them need review and fine-tuning, as documented in the table below. - Here is the current state of all services: | \ | BZip2 | LZ4 | LZ4HC | LZMA | LZO | Snappy | ZStandard | | Connect | * | | | | | | | | InnoDB | Y | Y | | Y | Y | Y | | | Mroonga | | Y | | | | | Y | | RocksDB | * | ? | ? | | | Y | ? | Y → Service is confirmed working, and tests are written. * → Service is working, but needs tweaking or additional tests. ? → Service is implemented, but it might not be worth the additional complexity. - Wrote tests for Connect and Mroonga. Next week - Work on the final report for GSoC. - Made a list of LZ4 and ZStandard versions on all supported platforms. RocksDB uses different sets of functions depending on the compile-time version of the library available, so this will allow us to make a more informed choice of the minimum version number. - Finish writing the HOWTO file. Thanks, Kartik

1 0

[Maria-developers] Building from source failing on Mac OS X complaining of librarymariadb
by Rodrigo Souza 21 Aug '20

21 Aug '20

Hello all, I’m a newbie, trying to familiarize myself with the source code, however, once I followed the steps outlined here https://mariadb.org/get-involved/getting-started-for-developers/get-code-bu… <https://mariadb.org/get-involved/getting-started-for-developers/get-code-bu…> to build the server on Mac OS X, currently the MAKE fails on me complaining about a deprecated GSSAPI.h as follows: /mariadb/server/libmariadb/plugins/auth/gssapi_client.c:72:12: error: 'gss_import_name' is deprecated: use GSS.framework [-Werror,-Wdeprecated-declarations] major= gss_import_name(&minor, &principal_name_buf, GSS_C_NT_USER_NAME, &service_name); ^ /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/gssapi/gssapi.h:591:19: note: 'gss_import_name' has been explicitly marked deprecated here gss_name_t *) GSSKRB_APPLE_DEPRECATED("use GSS.framework") __API_DEPRECATED("Use GSS.framework", macos(10.0, 10.8)) __API_UNAVAILABLE(macCatalyst); /* output_name */ ^ /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/gssapi/gssapi.h:57:51: note: expanded from macro 'GSSKRB_APPLE_DEPRECATED' #define GSSKRB_APPLE_DEPRECATED(x) __attribute__((deprecated(x))) My Mac OS X is Catalina 10.15.6 and XCode Tools 11.6. Have you guys ever had this kind of issue trying to build on Mac OS? Thank you, Rodrigo Souza

1 0

[Maria-developers] Weekly Report
by Kartik Soneji 17 Aug '20

17 Aug '20

Hi, This week - Worked out, fixed and documented some bugs in the Snappy service. My initial conclusion needed some fine-tuning, it might be possible to make RocksDB work with our modifications, if we are able to switch the default compression engine on startup. I also fixed the crash on startup for the Snappy service. - Implemented a solution to conditionally include the custom headers only for the relevant storage engines. This will also allow us to skip over a storage engine (for example RocksDB) if it does not work well with our modifications. # in cmake/plugin.cmake IF( plugin STREQUAL "CONNECT" OR plugin STREQUAL "INNOBASE" OR plugin STREQUAL "MROONGA" OR plugin STREQUAL "ROCKSDB" ) INCLUDE_DIRECTORIES(${CMAKE_SOURCE_DIR}/include/compression) ENDIF() - Reverted the implementation for the Snappy and LZO service. - Rebased the branches onto the latest 10.5 branch. Next week - Implement the BZip2 service. - Work on the ZStd and Snappy service to try and make it work with RocksDB. - Continue writing the HOWTO file. Thanks, Kartik

1 0

[Maria-developers] Progress Report GSoC 2020 (10-16 August)
by Anton Mikhaylenko 17 Aug '20

17 Aug '20

Good day, everybody. In this post, I want to describe what I did this week, August 10-16. This week I've running benchmarks for threadpool and compare results. Regards, Anton Mikhailenko

1 0

[Maria-developers] 5dc2ebae474: MDEV-22330: mysqlbinlog stops with an error Don't know how to handle column type: 255 meta: 4 (0004)
by sujatha 17 Aug '20

17 Aug '20

revision-id: 5dc2ebae47481afe5b0769f0e0463d64b77143c0 (mariadb-10.1.43-260-g5dc2ebae474) parent(s): 101ce10d0ddda1e38806b8d2c491298482e7ab78 author: Sujatha committer: Sujatha timestamp: 2020-08-13 09:58:25 +0530 message: MDEV-22330: mysqlbinlog stops with an error Don't know how to handle column type: 255 meta: 4 (0004) Analysis: ======== "mysqlbinlog -v" option will reconstruct row events and display them as commented SQL statements. If this option is given twice, the output includes comments to indicate column data types and some metadata. `log_event_print_value` is the function responsible for printing values and their types. This function doesn't handle GEOMETRY type. Hence the above error gets printed. Fix: === Add support for GEOMETRY datatype. --- .../suite/binlog/r/binlog_mysqlbinlog_row.result | 56 +++++++++++++++++++++- .../suite/binlog/t/binlog_mysqlbinlog_row.test | 14 +++++- sql/log_event.cc | 17 +++++-- 3 files changed, 80 insertions(+), 7 deletions(-) diff --git a/mysql-test/suite/binlog/r/binlog_mysqlbinlog_row.result b/mysql-test/suite/binlog/r/binlog_mysqlbinlog_row.result index 9a3f4751165..0b05c4f449f 100644 --- a/mysql-test/suite/binlog/r/binlog_mysqlbinlog_row.result +++ b/mysql-test/suite/binlog/r/binlog_mysqlbinlog_row.result @@ -339,7 +339,11 @@ a 123.47 999.99 DROP TABLE t1dec102; -flush logs; +CREATE TABLE t1 (a GEOMETRY DEFAULT NULL); +INSERT INTO t1 VALUES (NULL); +INSERT INTO t1 VALUES (POINT(10,10)); +DROP TABLE t1; +FLUSH LOGS; /*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=1*/; /*!40019 SET @@session.max_insert_delayed_threads=0*/; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; @@ -4623,6 +4627,56 @@ SET TIMESTAMP=1000000000/*!*/; DROP TABLE `t1dec102` /* generated by server */ /*!*/; # at # +#010909 4:46:40 server id 1 end_log_pos # GTID 0-1-321 ddl +/*!100001 SET @@session.gtid_seq_no=321*//*!*/; +# at # +#010909 4:46:40 server id 1 end_log_pos # Query thread_id=# exec_time=# error_code=0 +SET TIMESTAMP=1000000000/*!*/; +CREATE TABLE t1 (a GEOMETRY DEFAULT NULL) +/*!*/; +# at # +#010909 4:46:40 server id 1 end_log_pos # GTID 0-1-322 +/*!100001 SET @@session.gtid_seq_no=322*//*!*/; +BEGIN +/*!*/; +# at # +#010909 4:46:40 server id 1 end_log_pos # Table_map: `test`.`t1` mapped to number # +# at # +#010909 4:46:40 server id 1 end_log_pos # Write_rows: table id # flags: STMT_END_F +### INSERT INTO `test`.`t1` +### SET +### @1=NULL /* GEOMETRY meta=4 nullable=1 is_null=1 */ +# at # +#010909 4:46:40 server id 1 end_log_pos # Query thread_id=# exec_time=# error_code=0 +SET TIMESTAMP=1000000000/*!*/; +COMMIT +/*!*/; +# at # +#010909 4:46:40 server id 1 end_log_pos # GTID 0-1-323 +/*!100001 SET @@session.gtid_seq_no=323*//*!*/; +BEGIN +/*!*/; +# at # +#010909 4:46:40 server id 1 end_log_pos # Table_map: `test`.`t1` mapped to number # +# at # +#010909 4:46:40 server id 1 end_log_pos # Write_rows: table id # flags: STMT_END_F +### INSERT INTO `test`.`t1` +### SET +### @1='\x00\x00\x00\x00\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00$@\x00\x00\x00\x00\x00\x00$@' /* GEOMETRY meta=4 nullable=1 is_null=0 */ +# at # +#010909 4:46:40 server id 1 end_log_pos # Query thread_id=# exec_time=# error_code=0 +SET TIMESTAMP=1000000000/*!*/; +COMMIT +/*!*/; +# at # +#010909 4:46:40 server id 1 end_log_pos # GTID 0-1-324 ddl +/*!100001 SET @@session.gtid_seq_no=324*//*!*/; +# at # +#010909 4:46:40 server id 1 end_log_pos # Query thread_id=# exec_time=# error_code=0 +SET TIMESTAMP=1000000000/*!*/; +DROP TABLE `t1` /* generated by server */ +/*!*/; +# at # #010909 4:46:40 server id 1 end_log_pos # Rotate to master-bin.000002 pos: 4 DELIMITER ; # End of log file diff --git a/mysql-test/suite/binlog/t/binlog_mysqlbinlog_row.test b/mysql-test/suite/binlog/t/binlog_mysqlbinlog_row.test index 0c94d968338..93c26cc57a2 100644 --- a/mysql-test/suite/binlog/t/binlog_mysqlbinlog_row.test +++ b/mysql-test/suite/binlog/t/binlog_mysqlbinlog_row.test @@ -450,7 +450,19 @@ INSERT INTO t1dec102 VALUES (999.99); SELECT * FROM t1dec102 ORDER BY a; DROP TABLE t1dec102; -flush logs; + +# +# MDEV-22330: mysqlbinlog stops with an error Don't know how to handle column +# type: 255 meta: 4 (0004) +# Check support for GEOMETRY type with verbose mode. +# +CREATE TABLE t1 (a GEOMETRY DEFAULT NULL); + +INSERT INTO t1 VALUES (NULL); +INSERT INTO t1 VALUES (POINT(10,10)); +DROP TABLE t1; + +FLUSH LOGS; --replace_result $MYSQLTEST_VARDIR MYSQLTEST_VARDIR --replace_regex /SQL_LOAD_MB-[0-9]-[0-9]/SQL_LOAD_MB-#-#/ /exec_time=[0-9]*/exec_time=#/ /end_log_pos [0-9]*/end_log_pos #/ /# at [0-9]*/# at #/ /thread_id=[0-9]*/thread_id=#/ /table id [0-9]*/table id #/ /mapped to number [0-9]*/mapped to number #/ /server v [^ ]*/server v #.##.##/ /((a)[0-9]*=[0-9]*[.][0-9]{1,3})[0-9e+-]*[^ ]*(.*(FLOAT|DOUBLE).*[*].)/\1...\2/ diff --git a/sql/log_event.cc b/sql/log_event.cc index a1a442df43f..e8121ba35ff 100644 --- a/sql/log_event.cc +++ b/sql/log_event.cc @@ -2575,13 +2575,20 @@ log_event_print_value(IO_CACHE *file, const uchar *ptr, "Not enough metadata to display the value. "); break; + case MYSQL_TYPE_GEOMETRY: + strmake(typestr, "GEOMETRY", typestr_length); + if (!ptr) + goto return_null; + + length= uint4korr(ptr); + my_b_write_quoted(file, ptr + meta, length); + return length + meta; + default: { - char tmp[5]; - my_snprintf(tmp, sizeof(tmp), "%04x", meta); - my_b_printf(file, - "!! Don't know how to handle column type=%d meta=%d (%s)", - type, meta, tmp); + fprintf(stderr, + "\nError: Don't know how to handle column type: %d meta: %d (%04x)\n", + type, meta, meta); } break; }

1 0

[Maria-developers] Weekly Report
by Kartik Soneji 10 Aug '20

10 Aug '20

Hi, This week - Had a meeting with Robert about further actions regarding GSoC and discussed expectations for the end of the month. - Started extending the tests to include all combinations of engine + library. let %TEST_MATRIX = { "connect" => ("bzip2", ), "innodb" => ("bzip2", "lz4", "lzma", "lzo", "snappy" ), "mroonga" => ( "lz4", "zstd"), "rocksdb" => ("bzip2", "lz4", "lz4hc", "snappy", "zstd") }; - Started writing a HOWTO about adding a new compression library in sql/compression/HOWTO. - Finished the first draft of the ZStd service. Since ZStd and RocksDB are both written by Facebook, RocksDB makes extensive use of the ZStd functions. RocksDB also compiles with different sets of functions depending on the compile time version of ZStd, so we need to decide on a minimum version number and enforce it while loading the library. Next week - Finish the implementation for ZStd. - Continue writing the HOWTO file. - Update the old service implementations. - Continue replicating the implementation for other libraries. Thanks, Kartik

1 0

[Maria-developers] Progress Report GSoC 2020 (3-9 August)
by Anton Mikhaylenko 10 Aug '20

10 Aug '20

Good day, everybody. In this post, I want to describe what I did this week, August 3-9. This week, I wrote a function, that gets current CPU Utilization for Windows and Linux systems. Also, I merged commits with squash into one. Regards, Anton Mikhailenko

1 0

[Maria-developers] 0664e1e766d: MDEV-22338: Assertion `!current_stmt_is_commit || !rgi->tables_to_lock' failed in Query_log_event::do_apply_event
by sujatha 10 Aug '20

10 Aug '20

revision-id: 0664e1e766d303187b381ace1c53547d1821e35c (mariadb-10.1.43-252-g0664e1e766d) parent(s): 85bd5314c56c150d756066806d4a6cb5b682383f author: Sujatha committer: Sujatha timestamp: 2020-08-10 12:29:48 +0530 message: MDEV-22338: Assertion `!current_stmt_is_commit || !rgi->tables_to_lock' failed in Query_log_event::do_apply_event Analysis: ========= With row based replication, master can generate event groups like shown below. BEGIN TABLE_MAP COMMIT i.e A binlog event group, without any rows event in it. Such groups are possible when a query operating on transactional table, doesn't modify any data in the table. For example a transaction is trying to modify both trans and non-trans tables and a trigger exists on non-transtable which modifies a trans table. CREATE TABLE t1 (id int) engine=innodb; CREATE TABLE t2 (id int) engine=myisam; CREATE TRIGGER t2 BEFORE INSERT ON t2 FOR EACH ROW DELETE FROM t1 LIMIT 1; INSERT INTO t2 VALUES (1); Above insert will write a record in 't2' table and its corresponding binlog entry is generated. First table map events are added to appropriate caches. t2's Table_Map is written into statement_cache. t1's Table_Map is written into transaction_cache. 'Write_rows' event is added to t2's statement_cache. Since there are no rows to remove from table 't1' there will not be any 'Delete_rows' event in the transaction cache. There won't be any 'Xid' event as well. During flush operation 'COMMIT' will be added to the transaction cache to complete the group. "t2's statement_cache" "t1's transaction_cache" BEGIN BEGIN TABLE_MAP TABLE_MAP WRITE_ROWS COMMIT COMMIT On the slave side it has strict rules, that a 'Table_Map' event should be followed by a rows event. This rule gets violated for 't1' transaction and results in an assert. Fix: ==== Replace the assert with if-else block. In a case where TABLE_MAP event is not followed by rows event invoke 'rows_event_stmt_cleanup' to initiate appropriate cleanup. The fix is based on upstream patch: commit da01f3846b62a2e41111256f9d6f0a2bb2de23c4 Author: Andrei Elkin <aelkin(a)mysql.com> --- .../suite/rpl/r/rpl_no_rows_event_assert.result | 26 ++++++++++++ .../suite/rpl/t/rpl_no_rows_event_assert.test | 46 ++++++++++++++++++++++ sql/log_event.cc | 22 ++++++++++- 3 files changed, 92 insertions(+), 2 deletions(-) diff --git a/mysql-test/suite/rpl/r/rpl_no_rows_event_assert.result b/mysql-test/suite/rpl/r/rpl_no_rows_event_assert.result new file mode 100644 index 00000000000..7661bb62305 --- /dev/null +++ b/mysql-test/suite/rpl/r/rpl_no_rows_event_assert.result @@ -0,0 +1,26 @@ +include/master-slave.inc +[connection master] +######################################################################### +# Test case1: Transaction cache has 'Table_Map' event followed by # +# 'COMMIT'. # +######################################################################### +CREATE TABLE t1 (id int) ENGINE=INNODB; +CREATE TABLE t2 (id int) ENGINE=MYISAM; +CREATE TRIGGER t2 BEFORE INSERT ON t2 FOR EACH ROW DELETE FROM t1 LIMIT 1; +INSERT INTO t2 VALUES (1); +DROP TABLE t1,t2; +######################################################################### +# Test case2: Transaction cache has 'Query', 'Table_Map' followed by # +# 'COMMIT'. # +######################################################################### +CREATE TABLE t1(f INT) ENGINE=INNODB; +CREATE TABLE t2(f INT) ENGINE=MYISAM; +CREATE TABLE t3(id INT AUTO_INCREMENT, i INT, PRIMARY KEY (id)) ENGINE=INNODB; +CREATE TABLE t4(f INT) ENGINE=MYISAM; +CREATE TRIGGER trig1 BEFORE INSERT ON t2 FOR EACH ROW INSERT INTO t3(i) SELECT * FROM t4 LIMIT 0; +BEGIN; +DELETE FROM t1 WHERE f=0; +INSERT INTO t2 VALUES (2); +COMMIT; +DROP TABLE t1,t2,t3,t4; +include/rpl_end.inc diff --git a/mysql-test/suite/rpl/t/rpl_no_rows_event_assert.test b/mysql-test/suite/rpl/t/rpl_no_rows_event_assert.test new file mode 100644 index 00000000000..c5cc7634615 --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_no_rows_event_assert.test @@ -0,0 +1,46 @@ +--source include/have_innodb.inc +--source include/have_binlog_format_mixed_or_row.inc +--source include/master-slave.inc +# ==== References ==== +# +# MDEV-22338: Assertion `!current_stmt_is_commit || !rgi->tables_to_lock' +# failed in Query_log_event::do_apply_event +# + +--echo ######################################################################### +--echo # Test case1: Transaction cache has 'Table_Map' event followed by # +--echo # 'COMMIT'. # +--echo ######################################################################### +--connection master +CREATE TABLE t1 (id int) ENGINE=INNODB; +CREATE TABLE t2 (id int) ENGINE=MYISAM; +CREATE TRIGGER t2 BEFORE INSERT ON t2 FOR EACH ROW DELETE FROM t1 LIMIT 1; +INSERT INTO t2 VALUES (1); +--sync_slave_with_master + +# Cleanup +--connection master +DROP TABLE t1,t2; + +--echo ######################################################################### +--echo # Test case2: Transaction cache has 'Query', 'Table_Map' followed by # +--echo # 'COMMIT'. # +--echo ######################################################################### + +CREATE TABLE t1(f INT) ENGINE=INNODB; +CREATE TABLE t2(f INT) ENGINE=MYISAM; +CREATE TABLE t3(id INT AUTO_INCREMENT, i INT, PRIMARY KEY (id)) ENGINE=INNODB; +CREATE TABLE t4(f INT) ENGINE=MYISAM; +CREATE TRIGGER trig1 BEFORE INSERT ON t2 FOR EACH ROW INSERT INTO t3(i) SELECT * FROM t4 LIMIT 0; + +BEGIN; +DELETE FROM t1 WHERE f=0; +INSERT INTO t2 VALUES (2); +COMMIT; +--sync_slave_with_master + +# Cleanup +--connection master +DROP TABLE t1,t2,t3,t4; + +--source include/rpl_end.inc diff --git a/sql/log_event.cc b/sql/log_event.cc index a1a442df43f..c37512aa6c1 100644 --- a/sql/log_event.cc +++ b/sql/log_event.cc @@ -4329,8 +4329,26 @@ int Query_log_event::do_apply_event(rpl_group_info *rgi, thd->clear_error(1); current_stmt_is_commit= is_commit(); - DBUG_ASSERT(!current_stmt_is_commit || !rgi->tables_to_lock); - rgi->slave_close_thread_tables(thd); + if (strcmp("COMMIT", query) == 0 && rgi->tables_to_lock != NULL) + { + /* + Cleaning-up the last statement context: + Found a row based event group which didn't modfiy any rows. i.e. + BEGIN + TABLE_MAP + COMMIT + */ + int error; + if ((error = rows_event_stmt_cleanup(rgi, thd))) + { + rli->report(ERROR_LEVEL, error, "Error in cleaning up after an event " + "preceding the commit; the group log file/position: %s %llu", + rli->group_master_log_name, + (ulong) rli->group_master_log_pos); + } + } + else + rgi->slave_close_thread_tables(thd); /* Note: We do not need to execute reset_one_shot_variables() if this

1 0

Re: [Maria-developers] [Commits] c776cad5f8e: MDEV-22702: Assertion `!field->is_null()' failed in my_decimal::my_decimal
by Sergey Petrunia 06 Aug '20

06 Aug '20

Hi Varun, First: I can still get the assertion failure if I run this: set @@sql_mode='only_full_group_by'; select min(1 mod a1), bit_or(a2) over () from t1; As far as I understand, the query is not valid for only_full_group_by mode. (Window functions as such are ok, but here window function is using a2, which cannot be accessed in the post-grouping context). More input below. On Wed, Jul 08, 2020 at 02:58:31PM +0530, Varun wrote: > revision-id: c776cad5f8ecf2675510deeb55724d7255a52503 (mariadb-10.4.11-267-gc776cad5f8e) > parent(s): cc0dca366357651ddb549e31a12b1ecd39c7380e > author: Varun Gupta > committer: Varun Gupta > timestamp: 2020-07-08 14:58:17 +0530 > message: > > MDEV-22702: Assertion `!field->is_null()' failed in my_decimal::my_decimal > > With implicit grouping with window functions, we need to make sure that all the > fields inside the window functions are nullable as any non-aggregated field can > produce a NULL value. ... > diff --git a/sql/sql_lex.h b/sql/sql_lex.h > index 0983cea44d0..16a0b3b08a6 100644 > --- a/sql/sql_lex.h > +++ b/sql/sql_lex.h > @@ -1510,6 +1510,8 @@ class st_select_lex: public st_select_lex_node > } > > bool have_window_funcs() const { return (window_funcs.elements !=0); } > + uint32 get_number_of_window_funcs() const > + { return (uint32)window_funcs.elements; } Why use uint32 when n_sum_items is uint, and elements is also uint? I suggest just keep using uint. > ORDER *find_common_window_func_partition_fields(THD *thd); > > bool cond_pushdown_is_allowed() const > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index 0ca5ab23288..05463efe9a5 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -1201,6 +1201,18 @@ JOIN::prepare(TABLE_LIST *tables_init, > break; > } > } > + /* > + If the query has a window function with an aggregate function, > + then also we have a mix of elements with and without grouping. > + Window function can be in the ORDER BY clause too so the check > + is made separately. Wording could be better. I don't understand what did you mean to say about ORDER BY. can you elaborate? > + Window function is inherited from Item_sum so each window function is > + also registered as a sum item, so need to check that we have an > + explicit aggregate function also in the query. what's "explicit aggregate" ? A regular aggregate function (i.e. not a window function)? > + */ > + if (select_lex->have_window_funcs() && > + select_lex->get_number_of_window_funcs() < select_lex->n_sum_items) > + mixed_implicit_grouping= true; > } > > table_count= select_lex->leaf_tables.elements; BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

2 1

[Maria-developers] Optimizer call followup: for the parser, JSON_TABLE is not a "table function"
by Sergei Petrunia 05 Aug '20

05 Aug '20

Hello, A followup to the optimizer call discussion: Here is the production from the SQL Standard describing JSON_TABLE: section 7.6 <table reference>: <table primary> ::= <table or query name> [ <query system time period specification> ] [ <correlation or recognition> ] | <derived table> <correlation or recognition> | <lateral derived table> <correlation or recognition> | <collection derived table> <correlation or recognition> | <table function derived table> <correlation or recognition> | <only spec> [ <correlation or recognition> ] | <data change delta table> [ <correlation or recognition> ] | <JSON table> <correlation or recognition> | <JSON table primitive> <correlation name> | <parenthesized joined table> <table function derived table> ::= TABLE <left paren> <collection value expression> <right parent> Section 7.11 <JSON table>: <JSON table> ::= JSON_TABLE <left paren> ....... <right parent> That is, JSON_TABLE and TABLE (...) are two different productions that have nothing in common. So, as far as syntax is concerned, JSON_TABLE is not a "table function". BR -- Sergei P.

1 0

[Maria-developers] Weekly Report
by Kartik Soneji 03 Aug '20

03 Aug '20

Hi, This week - Finished writing common testing foundation that allows other services to be tested easily. For example: --let $test_library = lzma --source test_library.inc --let $test_library = lzo --source test_library.inc ... - Finished the first draft of the Snappy service. Snappy has a C++ api that is used by RocksDB, so we have two options: - Use the mangled C++ function names directly, and load them normally with dlopen. - Re-implement the C++ function from Snappy’s C exports. The mangled function names may vary across platforms, so we are going with the second option for now. - Implemented global status variables for the services. This lays the foundation for integration with the storage engines. - Finished a first draft of the auto logic for the --use-compression= switch. Next week - Need to review my refactoring and decide on next steps. Details of refactor: - Removed the service struct. - Removed the struct from sql/sql_plugin_services.ic. - Define an anonymous struct in sql/compression/<library>.cc. - Initialize the struct with dummy function pointers. - If the library is successfully loaded, replace them with pointers to the real functions. - Define wrapper functions that forward the call to the function pointers in the struct. - Continue replicating the implementation for other libraries. - Decide what to do when dynamic libraries have the versions suffixed to the .so file. For example, liblzma.so.5.2.4 is included with Ubuntu 20, but liblzma.so is not. The liblzma.so -> liblzma.so.5.2.4 symlink is only added after installing the liblzma-dev package. - Sync up with Robert to discuss further actions. Thanks, Kartik

1 0

[Maria-developers] Review for: MDEV-17399 Add support for JSON_TABLE, part #9
by Sergey Petrunia 03 Aug '20

03 Aug '20

Hi Alexey, More input: == Circular dependencies are allowed == This query will cause an assert in the optimizer, for obvious reasons: select * from json_table(JS3.size, '$' columns (size INT PATH '$.size')) as JS1, json_table(JS1.size, '$' columns (size INT PATH '$.size')) as JS2, json_table(JS1.size, '$' columns (size INT PATH '$.size')) as JS3 where 1; == Character set inference is wrong == create table t20 (json varchar(100) character set utf8); insert into t20 values ('{"value":"АБВ"}'); create table tj20 as select T.value from t20, json_table(t20.json, '$' columns (value varchar(32) PATH '$.value')) T; mysql> show create table tj20\G *************************** 1. row *************************** Table: tj20 Create Table: CREATE TABLE `tj20` ( `value` varchar(32) DEFAULT NULL ) ENGINE=InnoDB DEFAULT CHARSET=latin1 The source document uses UTF-8 but the values from it are in latin1? == Comments and code cleanup == There are a lot of comments missing, some duplicate code, and code using out-of-date ways of doing things. Please find attached a patch with obvious fixes for most of it. == Explain == Let's follow MySQL here: let the tabular EXPLAIN show "Table function: json_table" in the Extra column, let EXPLAIN FORMAT=JSON show "table_function": "json_table" in the "table" element. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Obtaining usernames in MariaDB UDFs
by Denis Scherbakov 02 Aug '20

02 Aug '20

Hello guys, I am building a set of MariaDB UDFs that require to know a name of a user who called them in each and every call. Users should not be able to pretend as someone else, hence usernames as an argument cannot be used. I compile UDFs against MariaDB headers and I was able to obtain the current_thd, but the security_ctx pointer inside is NULL. Could somebody give me a hint what I am doing wrong? Thank you for your answers! Best regards, Denis Scherbakov

2 2

Re: [Maria-developers] 7311586ca25: MDEV-15703 Crash in EXECUTE IMMEDIATE 'CREATE OR REPLACE TABLE t1 (a INT DEFAULT ?)' USING DEFAULT
by Sergei Golubchik 02 Aug '20

02 Aug '20

Hi, Oleksandr! On Jul 29, Oleksandr Byelkin wrote: > revision-id: 7311586ca25 (mariadb-10.2.31-317-g7311586ca25) > parent(s): a1e52e7f32c > author: Oleksandr Byelkin <sanja(a)mariadb.com> > committer: Oleksandr Byelkin <sanja(a)mariadb.com> > timestamp: 2020-07-17 13:47:26 +0200 > message: > > MDEV-15703 Crash in EXECUTE IMMEDIATE 'CREATE OR REPLACE TABLE t1 (a INT DEFAULT ?)' USING DEFAULT > > Part 1: make better asserts. > > diff --git a/sql/field.cc b/sql/field.cc > index 65bd9d22857..cf0cc655888 100644 > --- a/sql/field.cc > +++ b/sql/field.cc > @@ -11138,6 +11138,16 @@ bool Field::save_in_field_default_value(bool view_error_processing) > { > THD *thd= table->in_use; > > + if ( // table is not opened properly > + table == NULL || table->pos_in_table_list == NULL || > + table->pos_in_table_list->select_lex == NULL || > + // we are in subquery/derived/CTE > + !table->pos_in_table_list->top_table()->select_lex->is_top_level_select()) > + { > + DBUG_ASSERT(0); // shoud not happened > + my_error(ER_INVALID_DEFAULT_PARAM, MYF(0)); > + return false; > + } that's kind of questionable. if this condition can never be true, why do you have my_error() there? Just do asserts, like DBUG_ASSERT(table); DBUG_ASSERT(table->pos_in_table_list); DBUG_ASSERT(table->pos_in_table_list->select_lex); DBUG_ASSERT(table->pos_in_table_list->top_table()->select_lex->is_top_level_select()); > if (flags & NO_DEFAULT_VALUE_FLAG && > real_type() != MYSQL_TYPE_ENUM) > { > @@ -11177,12 +11187,29 @@ bool Field::save_in_field_default_value(bool view_error_processing) > bool Field::save_in_field_ignore_value(bool view_error_processing) > { > enum_sql_command com= table->in_use->lex->sql_command; > + > + if ( // table is not opened properly > + table == NULL || table->pos_in_table_list == NULL || > + table->pos_in_table_list->select_lex == NULL || > + // we are in subquery/derived/CTE > + !table->pos_in_table_list->top_table()->select_lex->is_top_level_select()) > + { > + DBUG_ASSERT(0); // shoud not happened > + my_error(ER_INVALID_DEFAULT_PARAM, MYF(0)); > + return false; > + } ditto > // All insert-like commands > if (com == SQLCOM_INSERT || com == SQLCOM_REPLACE || > com == SQLCOM_INSERT_SELECT || com == SQLCOM_REPLACE_SELECT || > com == SQLCOM_LOAD) > return save_in_field_default_value(view_error_processing); > - return 0; // ignore > + if (com == SQLCOM_UPDATE || com == SQLCOM_UPDATE_MULTI) > + return 0; // ignore > + > + // unexpected command > + DBUG_ASSERT(0); // shoud not happened ditto but why it should not happen? What if one does SELECT DEFAULT(f) FROM t? > + my_error(ER_INVALID_DEFAULT_PARAM, MYF(0)); > + return false; > } > > > diff --git a/sql/item.cc b/sql/item.cc > index 9451d4203ca..382eb3ac47a 100644 > --- a/sql/item.cc > +++ b/sql/item.cc > @@ -3930,13 +3930,19 @@ int Item_param::save_in_field(Field *field, bool no_conversions) > case NULL_VALUE: > return set_field_to_null_with_conversions(field, no_conversions); > case DEFAULT_VALUE: > - return field->save_in_field_default_value(field->table->pos_in_table_list-> > + return field->save_in_field_default_value((field->table && > + field->table->pos_in_table_list)? in what case can field->table be NULL here? > + field->table->pos_in_table_list-> > top_table() != > - field->table->pos_in_table_list); > + field->table->pos_in_table_list: > + FALSE); > case IGNORE_VALUE: > - return field->save_in_field_ignore_value(field->table->pos_in_table_list-> > + return field->save_in_field_ignore_value((field->table && > + field->table->pos_in_table_list)? > + field->table->pos_in_table_list-> > top_table() != > - field->table->pos_in_table_list); > + field->table->pos_in_table_list: > + FALSE); > case NO_VALUE: > DBUG_ASSERT(0); // Should not be possible > return true; > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 2

Re: [Maria-developers] f854ac4d9e0: MDEV-21916: COM_STMT_BULK_EXECUTE with RETURNING insert wrong values
by Sergei Golubchik 02 Aug '20

02 Aug '20

Hi, Oleksandr! On Jul 29, Oleksandr Byelkin wrote: > revision-id: f854ac4d9e0 (mariadb-10.5.4-20-gf854ac4d9e0) > parent(s): 6cee9b1953b > author: Oleksandr Byelkin <sanja(a)mariadb.com> > committer: Oleksandr Byelkin <sanja(a)mariadb.com> > timestamp: 2020-07-03 13:39:49 +0200 > message: > > MDEV-21916: COM_STMT_BULK_EXECUTE with RETURNING insert wrong values > > To allocate new net buffer to avoid changing bufer we are reading. please add a more descriptive comment. I'm still not sure I understood correctly what the bug was. > diff --git a/sql/net_serv.cc b/sql/net_serv.cc > --- a/sql/net_serv.cc > +++ b/sql/net_serv.cc > @@ -178,6 +178,14 @@ my_bool my_net_init(NET *net, Vio *vio, void *thd, uint my_flags) > DBUG_RETURN(0); > } > > + > +inline void net_reset_new_packet(NET *net) static > +{ > + net->buff_end= net->buff + net->max_packet; > + net->write_pos= net->read_pos= net->buff; > +} > + > + > my_bool net_allocate_new_packet(NET *net, void *thd, uint my_flags) > { > DBUG_ENTER("net_allocate_new_packet"); > @@ -186,11 +194,26 @@ my_bool net_allocate_new_packet(NET *net, void *thd, uint my_flags) > NET_HEADER_SIZE + COMP_HEADER_SIZE + 1, > MYF(MY_WME | my_flags)))) > DBUG_RETURN(1); > - net->buff_end=net->buff+net->max_packet; > - net->write_pos=net->read_pos = net->buff; > + net_reset_new_packet(net); > DBUG_RETURN(0); > } > > +unsigned char *net_try_allocate_new_packet(NET *net, void *thd, uint my_flags) > +{ > + unsigned char * old_buff= net->buff; > + if (net_allocate_new_packet(net, thd, my_flags)) > + { > + /* > + We failed to allocate a new buffer => return the old buffer and > + return an error > + */ > + net->buff= old_buff; > + net_reset_new_packet(net); > + old_buff= NULL; wouldn't it be better to make net_allocate_new_packet not to change net->buff on failure? because currently you have exactly the same problem in sql_parse.cc (in 10.4) and your patch doesn't fix it. simply if (!(buff= (uchar*) my_malloc(...))) DBUG_RETURN(1); net->buff= buff; net->buff_end= net->buff + net->max_packet; net->write_pos= net->read_pos= net->buff; DBUG_RETURN(0); > + } > + return old_buff; > +} > + > > void net_end(NET *net) > { > diff --git a/sql/sql_delete.cc b/sql/sql_delete.cc > --- a/sql/sql_delete.cc > +++ b/sql/sql_delete.cc > @@ -685,7 +685,7 @@ bool mysql_delete(THD *thd, TABLE_LIST *table_list, COND *conds, > !table->prepare_triggers_for_delete_stmt_or_event()) > will_batch= !table->file->start_bulk_delete(); > > - if (returning) > + if (returning && !thd->get_stmt_da()->is_set()) how are result sets for bulk delete returning sent? > { > if (result->send_result_set_metadata(returning->item_list, > Protocol::SEND_NUM_ROWS | Protocol::SEND_EOF)) > diff --git a/sql/sql_error.cc b/sql/sql_error.cc > --- a/sql/sql_error.cc > +++ b/sql/sql_error.cc > @@ -380,16 +380,33 @@ Diagnostics_area::set_eof_status(THD *thd) > if (unlikely(is_error() || is_disabled())) > return; > > - /* > - If inside a stored procedure, do not return the total > - number of warnings, since they are not available to the client > - anyway. > - */ > - m_statement_warn_count= (thd->spcont ? > - 0 : > - current_statement_warn_count()); > + if (m_status == DA_EOF_BULK) > + { > + /* > + If inside a stored procedure, do not return the total > + number of warnings, since they are not available to the client > + anyway. > + */ > + if (!thd->spcont) > + m_statement_warn_count+= current_statement_warn_count(); > + } > + else > + { > + /* > + If inside a stored procedure, do not return the total > + number of warnings, since they are not available to the client > + anyway. > + */ did you have to repeat the same comment twice? > + if (thd->spcont) > + { > + m_statement_warn_count= 0; > + m_affected_rows= 0; affected rows too? old code didn't do that > + } > + else > + m_statement_warn_count= current_statement_warn_count(); > + m_status= (is_bulk_op() ? DA_EOF_BULK : DA_EOF); > + } > > - m_status= DA_EOF; > DBUG_VOID_RETURN; > } > > diff --git a/sql/sql_prepare.cc b/sql/sql_prepare.cc > --- a/sql/sql_prepare.cc > +++ b/sql/sql_prepare.cc > @@ -4364,24 +4368,37 @@ Prepared_statement::execute_bulk_loop(String *expanded_query, > if (state == Query_arena::STMT_ERROR) > { > my_message(last_errno, last_error, MYF(0)); > - thd->set_bulk_execution(0); > - return TRUE; > + goto err; > } > /* Check for non zero parameter count*/ > if (param_count == 0) > { > DBUG_PRINT("error", ("Statement with no parameters for bulk execution.")); > my_error(ER_UNSUPPORTED_PS, MYF(0)); > - thd->set_bulk_execution(0); > - return TRUE; > + goto err; > } > > if (!(sql_command_flags[lex->sql_command] & CF_SP_BULK_SAFE)) > { > DBUG_PRINT("error", ("Command is not supported in bulk execution.")); > my_error(ER_UNSUPPORTED_PS, MYF(0)); > - thd->set_bulk_execution(0); > - return TRUE; > + goto err; > + } > + /* > + Here second buffer for not optimized commands, > + optimized commands do it inside thier internal loop. > + */ > + if (!(sql_command_flags[lex->sql_command] & CF_SP_BULK_OPTIMIZED) && > + this->lex->has_returning()) > + { > + // Above check can be true for SELECT in future > + DBUG_ASSERT(lex->sql_command != SQLCOM_SELECT); > + if ((readbuff= > + net_try_allocate_new_packet(&thd->net, thd, > + MYF(MY_THREAD_SPECIFIC))) == NULL) sorry, I'm confused. You allocate a buffer here, in Prepared_statement::execute_bulk_loop, and then again in mysql_insert() ? > + { > + goto err; > + } > } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 2

Re: [Maria-developers] ca7e6f8d572: MDEV-19632 Replication aborts with ER_SLAVE_CONVERSION_FAILED upon CREATE ... SELECT in ORACLE mode
by Sergei Golubchik 31 Jul '20

31 Jul '20

Hi, Alexander! On Jul 28, Alexander Barkov wrote: > revision-id: ca7e6f8d572 (mariadb-10.3.21-178-gca7e6f8d572) > parent(s): f3f23b5c4bd > author: Alexander Barkov <bar(a)mariadb.com> > committer: Alexander Barkov <bar(a)mariadb.com> > timestamp: 2020-07-08 12:37:17 +0400 > message: > > MDEV-19632 Replication aborts with ER_SLAVE_CONVERSION_FAILED upon CREATE ... SELECT in ORACLE mode > > diff --git a/mysql-test/main/lowercase_fs_off.result b/mysql-test/main/lowercase_fs_off.result > index f2a8ec14641..3f0b08a78c4 100644 > --- a/mysql-test/main/lowercase_fs_off.result > +++ b/mysql-test/main/lowercase_fs_off.result > @@ -158,3 +158,13 @@ show triggers like '%T1%'; > Trigger Event Table Statement Timing Created sql_mode Definer character_set_client collation_connection Database Collation > drop table t1; > set GLOBAL sql_mode=default; > +# > +# MDEV-19632 Replication aborts with ER_SLAVE_CONVERSION_FAILED upon CREATE ... SELECT in ORACLE mode > +# > +# Compatibility schema names respect the filesystem case sensitivity I agree, it should. But this is not consistent with information_schema. So I'd suggest a comment in eq_name method, explaining why compatibility schema names respect the filesystem case sensitivity. (because, e.g., we eventually turn them into real schemas, on disk). > +CREATE TABLE t1 (a MARIADB_SCHEMA.date); > +ERROR HY000: Unknown data type: 'MARIADB_SCHEMA.date' > +CREATE TABLE t1 (a Mariadb_schema.date); > +ERROR HY000: Unknown data type: 'Mariadb_schema.date' > +CREATE TABLE t1 (a mariadb_schema.date); > +DROP TABLE t1; > diff --git a/mysql-test/suite/compat/oracle/t/type_date.test b/mysql-test/suite/compat/oracle/t/type_date.test > index 61f7aa53944..34adc5889b4 100644 > --- a/mysql-test/suite/compat/oracle/t/type_date.test > +++ b/mysql-test/suite/compat/oracle/t/type_date.test > @@ -2,3 +2,101 @@ SET sql_mode=ORACLE; ... > +--echo # > +--echo # Qualified syntax is not supported yet in SP > +--echo # an MDEV for that? > diff --git a/sql/field.cc b/sql/field.cc > index 8accfb35b0b..4ed38f2c06f 100644 > --- a/sql/field.cc > +++ b/sql/field.cc > @@ -10407,6 +10407,19 @@ void Column_definition::set_attributes(const Lex_field_type_st &type, > set_handler(type.type_handler()); > charset= cs; > > +#if MYSQL_VERSION_ID > 100500 > +#error When merging to 10.5, please move the code below to > +#error Type_handler_timestamp_common::Column_definition_set_attributes() good > +#else > + /* > + Unlike other types TIMESTAMP fields are NOT NULL by default. > + Unless --explicit-defaults-for-timestamp is given. > + */ > + if (!opt_explicit_defaults_for_timestamp && > + type.type_handler()->field_type() == MYSQL_TYPE_TIMESTAMP) > + flags|= NOT_NULL_FLAG; > +#endif > + > if (type.length()) > { > int err; > diff --git a/sql/sql_lex.cc b/sql/sql_lex.cc > index 0b423355170..3fb07601f99 100644 > --- a/sql/sql_lex.cc > +++ b/sql/sql_lex.cc > @@ -2080,6 +2081,13 @@ int Lex_input_stream::scan_ident_start(THD *thd, Lex_ident_cli_st *str) > > uint length= yyLength(); > yyUnget(); // ptr points now after last token char > + > + if (thd->lex->parsing_options.lookup_keywords_after_qualifier) > + { > + if (int tokval= find_keyword(str, length, 0)) > + return tokval; > + } why is that? > + > str->set_ident(m_tok_start, length, is_8bit); > m_cpp_text_start= m_cpp_tok_start; > m_cpp_text_end= m_cpp_text_start + length; > @@ -8284,3 +8292,32 @@ bool LEX::tvc_finalize_derived() > current_select->linkage= DERIVED_TABLE_TYPE; > return tvc_finalize(); > } > + > + > +bool LEX::map_data_type(const Lex_ident_sys_st &schema_name, > + Lex_field_type_st *type) const > +{ > + const Schema *schema= schema_name.str ? > + Schema::find_by_name(schema_name) : > + Schema::find_implied(thd); > + if (!schema) > + { > + char buf[128]; > + const Name type_name= type->type_handler()->name(); > + my_snprintf(buf, sizeof(buf), "%.*s.%.*s", > + (int) schema_name.length, schema_name.str, > + (int) type_name.length(), type_name.ptr()); > +#if MYSQL_VERSION_ID > 100500 > +#error Please remove the old code > + my_error(ER_UNKNOWN_DATA_TYPE, MYF(0), buf); > +#else > + my_printf_error(ER_UNKNOWN_ERROR, "Unknown data type: '%-.64s'", > + MYF(0), buf); > +#endif > + return true; > + } > + const Type_handler *mapped= schema->map_data_type(thd, type->type_handler()); > + if (mapped != type->type_handler()) > + type->set_handler(mapped); why do you need an if() here? type->set_handler(mapped) is not that expensive. > + return false; > +} > diff --git a/sql/sql_show.cc b/sql/sql_show.cc > index 3dbc7724928..fcc2d44fdd6 100644 > --- a/sql/sql_show.cc > +++ b/sql/sql_show.cc > @@ -2224,6 +2224,13 @@ int show_create_table(THD *thd, TABLE_LIST *table_list, String *packet, > append_identifier(thd, packet, &field->field_name); > packet->append(' '); > > + const Type_handler *th= field->type_handler(); > + const Schema *implied_schema= Schema::find_implied(thd); > + if (th != implied_schema->map_data_type(thd, th)) > + { > + packet->append(th->schema()->name(), system_charset_info); sorry, I don't see it. How can th->schema() return not mariadb_schema? > + packet->append(STRING_WITH_LEN("."), system_charset_info); > + } > type.set(tmp, sizeof(tmp), system_charset_info); > field->sql_type(type); > packet->append(type.ptr(), type.length(), system_charset_info); > diff --git a/storage/test_sql_discovery/mysql-test/sql_discovery/simple.result b/storage/test_sql_discovery/mysql-test/sql_discovery/simple.result > index 23b7804638f..1feea5e47ee 100644 > --- a/storage/test_sql_discovery/mysql-test/sql_discovery/simple.result > +++ b/storage/test_sql_discovery/mysql-test/sql_discovery/simple.result > @@ -82,7 +82,7 @@ select * from t1; > ERROR HY000: Engine TEST_SQL_DISCOVERY failed to discover table `test`.`t1` with 'create table t1 (a uint)' > show warnings; > Level Code Message > -Error 1064 You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'uint)' at line 1 > +Error 1064 You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near ')' at line 1 this is not ideal :( can you try to keep the old error message? Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 3

[Maria-developers] Review for: MDEV-17399 Add support for JSON_TABLE, part #7
by Sergey Petrunia 30 Jul '20

30 Jul '20

Hi Alexey, On Thu, Jul 16, 2020 at 04:34:46PM +0400, Alexey Botchkov wrote: > > In the new patch i think i addressed most of your comments. > https://github.com/MariaDB/server/commit/d3a311a8e9508ef94dddb6b7f4d366337a… > ... > > The above doesn't do proper quoting, which is wrong. Testcase: > > > > select * > > from > > json_table('[{"co\\\\lor": "blue", "price": 50}]', > > '$[*]' columns( color varchar(100) path '$.co\\\\lor') > > ) as T; > > +-------+ > > | color | > > +-------+ > > | blue | > > +-------+ > > > > create view v2 as > > select * > > from > > json_table('[{"co\\\\lor": "blue", "price": 50}]', > > '$[*]' columns( color varchar(100) path '$.co\\\\lor') > > ) as T; > > select * from v2; Quoting is still an issue apparently: create view v1 as select * from json_table('[{"color": "blue", "price": 50}]', '$[*]' columns(color text path '$.nonexistent', seq for ordinality ) ) as `ALIAS NOT QUOTED`; select * from v1; ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'NOT QUOTED' at line 1 create view v2 as select * from json_table('[{"color": "blue", "price": 50}, {"color": "red", "price": 100}]', '$[*]' columns(color text path '$.QUOTES " HERE "', seq for ordinality ) ) as T; select * from v2; ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'HERE "", `seq` FOR ORDINALITY)) T' at line 1 Could you please go through the printout functions and make sure all printed literals are properly escaped/quoted? Printout code for other functions can be used as an example. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 1

Re: [Maria-developers] 2b527a4e858: MDEV-17729: Assertion `! is_set() || m_can_overwrite_status' failed in
by Sergei Golubchik 30 Jul '20

30 Jul '20

Hi, Rucha! On Jul 30, Rucha Deodhar wrote: > > > > > > - if ((level == Sql_condition::WARN_LEVEL_WARN) && > > > - really_abort_on_warning()) > > > + if ((killed == NOT_KILLED) && (level == Sql_condition::WARN_LEVEL_WARN) && > > > + really_abort_on_warning()) > > > > I don't understand this. I presume the assert that fails is in > > da->set_error_status(), called below as > > > > Yes, the assertion fails in set_error_status(). > > > if (!da->is_error()) > > { > > set_row_count_func(-1); > > da->set_error_status(sql_errno, msg, sqlstate, cond); > > } > > > > but if(!da->is_error()) should prevent set_error_status() from being > > called when it would need to overwrite the status. So how comes it is > > still called and triggers the assert? > > The status does not change to DA_ERROR yet, it changes in > set_error_status(), so !thd->is_error() becomes true and does not prevent > set_error_status() from being called. The reason we get assertion failure > in set_error_status() is because !is_set() is false (because status is > DA_OK) and m_can_overwrite_status is false too. I see. I also was confused by how can THD::raise_condition() be called after the status is already DA_OK? An error should not happen after DA_OK. But it turns out it was not just any error, it was push_warning() with the message "Query execution was interrupted. The query examined at least 4 rows, which exceeds LIMIT ROWS EXAMINED (2). The query result may be incomplete". This warning should not be converted to an error, obviously. It is converted, because thd->abort_on_warning is true. It's set to true in select_insert::prepare(). And the warning is pushed in handle_select(). Perhaps this warning should never have thd->abort_on_warning ? Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 2b527a4e858: MDEV-17729: Assertion `! is_set() || m_can_overwrite_status' failed in
by Sergei Golubchik 30 Jul '20

30 Jul '20

Hi, Rucha! On Jul 30, Rucha Deodhar wrote: > revision-id: 2b527a4e858 (mariadb-10.1.43-218-g2b527a4e858) > parent(s): 57ec42bc321 > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2020-07-27 09:50:00 +0530 > message: > > MDEV-17729: Assertion `! is_set() || m_can_overwrite_status' failed in > Diagnostics_area::set_error_status > > Analysis: When strict mode is enabled, all warnings are converted to errors > even when there is no bad data and the query is already killed. > Fix: If the query is already killed because examined rows exceeded the limit, > the warning is not because of bad data and should not be converted to error. > > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index 6897a26bda1..3591695a22d 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -1156,13 +1156,17 @@ Sql_condition* THD::raise_condition(uint sql_errno, > if (sqlstate == NULL) > sqlstate= mysql_errno_to_sqlstate(sql_errno); > > - if ((level == Sql_condition::WARN_LEVEL_WARN) && > - really_abort_on_warning()) > + if ((killed == NOT_KILLED) && (level == Sql_condition::WARN_LEVEL_WARN) && > + really_abort_on_warning()) I don't understand this. I presume the assert that fails is in da->set_error_status(), called below as if (!da->is_error()) { set_row_count_func(-1); da->set_error_status(sql_errno, msg, sqlstate, cond); } but if(!da->is_error()) should prevent set_error_status() from being called when it would need to overwrite the status. So how comes it is still called and triggers the assert? > { > - /* > - FIXME: > - push_warning and strict SQL_MODE case. > - */ > level= Sql_condition::WARN_LEVEL_ERROR; > set_killed(KILL_BAD_DATA); > } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] bc73b455ba2: MDEV-15703 Crash in EXECUTE IMMEDIATE 'CREATE OR REPLACE TABLE t1 (a INT DEFAULT ?)' USING DEFAULT
by Sergei Golubchik 29 Jul '20

29 Jul '20

Hi, Oleksandr! On Jul 29, Oleksandr Byelkin wrote: > revision-id: bc73b455ba2 (mariadb-10.2.31-318-gbc73b455ba2) > parent(s): 7311586ca25 > author: Oleksandr Byelkin <sanja(a)mariadb.com> > committer: Oleksandr Byelkin <sanja(a)mariadb.com> > timestamp: 2020-07-17 13:47:26 +0200 > message: > > MDEV-15703 Crash in EXECUTE IMMEDIATE 'CREATE OR REPLACE TABLE t1 (a INT DEFAULT ?)' USING DEFAULT > > part 2: ... > diff --git a/sql/field.h b/sql/field.h > index d5e2cb25788..2755f5f687c 100644 > --- a/sql/field.h > +++ b/sql/field.h > @@ -51,6 +51,16 @@ enum enum_check_fields > CHECK_FIELD_ERROR_FOR_NULL > }; > > + > +enum ignore_value_reaction > +{ > + IGNORE_MEANS_ERROR, > + IGNORE_MEANS_DEFAULT, > + IGNORE_MEANS_FIELD_VALUE I thought you'll have IGNORE_MEANS_IGNORE :) > +}; > + > +ignore_value_reaction find_ignore_reaction(THD *thd); > + > /* > Common declarations for Field and Item > */ > diff --git a/sql/item.h b/sql/item.h > index 7338c8be47b..dabf237712b 100644 > --- a/sql/item.h > +++ b/sql/item.h > @@ -1404,6 +1424,16 @@ class Item: public Value_source, > LOWEST_PRECEDENCE); > } > virtual void print(String *str, enum_query_type query_type); > + > + class Print: public String > + { > + public: > + Print(Item *item, enum_query_type type) > + { > + item->print(this, type); > + } no, I don't think this should be part of the bugfix. > + }; > + > void print_item_w_name(String *str, enum_query_type query_type); > void print_value(String *str); > > @@ -2077,6 +2107,15 @@ class Item: public Value_source, > { > marker &= ~EXTRACTION_MASK; > } > + > + virtual bool associate_with_target_field(THD *thd __attribute__((unused)), > + Item_field *field > + __attribute__((unused))) > + { > + DBUG_ASSERT(fixed); > + DBUG_ASSERT(fixed); you want to be double-sure? :) > + return false; > + } > }; > > > diff --git a/sql/item.cc b/sql/item.cc > index 382eb3ac47a..d11017f3ca6 100644 > --- a/sql/item.cc > +++ b/sql/item.cc > @@ -84,6 +85,15 @@ void item_init(void) > } > > > +void Item::raise_error_not_evaluable() > +{ > + Item::Print tmp(this, QT_ORDINARY); > + // TODO-10.5: add an error message to errmsg-utf8.txt > + my_printf_error(ER_UNKNOWN_ERROR, > + "'%s' is not allowed in this context", MYF(0), tmp.ptr()); I don't see any tests for this error > +} > + > + > void Item::push_note_converted_to_negative_complement(THD *thd) > { > push_warning(thd, Sql_condition::WARN_LEVEL_NOTE, ER_UNKNOWN_ERROR, > @@ -8945,6 +9022,14 @@ bool Item_default_value::fix_fields(THD *thd, Item **items) > return FALSE; > } > > + return tie_field(thd); > +} > + > + comment? what does tie_field do? > +bool Item_default_value::tie_field(THD *thd) > +{ > + Item *real_arg; > + Item_field *field_arg; > /* > DEFAULT() do not need table field so should not ask handler to bring > field value (mark column for read) > @@ -9131,8 +9241,54 @@ void Item_ignore_value::print(String *str, enum_query_type query_type) > str->append(STRING_WITH_LEN("ignore")); > } > > + > +bool Item_ignore_value::associate_with_target_field(THD *thd, > + Item_field *field) > +{ > + bitmap_set_bit(field->field->table->read_set, field->field->field_index); may be, remove it from the write_set instead? > + return Item_default_value::associate_with_target_field(thd, field); > +} > + > + > int Item_ignore_value::save_in_field(Field *field_arg, bool no_conversions) > { > + if (arg) > + { > + switch (find_ignore_reaction(field->table->in_use)) > + { > + case IGNORE_MEANS_DEFAULT: > + DBUG_ASSERT(0); // impossible now, but fully working code if needed Impossible? One cannot use IGNORE in INSERT? Not even in EXECUTE IMMEDIATE "INSERT ... " USING IGNORE? > + /* > + save default value > + Note: (((Field*)(arg()->real_item()) is Item_field* (checked in > + tie_field(). > + */ > + if ((((Item_field*)(arg->real_item()))->field->flags & > + NO_DEFAULT_VALUE_FLAG)) > + { > + my_error(ER_NO_DEFAULT_FOR_FIELD, MYF(0), > + ((Item_field*)(arg->real_item()))->field->field_name); > + return true; > + } > + calculate(); > + return Item_field::save_in_field(field_arg, no_conversions); > + case IGNORE_MEANS_FIELD_VALUE: > + // checked on tie_field > + DBUG_ASSERT(arg->real_item()->type() == FIELD_ITEM); > + /* > + save original field (Item::save_in_field is not applicable because > + result_field for the referenced field set to this temporary table). > + */ > + arg->save_val(field_arg); > + return false; > + default: > + ; // fall through to error > + } > + DBUG_ASSERT(0); //impossible > + my_error(ER_INVALID_DEFAULT_PARAM, MYF(0)); > + return true; > + } > + > return field_arg->save_in_field_ignore_value(context->error_processor == > &view_error_processor); > } > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > index 436f753557e..4dbda30708a 100644 > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -8340,6 +8341,9 @@ fill_record(THD *thd, TABLE *table, Field **ptr, List<Item> &values, > field->field_name, table->s->table_name.str); > } > > + if(evaluable && value->check_is_evaluable_expression_or_error()) > + goto err; this looks like the wrong way of doing it. whether an item is "evaluable" or not is metadata. It should be checked once before the statement execution > + > if (use_value) > value->save_val(field); > else > diff --git a/sql/sql_update.cc b/sql/sql_update.cc > index 65a828147ae..26ee427e3e7 100644 > --- a/sql/sql_update.cc > +++ b/sql/sql_update.cc > @@ -1796,6 +1796,9 @@ int multi_update::prepare(List<Item> &not_used_values, > while ((item= (Item_field *) field_it++)) > { > Item *value= value_it++; > + if (value->associate_with_target_field(thd, item)) > + DBUG_RETURN(1); what happens for single table updates? > + > uint offset= item->field->table->pos_in_table_list->shared; > fields_for_table[offset]->push_back(item, thd->mem_root); > values_for_table[offset]->push_back(value, thd->mem_root); > diff --git a/sql/unireg.cc b/sql/unireg.cc > index 083960523c1..a3bfc678e07 100644 > --- a/sql/unireg.cc > +++ b/sql/unireg.cc > @@ -1000,8 +1000,14 @@ static bool make_empty_rec(THD *thd, uchar *buff, uint table_options, > { > Item *expr= field->default_value->expr; > > - int res= !expr->fixed && // may be already fixed if ALTER TABLE > - expr->fix_fields(thd, &expr); > + int res= MY_TEST(!expr->fixed && // may be already fixed if ALTER TABLE > + expr->fix_fields(thd, &expr)); eh? what's the point of this change? > + if (!res && expr->check_is_evaluable_expression_or_error()) > + { > + error= 1; > + delete regfield; //To avoid memory leak > + goto err; > + } > if (!res) > res= expr->save_in_field(regfield, 1); > if (!res && (field->flags & BLOB_FLAG)) Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] Progress Report GSoC 2020 (20-26 Jule)
by Anton Mikhaylenko 28 Jul '20

28 Jul '20

Good day, everybody. In this post, I want to describe what I did this week, Jule 20-26. This week, I was refactoring. I added a flag for logging the number of threads and corrected the calculation of completed tasks. Also added comments to the code. Regards, Anton MIkhailenko

1 0

[Maria-developers] Weekly Report
by Kartik Soneji 27 Jul '20

27 Jul '20

Hi, This week - Resolved the abi_check issue with different compilers. Also added checks and updated documentation so that it does not happen again. - Started writing some tests for the lzma service. - Finished the first draft of the lzo service. - Worked on some refactors. - Discussed the idea of adding status variables to allow the storage engines to know which libraries are loaded. This will also allow the user to view the libraries that are loaded while the server is running. Next week - Finish writing tests for the lzma service. - Finalize the implementation of the service. - Start replicating the implementation for other libraries. - Discuss if we should change all to auto for the --use-compression= switch. That way, the server can throw an error if it is started with --use-compression=lzma but not if it is started normally. This will also allow for a configuration like --use-compression=lzma,auto so that the server will try to load all libraries but will not start unless lzma is loaded. Thanks, Kartik

1 0

[Maria-developers] 9b999e79a35: MDEV-23108: Point in time recovery of binary log fails when sql_mode=ORACLE
by sujatha 24 Jul '20

24 Jul '20

revision-id: 9b999e79a358de2c86ae44fe83cf6c62cff1d2ef (mariadb-10.3.21-180-g9b999e79a35) parent(s): 73aa31fbfd793bdb597bb19c52118ab4e637f4bc author: Sujatha committer: Sujatha timestamp: 2020-07-17 12:23:32 +0530 message: MDEV-23108: Point in time recovery of binary log fails when sql_mode=ORACLE Problem: ======== During point in time recovery of binary log syntax error is reported for BEGIN statement and recovery fails. Analysis: ========= In MariaDB 10.3 and later, setting the sql_mode system variable to Oracle allows the server to understand a subset of Oracle's PL/SQL language. When sql_mode=ORACLE is set, it switches the parser from the MariaDB parser to Oracle compatible parser. With this change 'BEGIN' is not considered as 'START TRANSACTION'. Hence the syntax error is reported. Fix: === Add a new option to 'mysqlbinlog' tool named 'sql_mode_oracle'. When 'sql_mode_oracle' option is specified 'BEGIN' statements will be replaced with 'START TRANSACTION' in the mysqlbinlog output. --- client/mysqlbinlog.cc | 7 +- .../compat/oracle/r/binlog_ptr_mysqlbinlog.result | 100 +++++++++++++++++ .../oracle/t/binlog_ptr_mysqlbinlog-master.opt | 1 + .../compat/oracle/t/binlog_ptr_mysqlbinlog.test | 120 +++++++++++++++++++++ sql/log_event.cc | 15 +-- sql/log_event.h | 1 + 6 files changed, 237 insertions(+), 7 deletions(-) diff --git a/client/mysqlbinlog.cc b/client/mysqlbinlog.cc index a732a6ab8c3..ad995629272 100644 --- a/client/mysqlbinlog.cc +++ b/client/mysqlbinlog.cc @@ -145,6 +145,7 @@ static MYSQL* mysql = NULL; static const char* dirname_for_local_load= 0; static bool opt_skip_annotate_row_events= 0; +static my_bool opt_sql_mode_oracle= 0; static my_bool opt_flashback; #ifdef WHEN_FLASHBACK_REVIEW_READY static my_bool opt_flashback_review; @@ -1045,7 +1046,8 @@ Exit_status process_event(PRINT_EVENT_INFO *print_event_info, Log_event *ev, DBUG_ENTER("process_event"); Exit_status retval= OK_CONTINUE; IO_CACHE *const head= &print_event_info->head_cache; - + /* Propogate sql_mode_oracle settings to event */ + ev->is_sql_mode_oracle= opt_sql_mode_oracle; /* Bypass flashback settings to event */ ev->is_flashback= opt_flashback; #ifdef WHEN_FLASHBACK_REVIEW_READY @@ -1704,6 +1706,9 @@ static struct my_option my_options[] = "use --base64-output=never", &short_form, &short_form, 0, GET_BOOL, NO_ARG, 0, 0, 0, 0, 0, 0}, + {"sql_mode_oracle", 0, "Converts BEGIN statement to 'ORALCE' " + "sql_mode compatible START TRANSACTION.", &opt_sql_mode_oracle, + &opt_sql_mode_oracle, 0, GET_BOOL, NO_ARG, 0, 0, 0, 0, 0, 0}, {"socket", 'S', "The socket file to use for connection.", &sock, &sock, 0, GET_STR, REQUIRED_ARG, 0, 0, 0, 0, 0, 0}, diff --git a/mysql-test/suite/compat/oracle/r/binlog_ptr_mysqlbinlog.result b/mysql-test/suite/compat/oracle/r/binlog_ptr_mysqlbinlog.result new file mode 100644 index 00000000000..3080a4b69a0 --- /dev/null +++ b/mysql-test/suite/compat/oracle/r/binlog_ptr_mysqlbinlog.result @@ -0,0 +1,100 @@ +SET @@SQL_MODE = 'ORACLE'; +########################################################################## +# Test verifies Gtid_log_event/Xid_log_event specific print # +########################################################################## +CREATE TABLE tm (f INT) ENGINE=MYISAM; +INSERT INTO tm VALUES (10); +CREATE TABLE t(f INT) ENGINE=INNODB; +INSERT INTO t VALUES (10); +CREATE OR REPLACE PROCEDURE simpleproc (param1 OUT INT) AS +BEGIN +SELECT COUNT(*) INTO param1 FROM t; +END; +/ +CREATE FUNCTION f1 RETURN INT +AS +BEGIN +RETURN 10; +END; +/ +FLUSH LOGS; +########################################################################## +# Delete data from master so that it can be restored from binlog # +########################################################################## +DROP FUNCTION f1; +DROP PROCEDURE simpleproc; +DROP TABLE tm; +DROP TABLE t; +########################################################################## +# Post recovery using mysqlbinlog --sql-mode-oracle # +########################################################################## +SHOW TABLES; +Tables_in_test +t +tm +SELECT * FROM tm; +f +10 +SELECT * FROM t; +f +10 +SELECT f1(); +f1() +10 +CALL simpleproc(@a); +SELECT @a; +@a +1 +"***** Clean Up *****" +DROP TABLE t,tm; +DROP PROCEDURE simpleproc; +DROP FUNCTION f1; +RESET MASTER; +########################################################################## +# Test verifies Gtid_log_event/Xid_log_event/Qery_log_event # +# specific print along with flashback option # +########################################################################## +CREATE TABLE tm(f INT) ENGINE=MYISAM; +INSERT INTO tm VALUES (10); +INSERT INTO tm VALUES (20); +CREATE TABLE t(f INT) ENGINE=INNODB; +INSERT INTO t VALUES (10); +INSERT INTO t VALUES (20); +########################################################################## +# Initial data # +########################################################################## +SELECT * FROM tm; +f +10 +20 +SELECT * FROM t; +f +10 +20 +FLUSH LOGS; +DELETE FROM tm WHERE f=20; +DELETE FROM t WHERE f=20; +FLUSH LOGS; +########################################################################## +# Data after deletion # +########################################################################## +SELECT * FROM tm; +f +10 +SELECT * FROM t; +f +10 +FOUND 2 /START TRANSACTION/ in test_sql_mode.sql +########################################################################## +# Data after recovery using flashback, sql-mode-oracle # +########################################################################## +SELECT * FROM tm; +f +10 +20 +SELECT * FROM t; +f +10 +20 +"***** Clean Up *****" +DROP TABLE t,tm; diff --git a/mysql-test/suite/compat/oracle/t/binlog_ptr_mysqlbinlog-master.opt b/mysql-test/suite/compat/oracle/t/binlog_ptr_mysqlbinlog-master.opt new file mode 100644 index 00000000000..8f0cc182f51 --- /dev/null +++ b/mysql-test/suite/compat/oracle/t/binlog_ptr_mysqlbinlog-master.opt @@ -0,0 +1 @@ +--flashback diff --git a/mysql-test/suite/compat/oracle/t/binlog_ptr_mysqlbinlog.test b/mysql-test/suite/compat/oracle/t/binlog_ptr_mysqlbinlog.test new file mode 100644 index 00000000000..07a1b5c4fdb --- /dev/null +++ b/mysql-test/suite/compat/oracle/t/binlog_ptr_mysqlbinlog.test @@ -0,0 +1,120 @@ +# ==== Purpose ==== +# +# Test verifies that a binary log which was generated by setting +# 'sql_mod=ORACLE' can be succefully used for point in time recovery with +# following options. +# 1) mysqlbinlog --sql-mode-oracle +# 2) mysqlbinlog --flashback --sql-mode-oracle +# +# BEGIN statement is printed in three places +# 1) "Gtid_log_event::print" +# 2) "Xid_log_event::print" if flashback is enabled +# 3) "Query_log_event::print" if flashback is enabled and engine is +# non-transacional. +# +# Test verifies all these cases. +# +# ==== References ==== +# +# MDEV-23108: Point in time recovery of binary log fails when sql_mode=ORACLE +# +--source include/have_log_bin.inc +--source include/have_innodb.inc + +let $MYSQLD_DATADIR= `select @@datadir`; +SET @@SQL_MODE = 'ORACLE'; + +--echo ########################################################################## +--echo # Test verifies Gtid_log_event/Xid_log_event specific print # +--echo ########################################################################## +CREATE TABLE tm (f INT) ENGINE=MYISAM; +INSERT INTO tm VALUES (10); + +CREATE TABLE t(f INT) ENGINE=INNODB; +INSERT INTO t VALUES (10); + +DELIMITER /; +CREATE OR REPLACE PROCEDURE simpleproc (param1 OUT INT) AS + BEGIN + SELECT COUNT(*) INTO param1 FROM t; + END; +/ +CREATE FUNCTION f1 RETURN INT +AS +BEGIN + RETURN 10; +END; +/ +DELIMITER ;/ + +FLUSH LOGS; +--echo ########################################################################## +--echo # Delete data from master so that it can be restored from binlog # +--echo ########################################################################## +DROP FUNCTION f1; +DROP PROCEDURE simpleproc; +DROP TABLE tm; +DROP TABLE t; + +--exec $MYSQL_BINLOG --sql-mode-oracle $MYSQLD_DATADIR/master-bin.000001 > $MYSQLTEST_VARDIR/tmp/test.sql +--exec $MYSQL < $MYSQLTEST_VARDIR/tmp/test.sql + +--echo ########################################################################## +--echo # Post recovery using mysqlbinlog --sql-mode-oracle # +--echo ########################################################################## +SHOW TABLES; +SELECT * FROM tm; +SELECT * FROM t; +--horizontal_results +SELECT f1(); +CALL simpleproc(@a); +SELECT @a; + +--echo "***** Clean Up *****" +DROP TABLE t,tm; +DROP PROCEDURE simpleproc; +DROP FUNCTION f1; +--remove_file $MYSQLTEST_VARDIR/tmp/test.sql +RESET MASTER; + +--echo ########################################################################## +--echo # Test verifies Gtid_log_event/Xid_log_event/Qery_log_event # +--echo # specific print along with flashback option # +--echo ########################################################################## +CREATE TABLE tm(f INT) ENGINE=MYISAM; +INSERT INTO tm VALUES (10); +INSERT INTO tm VALUES (20); +CREATE TABLE t(f INT) ENGINE=INNODB; +INSERT INTO t VALUES (10); +INSERT INTO t VALUES (20); +--echo ########################################################################## +--echo # Initial data # +--echo ########################################################################## +SELECT * FROM tm; +SELECT * FROM t; +FLUSH LOGS; +DELETE FROM tm WHERE f=20; +DELETE FROM t WHERE f=20; +FLUSH LOGS; + +--echo ########################################################################## +--echo # Data after deletion # +--echo ########################################################################## +SELECT * FROM tm; +SELECT * FROM t; +--exec $MYSQL_BINLOG --flashback --sql-mode-oracle $MYSQLD_DATADIR/master-bin.000002 > $MYSQLTEST_VARDIR/tmp/test_sql_mode.sql + +--let SEARCH_FILE=$MYSQLTEST_VARDIR/tmp/test_sql_mode.sql +--let SEARCH_PATTERN=START TRANSACTION +--source include/search_pattern_in_file.inc +--exec $MYSQL < $MYSQLTEST_VARDIR/tmp/test_sql_mode.sql + +--echo ########################################################################## +--echo # Data after recovery using flashback, sql-mode-oracle # +--echo ########################################################################## +SELECT * FROM tm; +SELECT * FROM t; + +--echo "***** Clean Up *****" +DROP TABLE t,tm; +--remove_file $MYSQLTEST_VARDIR/tmp/test_sql_mode.sql diff --git a/sql/log_event.cc b/sql/log_event.cc index 57a39085e86..e06e6a1b1d6 100644 --- a/sql/log_event.cc +++ b/sql/log_event.cc @@ -5358,8 +5358,9 @@ bool Query_log_event::print(FILE* file, PRINT_EVENT_INFO* print_event_info) } else if (strcmp("COMMIT", query) == 0) { - if (my_b_write(&cache, (uchar*) "BEGIN", 5) || - my_b_printf(&cache, "\n%s\n", print_event_info->delimiter)) + if (my_b_printf(&cache, is_sql_mode_oracle ? + "START TRANSACTION\n%s\n" : "BEGIN\n%s\n" , + print_event_info->delimiter)) goto err; } } @@ -8254,9 +8255,10 @@ Gtid_log_event::print(FILE *file, PRINT_EVENT_INFO *print_event_info) goto err; } if (!(flags2 & FL_STANDALONE)) - if (my_b_printf(&cache, is_flashback ? "COMMIT\n%s\n" : "BEGIN\n%s\n", print_event_info->delimiter)) + if (my_b_printf(&cache, is_flashback ? "COMMIT\n%s\n" : + (is_sql_mode_oracle ? "START TRANSACTION\n%s\n" : + "BEGIN\n%s\n"), print_event_info->delimiter)) goto err; - return cache.flush_data(); err: return 1; @@ -8937,8 +8939,9 @@ bool Xid_log_event::print(FILE* file, PRINT_EVENT_INFO* print_event_info) my_b_printf(&cache, "\tXid = %s\n", buf)) goto err; } - if (my_b_printf(&cache, is_flashback ? "BEGIN%s\n" : "COMMIT%s\n", - print_event_info->delimiter)) + if (my_b_printf(&cache, is_flashback ? (is_sql_mode_oracle ? + "START TRANSACTION\n%s\n" : "BEGIN\n%s\n") : + "COMMIT\n%s\n", print_event_info->delimiter)) goto err; return cache.flush_data(); diff --git a/sql/log_event.h b/sql/log_event.h index 73809953d70..66e29579fbc 100644 --- a/sql/log_event.h +++ b/sql/log_event.h @@ -1270,6 +1270,7 @@ class Log_event /* The following code used for Flashback */ #ifdef MYSQL_CLIENT + my_bool is_sql_mode_oracle; my_bool is_flashback; my_bool need_flashback_review; String output_buf; // Storing the event output

3 7

Re: [Maria-developers] 0dd0bbe54e1: MDEV-14836: Assertion `m_status == DA_ERROR' failed in
by Sergei Golubchik 22 Jul '20

22 Jul '20

Hi, Rucha! On Jul 22, Rucha Deodhar wrote: > revision-id: 0dd0bbe54e1 (mariadb-10.2.31-307-g0dd0bbe54e1) > parent(s): 0994af43e58 > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2020-07-16 22:29:06 +0530 > message: > > MDEV-14836: Assertion `m_status == DA_ERROR' failed in > Diagnostics_area::sql_errno upon query from I_S with LIMIT ROWS EXAMINED > > Query is aborted because number of examined rows exceeds LIMIT EXAMINED ROWS. > Even though this is not a real error, open_normal_and_derived_tables() > returns true. As a result, thd->is_errno() is called and thus the assertion > failure. > > diff --git a/mysql-test/r/information_schema.result b/mysql-test/r/information_schema.result > index 2d404aff6bc..7644ff4e7a2 100644 > --- a/mysql-test/r/information_schema.result > +++ b/mysql-test/r/information_schema.result > @@ -2184,3 +2184,16 @@ SCHEMA_NAME > # > # End of 10.1 tests > # > +# > +# Start of 10.2 Test > +# > +# MDEV-14836: Assertion `m_status == DA_ERROR' failed in > +# Diagnostics_area::sql_errno upon query from I_S with LIMIT ROWS EXAMINED > +# > +SELECT * FROM INFORMATION_SCHEMA.`COLUMNS` LIMIT ROWS EXAMINED 10; > +TABLE_CATALOG TABLE_SCHEMA TABLE_NAME COLUMN_NAME ORDINAL_POSITION COLUMN_DEFAULT IS_NULLABLE DATA_TYPE CHARACTER_MAXIMUM_LENGTH CHARACTER_OCTET_LENGTH NUMERIC_PRECISION NUMERIC_SCALE DATETIME_PRECISION CHARACTER_SET_NAME COLLATION_NAME COLUMN_TYPE COLUMN_KEY EXTRA PRIVILEGES COLUMN_COMMENT IS_GENERATED GENERATION_EXPRESSION > +Warnings: > +Warning 1931 Query execution was interrupted. The query examined at least 666 rows, which exceeds LIMIT ROWS EXAMINED (10). The query result may be incomplete I don't understand why there are no rows in the output here? It has examined quite a lot of rows, I would expect a truncated result, but not a completely empty one. Could you check that out, please? > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > index 436f753557e..8f1c0ebab29 100644 > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -1590,6 +1590,12 @@ bool open_table(THD *thd, TABLE_LIST *table_list, Open_table_context *ot_ctx) > if (!(flags & MYSQL_OPEN_IGNORE_KILLED) && thd->killed) > { > thd->send_kill_message(); > + /* > + Query was only aborted, but this is not an actual error. > + So return false. > + */ > + if (thd->killed == ABORT_QUERY) > + DBUG_RETURN(FALSE); > DBUG_RETURN(TRUE); This looks somewhat risky. open_table() returns TRUE on error and FALSE on success. You here make it to return FALSE even if the table wasn't opened. I am not sure the caller will always be able to handle that. On the other hand, normally all tables are opened first, and then the server starts reading the data. It's only in I_S the server opens tables one by one. So get_all_tables() could be the only possibility for ABORT_QUERY to happen before open_table(). In that case it is safer to add a check for thd->killed into get_all_tables(). Could you try that approach instead? Also, note, don't compare thd->killed with ABORT_QUERY, use thd->killed_errno() method instead. > } > > diff --git a/sql/sql_show.cc b/sql/sql_show.cc > index 18b7e92bca5..9f834ee118b 100644 > --- a/sql/sql_show.cc > +++ b/sql/sql_show.cc > @@ -5622,6 +5622,16 @@ static int get_schema_column_record(THD *thd, TABLE_LIST *tables, > } > DBUG_RETURN(res); > } > + /* > + Query was aborted. This could be because of number of > + examined rows exceeded LIMIT ROWS EXAMINED. But don't > + issue warning here. It is done later, in handle_select(). > + */ > + if(thd->killed == ABORT_QUERY) > + { > + res= 0; > + DBUG_RETURN(res); > + } > > show_table= tables->table; > count= 0; > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] 27b046d2e95: MDEV-14203: rpl.rpl_extra_col_master_myisam, rpl.rpl_slave_load_tmpdir_not_exist failed in buildbot with a warning
by sujatha 21 Jul '20

21 Jul '20

revision-id: 27b046d2e95add5ba87e231279d2a7774db5b0c7 (mariadb-10.2.31-317-g27b046d2e95) parent(s): a1e52e7f32cee7c204078db4c4beb88250d7e1f6 author: Sujatha committer: Sujatha timestamp: 2020-07-21 11:16:40 +0530 message: MDEV-14203: rpl.rpl_extra_col_master_myisam, rpl.rpl_slave_load_tmpdir_not_exist failed in buildbot with a warning Problem: ======= rpl.rpl_slave_load_tmpdir_not_exist 'stmt' w3 [ fail ] Found warnings/errors in server log file! Test ended at 2017-09-27 20:34:55 [Warning] Master is configured to log replication events with checksum, but will not send such events to slaves that cannot process them ^ Found warnings in /mnt/buildbot/build/mariadb-10.2.10/mysql-test/var/3/log/mysqld.1.err ok Analysis: ======== When slave tries to connect to master 'get_master_version_and_clock' function is invoked to perform elaborated slave-master handshake. During this process slave server queries master server, to know if it is checksum aware and at the same time master is notified about its CRC-awareness. The master's side instant value of @@global.binlog_checksum is stored in the dump thread's uservar area as well as cached locally to become known in consensus by master and slave. Post hand-shake slave requests master for binlog dump. It sends 'COM_BINLOG_DUMP'. This command is sent to master by 'cli_advanced_command' call. If there is some temporary network failure during this request_dump call, 'end_server' is invoked to close the current connection between master and slave. Upon connection close the dump thread on the master gets terminated and it clears the 'uservar' data it got through master-slave handshake. The 'COM_BINLOG_DUMP' command is sent once again without master-slave handshake. Since the checksum data is not available with new dump thread a warning gets reported. Fix: === Upon network write error donot attempt reconnect, proceed to master-slave handshake. This ensures that master is aware of slave's capability to use checksums. --- .../rpl/r/rpl_dump_request_retry_warning.result | 22 ++++++++ .../rpl/t/rpl_dump_request_retry_warning.test | 60 ++++++++++++++++++++++ sql-common/client.c | 2 + sql/net_serv.cc | 13 +++++ sql/slave.cc | 3 +- 5 files changed, 99 insertions(+), 1 deletion(-) diff --git a/mysql-test/suite/rpl/r/rpl_dump_request_retry_warning.result b/mysql-test/suite/rpl/r/rpl_dump_request_retry_warning.result new file mode 100644 index 00000000000..3a80d5b5f31 --- /dev/null +++ b/mysql-test/suite/rpl/r/rpl_dump_request_retry_warning.result @@ -0,0 +1,22 @@ +include/master-slave.inc +[connection master] +CREATE TABLE t1(a int); +INSERT INTO t1 VALUES(1); +connection slave; +SET @saved_dbug = @@GLOBAL.debug_dbug; +SET @@global.debug_dbug= 'd,simulate_error_on_packet_write'; +START SLAVE; +SET DEBUG_SYNC= 'now WAIT_FOR parked'; +SET @@GLOBAL.debug_dbug = @saved_dbug; +SET DEBUG_SYNC= 'now SIGNAL continue'; +SET DEBUG_SYNC= 'RESET'; +include/wait_for_slave_io_to_start.inc +include/wait_for_slave_sql_to_start.inc +connection master; +include/sync_slave_sql_with_master.inc +SELECT * FROM t1; +a +1 +connection master; +DROP TABLE t1; +include/rpl_end.inc diff --git a/mysql-test/suite/rpl/t/rpl_dump_request_retry_warning.test b/mysql-test/suite/rpl/t/rpl_dump_request_retry_warning.test new file mode 100644 index 00000000000..d9fb8258765 --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_dump_request_retry_warning.test @@ -0,0 +1,60 @@ +# ==== Purpose ==== +# +# Test verifies that, due to a temporary network error, if request dump +# command specific packet write operation fails then the write error gets +# handled appropriately. Further retry will be initiated with appropriate +# slave registration on master. This will ensure that master has all the +# details of slave and no warnings are reported on the master side. +# +# ==== Implementation ==== +# +# Steps: +# 0 - Skip the slave start. +# 1 - Enable debug simulation which will simulate packet write error during +# dump request command execution. +# 2 - Start the slave. Observe that slave is able to reconnect post +# temporary network write error. +# +# ==== References ==== +# +# MDEV-14203: rpl.rpl_extra_col_master_myisam, +# rpl.rpl_slave_load_tmpdir_not_exist failed in buildbot with a +# warning +# +# MDEV-13258: rpl.rpl_skip_replication, rpl.rpl_set_statement_default_master +# failed in buildbot +# + +--source include/have_debug.inc +--source include/have_debug_sync.inc +--let $rpl_skip_start_slave=1 +--source include/master-slave.inc + +# Do an insert on master +CREATE TABLE t1(a int); +INSERT INTO t1 VALUES(1); + +# Add a debug point and start the slave so that dump request fails. +connection slave; +SET @saved_dbug = @@GLOBAL.debug_dbug; +SET @@global.debug_dbug= 'd,simulate_error_on_packet_write'; + +START SLAVE; +SET DEBUG_SYNC= 'now WAIT_FOR parked'; +SET @@GLOBAL.debug_dbug = @saved_dbug; +SET DEBUG_SYNC= 'now SIGNAL continue'; +SET DEBUG_SYNC= 'RESET'; + +--source include/wait_for_slave_io_to_start.inc +--source include/wait_for_slave_sql_to_start.inc + +# Sync the slave and verify that slave has caught up with the master. +connection master; +--source include/sync_slave_sql_with_master.inc +SELECT * FROM t1; + +# Cleanup +connection master; +DROP TABLE t1; + +--source include/rpl_end.inc diff --git a/sql-common/client.c b/sql-common/client.c index 227759f4163..b66eee2a508 100644 --- a/sql-common/client.c +++ b/sql-common/client.c @@ -721,6 +721,8 @@ cli_advanced_command(MYSQL *mysql, enum enum_server_command command, set_mysql_error(mysql, CR_NET_PACKET_TOO_LARGE, unknown_sqlstate); goto end; } + if (net->last_errno == ER_NET_ERROR_ON_WRITE && command == COM_BINLOG_DUMP) + goto end; end_server(mysql); if (mysql_reconnect(mysql) || stmt_skip) goto end; diff --git a/sql/net_serv.cc b/sql/net_serv.cc index f39eaaadf46..e625b307343 100644 --- a/sql/net_serv.cc +++ b/sql/net_serv.cc @@ -45,6 +45,7 @@ #include <violite.h> #include <signal.h> #include "probes_mysql.h" +#include <debug_sync.h> #ifdef EMBEDDED_LIBRARY #undef MYSQL_SERVER @@ -486,6 +487,18 @@ net_write_command(NET *net,uchar command, DBUG_ENTER("net_write_command"); DBUG_PRINT("enter",("length: %lu", (ulong) len)); + DBUG_EXECUTE_IF("simulate_error_on_packet_write", + { + if (command == COM_BINLOG_DUMP) + { + net->last_errno = ER_NET_ERROR_ON_WRITE; + socket_errno = 2; + DBUG_ASSERT(!debug_sync_set_action( + (THD *)net->thd, + STRING_WITH_LEN("now SIGNAL parked WAIT_FOR continue"))); + DBUG_RETURN(true); + } + };); MYSQL_NET_WRITE_START(length); buff[4]=command; /* For first packet */ diff --git a/sql/slave.cc b/sql/slave.cc index 1bf83aa9652..87eacfcfd0a 100644 --- a/sql/slave.cc +++ b/sql/slave.cc @@ -3297,7 +3297,8 @@ static int request_dump(THD *thd, MYSQL* mysql, Master_info* mi, in the future, we should do a better error analysis, but for now we just fill up the error log :-) */ - if (mysql_errno(mysql) == ER_NET_READ_INTERRUPTED) + if (mysql_errno(mysql) == ER_NET_READ_INTERRUPTED || + mysql_errno(mysql) == ER_NET_ERROR_ON_WRITE) *suppress_warnings= TRUE; // Suppress reconnect warning else sql_print_error("Error on COM_BINLOG_DUMP: %d %s, will retry in %d secs",

1 0

[Maria-developers] Weekly Report
by Kartik Soneji 20 Jul '20

20 Jul '20

Hi, This week - Changed the build system to use our headers instead of the libraries’ headers. - I found that running make abi_update on gcc and compiling on clang (or vice versa) fails the abi_check. - During cmake documentation review, I discovered that the existing cmake configuration uses a lot of depricated commands. Even for this project, it might be easier and more stable to rewrite the relavent portions of the cmake files instead of trying to work around the depricated commands. - Removed the test cases and printf statements from the code. Next week - Resolve the abi_check issue with different compilers. - Write tests for the lzma service. - Start work on the next service. Thanks, Kartik

1 0

[Maria-developers] Progress Report GSoC 2020 (13-19 Jule)
by Anton Mikhaylenko 20 Jul '20

20 Jul '20

Good day, everybody. In this post, I want to describe what I did this week, Jule 13-19. I kept working on the benchmarking for a treadpool. I ran several configurations, which included the version without the algorithm and the version with the algorithm, also changed the time parameters, a number of clients, types of requests. Regards, Anton MIkhailenko

1 0

[Maria-developers] f963fa52ed0: MDEV-23108: Point in time recovery of binary log fails when sql_mode=ORACLE
by sujatha 17 Jul '20

17 Jul '20

revision-id: f963fa52ed07469881e75349e73efede8240fcea (mariadb-10.3.21-180-gf963fa52ed0) parent(s): 73aa31fbfd793bdb597bb19c52118ab4e637f4bc author: Sujatha committer: Sujatha timestamp: 2020-07-17 19:53:43 +0530 message: MDEV-23108: Point in time recovery of binary log fails when sql_mode=ORACLE Problem: ======== During point in time recovery of binary log syntax error is reported for BEGIN statement and recovery fails. Analysis: ========= In MariaDB 10.3 and later, setting the sql_mode system variable to Oracle allows the server to understand a subset of Oracle's PL/SQL language. When sql_mode=ORACLE is set, it switches the parser from the MariaDB parser to Oracle compatible parser. With this change 'BEGIN' is not considered as 'START TRANSACTION'. Hence the syntax error is reported. Fix: === 'mysqlbinlog' tool will convert all 'BEGIN' statements to 'START TRANSACTION' as 'START TRANSACTION' works fine in all sql_modes. --- .../include/binlog_parallel_replication_marks.test | 2 +- mysql-test/main/mysqlbinlog-innodb.result | 8 +- mysql-test/main/mysqlbinlog.result | 74 ++--- mysql-test/main/mysqlbinlog_row_compressed.result | 16 +- mysql-test/main/mysqlbinlog_row_minimal.result | 18 +- mysql-test/main/mysqlbinlog_stmt_compressed.result | 16 +- mysql-test/main/user_var-binlog.result | 4 +- .../suite/binlog/r/binlog_mysqlbinlog2.result | 206 ++++++------ .../suite/binlog/r/binlog_mysqlbinlog_row.result | 358 ++++++++++----------- .../binlog/r/binlog_mysqlbinlog_row_innodb.result | 40 +-- .../binlog/r/binlog_mysqlbinlog_row_myisam.result | 40 +-- .../binlog/r/binlog_mysqlbinlog_row_trans.result | 16 +- .../r/binlog_parallel_replication_marks_row.result | 18 +- ...inlog_parallel_replication_marks_stm_mix.result | 18 +- .../suite/binlog/r/binlog_row_annotate.result | 72 ++--- .../suite/binlog/r/binlog_row_ctype_ucs.result | 14 +- .../binlog/r/binlog_row_mysqlbinlog_options.result | 28 +- .../suite/binlog/r/binlog_stm_ctype_ucs.result | 14 +- mysql-test/suite/binlog/r/flashback.result | 20 +- .../binlog_encryption/binlog_row_annotate.result | 36 +-- .../compat/oracle/r/binlog_ptr_mysqlbinlog.result | 100 ++++++ .../oracle/t/binlog_ptr_mysqlbinlog-master.opt | 1 + .../compat/oracle/t/binlog_ptr_mysqlbinlog.test | 117 +++++++ .../suite/rpl/r/rpl_blackhole_row_annotate.result | 22 +- mysql-test/suite/rpl/r/rpl_hrtime.result | 10 +- mysql-test/suite/rpl/r/rpl_mdev382.result | 10 +- mysql-test/suite/rpl/r/rpl_sp.result | 52 +-- sql/log_event.cc | 8 +- 28 files changed, 778 insertions(+), 560 deletions(-) diff --git a/mysql-test/include/binlog_parallel_replication_marks.test b/mysql-test/include/binlog_parallel_replication_marks.test index 3976088ca43..29e86d64d3e 100644 --- a/mysql-test/include/binlog_parallel_replication_marks.test +++ b/mysql-test/include/binlog_parallel_replication_marks.test @@ -80,7 +80,7 @@ while (<F>) { s/table id \d+/table id #/; s/mapped to number \d+/mapped to number #/; s/CRC32 0x[0-9a-f]+/CRC32 0x########/; - print if /\b(GTID|BEGIN|COMMIT|Table_map|Write_rows|Update_rows|Delete_rows|generated by server|40005 TEMPORARY)\b/; + print if /\b(GTID|START TRANSACTION|COMMIT|Table_map|Write_rows|Update_rows|Delete_rows|generated by server|40005 TEMPORARY)\b/; } close F; EOF diff --git a/mysql-test/main/mysqlbinlog-innodb.result b/mysql-test/main/mysqlbinlog-innodb.result index f9f48299f41..9d3bcc7c776 100644 --- a/mysql-test/main/mysqlbinlog-innodb.result +++ b/mysql-test/main/mysqlbinlog-innodb.result @@ -24,7 +24,7 @@ FLUSH LOGS; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; ROLLBACK/*!*/; -BEGIN +START TRANSACTION /*!*/; use `test`/*!*/; SET TIMESTAMP=1000000000/*!*/; @@ -39,7 +39,7 @@ SET @@session.collation_database=DEFAULT/*!*/; INSERT INTO t1 VALUES (1) /*!*/; COMMIT/*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; INSERT INTO t1 VALUES (2) @@ -55,7 +55,7 @@ ROLLBACK /* added by mysqlbinlog */; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; ROLLBACK/*!*/; -BEGIN +START TRANSACTION /*!*/; use `foo`/*!*/; SET TIMESTAMP=1000000000/*!*/; @@ -70,7 +70,7 @@ SET @@session.collation_database=DEFAULT/*!*/; INSERT INTO t1 VALUES (1) /*!*/; COMMIT/*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; INSERT INTO t1 VALUES (2) diff --git a/mysql-test/main/mysqlbinlog.result b/mysql-test/main/mysqlbinlog.result index decd095bbdf..4c82563e0f6 100644 --- a/mysql-test/main/mysqlbinlog.result +++ b/mysql-test/main/mysqlbinlog.result @@ -37,7 +37,7 @@ create table t1 (word varchar(20)) SET TIMESTAMP=1000000000/*!*/; create table t2 (id int auto_increment not null primary key) /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; insert into t1 values ("abirvalg") @@ -45,7 +45,7 @@ insert into t1 values ("abirvalg") SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=1/*!*/; SET TIMESTAMP=1000000000/*!*/; @@ -54,7 +54,7 @@ insert into t2 values () SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; LOAD DATA LOCAL INFILE 'MYSQLTEST_VARDIR/tmp/SQL_LOAD_MB-#-#' INTO TABLE `t1` FIELDS TERMINATED BY '\t' ENCLOSED BY '' ESCAPED BY '\\' LINES TERMINATED BY '\n' (`word`) @@ -62,7 +62,7 @@ LOAD DATA LOCAL INFILE 'MYSQLTEST_VARDIR/tmp/SQL_LOAD_MB-#-#' INTO TABLE `t1` FI SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; LOAD DATA LOCAL INFILE 'MYSQLTEST_VARDIR/tmp/SQL_LOAD_MB-#-#' INTO TABLE `t1` FIELDS TERMINATED BY '\t' ENCLOSED BY '' ESCAPED BY '\\' LINES TERMINATED BY '\n' (`word`) @@ -70,7 +70,7 @@ LOAD DATA LOCAL INFILE 'MYSQLTEST_VARDIR/tmp/SQL_LOAD_MB-#-#' INTO TABLE `t1` FI SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; LOAD DATA LOCAL INFILE 'MYSQLTEST_VARDIR/tmp/SQL_LOAD_MB-#-#' INTO TABLE `t1` FIELDS TERMINATED BY '\t' ENCLOSED BY '' ESCAPED BY '\\' LINES TERMINATED BY '\n' (`word`) @@ -78,7 +78,7 @@ LOAD DATA LOCAL INFILE 'MYSQLTEST_VARDIR/tmp/SQL_LOAD_MB-#-#' INTO TABLE `t1` FI SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; LOAD DATA LOCAL INFILE 'MYSQLTEST_VARDIR/tmp/SQL_LOAD_MB-#-#' INTO TABLE `t1` FIELDS TERMINATED BY '\t' ENCLOSED BY '' ESCAPED BY '\\' LINES TERMINATED BY '\n' (`word`) @@ -97,7 +97,7 @@ ROLLBACK /* added by mysqlbinlog */; /*!40019 SET @@session.max_insert_delayed_threads=0*/; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; -BEGIN +START TRANSACTION /*!*/; use `test`/*!*/; SET TIMESTAMP=1000000000/*!*/; @@ -114,7 +114,7 @@ LOAD DATA LOCAL INFILE 'MYSQLTEST_VARDIR/tmp/SQL_LOAD_MB-#-#' INTO TABLE `t1` FI SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; insert into t1 values ("Alas") @@ -134,7 +134,7 @@ ROLLBACK /* added by mysqlbinlog */; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; ROLLBACK/*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; SET @@session.pseudo_thread_id=999999999/*!*/; @@ -147,28 +147,28 @@ SET @@session.lc_time_names=0/*!*/; SET @@session.collation_database=DEFAULT/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=1/*!*/; SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; COMMIT @@ -184,7 +184,7 @@ ROLLBACK /* added by mysqlbinlog */; /*!40019 SET @@session.max_insert_delayed_threads=0*/; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; -BEGIN +START TRANSACTION /*!*/; use `test`/*!*/; SET TIMESTAMP=1000000000/*!*/; @@ -231,7 +231,7 @@ create table t1 (word varchar(20)) SET TIMESTAMP=1000000000/*!*/; create table t2 (id int auto_increment not null primary key) /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; insert into t1 values ("abirvalg") @@ -239,7 +239,7 @@ insert into t1 values ("abirvalg") SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=1/*!*/; SET TIMESTAMP=1000000000/*!*/; @@ -248,7 +248,7 @@ insert into t2 values () SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; LOAD DATA LOCAL INFILE 'MYSQLTEST_VARDIR/tmp/SQL_LOAD_MB-#-#' INTO TABLE `t1` FIELDS TERMINATED BY '\t' ENCLOSED BY '' ESCAPED BY '\\' LINES TERMINATED BY '\n' (`word`) @@ -256,7 +256,7 @@ LOAD DATA LOCAL INFILE 'MYSQLTEST_VARDIR/tmp/SQL_LOAD_MB-#-#' INTO TABLE `t1` FI SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; LOAD DATA LOCAL INFILE 'MYSQLTEST_VARDIR/tmp/SQL_LOAD_MB-#-#' INTO TABLE `t1` FIELDS TERMINATED BY '\t' ENCLOSED BY '' ESCAPED BY '\\' LINES TERMINATED BY '\n' (`word`) @@ -264,7 +264,7 @@ LOAD DATA LOCAL INFILE 'MYSQLTEST_VARDIR/tmp/SQL_LOAD_MB-#-#' INTO TABLE `t1` FI SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; LOAD DATA LOCAL INFILE 'MYSQLTEST_VARDIR/tmp/SQL_LOAD_MB-#-#' INTO TABLE `t1` FIELDS TERMINATED BY '\t' ENCLOSED BY '' ESCAPED BY '\\' LINES TERMINATED BY '\n' (`word`) @@ -272,7 +272,7 @@ LOAD DATA LOCAL INFILE 'MYSQLTEST_VARDIR/tmp/SQL_LOAD_MB-#-#' INTO TABLE `t1` FI SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; LOAD DATA LOCAL INFILE 'MYSQLTEST_VARDIR/tmp/SQL_LOAD_MB-#-#' INTO TABLE `t1` FIELDS TERMINATED BY '\t' ENCLOSED BY '' ESCAPED BY '\\' LINES TERMINATED BY '\n' (`word`) @@ -291,7 +291,7 @@ ROLLBACK /* added by mysqlbinlog */; /*!40019 SET @@session.max_insert_delayed_threads=0*/; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; -BEGIN +START TRANSACTION /*!*/; use `test`/*!*/; SET TIMESTAMP=1000000000/*!*/; @@ -308,7 +308,7 @@ LOAD DATA LOCAL INFILE 'MYSQLTEST_VARDIR/tmp/SQL_LOAD_MB-#-#' INTO TABLE `t1` FI SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; insert into t1 values ("Alas") @@ -328,7 +328,7 @@ ROLLBACK /* added by mysqlbinlog */; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; ROLLBACK/*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; SET @@session.pseudo_thread_id=999999999/*!*/; @@ -341,28 +341,28 @@ SET @@session.lc_time_names=0/*!*/; SET @@session.collation_database=DEFAULT/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=1/*!*/; SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; COMMIT @@ -378,7 +378,7 @@ ROLLBACK /* added by mysqlbinlog */; /*!40019 SET @@session.max_insert_delayed_threads=0*/; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; -BEGIN +START TRANSACTION /*!*/; use `test`/*!*/; SET TIMESTAMP=1000000000/*!*/; @@ -553,7 +553,7 @@ SET @@session.lc_time_names=0/*!*/; SET @@session.collation_database=DEFAULT/*!*/; create table t1 (a varchar(64) character set utf8) /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; LOAD DATA LOCAL INFILE 'MYSQLTEST_VARDIR/tmp/SQL_LOAD_MB-#-#' INTO TABLE `t1` FIELDS TERMINATED BY '\t' ENCLOSED BY '' ESCAPED BY '\\' LINES TERMINATED BY '\n' (`a`) @@ -561,7 +561,7 @@ LOAD DATA LOCAL INFILE 'MYSQLTEST_VARDIR/tmp/SQL_LOAD_MB-#-#' INTO TABLE `t1` FI SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; SET @@session.collation_database=7/*!*/; @@ -570,7 +570,7 @@ LOAD DATA LOCAL INFILE 'MYSQLTEST_VARDIR/tmp/SQL_LOAD_MB-#-#' INTO TABLE `t1` FI SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; SET @@session.collation_database=DEFAULT/*!*/; @@ -579,7 +579,7 @@ LOAD DATA LOCAL INFILE 'MYSQLTEST_VARDIR/tmp/SQL_LOAD_MB-#-#' INTO TABLE `t1` FI SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; LOAD DATA LOCAL INFILE 'MYSQLTEST_VARDIR/tmp/SQL_LOAD_MB-#-#' INTO TABLE `t1` FIELDS TERMINATED BY '\t' ENCLOSED BY '' ESCAPED BY '\\' LINES TERMINATED BY '\n' (`a`) @@ -587,7 +587,7 @@ LOAD DATA LOCAL INFILE 'MYSQLTEST_VARDIR/tmp/SQL_LOAD_MB-#-#' INTO TABLE `t1` FI SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; SET @@session.collation_database=7/*!*/; @@ -596,7 +596,7 @@ LOAD DATA LOCAL INFILE 'MYSQLTEST_VARDIR/tmp/SQL_LOAD_MB-#-#' INTO TABLE `t1` FI SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; SET @@session.collation_database=DEFAULT/*!*/; @@ -605,7 +605,7 @@ LOAD DATA LOCAL INFILE 'MYSQLTEST_VARDIR/tmp/SQL_LOAD_MB-#-#' INTO TABLE `t1` FI SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; LOAD DATA LOCAL INFILE 'MYSQLTEST_VARDIR/tmp/SQL_LOAD_MB-#-#' INTO TABLE `t1` CHARACTER SET koi8r FIELDS TERMINATED BY '\t' ENCLOSED BY '' ESCAPED BY '\\' LINES TERMINATED BY '\n' (`a`) diff --git a/mysql-test/main/mysqlbinlog_row_compressed.result b/mysql-test/main/mysqlbinlog_row_compressed.result index 94125352654..cb8d60704f9 100644 --- a/mysql-test/main/mysqlbinlog_row_compressed.result +++ b/mysql-test/main/mysqlbinlog_row_compressed.result @@ -53,7 +53,7 @@ CREATE TABLE t2 (pk INT PRIMARY KEY, f1 INT, f2 INT, f3 INT, f4 INT, f5 MEDIUMIN # at 727 #<date> server id 1 end_log_pos 769 CRC32 XXX GTID 0-1-3 /*!100001 SET @@session.gtid_seq_no=3*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at 769 # at 843 @@ -82,7 +82,7 @@ COMMIT # at 1040 #<date> server id 1 end_log_pos 1082 CRC32 XXX GTID 0-1-4 /*!100001 SET @@session.gtid_seq_no=4*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at 1082 # at 1158 @@ -111,7 +111,7 @@ COMMIT # at 1354 #<date> server id 1 end_log_pos 1396 CRC32 XXX GTID 0-1-5 /*!100001 SET @@session.gtid_seq_no=5*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at 1396 # at 1474 @@ -140,7 +140,7 @@ COMMIT # at 1669 #<date> server id 1 end_log_pos 1711 CRC32 XXX GTID 0-1-6 /*!100001 SET @@session.gtid_seq_no=6*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at 1711 # at 1786 @@ -169,7 +169,7 @@ COMMIT # at 1982 #<date> server id 1 end_log_pos 2024 CRC32 XXX GTID 0-1-7 /*!100001 SET @@session.gtid_seq_no=7*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at 2024 # at 2078 @@ -231,7 +231,7 @@ COMMIT # at 2298 #<date> server id 1 end_log_pos 2340 CRC32 XXX GTID 0-1-8 /*!100001 SET @@session.gtid_seq_no=8*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at 2340 # at 2406 @@ -312,7 +312,7 @@ COMMIT # at 2634 #<date> server id 1 end_log_pos 2676 CRC32 XXX GTID 0-1-9 /*!100001 SET @@session.gtid_seq_no=9*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at 2676 # at 2713 @@ -374,7 +374,7 @@ COMMIT # at 2934 #<date> server id 1 end_log_pos 2976 CRC32 XXX GTID 0-1-10 /*!100001 SET @@session.gtid_seq_no=10*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at 2976 # at 3013 diff --git a/mysql-test/main/mysqlbinlog_row_minimal.result b/mysql-test/main/mysqlbinlog_row_minimal.result index 6f3dbab55a8..933457edc25 100644 --- a/mysql-test/main/mysqlbinlog_row_minimal.result +++ b/mysql-test/main/mysqlbinlog_row_minimal.result @@ -51,7 +51,7 @@ CREATE TABLE t2 (pk INT PRIMARY KEY, f1 INT, f2 INT, f3 INT, f4 INT, f5 MEDIUMIN # at 774 #<date> server id 1 end_log_pos 816 CRC32 XXX GTID 0-1-3 /*!100001 SET @@session.gtid_seq_no=3*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at 816 # at 890 @@ -80,7 +80,7 @@ COMMIT # at 1088 #<date> server id 1 end_log_pos 1130 CRC32 XXX GTID 0-1-4 /*!100001 SET @@session.gtid_seq_no=4*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at 1130 # at 1206 @@ -109,7 +109,7 @@ COMMIT # at 1403 #<date> server id 1 end_log_pos 1445 CRC32 XXX GTID 0-1-5 /*!100001 SET @@session.gtid_seq_no=5*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at 1445 # at 1523 @@ -138,7 +138,7 @@ COMMIT # at 1719 #<date> server id 1 end_log_pos 1761 CRC32 XXX GTID 0-1-6 /*!100001 SET @@session.gtid_seq_no=6*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at 1761 # at 1836 @@ -167,7 +167,7 @@ COMMIT # at 2035 #<date> server id 1 end_log_pos 2077 CRC32 XXX GTID 0-1-7 /*!100001 SET @@session.gtid_seq_no=7*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at 2077 # at 2131 @@ -229,7 +229,7 @@ COMMIT # at 2427 #<date> server id 1 end_log_pos 2469 CRC32 XXX GTID 0-1-8 /*!100001 SET @@session.gtid_seq_no=8*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at 2469 # at 2535 @@ -262,7 +262,7 @@ COMMIT # at 2730 #<date> server id 1 end_log_pos 2772 CRC32 XXX GTID 0-1-9 /*!100001 SET @@session.gtid_seq_no=9*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at 2772 # at 2809 @@ -292,7 +292,7 @@ COMMIT # at 2992 #<date> server id 1 end_log_pos 3034 CRC32 XXX GTID 0-1-10 /*!100001 SET @@session.gtid_seq_no=10*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at 3034 # at 3071 @@ -357,7 +357,7 @@ DELIMITER /*!*/; /*!100001 SET @@session.gtid_domain_id=0*//*!*/; /*!100001 SET @@session.server_id=1*//*!*/; /*!100001 SET @@session.gtid_seq_no=16*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at 429 # at 543 diff --git a/mysql-test/main/mysqlbinlog_stmt_compressed.result b/mysql-test/main/mysqlbinlog_stmt_compressed.result index 42717bccd01..654dff9e90d 100644 --- a/mysql-test/main/mysqlbinlog_stmt_compressed.result +++ b/mysql-test/main/mysqlbinlog_stmt_compressed.result @@ -53,7 +53,7 @@ CREATE TABLE t2 (pk INT PRIMARY KEY, f1 INT, f2 INT, f3 INT, f4 INT, f5 MEDIUMIN # at 727 #<date> server id 1 end_log_pos 769 CRC32 XXX GTID 0-1-3 /*!100001 SET @@session.gtid_seq_no=3*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at 769 #<date> server id 1 end_log_pos 897 CRC32 XXX Query_compressed thread_id=5 exec_time=x error_code=0 @@ -68,7 +68,7 @@ COMMIT # at 970 #<date> server id 1 end_log_pos 1012 CRC32 XXX GTID 0-1-4 /*!100001 SET @@session.gtid_seq_no=4*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at 1012 #<date> server id 1 end_log_pos 1140 CRC32 XXX Query_compressed thread_id=5 exec_time=x error_code=0 @@ -83,7 +83,7 @@ COMMIT # at 1213 #<date> server id 1 end_log_pos 1255 CRC32 XXX GTID 0-1-5 /*!100001 SET @@session.gtid_seq_no=5*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at 1255 #<date> server id 1 end_log_pos 1385 CRC32 XXX Query_compressed thread_id=5 exec_time=x error_code=0 @@ -98,7 +98,7 @@ COMMIT # at 1458 #<date> server id 1 end_log_pos 1500 CRC32 XXX GTID 0-1-6 /*!100001 SET @@session.gtid_seq_no=6*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at 1500 #<date> server id 1 end_log_pos 1627 CRC32 XXX Query_compressed thread_id=5 exec_time=x error_code=0 @@ -113,7 +113,7 @@ COMMIT # at 1700 #<date> server id 1 end_log_pos 1742 CRC32 XXX GTID 0-1-7 /*!100001 SET @@session.gtid_seq_no=7*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at 1742 #<date> server id 1 end_log_pos 1850 CRC32 XXX Query_compressed thread_id=5 exec_time=x error_code=0 @@ -128,7 +128,7 @@ COMMIT # at 1923 #<date> server id 1 end_log_pos 1965 CRC32 XXX GTID 0-1-8 /*!100001 SET @@session.gtid_seq_no=8*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at 1965 #<date> server id 1 end_log_pos 2082 CRC32 XXX Query_compressed thread_id=5 exec_time=x error_code=0 @@ -143,7 +143,7 @@ COMMIT # at 2155 #<date> server id 1 end_log_pos 2197 CRC32 XXX GTID 0-1-9 /*!100001 SET @@session.gtid_seq_no=9*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at 2197 #<date> server id 1 end_log_pos 2288 CRC32 XXX Query_compressed thread_id=5 exec_time=x error_code=0 @@ -158,7 +158,7 @@ COMMIT # at 2361 #<date> server id 1 end_log_pos 2403 CRC32 XXX GTID 0-1-10 /*!100001 SET @@session.gtid_seq_no=10*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at 2403 #<date> server id 1 end_log_pos 2494 CRC32 XXX Query_compressed thread_id=5 exec_time=x error_code=0 diff --git a/mysql-test/main/user_var-binlog.result b/mysql-test/main/user_var-binlog.result index 0d8732399c7..0a9fe956aee 100644 --- a/mysql-test/main/user_var-binlog.result +++ b/mysql-test/main/user_var-binlog.result @@ -23,7 +23,7 @@ flush logs; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; ROLLBACK/*!*/; -BEGIN +START TRANSACTION /*!*/; SET @`a b`:=_latin1 X'68656C6C6F' COLLATE `latin1_swedish_ci`/*!*/; use `test`/*!*/; @@ -41,7 +41,7 @@ INSERT INTO t1 VALUES(@`a b`) SET TIMESTAMP=10000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET @`var1`:=_latin1 X'273B616161' COLLATE `latin1_swedish_ci`/*!*/; SET @`var2`:=_binary X'61' COLLATE `binary`/*!*/; diff --git a/mysql-test/suite/binlog/r/binlog_mysqlbinlog2.result b/mysql-test/suite/binlog/r/binlog_mysqlbinlog2.result index 8ecd4d37b1f..2a0e1d3c810 100644 --- a/mysql-test/suite/binlog/r/binlog_mysqlbinlog2.result +++ b/mysql-test/suite/binlog/r/binlog_mysqlbinlog2.result @@ -34,7 +34,7 @@ SET @@session.lc_time_names=0/*!*/; SET @@session.collation_database=DEFAULT/*!*/; create table t1 (a int auto_increment not null primary key, b char(3)) /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=1/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -43,7 +43,7 @@ insert into t1 values(null, "a") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=2/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -52,7 +52,7 @@ insert into t1 values(null, "b") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=3/*!*/; SET TIMESTAMP=1773144/*!*/; @@ -61,7 +61,7 @@ insert into t1 values(null, "c") SET TIMESTAMP=1773144/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=4/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -70,7 +70,7 @@ insert into t1 values(null, "d") SET TIMESTAMP=1773146/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=5/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -91,7 +91,7 @@ ROLLBACK /* added by mysqlbinlog */; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; ROLLBACK/*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=1/*!*/; use `test`/*!*/; @@ -109,7 +109,7 @@ insert into t1 values(null, "a") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=2/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -118,7 +118,7 @@ insert into t1 values(null, "b") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=3/*!*/; SET TIMESTAMP=1773144/*!*/; @@ -127,7 +127,7 @@ insert into t1 values(null, "c") SET TIMESTAMP=1773144/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=4/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -136,7 +136,7 @@ insert into t1 values(null, "d") SET TIMESTAMP=1773146/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=5/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -157,7 +157,7 @@ ROLLBACK /* added by mysqlbinlog */; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; ROLLBACK/*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=3/*!*/; use `test`/*!*/; @@ -175,7 +175,7 @@ insert into t1 values(null, "c") SET TIMESTAMP=1773144/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=4/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -184,7 +184,7 @@ insert into t1 values(null, "d") SET TIMESTAMP=1773146/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=5/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -217,7 +217,7 @@ SET @@session.lc_time_names=0/*!*/; SET @@session.collation_database=DEFAULT/*!*/; create table t1 (a int auto_increment not null primary key, b char(3)) /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=1/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -226,7 +226,7 @@ insert into t1 values(null, "a") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=2/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -247,7 +247,7 @@ ROLLBACK /* added by mysqlbinlog */; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; ROLLBACK/*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=3/*!*/; use `test`/*!*/; @@ -274,7 +274,7 @@ ROLLBACK /* added by mysqlbinlog */; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; ROLLBACK/*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=3/*!*/; use `test`/*!*/; @@ -292,7 +292,7 @@ insert into t1 values(null, "c") SET TIMESTAMP=1773144/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=4/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -301,7 +301,7 @@ insert into t1 values(null, "d") SET TIMESTAMP=1773146/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=5/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -334,7 +334,7 @@ SET @@session.lc_time_names=0/*!*/; SET @@session.collation_database=DEFAULT/*!*/; create table t1 (a int auto_increment not null primary key, b char(3)) /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=1/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -343,7 +343,7 @@ insert into t1 values(null, "a") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=2/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -377,7 +377,7 @@ SET @@session.lc_time_names=0/*!*/; SET @@session.collation_database=DEFAULT/*!*/; create table t1 (a int auto_increment not null primary key, b char(3)) /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=1/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -386,7 +386,7 @@ insert into t1 values(null, "a") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=2/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -395,7 +395,7 @@ insert into t1 values(null, "b") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=3/*!*/; SET TIMESTAMP=1773144/*!*/; @@ -404,7 +404,7 @@ insert into t1 values(null, "c") SET TIMESTAMP=1773144/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=4/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -413,7 +413,7 @@ insert into t1 values(null, "d") SET TIMESTAMP=1773146/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=5/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -424,7 +424,7 @@ COMMIT /*!*/; DELIMITER ; DELIMITER /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=6/*!*/; use `test`/*!*/; @@ -454,7 +454,7 @@ ROLLBACK /* added by mysqlbinlog */; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; ROLLBACK/*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=1/*!*/; use `test`/*!*/; @@ -472,7 +472,7 @@ insert into t1 values(null, "a") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=2/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -481,7 +481,7 @@ insert into t1 values(null, "b") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=3/*!*/; SET TIMESTAMP=1773144/*!*/; @@ -490,7 +490,7 @@ insert into t1 values(null, "c") SET TIMESTAMP=1773144/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=4/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -499,7 +499,7 @@ insert into t1 values(null, "d") SET TIMESTAMP=1773146/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=5/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -510,7 +510,7 @@ COMMIT /*!*/; DELIMITER ; DELIMITER /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=6/*!*/; use `test`/*!*/; @@ -540,7 +540,7 @@ ROLLBACK /* added by mysqlbinlog */; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; ROLLBACK/*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=3/*!*/; use `test`/*!*/; @@ -558,7 +558,7 @@ insert into t1 values(null, "c") SET TIMESTAMP=1773144/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=4/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -567,7 +567,7 @@ insert into t1 values(null, "d") SET TIMESTAMP=1773146/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=5/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -578,7 +578,7 @@ COMMIT /*!*/; DELIMITER ; DELIMITER /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=6/*!*/; use `test`/*!*/; @@ -620,7 +620,7 @@ SET @@session.lc_time_names=0/*!*/; SET @@session.collation_database=DEFAULT/*!*/; create table t1 (a int auto_increment not null primary key, b char(3)) /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=1/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -629,7 +629,7 @@ insert into t1 values(null, "a") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=2/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -638,7 +638,7 @@ insert into t1 values(null, "b") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=3/*!*/; SET TIMESTAMP=1773144/*!*/; @@ -647,7 +647,7 @@ insert into t1 values(null, "c") SET TIMESTAMP=1773144/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=4/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -656,7 +656,7 @@ insert into t1 values(null, "d") SET TIMESTAMP=1773146/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=5/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -667,7 +667,7 @@ COMMIT /*!*/; DELIMITER ; DELIMITER /*!*/; -BEGIN +START TRANSACTION /*!*/; DELIMITER ; # End of log file @@ -681,7 +681,7 @@ ROLLBACK /* added by mysqlbinlog */; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; ROLLBACK/*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=3/*!*/; use `test`/*!*/; @@ -699,7 +699,7 @@ insert into t1 values(null, "c") SET TIMESTAMP=1773144/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=4/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -708,7 +708,7 @@ insert into t1 values(null, "d") SET TIMESTAMP=1773146/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=5/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -719,7 +719,7 @@ COMMIT /*!*/; DELIMITER ; DELIMITER /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=6/*!*/; use `test`/*!*/; @@ -761,7 +761,7 @@ SET @@session.lc_time_names=0/*!*/; SET @@session.collation_database=DEFAULT/*!*/; create table t1 (a int auto_increment not null primary key, b char(3)) /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=1/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -770,7 +770,7 @@ insert into t1 values(null, "a") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=2/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -803,7 +803,7 @@ SET @@session.lc_time_names=0/*!*/; SET @@session.collation_database=DEFAULT/*!*/; create table t1 (a int auto_increment not null primary key, b char(3)) /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=1/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -812,7 +812,7 @@ insert into t1 values(null, "a") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=2/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -821,7 +821,7 @@ insert into t1 values(null, "b") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=3/*!*/; SET TIMESTAMP=1773144/*!*/; @@ -830,7 +830,7 @@ insert into t1 values(null, "c") SET TIMESTAMP=1773144/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=4/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -839,7 +839,7 @@ insert into t1 values(null, "d") SET TIMESTAMP=1773146/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=5/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -860,7 +860,7 @@ ROLLBACK /* added by mysqlbinlog */; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; ROLLBACK/*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=1/*!*/; use `test`/*!*/; @@ -878,7 +878,7 @@ insert into t1 values(null, "a") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=2/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -887,7 +887,7 @@ insert into t1 values(null, "b") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=3/*!*/; SET TIMESTAMP=1773144/*!*/; @@ -896,7 +896,7 @@ insert into t1 values(null, "c") SET TIMESTAMP=1773144/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=4/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -905,7 +905,7 @@ insert into t1 values(null, "d") SET TIMESTAMP=1773146/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=5/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -925,7 +925,7 @@ ROLLBACK /* added by mysqlbinlog */; /*!40019 SET @@session.max_insert_delayed_threads=0*/; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=3/*!*/; use `test`/*!*/; @@ -943,7 +943,7 @@ insert into t1 values(null, "c") SET TIMESTAMP=1773144/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=4/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -952,7 +952,7 @@ insert into t1 values(null, "d") SET TIMESTAMP=1773146/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=5/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -985,7 +985,7 @@ SET @@session.lc_time_names=0/*!*/; SET @@session.collation_database=DEFAULT/*!*/; create table t1 (a int auto_increment not null primary key, b char(3)) /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=1/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -994,7 +994,7 @@ insert into t1 values(null, "a") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=2/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -1014,7 +1014,7 @@ ROLLBACK /* added by mysqlbinlog */; /*!40019 SET @@session.max_insert_delayed_threads=0*/; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=3/*!*/; use `test`/*!*/; @@ -1041,7 +1041,7 @@ ROLLBACK /* added by mysqlbinlog */; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; ROLLBACK/*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=3/*!*/; use `test`/*!*/; @@ -1059,7 +1059,7 @@ insert into t1 values(null, "c") SET TIMESTAMP=1773144/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=4/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -1068,7 +1068,7 @@ insert into t1 values(null, "d") SET TIMESTAMP=1773146/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=5/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -1101,7 +1101,7 @@ SET @@session.lc_time_names=0/*!*/; SET @@session.collation_database=DEFAULT/*!*/; create table t1 (a int auto_increment not null primary key, b char(3)) /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=1/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -1110,7 +1110,7 @@ insert into t1 values(null, "a") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=2/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -1143,7 +1143,7 @@ SET @@session.lc_time_names=0/*!*/; SET @@session.collation_database=DEFAULT/*!*/; create table t1 (a int auto_increment not null primary key, b char(3)) /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=1/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -1152,7 +1152,7 @@ insert into t1 values(null, "a") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=2/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -1161,7 +1161,7 @@ insert into t1 values(null, "b") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=3/*!*/; SET TIMESTAMP=1773144/*!*/; @@ -1170,7 +1170,7 @@ insert into t1 values(null, "c") SET TIMESTAMP=1773144/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=4/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -1179,7 +1179,7 @@ insert into t1 values(null, "d") SET TIMESTAMP=1773146/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=5/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -1190,7 +1190,7 @@ COMMIT /*!*/; DELIMITER ; DELIMITER /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=6/*!*/; use `test`/*!*/; @@ -1220,7 +1220,7 @@ ROLLBACK /* added by mysqlbinlog */; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; ROLLBACK/*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=1/*!*/; use `test`/*!*/; @@ -1238,7 +1238,7 @@ insert into t1 values(null, "a") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=2/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -1247,7 +1247,7 @@ insert into t1 values(null, "b") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=3/*!*/; SET TIMESTAMP=1773144/*!*/; @@ -1256,7 +1256,7 @@ insert into t1 values(null, "c") SET TIMESTAMP=1773144/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=4/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -1265,7 +1265,7 @@ insert into t1 values(null, "d") SET TIMESTAMP=1773146/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=5/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -1276,7 +1276,7 @@ COMMIT /*!*/; DELIMITER ; DELIMITER /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=6/*!*/; use `test`/*!*/; @@ -1305,7 +1305,7 @@ ROLLBACK /* added by mysqlbinlog */; /*!40019 SET @@session.max_insert_delayed_threads=0*/; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=3/*!*/; use `test`/*!*/; @@ -1323,7 +1323,7 @@ insert into t1 values(null, "c") SET TIMESTAMP=1773144/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=4/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -1332,7 +1332,7 @@ insert into t1 values(null, "d") SET TIMESTAMP=1773146/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=5/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -1343,7 +1343,7 @@ COMMIT /*!*/; DELIMITER ; DELIMITER /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=6/*!*/; use `test`/*!*/; @@ -1385,7 +1385,7 @@ SET @@session.lc_time_names=0/*!*/; SET @@session.collation_database=DEFAULT/*!*/; create table t1 (a int auto_increment not null primary key, b char(3)) /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=1/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -1394,7 +1394,7 @@ insert into t1 values(null, "a") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=2/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -1403,7 +1403,7 @@ insert into t1 values(null, "b") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=3/*!*/; SET TIMESTAMP=1773144/*!*/; @@ -1412,7 +1412,7 @@ insert into t1 values(null, "c") SET TIMESTAMP=1773144/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=4/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -1421,7 +1421,7 @@ insert into t1 values(null, "d") SET TIMESTAMP=1773146/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=5/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -1444,7 +1444,7 @@ ROLLBACK /* added by mysqlbinlog */; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; ROLLBACK/*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=3/*!*/; use `test`/*!*/; @@ -1462,7 +1462,7 @@ insert into t1 values(null, "c") SET TIMESTAMP=1773144/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=4/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -1471,7 +1471,7 @@ insert into t1 values(null, "d") SET TIMESTAMP=1773146/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=5/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -1482,7 +1482,7 @@ COMMIT /*!*/; DELIMITER ; DELIMITER /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=6/*!*/; use `test`/*!*/; @@ -1524,7 +1524,7 @@ SET @@session.lc_time_names=0/*!*/; SET @@session.collation_database=DEFAULT/*!*/; create table t1 (a int auto_increment not null primary key, b char(3)) /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=1/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -1533,7 +1533,7 @@ insert into t1 values(null, "a") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=2/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -1566,7 +1566,7 @@ SET @@session.lc_time_names=0/*!*/; SET @@session.collation_database=DEFAULT/*!*/; create table t1 (a int auto_increment not null primary key, b char(3)) /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=1/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -1575,7 +1575,7 @@ insert into t1 values(null, "a") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=2/*!*/; SET TIMESTAMP=1773142/*!*/; @@ -1584,7 +1584,7 @@ insert into t1 values(null, "b") SET TIMESTAMP=1773142/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=3/*!*/; SET TIMESTAMP=1773144/*!*/; @@ -1593,7 +1593,7 @@ insert into t1 values(null, "c") SET TIMESTAMP=1773144/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=4/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -1602,7 +1602,7 @@ insert into t1 values(null, "d") SET TIMESTAMP=1773146/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=5/*!*/; SET TIMESTAMP=1773146/*!*/; @@ -1611,7 +1611,7 @@ insert into t1 values(null, "e") SET TIMESTAMP=1773146/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET INSERT_ID=6/*!*/; SET TIMESTAMP=1773143/*!*/; diff --git a/mysql-test/suite/binlog/r/binlog_mysqlbinlog_row.result b/mysql-test/suite/binlog/r/binlog_mysqlbinlog_row.result index 0cc1805eadc..469eb97badc 100644 --- a/mysql-test/suite/binlog/r/binlog_mysqlbinlog_row.result +++ b/mysql-test/suite/binlog/r/binlog_mysqlbinlog_row.result @@ -383,7 +383,7 @@ CREATE TABLE t1 (c01 BIT) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-2 /*!100001 SET @@session.gtid_seq_no=2*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -404,7 +404,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-3 /*!100001 SET @@session.gtid_seq_no=3*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -441,7 +441,7 @@ CREATE TABLE t1 (c01 BIT(7)) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-6 /*!100001 SET @@session.gtid_seq_no=6*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -462,7 +462,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-7 /*!100001 SET @@session.gtid_seq_no=7*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -483,7 +483,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-8 /*!100001 SET @@session.gtid_seq_no=8*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -504,7 +504,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-9 /*!100001 SET @@session.gtid_seq_no=9*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -525,7 +525,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-10 /*!100001 SET @@session.gtid_seq_no=10*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -546,7 +546,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-11 /*!100001 SET @@session.gtid_seq_no=11*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -567,7 +567,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-12 /*!100001 SET @@session.gtid_seq_no=12*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -588,7 +588,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-13 /*!100001 SET @@session.gtid_seq_no=13*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -609,7 +609,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-14 /*!100001 SET @@session.gtid_seq_no=14*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -630,7 +630,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-15 /*!100001 SET @@session.gtid_seq_no=15*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -669,7 +669,7 @@ CREATE TABLE t1 (a BIT(20), b CHAR(2)) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-18 /*!100001 SET @@session.gtid_seq_no=18*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -707,7 +707,7 @@ CREATE TABLE t1 (c02 BIT(64)) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-21 /*!100001 SET @@session.gtid_seq_no=21*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -728,7 +728,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-22 /*!100001 SET @@session.gtid_seq_no=22*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -749,7 +749,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-23 /*!100001 SET @@session.gtid_seq_no=23*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -770,7 +770,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-24 /*!100001 SET @@session.gtid_seq_no=24*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -807,7 +807,7 @@ CREATE TABLE t1 (c03 TINYINT) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-27 /*!100001 SET @@session.gtid_seq_no=27*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -834,7 +834,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-28 /*!100001 SET @@session.gtid_seq_no=28*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -855,7 +855,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-29 /*!100001 SET @@session.gtid_seq_no=29*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -878,7 +878,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-30 /*!100001 SET @@session.gtid_seq_no=30*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -915,7 +915,7 @@ CREATE TABLE t1 (c04 TINYINT UNSIGNED) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-33 /*!100001 SET @@session.gtid_seq_no=33*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -939,7 +939,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-34 /*!100001 SET @@session.gtid_seq_no=34*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -976,7 +976,7 @@ CREATE TABLE t1 (c06 BOOL) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-37 /*!100001 SET @@session.gtid_seq_no=37*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -997,7 +997,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-38 /*!100001 SET @@session.gtid_seq_no=38*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1034,7 +1034,7 @@ CREATE TABLE t1 (c07 SMALLINT) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-41 /*!100001 SET @@session.gtid_seq_no=41*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1055,7 +1055,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-42 /*!100001 SET @@session.gtid_seq_no=42*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1092,7 +1092,7 @@ CREATE TABLE t1 (c08 SMALLINT UNSIGNED) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-45 /*!100001 SET @@session.gtid_seq_no=45*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1116,7 +1116,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-46 /*!100001 SET @@session.gtid_seq_no=46*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1139,7 +1139,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-47 /*!100001 SET @@session.gtid_seq_no=47*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1176,7 +1176,7 @@ CREATE TABLE t1 (c10 MEDIUMINT) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-50 /*!100001 SET @@session.gtid_seq_no=50*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1197,7 +1197,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-51 /*!100001 SET @@session.gtid_seq_no=51*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1234,7 +1234,7 @@ CREATE TABLE t1 (c11 MEDIUMINT UNSIGNED) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-54 /*!100001 SET @@session.gtid_seq_no=54*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1258,7 +1258,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-55 /*!100001 SET @@session.gtid_seq_no=55*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1281,7 +1281,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-56 /*!100001 SET @@session.gtid_seq_no=56*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1318,7 +1318,7 @@ CREATE TABLE t1 (c13 INT) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-59 /*!100001 SET @@session.gtid_seq_no=59*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1339,7 +1339,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-60 /*!100001 SET @@session.gtid_seq_no=60*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1376,7 +1376,7 @@ CREATE TABLE t1 (c14 INT UNSIGNED) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-63 /*!100001 SET @@session.gtid_seq_no=63*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1400,7 +1400,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-64 /*!100001 SET @@session.gtid_seq_no=64*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1423,7 +1423,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-65 /*!100001 SET @@session.gtid_seq_no=65*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1460,7 +1460,7 @@ CREATE TABLE t1 (c16 BIGINT) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-68 /*!100001 SET @@session.gtid_seq_no=68*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1481,7 +1481,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-69 /*!100001 SET @@session.gtid_seq_no=69*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1518,7 +1518,7 @@ CREATE TABLE t1 (c17 BIGINT UNSIGNED) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-72 /*!100001 SET @@session.gtid_seq_no=72*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1542,7 +1542,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-73 /*!100001 SET @@session.gtid_seq_no=73*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1565,7 +1565,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-74 /*!100001 SET @@session.gtid_seq_no=74*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1602,7 +1602,7 @@ CREATE TABLE t1 (c19 FLOAT) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-77 /*!100001 SET @@session.gtid_seq_no=77*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1623,7 +1623,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-78 /*!100001 SET @@session.gtid_seq_no=78*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1660,7 +1660,7 @@ CREATE TABLE t1 (c22 DOUBLE) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-81 /*!100001 SET @@session.gtid_seq_no=81*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1681,7 +1681,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-82 /*!100001 SET @@session.gtid_seq_no=82*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1718,7 +1718,7 @@ CREATE TABLE t1 (c25 DECIMAL(10,5)) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-85 /*!100001 SET @@session.gtid_seq_no=85*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1739,7 +1739,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-86 /*!100001 SET @@session.gtid_seq_no=86*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1760,7 +1760,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-87 /*!100001 SET @@session.gtid_seq_no=87*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1797,7 +1797,7 @@ CREATE TABLE t1 (c28 DATE) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-90 /*!100001 SET @@session.gtid_seq_no=90*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1818,7 +1818,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-91 /*!100001 SET @@session.gtid_seq_no=91*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1855,7 +1855,7 @@ CREATE TABLE t1 (c29 DATETIME) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-94 /*!100001 SET @@session.gtid_seq_no=94*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1876,7 +1876,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-95 /*!100001 SET @@session.gtid_seq_no=95*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1913,7 +1913,7 @@ CREATE TABLE t1 (c30 TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURR # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-98 /*!100001 SET @@session.gtid_seq_no=98*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1935,7 +1935,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-99 /*!100001 SET @@session.gtid_seq_no=99*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1972,7 +1972,7 @@ CREATE TABLE t1 (c31 TIME) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-102 /*!100001 SET @@session.gtid_seq_no=102*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1993,7 +1993,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-103 /*!100001 SET @@session.gtid_seq_no=103*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2030,7 +2030,7 @@ CREATE TABLE t1 (c32 YEAR) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-106 /*!100001 SET @@session.gtid_seq_no=106*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2051,7 +2051,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-107 /*!100001 SET @@session.gtid_seq_no=107*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2088,7 +2088,7 @@ CREATE TABLE t1 (c33 CHAR) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-110 /*!100001 SET @@session.gtid_seq_no=110*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2109,7 +2109,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-111 /*!100001 SET @@session.gtid_seq_no=111*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2146,7 +2146,7 @@ CREATE TABLE t1 (c34 CHAR(0)) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-114 /*!100001 SET @@session.gtid_seq_no=114*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2167,7 +2167,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-115 /*!100001 SET @@session.gtid_seq_no=115*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2204,7 +2204,7 @@ CREATE TABLE t1 (c35 CHAR(1)) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-118 /*!100001 SET @@session.gtid_seq_no=118*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2225,7 +2225,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-119 /*!100001 SET @@session.gtid_seq_no=119*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2262,7 +2262,7 @@ CREATE TABLE t1 (c36 CHAR(255)) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-122 /*!100001 SET @@session.gtid_seq_no=122*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2283,7 +2283,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-123 /*!100001 SET @@session.gtid_seq_no=123*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2320,7 +2320,7 @@ CREATE TABLE t1 (c37 NATIONAL CHAR) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-126 /*!100001 SET @@session.gtid_seq_no=126*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2341,7 +2341,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-127 /*!100001 SET @@session.gtid_seq_no=127*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2378,7 +2378,7 @@ CREATE TABLE t1 (c38 NATIONAL CHAR(0)) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-130 /*!100001 SET @@session.gtid_seq_no=130*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2399,7 +2399,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-131 /*!100001 SET @@session.gtid_seq_no=131*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2436,7 +2436,7 @@ CREATE TABLE t1 (c39 NATIONAL CHAR(1)) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-134 /*!100001 SET @@session.gtid_seq_no=134*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2457,7 +2457,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-135 /*!100001 SET @@session.gtid_seq_no=135*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2494,7 +2494,7 @@ CREATE TABLE t1 (c40 NATIONAL CHAR(255)) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-138 /*!100001 SET @@session.gtid_seq_no=138*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2515,7 +2515,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-139 /*!100001 SET @@session.gtid_seq_no=139*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2536,7 +2536,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-140 /*!100001 SET @@session.gtid_seq_no=140*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2576,7 +2576,7 @@ CREATE TABLE t1 (c41 CHAR CHARACTER SET UCS2) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-143 /*!100001 SET @@session.gtid_seq_no=143*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2597,7 +2597,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-144 /*!100001 SET @@session.gtid_seq_no=144*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2634,7 +2634,7 @@ CREATE TABLE t1 (c42 CHAR(0) CHARACTER SET UCS2) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-147 /*!100001 SET @@session.gtid_seq_no=147*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2655,7 +2655,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-148 /*!100001 SET @@session.gtid_seq_no=148*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2692,7 +2692,7 @@ CREATE TABLE t1 (c43 CHAR(1) CHARACTER SET UCS2) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-151 /*!100001 SET @@session.gtid_seq_no=151*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2713,7 +2713,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-152 /*!100001 SET @@session.gtid_seq_no=152*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2750,7 +2750,7 @@ CREATE TABLE t1 (c44 CHAR(255) CHARACTER SET UCS2) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-155 /*!100001 SET @@session.gtid_seq_no=155*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2771,7 +2771,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-156 /*!100001 SET @@session.gtid_seq_no=156*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2792,7 +2792,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-157 /*!100001 SET @@session.gtid_seq_no=157*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2832,7 +2832,7 @@ CREATE TABLE t1 (c45 VARCHAR(0)) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-160 /*!100001 SET @@session.gtid_seq_no=160*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2853,7 +2853,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-161 /*!100001 SET @@session.gtid_seq_no=161*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2890,7 +2890,7 @@ CREATE TABLE t1 (c46 VARCHAR(1)) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-164 /*!100001 SET @@session.gtid_seq_no=164*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2911,7 +2911,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-165 /*!100001 SET @@session.gtid_seq_no=165*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2948,7 +2948,7 @@ CREATE TABLE t1 (c47 VARCHAR(255)) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-168 /*!100001 SET @@session.gtid_seq_no=168*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2969,7 +2969,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-169 /*!100001 SET @@session.gtid_seq_no=169*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3006,7 +3006,7 @@ CREATE TABLE t1 (c48 VARCHAR(261)) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-172 /*!100001 SET @@session.gtid_seq_no=172*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3027,7 +3027,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-173 /*!100001 SET @@session.gtid_seq_no=173*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3064,7 +3064,7 @@ CREATE TABLE t1 (c49 NATIONAL VARCHAR(0)) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-176 /*!100001 SET @@session.gtid_seq_no=176*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3085,7 +3085,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-177 /*!100001 SET @@session.gtid_seq_no=177*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3122,7 +3122,7 @@ CREATE TABLE t1 (c50 NATIONAL VARCHAR(1)) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-180 /*!100001 SET @@session.gtid_seq_no=180*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3143,7 +3143,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-181 /*!100001 SET @@session.gtid_seq_no=181*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3180,7 +3180,7 @@ CREATE TABLE t1 (c51 NATIONAL VARCHAR(255)) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-184 /*!100001 SET @@session.gtid_seq_no=184*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3201,7 +3201,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-185 /*!100001 SET @@session.gtid_seq_no=185*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3222,7 +3222,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-186 /*!100001 SET @@session.gtid_seq_no=186*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3262,7 +3262,7 @@ CREATE TABLE t1 (c52 NATIONAL VARCHAR(261)) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-189 /*!100001 SET @@session.gtid_seq_no=189*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3283,7 +3283,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-190 /*!100001 SET @@session.gtid_seq_no=190*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3304,7 +3304,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-191 /*!100001 SET @@session.gtid_seq_no=191*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3344,7 +3344,7 @@ CREATE TABLE t1 (c53 VARCHAR(0) CHARACTER SET ucs2) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-194 /*!100001 SET @@session.gtid_seq_no=194*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3365,7 +3365,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-195 /*!100001 SET @@session.gtid_seq_no=195*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3402,7 +3402,7 @@ CREATE TABLE t1 (c54 VARCHAR(1) CHARACTER SET ucs2) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-198 /*!100001 SET @@session.gtid_seq_no=198*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3423,7 +3423,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-199 /*!100001 SET @@session.gtid_seq_no=199*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3460,7 +3460,7 @@ CREATE TABLE t1 (c55 VARCHAR(255) CHARACTER SET ucs2) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-202 /*!100001 SET @@session.gtid_seq_no=202*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3481,7 +3481,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-203 /*!100001 SET @@session.gtid_seq_no=203*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3518,7 +3518,7 @@ CREATE TABLE t1 (c56 VARCHAR(261) CHARACTER SET ucs2) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-206 /*!100001 SET @@session.gtid_seq_no=206*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3539,7 +3539,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-207 /*!100001 SET @@session.gtid_seq_no=207*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3576,7 +3576,7 @@ CREATE TABLE t1 (c57 BINARY) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-210 /*!100001 SET @@session.gtid_seq_no=210*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3597,7 +3597,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-211 /*!100001 SET @@session.gtid_seq_no=211*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3618,7 +3618,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-212 /*!100001 SET @@session.gtid_seq_no=212*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3639,7 +3639,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-213 /*!100001 SET @@session.gtid_seq_no=213*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3676,7 +3676,7 @@ CREATE TABLE t1 (c58 BINARY(0)) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-216 /*!100001 SET @@session.gtid_seq_no=216*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3697,7 +3697,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-217 /*!100001 SET @@session.gtid_seq_no=217*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3734,7 +3734,7 @@ CREATE TABLE t1 (c59 BINARY(1)) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-220 /*!100001 SET @@session.gtid_seq_no=220*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3755,7 +3755,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-221 /*!100001 SET @@session.gtid_seq_no=221*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3776,7 +3776,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-222 /*!100001 SET @@session.gtid_seq_no=222*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3797,7 +3797,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-223 /*!100001 SET @@session.gtid_seq_no=223*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3834,7 +3834,7 @@ CREATE TABLE t1 (c60 BINARY(255)) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-226 /*!100001 SET @@session.gtid_seq_no=226*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3855,7 +3855,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-227 /*!100001 SET @@session.gtid_seq_no=227*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3876,7 +3876,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-228 /*!100001 SET @@session.gtid_seq_no=228*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3897,7 +3897,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-229 /*!100001 SET @@session.gtid_seq_no=229*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3934,7 +3934,7 @@ CREATE TABLE t1 (c61 VARBINARY(0)) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-232 /*!100001 SET @@session.gtid_seq_no=232*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3955,7 +3955,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-233 /*!100001 SET @@session.gtid_seq_no=233*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3992,7 +3992,7 @@ CREATE TABLE t1 (c62 VARBINARY(1)) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-236 /*!100001 SET @@session.gtid_seq_no=236*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4013,7 +4013,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-237 /*!100001 SET @@session.gtid_seq_no=237*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4034,7 +4034,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-238 /*!100001 SET @@session.gtid_seq_no=238*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4055,7 +4055,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-239 /*!100001 SET @@session.gtid_seq_no=239*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4092,7 +4092,7 @@ CREATE TABLE t1 (c63 VARBINARY(255)) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-242 /*!100001 SET @@session.gtid_seq_no=242*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4113,7 +4113,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-243 /*!100001 SET @@session.gtid_seq_no=243*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4134,7 +4134,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-244 /*!100001 SET @@session.gtid_seq_no=244*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4155,7 +4155,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-245 /*!100001 SET @@session.gtid_seq_no=245*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4192,7 +4192,7 @@ CREATE TABLE t1 (c65 TINYBLOB) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-248 /*!100001 SET @@session.gtid_seq_no=248*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4213,7 +4213,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-249 /*!100001 SET @@session.gtid_seq_no=249*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4250,7 +4250,7 @@ CREATE TABLE t1 (c68 BLOB) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-252 /*!100001 SET @@session.gtid_seq_no=252*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4271,7 +4271,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-253 /*!100001 SET @@session.gtid_seq_no=253*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4308,7 +4308,7 @@ CREATE TABLE t1 (c71 MEDIUMBLOB) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-256 /*!100001 SET @@session.gtid_seq_no=256*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4329,7 +4329,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-257 /*!100001 SET @@session.gtid_seq_no=257*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4366,7 +4366,7 @@ CREATE TABLE t1 (c74 LONGBLOB) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-260 /*!100001 SET @@session.gtid_seq_no=260*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4387,7 +4387,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-261 /*!100001 SET @@session.gtid_seq_no=261*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4424,7 +4424,7 @@ CREATE TABLE t1 (c66 TINYTEXT) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-264 /*!100001 SET @@session.gtid_seq_no=264*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4445,7 +4445,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-265 /*!100001 SET @@session.gtid_seq_no=265*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4482,7 +4482,7 @@ CREATE TABLE t1 (c69 TEXT) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-268 /*!100001 SET @@session.gtid_seq_no=268*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4503,7 +4503,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-269 /*!100001 SET @@session.gtid_seq_no=269*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4540,7 +4540,7 @@ CREATE TABLE t1 (c72 MEDIUMTEXT) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-272 /*!100001 SET @@session.gtid_seq_no=272*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4561,7 +4561,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-273 /*!100001 SET @@session.gtid_seq_no=273*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4598,7 +4598,7 @@ CREATE TABLE t1 (c75 LONGTEXT) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-276 /*!100001 SET @@session.gtid_seq_no=276*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4619,7 +4619,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-277 /*!100001 SET @@session.gtid_seq_no=277*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4656,7 +4656,7 @@ CREATE TABLE t1 (c67 TINYTEXT CHARACTER SET UCS2) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-280 /*!100001 SET @@session.gtid_seq_no=280*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4677,7 +4677,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-281 /*!100001 SET @@session.gtid_seq_no=281*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4714,7 +4714,7 @@ CREATE TABLE t1 (c70 TEXT CHARACTER SET UCS2) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-284 /*!100001 SET @@session.gtid_seq_no=284*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4735,7 +4735,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-285 /*!100001 SET @@session.gtid_seq_no=285*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4772,7 +4772,7 @@ CREATE TABLE t1 (c73 MEDIUMTEXT CHARACTER SET UCS2) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-288 /*!100001 SET @@session.gtid_seq_no=288*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4793,7 +4793,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-289 /*!100001 SET @@session.gtid_seq_no=289*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4830,7 +4830,7 @@ CREATE TABLE t1 (c76 LONGTEXT CHARACTER SET UCS2) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-292 /*!100001 SET @@session.gtid_seq_no=292*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4851,7 +4851,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-293 /*!100001 SET @@session.gtid_seq_no=293*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4888,7 +4888,7 @@ CREATE TABLE t1 (c77 ENUM('a','b','c')) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-296 /*!100001 SET @@session.gtid_seq_no=296*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4909,7 +4909,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-297 /*!100001 SET @@session.gtid_seq_no=297*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4946,7 +4946,7 @@ CREATE TABLE t1 (c78 SET('a','b','c','d','e','f')) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-300 /*!100001 SET @@session.gtid_seq_no=300*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4967,7 +4967,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-301 /*!100001 SET @@session.gtid_seq_no=301*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4988,7 +4988,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-302 /*!100001 SET @@session.gtid_seq_no=302*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5009,7 +5009,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-303 /*!100001 SET @@session.gtid_seq_no=303*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5030,7 +5030,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-304 /*!100001 SET @@session.gtid_seq_no=304*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5051,7 +5051,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-305 /*!100001 SET @@session.gtid_seq_no=305*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5072,7 +5072,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-306 /*!100001 SET @@session.gtid_seq_no=306*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5093,7 +5093,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-307 /*!100001 SET @@session.gtid_seq_no=307*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5138,7 +5138,7 @@ CREATE TABLE t2 (a int NOT NULL DEFAULT 0, b int NOT NULL DEFAULT 0) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-311 /*!100001 SET @@session.gtid_seq_no=311*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5160,7 +5160,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-312 /*!100001 SET @@session.gtid_seq_no=312*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5182,7 +5182,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-313 /*!100001 SET @@session.gtid_seq_no=313*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5204,7 +5204,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-314 /*!100001 SET @@session.gtid_seq_no=314*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5226,7 +5226,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-315 /*!100001 SET @@session.gtid_seq_no=315*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5294,7 +5294,7 @@ c_text_utf8 blob ) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-318 /*!100001 SET @@session.gtid_seq_no=318*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5347,7 +5347,7 @@ DELIMITER /*!*/; /*!100001 SET @@session.gtid_domain_id=0*//*!*/; /*!100001 SET @@session.server_id=1*//*!*/; /*!100001 SET @@session.gtid_seq_no=320*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # diff --git a/mysql-test/suite/binlog/r/binlog_mysqlbinlog_row_innodb.result b/mysql-test/suite/binlog/r/binlog_mysqlbinlog_row_innodb.result index 22309e964be..d8a5864ba3d 100644 --- a/mysql-test/suite/binlog/r/binlog_mysqlbinlog_row_innodb.result +++ b/mysql-test/suite/binlog/r/binlog_mysqlbinlog_row_innodb.result @@ -2368,7 +2368,7 @@ crn INT -- row number # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-2 trans /*!100001 SET @@session.gtid_seq_no=2*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2550,7 +2550,7 @@ COMMIT/*!*/; # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-3 trans /*!100001 SET @@session.gtid_seq_no=3*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2732,7 +2732,7 @@ COMMIT/*!*/; # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-4 trans /*!100001 SET @@session.gtid_seq_no=4*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3080,7 +3080,7 @@ COMMIT/*!*/; # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-5 trans /*!100001 SET @@session.gtid_seq_no=5*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3420,7 +3420,7 @@ COMMIT/*!*/; # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-6 trans /*!100001 SET @@session.gtid_seq_no=6*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3765,7 +3765,7 @@ COMMIT/*!*/; # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-7 trans /*!100001 SET @@session.gtid_seq_no=7*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4108,7 +4108,7 @@ COMMIT/*!*/; # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-8 trans /*!100001 SET @@session.gtid_seq_no=8*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4454,7 +4454,7 @@ COMMIT/*!*/; # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-9 trans /*!100001 SET @@session.gtid_seq_no=9*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4634,7 +4634,7 @@ COMMIT/*!*/; # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-10 trans /*!100001 SET @@session.gtid_seq_no=10*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4811,7 +4811,7 @@ COMMIT/*!*/; # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-11 trans /*!100001 SET @@session.gtid_seq_no=11*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4991,7 +4991,7 @@ COMMIT/*!*/; # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-12 trans /*!100001 SET @@session.gtid_seq_no=12*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5293,7 +5293,7 @@ crn INT -- row number # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-2 trans /*!100001 SET @@session.gtid_seq_no=2*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5363,7 +5363,7 @@ COMMIT/*!*/; # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-3 trans /*!100001 SET @@session.gtid_seq_no=3*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5442,7 +5442,7 @@ COMMIT/*!*/; # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-4 trans /*!100001 SET @@session.gtid_seq_no=4*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5717,7 +5717,7 @@ c_3_n INT -- row number # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-4 trans /*!100001 SET @@session.gtid_seq_no=4*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5787,7 +5787,7 @@ COMMIT/*!*/; # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-5 trans /*!100001 SET @@session.gtid_seq_no=5*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5857,7 +5857,7 @@ COMMIT/*!*/; # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-6 trans /*!100001 SET @@session.gtid_seq_no=6*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5927,7 +5927,7 @@ COMMIT/*!*/; # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-7 trans /*!100001 SET @@session.gtid_seq_no=7*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -6117,7 +6117,7 @@ COMMIT/*!*/; # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-8 trans /*!100001 SET @@session.gtid_seq_no=8*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -6331,7 +6331,7 @@ c3 VARCHAR(60) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-2 trans /*!100001 SET @@session.gtid_seq_no=2*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # diff --git a/mysql-test/suite/binlog/r/binlog_mysqlbinlog_row_myisam.result b/mysql-test/suite/binlog/r/binlog_mysqlbinlog_row_myisam.result index 8091242dab7..a75593055d7 100644 --- a/mysql-test/suite/binlog/r/binlog_mysqlbinlog_row_myisam.result +++ b/mysql-test/suite/binlog/r/binlog_mysqlbinlog_row_myisam.result @@ -2368,7 +2368,7 @@ crn INT -- row number # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-2 /*!100001 SET @@session.gtid_seq_no=2*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2553,7 +2553,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-3 /*!100001 SET @@session.gtid_seq_no=3*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -2737,7 +2737,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-4 /*!100001 SET @@session.gtid_seq_no=4*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3087,7 +3087,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-5 /*!100001 SET @@session.gtid_seq_no=5*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3429,7 +3429,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-6 /*!100001 SET @@session.gtid_seq_no=6*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -3776,7 +3776,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-7 /*!100001 SET @@session.gtid_seq_no=7*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4121,7 +4121,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-8 /*!100001 SET @@session.gtid_seq_no=8*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4469,7 +4469,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-9 /*!100001 SET @@session.gtid_seq_no=9*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4651,7 +4651,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-10 /*!100001 SET @@session.gtid_seq_no=10*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -4830,7 +4830,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-11 /*!100001 SET @@session.gtid_seq_no=11*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5012,7 +5012,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-12 /*!100001 SET @@session.gtid_seq_no=12*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5316,7 +5316,7 @@ crn INT -- row number # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-2 /*!100001 SET @@session.gtid_seq_no=2*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5388,7 +5388,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-3 /*!100001 SET @@session.gtid_seq_no=3*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5469,7 +5469,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-4 /*!100001 SET @@session.gtid_seq_no=4*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5746,7 +5746,7 @@ c_3_n INT -- row number # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-4 /*!100001 SET @@session.gtid_seq_no=4*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5818,7 +5818,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-5 /*!100001 SET @@session.gtid_seq_no=5*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5890,7 +5890,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-6 /*!100001 SET @@session.gtid_seq_no=6*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -5962,7 +5962,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-7 /*!100001 SET @@session.gtid_seq_no=7*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -6154,7 +6154,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-8 /*!100001 SET @@session.gtid_seq_no=8*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -6370,7 +6370,7 @@ c3 VARCHAR(60) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-2 /*!100001 SET @@session.gtid_seq_no=2*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # diff --git a/mysql-test/suite/binlog/r/binlog_mysqlbinlog_row_trans.result b/mysql-test/suite/binlog/r/binlog_mysqlbinlog_row_trans.result index 28070fab374..545cb8f7857 100644 --- a/mysql-test/suite/binlog/r/binlog_mysqlbinlog_row_trans.result +++ b/mysql-test/suite/binlog/r/binlog_mysqlbinlog_row_trans.result @@ -172,7 +172,7 @@ c2 VARCHAR(20) # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-3 trans /*!100001 SET @@session.gtid_seq_no=3*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -257,7 +257,7 @@ TRUNCATE TABLE t1 # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-6 /*!100001 SET @@session.gtid_seq_no=6*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -287,7 +287,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-7 /*!100001 SET @@session.gtid_seq_no=7*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -326,7 +326,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-8 /*!100001 SET @@session.gtid_seq_no=8*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -348,7 +348,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-9 trans /*!100001 SET @@session.gtid_seq_no=9*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -433,7 +433,7 @@ TRUNCATE TABLE t2 # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-12 /*!100001 SET @@session.gtid_seq_no=12*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -463,7 +463,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-13 /*!100001 SET @@session.gtid_seq_no=13*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -502,7 +502,7 @@ COMMIT # at # #010909 4:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-14 /*!100001 SET @@session.gtid_seq_no=14*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # diff --git a/mysql-test/suite/binlog/r/binlog_parallel_replication_marks_row.result b/mysql-test/suite/binlog/r/binlog_parallel_replication_marks_row.result index c4a1ba9b83b..bca96dd5fd0 100644 --- a/mysql-test/suite/binlog/r/binlog_parallel_replication_marks_row.result +++ b/mysql-test/suite/binlog/r/binlog_parallel_replication_marks_row.result @@ -38,13 +38,13 @@ disconnect tmp_con; connection default; FLUSH LOGS; # server id 1 end_log_pos # CRC32 0x######## GTID #-#-# trans -BEGIN +START TRANSACTION #Q> /* GTID */ INSERT INTO t1 VALUES (1,0) # server id 1 end_log_pos # CRC32 0x######## Table_map: `test`.`t1` mapped to number # # server id 1 end_log_pos # CRC32 0x######## Write_rows: table id # flags: STMT_END_F COMMIT/*!*/; # server id 1 end_log_pos # CRC32 0x######## GTID #-#-# trans -BEGIN +START TRANSACTION #Q> /* GTID */ INSERT INTO t1 VALUES (2,0) # server id 1 end_log_pos # CRC32 0x######## Table_map: `test`.`t1` mapped to number # # server id 1 end_log_pos # CRC32 0x######## Write_rows: table id # flags: STMT_END_F @@ -52,13 +52,13 @@ COMMIT/*!*/; # server id 1 end_log_pos # CRC32 0x######## GTID #-#-# ddl /* GTID */ ALTER TABLE t1 ADD c INT # server id 1 end_log_pos # CRC32 0x######## GTID #-#-# trans -BEGIN +START TRANSACTION #Q> /* GTID */ INSERT INTO t1 VALUES (3,0,0) # server id 1 end_log_pos # CRC32 0x######## Table_map: `test`.`t1` mapped to number # # server id 1 end_log_pos # CRC32 0x######## Write_rows: table id # flags: STMT_END_F COMMIT/*!*/; # server id 1 end_log_pos # CRC32 0x######## GTID #-#-# trans -BEGIN +START TRANSACTION #Q> /* GTID */ UPDATE t1 SET b=1, c=1 WHERE a=2 # server id 1 end_log_pos # CRC32 0x######## Table_map: `test`.`t1` mapped to number # # server id 1 end_log_pos # CRC32 0x######## Update_rows: table id # flags: STMT_END_F @@ -70,7 +70,7 @@ BEGIN # server id 1 end_log_pos # CRC32 0x######## Write_rows: table id # flags: STMT_END_F COMMIT/*!*/; # server id 1 end_log_pos # CRC32 0x######## GTID #-#-# trans -BEGIN +START TRANSACTION #Q> /* GTID */ DELETE FROM t1 WHERE a=5 # server id 1 end_log_pos # CRC32 0x######## Table_map: `test`.`t1` mapped to number # # server id 1 end_log_pos # CRC32 0x######## Delete_rows: table id # flags: STMT_END_F @@ -82,25 +82,25 @@ BEGIN # server id 1 end_log_pos # CRC32 0x######## Update_rows: table id # flags: STMT_END_F COMMIT/*!*/; # server id 1 end_log_pos # CRC32 0x######## GTID #-#-# trans -BEGIN +START TRANSACTION #Q> /* GTID */ INSERT INTO t1 VALUES (8, 5, 0) # server id 1 end_log_pos # CRC32 0x######## Table_map: `test`.`t1` mapped to number # # server id 1 end_log_pos # CRC32 0x######## Write_rows: table id # flags: STMT_END_F COMMIT/*!*/; # server id 1 end_log_pos # CRC32 0x######## GTID #-#-# trans -BEGIN +START TRANSACTION #Q> /* GTID */ INSERT INTO t1 VALUES (9, 5, 1) # server id 1 end_log_pos # CRC32 0x######## Table_map: `test`.`t1` mapped to number # # server id 1 end_log_pos # CRC32 0x######## Write_rows: table id # flags: STMT_END_F COMMIT/*!*/; # server id 1 end_log_pos # CRC32 0x######## GTID #-#-# trans -BEGIN +START TRANSACTION #Q> /* GTID */ INSERT INTO t1 VALUES (10, 6, 0) # server id 1 end_log_pos # CRC32 0x######## Table_map: `test`.`t1` mapped to number # # server id 1 end_log_pos # CRC32 0x######## Write_rows: table id # flags: STMT_END_F COMMIT/*!*/; # server id 1 end_log_pos # CRC32 0x######## GTID #-#-# trans -BEGIN +START TRANSACTION #Q> /* GTID */ INSERT INTO t1 VALUES (11, 7, 0) # server id 1 end_log_pos # CRC32 0x######## Table_map: `test`.`t1` mapped to number # # server id 1 end_log_pos # CRC32 0x######## Write_rows: table id # flags: STMT_END_F diff --git a/mysql-test/suite/binlog/r/binlog_parallel_replication_marks_stm_mix.result b/mysql-test/suite/binlog/r/binlog_parallel_replication_marks_stm_mix.result index c56dff9c8a0..4a6cd6f4ac0 100644 --- a/mysql-test/suite/binlog/r/binlog_parallel_replication_marks_stm_mix.result +++ b/mysql-test/suite/binlog/r/binlog_parallel_replication_marks_stm_mix.result @@ -38,21 +38,21 @@ disconnect tmp_con; connection default; FLUSH LOGS; # server id 1 end_log_pos # CRC32 0x######## GTID #-#-# trans -BEGIN +START TRANSACTION /* GTID */ INSERT INTO t1 VALUES (1,0) COMMIT/*!*/; # server id 1 end_log_pos # CRC32 0x######## GTID #-#-# trans -BEGIN +START TRANSACTION /* GTID */ INSERT INTO t1 VALUES (2,0) COMMIT/*!*/; # server id 1 end_log_pos # CRC32 0x######## GTID #-#-# ddl /* GTID */ ALTER TABLE t1 ADD c INT # server id 1 end_log_pos # CRC32 0x######## GTID #-#-# trans -BEGIN +START TRANSACTION /* GTID */ INSERT INTO t1 VALUES (3,0,0) COMMIT/*!*/; # server id 1 end_log_pos # CRC32 0x######## GTID #-#-# ddl -BEGIN +START TRANSACTION /* GTID */ UPDATE t1 SET b=1, c=1 WHERE a=2 /* GTID */ CREATE TEMPORARY TABLE t2 (a INT PRIMARY KEY, b INT) ENGINE=InnoDB /* GTID */ INSERT INTO t2 VALUES (4,10), (5,20) @@ -63,7 +63,7 @@ COMMIT/*!*/; # server id 1 end_log_pos # CRC32 0x######## GTID #-#-# ddl /* GTID */ CREATE TEMPORARY TABLE t3 (a INT PRIMARY KEY) ENGINE=InnoDB # server id 1 end_log_pos # CRC32 0x######## GTID #-#-# ddl -BEGIN +START TRANSACTION /* GTID */ DELETE FROM t1 WHERE a=5 /* GTID */ INSERT INTO t3 VALUES (7) /* GTID */ INSERT INTO t1 SELECT a, 4, 0 FROM t3 @@ -73,21 +73,21 @@ COMMIT/*!*/; # server id 1 end_log_pos # CRC32 0x######## GTID #-#-# ddl /* GTID */ CREATE TEMPORARY TABLE t4 (a INT PRIMARY KEY) ENGINE=InnoDB # server id 1 end_log_pos # CRC32 0x######## GTID #-#-# trans -BEGIN +START TRANSACTION /* GTID */ INSERT INTO t1 VALUES (8, 5, 0) COMMIT/*!*/; # server id 1 end_log_pos # CRC32 0x######## GTID #-#-# ddl /* GTID */ ALTER TABLE t4 ADD b INT # server id 1 end_log_pos # CRC32 0x######## GTID #-#-# trans -BEGIN +START TRANSACTION /* GTID */ INSERT INTO t1 VALUES (9, 5, 1) COMMIT/*!*/; # server id 1 end_log_pos # CRC32 0x######## GTID #-#-# trans -BEGIN +START TRANSACTION /* GTID */ INSERT INTO t1 VALUES (10, 6, 0) COMMIT/*!*/; # server id 1 end_log_pos # CRC32 0x######## GTID #-#-# ddl -BEGIN +START TRANSACTION /* GTID */ CREATE TEMPORARY TABLE t5 (a INT PRIMARY KEY) ENGINE=InnoDB /* GTID */ INSERT INTO t1 VALUES (11, 7, 0) COMMIT/*!*/; diff --git a/mysql-test/suite/binlog/r/binlog_row_annotate.result b/mysql-test/suite/binlog/r/binlog_row_annotate.result index bbdfe547b18..e65569853aa 100644 --- a/mysql-test/suite/binlog/r/binlog_row_annotate.result +++ b/mysql-test/suite/binlog/r/binlog_row_annotate.result @@ -144,7 +144,7 @@ CREATE DATABASE test3 # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-4 /*!100001 SET @@session.gtid_seq_no=4*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -171,7 +171,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-5 /*!100001 SET @@session.gtid_seq_no=5*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -198,7 +198,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-6 /*!100001 SET @@session.gtid_seq_no=6*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -225,7 +225,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-7 /*!100001 SET @@session.gtid_seq_no=7*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -267,7 +267,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-8 /*!100001 SET @@session.gtid_seq_no=8*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -294,7 +294,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-9 /*!100001 SET @@session.gtid_seq_no=9*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -374,7 +374,7 @@ CREATE DATABASE test1 # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-4 /*!100001 SET @@session.gtid_seq_no=4*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -401,7 +401,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-5 /*!100001 SET @@session.gtid_seq_no=5*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -414,7 +414,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-6 /*!100001 SET @@session.gtid_seq_no=6*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -427,7 +427,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-7 /*!100001 SET @@session.gtid_seq_no=7*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -458,7 +458,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-8 /*!100001 SET @@session.gtid_seq_no=8*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -471,7 +471,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-9 /*!100001 SET @@session.gtid_seq_no=9*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -542,7 +542,7 @@ CREATE DATABASE test3 # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-4 /*!100001 SET @@session.gtid_seq_no=4*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -567,7 +567,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-5 /*!100001 SET @@session.gtid_seq_no=5*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -592,7 +592,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-6 /*!100001 SET @@session.gtid_seq_no=6*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -617,7 +617,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-7 /*!100001 SET @@session.gtid_seq_no=7*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -655,7 +655,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-8 /*!100001 SET @@session.gtid_seq_no=8*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -680,7 +680,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-9 /*!100001 SET @@session.gtid_seq_no=9*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -768,7 +768,7 @@ CREATE DATABASE test3 # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-4 /*!100001 SET @@session.gtid_seq_no=4*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -795,7 +795,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-5 /*!100001 SET @@session.gtid_seq_no=5*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -822,7 +822,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-6 /*!100001 SET @@session.gtid_seq_no=6*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -849,7 +849,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-7 /*!100001 SET @@session.gtid_seq_no=7*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -891,7 +891,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-8 /*!100001 SET @@session.gtid_seq_no=8*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -918,7 +918,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-9 /*!100001 SET @@session.gtid_seq_no=9*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -998,7 +998,7 @@ CREATE DATABASE test1 # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-4 /*!100001 SET @@session.gtid_seq_no=4*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1025,7 +1025,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-5 /*!100001 SET @@session.gtid_seq_no=5*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1038,7 +1038,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-6 /*!100001 SET @@session.gtid_seq_no=6*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1051,7 +1051,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-7 /*!100001 SET @@session.gtid_seq_no=7*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1082,7 +1082,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-8 /*!100001 SET @@session.gtid_seq_no=8*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1095,7 +1095,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-9 /*!100001 SET @@session.gtid_seq_no=9*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -1166,7 +1166,7 @@ CREATE DATABASE test3 # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-4 /*!100001 SET @@session.gtid_seq_no=4*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # #010909 4:46:40 server id # end_log_pos # Table_map: `test1`.`t1` mapped to number # @@ -1190,7 +1190,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-5 /*!100001 SET @@session.gtid_seq_no=5*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # #010909 4:46:40 server id # end_log_pos # Table_map: `test2`.`t2` mapped to number # @@ -1214,7 +1214,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-6 /*!100001 SET @@session.gtid_seq_no=6*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # #010909 4:46:40 server id # end_log_pos # Table_map: `test3`.`t3` mapped to number # @@ -1238,7 +1238,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-7 /*!100001 SET @@session.gtid_seq_no=7*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # #010909 4:46:40 server id # end_log_pos # Table_map: `test1`.`t1` mapped to number # @@ -1275,7 +1275,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-8 /*!100001 SET @@session.gtid_seq_no=8*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # #010909 4:46:40 server id # end_log_pos # Table_map: `test2`.`t2` mapped to number # @@ -1299,7 +1299,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-9 /*!100001 SET @@session.gtid_seq_no=9*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # #010909 4:46:40 server id # end_log_pos # Table_map: `test2`.`t2` mapped to number # diff --git a/mysql-test/suite/binlog/r/binlog_row_ctype_ucs.result b/mysql-test/suite/binlog/r/binlog_row_ctype_ucs.result index 9e95f0a365e..0fe726766d2 100644 --- a/mysql-test/suite/binlog/r/binlog_row_ctype_ucs.result +++ b/mysql-test/suite/binlog/r/binlog_row_ctype_ucs.result @@ -16,7 +16,7 @@ flush logs; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; ROLLBACK/*!*/; -BEGIN +START TRANSACTION /*!*/; # Annotate_rows: #Q> insert into t2 values (@v) @@ -104,7 +104,7 @@ CREATE TABLE t1 (a VARCHAR(10) CHARACTER SET utf8) # at # #YYMMDD HH:MM:SS server id # end_log_pos # CRC32 XXX GTID #-#-# /*!100001 SET @@session.gtid_seq_no=#*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -125,7 +125,7 @@ COMMIT # at # #YYMMDD HH:MM:SS server id # end_log_pos # CRC32 XXX GTID #-#-# /*!100001 SET @@session.gtid_seq_no=#*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -146,7 +146,7 @@ COMMIT # at # #YYMMDD HH:MM:SS server id # end_log_pos # CRC32 XXX GTID #-#-# /*!100001 SET @@session.gtid_seq_no=#*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -167,7 +167,7 @@ COMMIT # at # #YYMMDD HH:MM:SS server id # end_log_pos # CRC32 XXX GTID #-#-# /*!100001 SET @@session.gtid_seq_no=#*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -188,7 +188,7 @@ COMMIT # at # #YYMMDD HH:MM:SS server id # end_log_pos # CRC32 XXX GTID #-#-# /*!100001 SET @@session.gtid_seq_no=#*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -209,7 +209,7 @@ COMMIT # at # #YYMMDD HH:MM:SS server id # end_log_pos # CRC32 XXX GTID #-#-# /*!100001 SET @@session.gtid_seq_no=#*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # diff --git a/mysql-test/suite/binlog/r/binlog_row_mysqlbinlog_options.result b/mysql-test/suite/binlog/r/binlog_row_mysqlbinlog_options.result index 8b138c461ab..0232443573a 100644 --- a/mysql-test/suite/binlog/r/binlog_row_mysqlbinlog_options.result +++ b/mysql-test/suite/binlog/r/binlog_row_mysqlbinlog_options.result @@ -62,7 +62,7 @@ CREATE TABLE t1 (a INT, b INT) # at # #010909 4:46:40 server id # end_log_pos # CRC32 XXX GTID 0-1-2 /*!100001 SET @@session.gtid_seq_no=2*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -97,7 +97,7 @@ CREATE TABLE t2 (a INT) # at # #010909 4:46:40 server id # end_log_pos # CRC32 XXX GTID 0-1-4 /*!100001 SET @@session.gtid_seq_no=4*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -121,7 +121,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # CRC32 XXX GTID 0-1-5 /*!100001 SET @@session.gtid_seq_no=5*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -152,7 +152,7 @@ CREATE TABLE t3 (a INT) # at # #010909 4:46:40 server id # end_log_pos # CRC32 XXX GTID 0-1-7 /*!100001 SET @@session.gtid_seq_no=7*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -176,7 +176,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # CRC32 XXX GTID 0-1-8 /*!100001 SET @@session.gtid_seq_no=8*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -198,7 +198,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # CRC32 XXX GTID 0-1-9 /*!100001 SET @@session.gtid_seq_no=9*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -237,7 +237,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # CRC32 XXX GTID 0-1-10 /*!100001 SET @@session.gtid_seq_no=10*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -303,7 +303,7 @@ CREATE TABLE t1 (a INT, b INT) # at # #010909 4:46:40 server id # end_log_pos # CRC32 XXX GTID 0-1-2 /*!100001 SET @@session.gtid_seq_no=2*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -338,7 +338,7 @@ CREATE TABLE t2 (a INT) # at # #010909 4:46:40 server id # end_log_pos # CRC32 XXX GTID 0-1-4 /*!100001 SET @@session.gtid_seq_no=4*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -362,7 +362,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # CRC32 XXX GTID 0-1-5 /*!100001 SET @@session.gtid_seq_no=5*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -393,7 +393,7 @@ CREATE TABLE t3 (a INT) # at # #010909 4:46:40 server id # end_log_pos # CRC32 XXX GTID 0-1-7 /*!100001 SET @@session.gtid_seq_no=7*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -417,7 +417,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # CRC32 XXX GTID 0-1-8 /*!100001 SET @@session.gtid_seq_no=8*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -439,7 +439,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # CRC32 XXX GTID 0-1-9 /*!100001 SET @@session.gtid_seq_no=9*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -478,7 +478,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # CRC32 XXX GTID 0-1-10 /*!100001 SET @@session.gtid_seq_no=10*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # diff --git a/mysql-test/suite/binlog/r/binlog_stm_ctype_ucs.result b/mysql-test/suite/binlog/r/binlog_stm_ctype_ucs.result index 00f7ce6ead0..05b1520724a 100644 --- a/mysql-test/suite/binlog/r/binlog_stm_ctype_ucs.result +++ b/mysql-test/suite/binlog/r/binlog_stm_ctype_ucs.result @@ -15,7 +15,7 @@ flush logs; /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; ROLLBACK/*!*/; -BEGIN +START TRANSACTION /*!*/; SET @`v`:=_ucs2 X'006100620063' COLLATE `ucs2_general_ci`/*!*/; use `test`/*!*/; @@ -106,7 +106,7 @@ CREATE TABLE t1 (a VARCHAR(10) CHARACTER SET utf8) # at # #YYMMDD HH:MM:SS server id # end_log_pos # CRC32 XXX GTID #-#-# /*!100001 SET @@session.gtid_seq_no=#*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # #YYMMDD HH:MM:SS server id # end_log_pos # CRC32 XXX Query thread_id=# exec_time=# error_code=0 @@ -121,7 +121,7 @@ COMMIT # at # #YYMMDD HH:MM:SS server id # end_log_pos # CRC32 XXX GTID #-#-# /*!100001 SET @@session.gtid_seq_no=#*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # #YYMMDD HH:MM:SS server id # end_log_pos # CRC32 XXX Query thread_id=# exec_time=# error_code=0 @@ -136,7 +136,7 @@ COMMIT # at # #YYMMDD HH:MM:SS server id # end_log_pos # CRC32 XXX GTID #-#-# /*!100001 SET @@session.gtid_seq_no=#*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # #YYMMDD HH:MM:SS server id # end_log_pos # CRC32 XXX Query thread_id=# exec_time=# error_code=0 @@ -151,7 +151,7 @@ COMMIT # at # #YYMMDD HH:MM:SS server id # end_log_pos # CRC32 XXX GTID #-#-# /*!100001 SET @@session.gtid_seq_no=#*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # #YYMMDD HH:MM:SS server id # end_log_pos # CRC32 XXX Query thread_id=# exec_time=# error_code=0 @@ -166,7 +166,7 @@ COMMIT # at # #YYMMDD HH:MM:SS server id # end_log_pos # CRC32 XXX GTID #-#-# /*!100001 SET @@session.gtid_seq_no=#*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # #YYMMDD HH:MM:SS server id # end_log_pos # CRC32 XXX Query thread_id=# exec_time=# error_code=0 @@ -181,7 +181,7 @@ COMMIT # at # #YYMMDD HH:MM:SS server id # end_log_pos # CRC32 XXX GTID #-#-# /*!100001 SET @@session.gtid_seq_no=#*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # #YYMMDD HH:MM:SS server id # end_log_pos # CRC32 XXX Query thread_id=# exec_time=# error_code=0 diff --git a/mysql-test/suite/binlog/r/flashback.result b/mysql-test/suite/binlog/r/flashback.result index bdffaabd673..eaced02b7a9 100644 --- a/mysql-test/suite/binlog/r/flashback.result +++ b/mysql-test/suite/binlog/r/flashback.result @@ -81,7 +81,7 @@ c08 TEXT # at # #010909 9:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-2 trans /*!100001 SET @@session.gtid_seq_no=2*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -107,7 +107,7 @@ COMMIT/*!*/; # at # #010909 9:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-3 trans /*!100001 SET @@session.gtid_seq_no=3*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -133,7 +133,7 @@ COMMIT/*!*/; # at # #010909 9:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-4 trans /*!100001 SET @@session.gtid_seq_no=4*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -159,7 +159,7 @@ COMMIT/*!*/; # at # #010909 9:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-5 trans /*!100001 SET @@session.gtid_seq_no=5*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -213,7 +213,7 @@ COMMIT/*!*/; # at # #010909 9:46:40 server id 1 end_log_pos # CRC32 XXX GTID 0-1-6 trans /*!100001 SET @@session.gtid_seq_no=6*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -296,7 +296,7 @@ ROLLBACK/*!*/; # Number of rows: 3 #010909 9:46:40 server id 1 end_log_pos # CRC32 XXX Rotate to master-bin.000002 pos: 4 #010909 9:46:40 server id 1 end_log_pos # CRC32 XXX Xid = # -BEGIN/*!*/; +START TRANSACTION/*!*/; #010909 9:46:40 server id 1 end_log_pos # CRC32 XXX Write_rows: table id # flags: STMT_END_F ### INSERT INTO `test`.`t1` ### SET @@ -331,7 +331,7 @@ BEGIN/*!*/; COMMIT /*!*/; #010909 9:46:40 server id 1 end_log_pos # CRC32 XXX Xid = # -BEGIN/*!*/; +START TRANSACTION/*!*/; #010909 9:46:40 server id 1 end_log_pos # CRC32 XXX Update_rows: table id # flags: STMT_END_F ### UPDATE `test`.`t1` ### WHERE @@ -374,7 +374,7 @@ BEGIN/*!*/; COMMIT /*!*/; #010909 9:46:40 server id 1 end_log_pos # CRC32 XXX Xid = # -BEGIN/*!*/; +START TRANSACTION/*!*/; #010909 9:46:40 server id 1 end_log_pos # CRC32 XXX Delete_rows: table id # flags: STMT_END_F ### DELETE FROM `test`.`t1` ### WHERE @@ -389,7 +389,7 @@ BEGIN/*!*/; COMMIT /*!*/; #010909 9:46:40 server id 1 end_log_pos # CRC32 XXX Xid = # -BEGIN/*!*/; +START TRANSACTION/*!*/; #010909 9:46:40 server id 1 end_log_pos # CRC32 XXX Delete_rows: table id # flags: STMT_END_F ### DELETE FROM `test`.`t1` ### WHERE @@ -404,7 +404,7 @@ BEGIN/*!*/; COMMIT /*!*/; #010909 9:46:40 server id 1 end_log_pos # CRC32 XXX Xid = # -BEGIN/*!*/; +START TRANSACTION/*!*/; #010909 9:46:40 server id 1 end_log_pos # CRC32 XXX Delete_rows: table id # flags: STMT_END_F ### DELETE FROM `test`.`t1` ### WHERE diff --git a/mysql-test/suite/binlog_encryption/binlog_row_annotate.result b/mysql-test/suite/binlog_encryption/binlog_row_annotate.result index b783488b07a..004f163f192 100644 --- a/mysql-test/suite/binlog_encryption/binlog_row_annotate.result +++ b/mysql-test/suite/binlog_encryption/binlog_row_annotate.result @@ -148,7 +148,7 @@ CREATE DATABASE test3 # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-4 /*!100001 SET @@session.gtid_seq_no=4*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -175,7 +175,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-5 /*!100001 SET @@session.gtid_seq_no=5*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -202,7 +202,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-6 /*!100001 SET @@session.gtid_seq_no=6*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -229,7 +229,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-7 /*!100001 SET @@session.gtid_seq_no=7*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -271,7 +271,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-8 /*!100001 SET @@session.gtid_seq_no=8*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -298,7 +298,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-9 /*!100001 SET @@session.gtid_seq_no=9*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -381,7 +381,7 @@ CREATE DATABASE test1 # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-4 /*!100001 SET @@session.gtid_seq_no=4*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -408,7 +408,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-5 /*!100001 SET @@session.gtid_seq_no=5*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -421,7 +421,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-6 /*!100001 SET @@session.gtid_seq_no=6*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -434,7 +434,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-7 /*!100001 SET @@session.gtid_seq_no=7*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -465,7 +465,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-8 /*!100001 SET @@session.gtid_seq_no=8*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -478,7 +478,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-9 /*!100001 SET @@session.gtid_seq_no=9*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -552,7 +552,7 @@ CREATE DATABASE test3 # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-4 /*!100001 SET @@session.gtid_seq_no=4*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # #010909 4:46:40 server id # end_log_pos # Table_map: `test1`.`t1` mapped to number # @@ -576,7 +576,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-5 /*!100001 SET @@session.gtid_seq_no=5*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # #010909 4:46:40 server id # end_log_pos # Table_map: `test2`.`t2` mapped to number # @@ -600,7 +600,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-6 /*!100001 SET @@session.gtid_seq_no=6*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # #010909 4:46:40 server id # end_log_pos # Table_map: `test3`.`t3` mapped to number # @@ -624,7 +624,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-7 /*!100001 SET @@session.gtid_seq_no=7*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # #010909 4:46:40 server id # end_log_pos # Table_map: `test1`.`t1` mapped to number # @@ -661,7 +661,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-8 /*!100001 SET @@session.gtid_seq_no=8*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # #010909 4:46:40 server id # end_log_pos # Table_map: `test2`.`t2` mapped to number # @@ -685,7 +685,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # GTID 0-1-9 /*!100001 SET @@session.gtid_seq_no=9*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # #010909 4:46:40 server id # end_log_pos # Table_map: `test2`.`t2` mapped to number # diff --git a/mysql-test/suite/compat/oracle/r/binlog_ptr_mysqlbinlog.result b/mysql-test/suite/compat/oracle/r/binlog_ptr_mysqlbinlog.result new file mode 100644 index 00000000000..0656a685976 --- /dev/null +++ b/mysql-test/suite/compat/oracle/r/binlog_ptr_mysqlbinlog.result @@ -0,0 +1,100 @@ +SET @@SQL_MODE = 'ORACLE'; +########################################################################## +# Test verifies Gtid_log_event/Xid_log_event specific print # +########################################################################## +CREATE TABLE tm (f INT) ENGINE=MYISAM; +INSERT INTO tm VALUES (10); +CREATE TABLE t(f INT) ENGINE=INNODB; +INSERT INTO t VALUES (10); +CREATE OR REPLACE PROCEDURE simpleproc (param1 OUT INT) AS +BEGIN +SELECT COUNT(*) INTO param1 FROM t; +END; +/ +CREATE FUNCTION f1 RETURN INT +AS +BEGIN +RETURN 10; +END; +/ +FLUSH LOGS; +########################################################################## +# Delete data from master so that it can be restored from binlog # +########################################################################## +DROP FUNCTION f1; +DROP PROCEDURE simpleproc; +DROP TABLE tm; +DROP TABLE t; +########################################################################## +# Post recovery using mysqlbinlog # +########################################################################## +SHOW TABLES; +Tables_in_test +t +tm +SELECT * FROM tm; +f +10 +SELECT * FROM t; +f +10 +SELECT f1(); +f1() +10 +CALL simpleproc(@a); +SELECT @a; +@a +1 +"***** Clean Up *****" +DROP TABLE t,tm; +DROP PROCEDURE simpleproc; +DROP FUNCTION f1; +RESET MASTER; +########################################################################## +# Test verifies Gtid_log_event/Xid_log_event/Qery_log_event # +# specific print along with flashback option # +########################################################################## +CREATE TABLE tm(f INT) ENGINE=MYISAM; +INSERT INTO tm VALUES (10); +INSERT INTO tm VALUES (20); +CREATE TABLE t(f INT) ENGINE=INNODB; +INSERT INTO t VALUES (10); +INSERT INTO t VALUES (20); +########################################################################## +# Initial data # +########################################################################## +SELECT * FROM tm; +f +10 +20 +SELECT * FROM t; +f +10 +20 +FLUSH LOGS; +DELETE FROM tm WHERE f=20; +DELETE FROM t WHERE f=20; +FLUSH LOGS; +########################################################################## +# Data after deletion # +########################################################################## +SELECT * FROM tm; +f +10 +SELECT * FROM t; +f +10 +FOUND 2 /START TRANSACTION/ in test.sql +########################################################################## +# Data after recovery using flashback # +########################################################################## +SELECT * FROM tm; +f +10 +20 +SELECT * FROM t; +f +10 +20 +"***** Clean Up *****" +DROP TABLE t,tm; diff --git a/mysql-test/suite/compat/oracle/t/binlog_ptr_mysqlbinlog-master.opt b/mysql-test/suite/compat/oracle/t/binlog_ptr_mysqlbinlog-master.opt new file mode 100644 index 00000000000..8f0cc182f51 --- /dev/null +++ b/mysql-test/suite/compat/oracle/t/binlog_ptr_mysqlbinlog-master.opt @@ -0,0 +1 @@ +--flashback diff --git a/mysql-test/suite/compat/oracle/t/binlog_ptr_mysqlbinlog.test b/mysql-test/suite/compat/oracle/t/binlog_ptr_mysqlbinlog.test new file mode 100644 index 00000000000..bda32af5d4e --- /dev/null +++ b/mysql-test/suite/compat/oracle/t/binlog_ptr_mysqlbinlog.test @@ -0,0 +1,117 @@ +# ==== Purpose ==== +# +# Test verifies that point in time recovery of binary log works when +# sql_mode='ORACLE'. +# +# BEGIN statement is printed in three places +# 1) "Gtid_log_event::print" +# 2) "Xid_log_event::print" if flashback is enabled +# 3) "Query_log_event::print" if flashback is enabled and engine is +# non-transacional. +# +# Test verifies all these cases. +# +# ==== References ==== +# +# MDEV-23108: Point in time recovery of binary log fails when sql_mode=ORACLE +# +--source include/have_log_bin.inc +--source include/have_innodb.inc + +let $MYSQLD_DATADIR= `select @@datadir`; +SET @@SQL_MODE = 'ORACLE'; + +--echo ########################################################################## +--echo # Test verifies Gtid_log_event/Xid_log_event specific print # +--echo ########################################################################## +CREATE TABLE tm (f INT) ENGINE=MYISAM; +INSERT INTO tm VALUES (10); + +CREATE TABLE t(f INT) ENGINE=INNODB; +INSERT INTO t VALUES (10); + +DELIMITER /; +CREATE OR REPLACE PROCEDURE simpleproc (param1 OUT INT) AS + BEGIN + SELECT COUNT(*) INTO param1 FROM t; + END; +/ +CREATE FUNCTION f1 RETURN INT +AS +BEGIN + RETURN 10; +END; +/ +DELIMITER ;/ + +FLUSH LOGS; +--echo ########################################################################## +--echo # Delete data from master so that it can be restored from binlog # +--echo ########################################################################## +DROP FUNCTION f1; +DROP PROCEDURE simpleproc; +DROP TABLE tm; +DROP TABLE t; + +--exec $MYSQL_BINLOG $MYSQLD_DATADIR/master-bin.000001 > $MYSQLTEST_VARDIR/tmp/test.sql +--exec $MYSQL < $MYSQLTEST_VARDIR/tmp/test.sql + +--echo ########################################################################## +--echo # Post recovery using mysqlbinlog # +--echo ########################################################################## +SHOW TABLES; +SELECT * FROM tm; +SELECT * FROM t; +--horizontal_results +SELECT f1(); +CALL simpleproc(@a); +SELECT @a; + +--echo "***** Clean Up *****" +DROP TABLE t,tm; +DROP PROCEDURE simpleproc; +DROP FUNCTION f1; +--remove_file $MYSQLTEST_VARDIR/tmp/test.sql +RESET MASTER; + +--echo ########################################################################## +--echo # Test verifies Gtid_log_event/Xid_log_event/Qery_log_event # +--echo # specific print along with flashback option # +--echo ########################################################################## +CREATE TABLE tm(f INT) ENGINE=MYISAM; +INSERT INTO tm VALUES (10); +INSERT INTO tm VALUES (20); +CREATE TABLE t(f INT) ENGINE=INNODB; +INSERT INTO t VALUES (10); +INSERT INTO t VALUES (20); +--echo ########################################################################## +--echo # Initial data # +--echo ########################################################################## +SELECT * FROM tm; +SELECT * FROM t; +FLUSH LOGS; +DELETE FROM tm WHERE f=20; +DELETE FROM t WHERE f=20; +FLUSH LOGS; + +--echo ########################################################################## +--echo # Data after deletion # +--echo ########################################################################## +SELECT * FROM tm; +SELECT * FROM t; +--exec $MYSQL_BINLOG --flashback $MYSQLD_DATADIR/master-bin.000002 > $MYSQLTEST_VARDIR/tmp/test.sql + +--let SEARCH_FILE=$MYSQLTEST_VARDIR/tmp/test.sql +--let SEARCH_PATTERN=START TRANSACTION +--source include/search_pattern_in_file.inc +--exec $MYSQL < $MYSQLTEST_VARDIR/tmp/test.sql + +--echo ########################################################################## +--echo # Data after recovery using flashback # +--echo ########################################################################## +SELECT * FROM tm; +SELECT * FROM t; + +--echo "***** Clean Up *****" +DROP TABLE t,tm; +--remove_file $MYSQLTEST_VARDIR/tmp/test.sql diff --git a/mysql-test/suite/rpl/r/rpl_blackhole_row_annotate.result b/mysql-test/suite/rpl/r/rpl_blackhole_row_annotate.result index 3c5711855d8..2e75080810b 100644 --- a/mysql-test/suite/rpl/r/rpl_blackhole_row_annotate.result +++ b/mysql-test/suite/rpl/r/rpl_blackhole_row_annotate.result @@ -230,7 +230,7 @@ ALTER TABLE t1 ENGINE=BLACKHOLE #010909 4:46:40 server id # end_log_pos # CRC32 XXX GTID 0-1-3 trans /*!100001 SET @@session.server_id=1*//*!*/; /*!100001 SET @@session.gtid_seq_no=3*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -248,7 +248,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # CRC32 XXX GTID 0-1-4 trans /*!100001 SET @@session.gtid_seq_no=4*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -266,7 +266,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # CRC32 XXX GTID 0-1-5 trans /*!100001 SET @@session.gtid_seq_no=5*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -284,7 +284,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # CRC32 XXX GTID 0-1-6 trans /*!100001 SET @@session.gtid_seq_no=6*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -302,7 +302,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # CRC32 XXX GTID 0-1-7 trans /*!100001 SET @@session.gtid_seq_no=7*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -328,7 +328,7 @@ ALTER TABLE t1 ADD PRIMARY KEY pk_t1 (a,b) # at # #010909 4:46:40 server id # end_log_pos # CRC32 XXX GTID 0-1-9 trans /*!100001 SET @@session.gtid_seq_no=9*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -346,7 +346,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # CRC32 XXX GTID 0-1-10 trans /*!100001 SET @@session.gtid_seq_no=10*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -364,7 +364,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # CRC32 XXX GTID 0-1-11 trans /*!100001 SET @@session.gtid_seq_no=11*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -390,7 +390,7 @@ ALTER TABLE t1 DROP PRIMARY KEY, ADD KEY key_t1 (a) # at # #010909 4:46:40 server id # end_log_pos # CRC32 XXX GTID 0-1-13 trans /*!100001 SET @@session.gtid_seq_no=13*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -408,7 +408,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # CRC32 XXX GTID 0-1-14 trans /*!100001 SET @@session.gtid_seq_no=14*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # @@ -426,7 +426,7 @@ COMMIT # at # #010909 4:46:40 server id # end_log_pos # CRC32 XXX GTID 0-1-15 trans /*!100001 SET @@session.gtid_seq_no=15*//*!*/; -BEGIN +START TRANSACTION /*!*/; # at # # at # diff --git a/mysql-test/suite/rpl/r/rpl_hrtime.result b/mysql-test/suite/rpl/r/rpl_hrtime.result index 7717dcf6700..d878a8d9df9 100644 --- a/mysql-test/suite/rpl/r/rpl_hrtime.result +++ b/mysql-test/suite/rpl/r/rpl_hrtime.result @@ -46,7 +46,7 @@ SET @@session.lc_time_names=0/*!*/; SET @@session.collation_database=DEFAULT/*!*/; create table t1 (a timestamp(4) NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, b varchar(100), c datetime(2)) /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1293832861.123456/*!*/; SET @@session.time_zone='+03:00'/*!*/; @@ -55,7 +55,7 @@ insert t1 (b,c) values (now(6), now(6)) SET TIMESTAMP=1293832861.123456/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1293832861/*!*/; insert t1 values ('2010-10-10 10:10:10.101010','2010-10-10 10:10:10.101010','2010-10-10 10:10:10.101010') @@ -63,7 +63,7 @@ insert t1 values ('2010-10-10 10:10:10.101010','2010-10-10 10:10:10.101010','201 SET TIMESTAMP=1293832861/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1643756522.654321/*!*/; insert t1 (b,c) values (now(), now()) @@ -71,7 +71,7 @@ insert t1 (b,c) values (now(), now()) SET TIMESTAMP=1643756522.654321/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1643756522.654321/*!*/; insert t1 (b,c) values (0,0) @@ -79,7 +79,7 @@ insert t1 (b,c) values (0,0) SET TIMESTAMP=1643756522.654321/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1643756522.654321/*!*/; insert t1 (a,b,c) values (0,0,now(6)) diff --git a/mysql-test/suite/rpl/r/rpl_mdev382.result b/mysql-test/suite/rpl/r/rpl_mdev382.result index ee9377906ad..7e26e82d8c9 100644 --- a/mysql-test/suite/rpl/r/rpl_mdev382.result +++ b/mysql-test/suite/rpl/r/rpl_mdev382.result @@ -189,7 +189,7 @@ SET @@session.collation_database=DEFAULT/*!*/; CREATE TABLE `t``1` (`a``1` VARCHAR(4) PRIMARY KEY, `b``2` VARCHAR(3), `c``3` VARCHAR(7)) /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; LOAD DATA LOCAL INFILE '<name>' INTO TABLE `t``1` FIELDS TERMINATED BY ',' ENCLOSED BY '\'' ESCAPED BY '\\' LINES TERMINATED BY '\n' (`a``1`, @`b```) SET `b``2`= @`b```, `c``3`= concat('|', "b""a'z", "!") @@ -200,7 +200,7 @@ COMMIT SET TIMESTAMP=1000000000/*!*/; truncate `t``1` /*!*/; -BEGIN +START TRANSACTION /*!*/; use `test`/*!*/; SET TIMESTAMP=1000000000/*!*/; @@ -288,7 +288,7 @@ SET @@session.lc_time_names=0/*!*/; SET @@session.collation_database=DEFAULT/*!*/; CREATE TABLE t1 (a1 BIGINT PRIMARY KEY, a2 BIGINT, a3 BIGINT, a4 BIGINT UNSIGNED, b DOUBLE, c DECIMAL(65,10), d VARCHAR(100)) /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=1000000000/*!*/; INSERT INTO t1 VALUES (-9223372036854775808,42,9223372036854775807,18446744073709551615,-1234560123456789e110, -1234501234567890123456789012345678901234567890123456789.0123456789, REPEAT("x", 100)) @@ -296,7 +296,7 @@ INSERT INTO t1 VALUES (-9223372036854775808,42,9223372036854775807,1844674407370 SET TIMESTAMP=1000000000/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET @`a``1`:=-9223372036854775808/*!*/; SET @`a``2`:=42/*!*/; @@ -383,7 +383,7 @@ master-bin.000002 # Query 1 # COMMIT /*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/; DELIMITER /*!*/; ROLLBACK/*!*/; -BEGIN +START TRANSACTION /*!*/; use `ts``et`/*!*/; SET TIMESTAMP=1000000000/*!*/; diff --git a/mysql-test/suite/rpl/r/rpl_sp.result b/mysql-test/suite/rpl/r/rpl_sp.result index 97cc4dad8ad..686380427b5 100644 --- a/mysql-test/suite/rpl/r/rpl_sp.result +++ b/mysql-test/suite/rpl/r/rpl_sp.result @@ -810,7 +810,7 @@ insert into t1 values (b); insert into t1 values (unix_timestamp()); end /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=t/*!*/; insert into t1 values ( NAME_CONST('b',8)) @@ -818,7 +818,7 @@ insert into t1 values ( NAME_CONST('b',8)) SET TIMESTAMP=t/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=t/*!*/; insert into t1 values (unix_timestamp()) @@ -826,7 +826,7 @@ insert into t1 values (unix_timestamp()) SET TIMESTAMP=t/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=t/*!*/; delete from t1 @@ -872,7 +872,7 @@ insert into t2 values(3); insert into t1 values (5); end /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=t/*!*/; insert into t2 values(3) @@ -880,7 +880,7 @@ insert into t2 values(3) SET TIMESTAMP=t/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=t/*!*/; insert into t1 values (15) @@ -888,7 +888,7 @@ insert into t1 values (15) SET TIMESTAMP=t/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=t/*!*/; insert into t2 values(3) @@ -899,7 +899,7 @@ COMMIT SET TIMESTAMP=t/*!*/; alter procedure foo4 sql security invoker /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=t/*!*/; insert into t2 values(3) @@ -907,7 +907,7 @@ insert into t2 values(3) SET TIMESTAMP=t/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=t/*!*/; insert into t1 values (5) @@ -915,7 +915,7 @@ insert into t1 values (5) SET TIMESTAMP=t/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=t/*!*/; delete from t2 @@ -936,7 +936,7 @@ begin insert into t2 values(20),(20); end /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=t/*!*/; insert into t2 values(20),(20) @@ -964,7 +964,7 @@ insert into t1 values (x); return x+2; end /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=t/*!*/; delete t1,t2 from t1,t2 @@ -972,7 +972,7 @@ delete t1,t2 from t1,t2 SET TIMESTAMP=t/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=t/*!*/; SELECT `mysqltest1`.`fn1`(20) @@ -980,7 +980,7 @@ SELECT `mysqltest1`.`fn1`(20) SET TIMESTAMP=t/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=t/*!*/; insert into t2 values(fn1(21)) @@ -998,7 +998,7 @@ begin return unix_timestamp(); end /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=t/*!*/; delete from t1 @@ -1006,7 +1006,7 @@ delete from t1 SET TIMESTAMP=t/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=t/*!*/; insert into t1 values(fn1()) @@ -1028,7 +1028,7 @@ begin return 0; end /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=t/*!*/; delete from t2 @@ -1049,7 +1049,7 @@ insert into t2 values(x),(x); return 10; end /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=t/*!*/; SELECT `mysqltest1`.`fn1`(100) @@ -1057,7 +1057,7 @@ SELECT `mysqltest1`.`fn1`(100) SET TIMESTAMP=t/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=t/*!*/; SELECT `mysqltest1`.`fn1`(20) @@ -1065,7 +1065,7 @@ SELECT `mysqltest1`.`fn1`(20) SET TIMESTAMP=t/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=t/*!*/; delete from t1 @@ -1076,7 +1076,7 @@ COMMIT SET TIMESTAMP=t/*!*/; CREATE DEFINER=`root`@`localhost` trigger trg before insert on t1 for each row set new.a= 10 /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=t/*!*/; insert into t1 values (1) @@ -1084,7 +1084,7 @@ insert into t1 values (1) SET TIMESTAMP=t/*!*/; COMMIT /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=t/*!*/; delete from t1 @@ -1095,7 +1095,7 @@ COMMIT SET TIMESTAMP=t/*!*/; drop trigger trg /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=t/*!*/; insert into t1 values (1) @@ -1142,7 +1142,7 @@ CREATE ALGORITHM=UNDEFINED DEFINER=`root`@`localhost` SQL SECURITY DEFINER VIEW SET TIMESTAMP=t/*!*/; create table t1 (a int) /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=t/*!*/; insert into t1 (a) values (f1()) @@ -1169,7 +1169,7 @@ SET TIMESTAMP=t/*!*/; CREATE DEFINER=`root`@`localhost` PROCEDURE `p1`(arg VARCHAR(10)) INSERT INTO t1 VALUES(arg) /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=t/*!*/; INSERT INTO t1 VALUES( NAME_CONST('arg',_latin1'test' COLLATE 'latin1_swedish_ci')) @@ -1223,7 +1223,7 @@ SET TIMESTAMP=t/*!*/; CREATE DEFINER=`root`@`localhost` PROCEDURE `mysqltest`.`test`() begin end /*!*/; -BEGIN +START TRANSACTION /*!*/; SET TIMESTAMP=t/*!*/; insert into t values ( 1 ) @@ -1238,7 +1238,7 @@ insert into t values (1); return 0; end /*!*/; -BEGIN +START TRANSACTION /*!*/; use `mysqltest`/*!*/; SET TIMESTAMP=t/*!*/; diff --git a/sql/log_event.cc b/sql/log_event.cc index 57a39085e86..964f451a766 100644 --- a/sql/log_event.cc +++ b/sql/log_event.cc @@ -5358,8 +5358,7 @@ bool Query_log_event::print(FILE* file, PRINT_EVENT_INFO* print_event_info) } else if (strcmp("COMMIT", query) == 0) { - if (my_b_write(&cache, (uchar*) "BEGIN", 5) || - my_b_printf(&cache, "\n%s\n", print_event_info->delimiter)) + if (my_b_printf(&cache, "START TRANSACTION\n%s\n", print_event_info->delimiter)) goto err; } } @@ -8254,7 +8253,8 @@ Gtid_log_event::print(FILE *file, PRINT_EVENT_INFO *print_event_info) goto err; } if (!(flags2 & FL_STANDALONE)) - if (my_b_printf(&cache, is_flashback ? "COMMIT\n%s\n" : "BEGIN\n%s\n", print_event_info->delimiter)) + if (my_b_printf(&cache, is_flashback ? "COMMIT\n%s\n" : + "START TRANSACTION\n%s\n", print_event_info->delimiter)) goto err; return cache.flush_data(); @@ -8937,7 +8937,7 @@ bool Xid_log_event::print(FILE* file, PRINT_EVENT_INFO* print_event_info) my_b_printf(&cache, "\tXid = %s\n", buf)) goto err; } - if (my_b_printf(&cache, is_flashback ? "BEGIN%s\n" : "COMMIT%s\n", + if (my_b_printf(&cache, is_flashback ? "START TRANSACTION%s\n" : "COMMIT%s\n", print_event_info->delimiter)) goto err;

1 0

[Maria-developers] Progress Report GSoC 2020 (6-12 Jule)
by Anton Mikhaylenko 16 Jul '20

16 Jul '20

Good day, everybody. In this post, I want to describe what I did this week, Jule 6-12. I was dealing with the benchmarking for a treadpool. I used https://github.com/vaintroub/benchmark_tools. I installed it and made initial launches. I had to run several commands to launch. 1. Compile the server with RelWithDebInfo. cmake --build. -config RelWithDebInfo -- -m 2. cd sql\RelWithDebInfo start mysqld.exe --defaults-file=C:\path\to\sysbench.ini 3. cd C:\path\to\benchmark_tools mysql -uroot -e "create database sbtest" 4. update_index prepare 1024 5. update_index run 1024 1024 is the number of clients. Regards, Anton MIkhailenko

1 0

[Maria-developers] Weekly Report
by Kartik Soneji 14 Jul '20

14 Jul '20

Hi, This week - Started working on changing the build system to make the plugins call our functions instead of the library functions as per step 4 of MDEV-22895. - I faced a lot of issues with cmake, as INCLUDE_DIRECTORIES does not work with CHECK_INCLUDE_FILES, and setting CMAKE_REQUIRED_INCLUDES in cmake/plugins.cmake did not affect storage/innobase/lzma.cmake. The only thing that worked was STRING(APPEND CMAKE_C_FLAGS " -I${CMAKE_SOURCE_DIR}/include/mysql"), but that sets the flag globally, so is not ideal. - I spoke with some devs on the cmake IRC (#cmake), and it seems that InnoDB uses improper Cmake commands. It might be easier for the project to rewrite them instead of trying to work around the issues. Next week - Review the CMake files and finish updating the build system as per current standards. - Continue work on step 5 of MDEV-22895. Thanks, Kartik

1 0

[Maria-developers] Weekly Report
by Kartik Soneji 06 Jul '20

06 Jul '20

Hi, This week - Started working on changing the build system to make the plugins call our functions instead of the library functions as per step 4 of MDEV-22895. - Worked on a proof-of-concept (poc-noservice-lzma) that bypassed the fake dynamic linker, and thus simplified the code greatly. Decided against using it because of stability and compatibility concerns. Next week - Finish tweaking the build system. - Start work step 5 on MDEV-22895. Thanks, Kartik

1 0

Re: [Maria-developers] cb2cd9c3a43: MDEV-22058: Assertion `!is_set() || (m_status == DA_OK_BULK && is_bulk_op())' failed in Diagnostics_area::set_ok_status
by Sergei Golubchik 02 Jul '20

02 Jul '20

Hi, Rucha! On Jul 02, Rucha Deodhar wrote: > revision-id: cb2cd9c3a43 (mariadb-10.2.31-302-gcb2cd9c3a43) > parent(s): dba7e1e8e14 > author: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > committer: Rucha Deodhar <rucha.deodhar(a)mariadb.com> > timestamp: 2020-07-02 19:51:55 +0530 > message: > > MDEV-22058: Assertion `!is_set() || (m_status == DA_OK_BULK && is_bulk_op())' failed in Diagnostics_area::set_ok_status > > Error state is not stored in check_and_do_in_subquery_rewrites() when there is > illegal combination of optimizer switches. So all the functions eventually > return false. Thus the assetion failure. > > diff --git a/sql/opt_subselect.cc b/sql/opt_subselect.cc > index 2fb5b88687f..e37b389836c 100644 > --- a/sql/opt_subselect.cc > +++ b/sql/opt_subselect.cc > @@ -559,6 +559,7 @@ bool is_materialization_applicable(THD *thd, Item_in_subselect *in_subs, > int check_and_do_in_subquery_rewrites(JOIN *join) > { > THD *thd=join->thd; > + int res= 0; > st_select_lex *select_lex= join->select_lex; > st_select_lex_unit* parent_unit= select_lex->master_unit(); > DBUG_ENTER("check_and_do_in_subquery_rewrites"); > @@ -703,7 +704,11 @@ int check_and_do_in_subquery_rewrites(JOIN *join) > /* Test if the user has set a legal combination of optimizer switches. */ > if (!optimizer_flag(thd, OPTIMIZER_SWITCH_IN_TO_EXISTS) && > !optimizer_flag(thd, OPTIMIZER_SWITCH_MATERIALIZATION)) > + { > my_error(ER_ILLEGAL_SUBQUERY_OPTIMIZER_SWITCHES, MYF(0)); I think this is very weird behavior. A user sets incorrect optimizer switch, SET statement succeeds. But later at some point SELECT returns an error. You have correctly found the place where the error status is not properly returned and fixed it. But I'd rather suggest to move the check to where it should've been. Here instead of the if() you put an assert: DBUG_ASSERT(optimizer_flag(thd, OPTIMIZER_SWITCH_IN_TO_EXISTS | OPTIMIZER_SWITCH_MATERIALIZATION)); and the if() with an error should be in sys_vars.cc. So that the user would get an error from the SET statement. Look for ON_CHECK in the sys_vars.cc > + res= -1; > + } > + > /* > Transform each subquery predicate according to its overloaded > transformer. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] cf0fa125877: MENT-471: Plugin implementation for the Hashicorp Vault KMS
by Sergei Golubchik 01 Jul '20

01 Jul '20

Hi, Julius! On Jul 01, Julius Goryavsky wrote: > revision-id: cf0fa125877 (mariadb-10.5.2-177-gcf0fa125877) > parent(s): e7ca9ec6f6b > author: Julius Goryavsky <julius.goryavsky(a)mariadb.com> > committer: Julius Goryavsky <julius.goryavsky(a)mariadb.com> > timestamp: 2020-04-20 15:28:49 +0200 > message: > > MENT-471: Plugin implementation for the Hashicorp Vault KMS > > - Authentication is done using the Hashicorp Vault's token > authentication method; > - If additional client authentication is required, then the > path to the CA authentication bundle file may be passed > as a plugin parameter; > - The creation of the keys and their management is carried > out using the Hashicorp Vault KMS and their tools; > - Key values stored as hexadecimal strings; > - Key values caching is supported. > - The plugin uses libcurl (https) as an interface to > the HashiCorp Vault server; > - JSON parsing is performed through the JSON service > (through the include/mysql/service_json.h); > - HashiCorp Vault 1.2.4 was used for development and testing. > > --- > mysql-test/suite/vault/disabled.def | 11 + > mysql-test/suite/vault/my.cnf | 1 + > mysql-test/suite/vault/r/hashicorp_encode.result | 45 ++ > mysql-test/suite/vault/suite.pm | 27 + > mysql-test/suite/vault/t/hashicorp_deinit.inc | 6 + > mysql-test/suite/vault/t/hashicorp_encode.opt | 5 + > mysql-test/suite/vault/t/hashicorp_encode.test | 3 + > mysql-test/suite/vault/t/hashicorp_goodtest.inc | 20 + > mysql-test/suite/vault/t/hashicorp_init.inc | 32 + > mysql-test/suite/vault/t/hashicorp_plugin.inc | 1 + please, put the plugin specific test suite into the plugin dir: plugin/hashicorp_key_management/mysql-test/vault > plugin/hashicorp_key_management/CMakeLists.txt | 11 + > .../hashicorp_key_management_plugin.cc | 899 +++++++++++++++++++++ > plugin/hashicorp_key_management/readme.txt | 138 ++++ > 14 files changed, 1203 insertions(+) > > diff --git a/mysql-test/suite/vault/disabled.def b/mysql-test/suite/vault/disabled.def > --- /dev/null > +++ b/mysql-test/suite/vault/disabled.def > @@ -0,0 +1,11 @@ > +############################################################################## > +# > +# List the test cases that are to be disabled temporarily. > +# > +# Separate the test case name and the comment with ':'. > +# > +# <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> > +# > +# Do not use any TAB characters for whitespace. > +# > +############################################################################## you don't have to create and commit this template > diff --git a/mysql-test/suite/vault/include/have_hashicorp_key_management_plugin.inc b/mysql-test/suite/vault/include/have_hashicorp_key_management_plugin.inc > --- /dev/null > +++ b/mysql-test/suite/vault/include/have_hashicorp_key_management_plugin.inc > @@ -0,0 +1,4 @@ > +if (`SELECT COUNT(*)=0 FROM INFORMATION_SCHEMA.PLUGINS WHERE PLUGIN_NAME = 'hashicorp_key_management' AND PLUGIN_STATUS='ACTIVE'`) > +{ > + --skip Test requires active hashicorp_key_management plugin > +} > diff --git a/mysql-test/suite/vault/my.cnf b/mysql-test/suite/vault/my.cnf > --- /dev/null > +++ b/mysql-test/suite/vault/my.cnf > @@ -0,0 +1 @@ > +!include include/default_my.cnf and you definitely should not create suite my.cnf that only includes the default my.cnf. You only need a custom my.cnf if it adds some extra options that are not default. > diff --git a/mysql-test/suite/vault/suite.pm b/mysql-test/suite/vault/suite.pm > --- /dev/null > +++ b/mysql-test/suite/vault/suite.pm > @@ -0,0 +1,27 @@ > +package My::Suite::Vault; > + > +@ISA = qw(My::Suite); > + > +use strict; > + > +if (::IS_WINDOWS) { > + return "The test suite for Hashicorp Vault plugin currently does not support Windows"; > +} instead of a custom check, you'd better add source include/not_windows.inc into your have_hashicorp_key_management_plugin.inc > + > +return "Hashicorp Vault management utility not found" > + unless ::which("vault"); > + > +return "You need to set the value of the VAULT_ADDR variable" > + unless $ENV{VAULT_ADDR}; > + > +return "You need to set the value of the VAULT_TOKEN variable" > + unless $ENV{VAULT_TOKEN}; > + > +sub skip_combinations { > + my %skip; > + $skip{'include/have_hashicorp_key_management_plugin.inc'} = 'Needs dynamic hashicorp_key_management plugin' > + unless $ENV{HASHICORP_KEY_MANAGEMENT_SO}; this is rather redundant, isn't it? have_hashicorp_key_management_plugin.inc itself checks for the plugin and skips. If you remove your skip_combinations() function, the behavior will be exactly the same. Normally one needs to skip tests from suite.pm for conditions that cannot be checked from inside the test. > + %skip; > +} > + > +bless {}; > diff --git a/mysql-test/suite/vault/t/hashicorp_deinit.inc b/mysql-test/suite/vault/t/hashicorp_deinit.inc > --- /dev/null > +++ b/mysql-test/suite/vault/t/hashicorp_deinit.inc > @@ -0,0 +1,6 @@ > +perl; > + `vault secrets disable test`; 1. perl has "system" command, one normally uses backticks when one needs the output. 2. mysqltest will write ths small perl script into a file and will run perl on it. How is this overhead better than just exec vault secrets disable test; with no perl at all? 3. why did you create a separate hashicorp_deinit.inc if you only use it once? > + if (($? >> 8) != 0) { > + die "Vault: unable to destroy path for test keys"; > + } > +EOF > diff --git a/mysql-test/suite/vault/t/hashicorp_init.inc b/mysql-test/suite/vault/t/hashicorp_init.inc > --- /dev/null > +++ b/mysql-test/suite/vault/t/hashicorp_init.inc > @@ -0,0 +1,32 @@ > +perl; > +my $list = `vault secrets list`; > +if (($? >> 8) != 0) { > + die "Unable to retrieve secrets list - Hashicorp Vault not configured properly or not enough permissions"; > +} > +if ($list !~ /(^|\n|\r)test\/[\s\t]+kv[\s\t]+/) { > + `vault secrets enable -path /test -version=2 kv`; > + if (($? >> 8) != 0) { > + die "Vault: unable to create path for test keys"; > + } > + `vault kv put /test/1 data="123456789ABCDEF0123456789ABCDEF0"`; > + if (($? >> 8) != 0) { > + die "Vault: unable to create first secret key"; > + } > + `vault kv put /test/2 data="23456789ABCDEF0123456789ABCDEF01"`; > + if (($? >> 8) != 0) { > + die "Vault: unable to create second secret key"; > + } > + `vault kv put /test/33 data="00000000000000000000000000000000"`; > + if (($? >> 8) != 0) { > + die "Vault: unable to create third secret key"; > + } > + `vault kv put /test/33 data="3456789ABCDEF0123456789ABCDEF012"`; > + if (($? >> 8) != 0) { > + die "Vault: unable to update third secret key"; > + } > + `vault kv put /test/4 data="456789ABCDEF0123456789ABCDEF0123"`; > + if (($? >> 8) != 0) { > + die "Vault: unable to create fourth secret key"; > + } > +} > +EOF same as above. you shouldn't abuse backticks, don't need perl and you don't need a separate file. Use something like directly in your hashicorp_goodtest.test exec vault secrets disable test; # start from a clean state exec vault secrets enable -path /test -version=2 kv; exec vault kv put /test/1 data="123456789ABCDEF0123456789ABCDEF0"; exec vault kv put /test/2 data="23456789ABCDEF0123456789ABCDEF01"; exec vault kv put /test/33 data="00000000000000000000000000000000"; exec vault kv put /test/33 data="3456789ABCDEF0123456789ABCDEF012"; exec vault kv put /test/4 data="456789ABCDEF0123456789ABCDEF0123"; also, I'd recommend to use a more specific secret name, like mariadbtest > diff --git a/mysql-test/suite/vault/t/hashicorp_goodtest.inc b/mysql-test/suite/vault/t/hashicorp_goodtest.inc > --- /dev/null > +++ b/mysql-test/suite/vault/t/hashicorp_goodtest.inc > @@ -0,0 +1,20 @@ > +--source include/have_innodb.inc > +--source hashicorp_plugin.inc > + > +SHOW GLOBAL variables WHERE Variable_name LIKE "hashicorp%" AND Variable_name NOT LIKE "%vault_url"; normally it's done like replace_result $VAULT_ADDR VAULT_ADDR; SHOW GLOBAL variables LIKE "hashicorp%"; that is, you show all variables and replace away the dynamic part. > + > +create table t1(c1 bigint not null, b char(200)) engine=innodb encrypted=yes encryption_key_id=1; > +show create table t1; > +insert t1 values (12345, repeat('1234567890', 20)); > + > +alter table t1 encryption_key_id=2; > +show create table t1; > +--error ER_ILLEGAL_HA_CREATE_OPTION > +alter table t1 encryption_key_id=3; > +show create table t1; > +alter table t1 encryption_key_id=33; > +show create table t1; > +alter table t1 encryption_key_id=4; > +show create table t1; where's a test for key rotation? where's an encrypt/restart/decrypt test? > + > +drop table t1; > diff --git a/mysql-test/suite/vault/t/hashicorp_encode.opt b/mysql-test/suite/vault/t/hashicorp_encode.opt > --- /dev/null > +++ b/mysql-test/suite/vault/t/hashicorp_encode.opt > @@ -0,0 +1,5 @@ > +--plugin-load-add=$HASHICORP_KEY_MANAGEMENT_SO > +--loose-hashicorp-key-management > +--loose-hashicorp-key-management-vault-url="$VAULT_ADDR/v1/test/" > +--loose-hashicorp-key-management-token="$VAULT_TOKEN" > +--loose-hashicorp-key-management-timeout=60 > diff --git a/mysql-test/suite/vault/t/hashicorp_encode.test b/mysql-test/suite/vault/t/hashicorp_encode.test > --- /dev/null > +++ b/mysql-test/suite/vault/t/hashicorp_encode.test > @@ -0,0 +1,3 @@ > +source hashicorp_init.inc; > +source hashicorp_goodtest.inc; > +source hashicorp_deinit.inc; > diff --git a/mysql-test/suite/vault/t/hashicorp_plugin.inc b/mysql-test/suite/vault/t/hashicorp_plugin.inc > --- /dev/null > +++ b/mysql-test/suite/vault/t/hashicorp_plugin.inc > @@ -0,0 +1 @@ > +--source include/have_hashicorp_key_management_plugin.inc another file that it's only used once? what's the point? include have_hashicorp_key_management_plugin.inc directly into your hashicorp_encode.test > diff --git a/plugin/hashicorp_key_management/hashicorp_key_management_plugin.cc b/plugin/hashicorp_key_management/hashicorp_key_management_plugin.cc > --- /dev/null > +++ b/plugin/hashicorp_key_management/hashicorp_key_management_plugin.cc > @@ -0,0 +1,899 @@ > +/* Copyright (C) 2019 MariaDB Corporation 2019, 2020 > + > + This program is free software; you can redistribute it and/or modify > + it under the terms of the GNU General Public License as published by > + the Free Software Foundation; version 2 of the License. > + > + This program is distributed in the hope that it will be useful, > + but WITHOUT ANY WARRANTY; without even the implied warranty of > + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + GNU General Public License for more details. > + > + You should have received a copy of the GNU General Public License > + along with this program; if not, write to the Free Software > + Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1335 USA */ > + > +#include <mysql/plugin_encryption.h> > +#include <mysqld_error.h> > +#include <string.h> > +#include <stdlib.h> > +#include <limits.h> > +#include <errno.h> > +#include <string> > +#include <sstream> > +#include <curl/curl.h> > +#ifdef _WIN32 > +#include <malloc.h> > +#define alloca _alloca > +#else > +#include <alloca.h> > +#endif > +#include <algorithm> > +#include <map> > +#include <mutex> > + > +static std::mutex mtx; > + > +/* Key information structure: */ > +typedef struct KEY_INFO > +{ > + unsigned int key_id; > + unsigned int key_version; > + unsigned int length; > + unsigned char data [MY_AES_MAX_KEY_LENGTH]; > +public: > + KEY_INFO() : key_id(0), key_version(0), length(0) {}; > +} KEY_INFO; > + > +/* Cache for the latest version, per key id: */ > +static std::map<unsigned int, unsigned int> latest_version_cache; > + > +/* Cache for key information: */ > +static std::map<unsigned long long, KEY_INFO> key_info_cache; why not unordered_map here and above? where do you use the ordering property? > + > +#define KEY_ID_AND_VERSION(key_id, version) \ > + ((unsigned long long)key_id << 32 | version) > + > +static void cache_add (unsigned int key_id, unsigned int key_version, > + const KEY_INFO* info) > +{ > + mtx.lock(); > + latest_version_cache[key_id]= > + std::max(latest_version_cache[key_id], key_version); > + key_info_cache[KEY_ID_AND_VERSION(key_id, key_version)]= *info; > + mtx.unlock(); > +} > + > +static unsigned int > + cache_get (unsigned int key_id, unsigned int key_version, > + unsigned char* data, unsigned int* buflen) > +{ > + unsigned int ver= key_version; > + mtx.lock(); > + if (key_version == ENCRYPTION_KEY_VERSION_INVALID) > + { > + ver= latest_version_cache[key_id]; > + if (ver == 0) > + { > + mtx.unlock(); > + return ENCRYPTION_KEY_VERSION_INVALID; > + } > + } > + KEY_INFO info= key_info_cache[KEY_ID_AND_VERSION(key_id, ver)]; > + mtx.unlock(); > + unsigned int length= info.length; > + if (length == 0) > + { > + return ENCRYPTION_KEY_VERSION_INVALID; > + } > + unsigned int max_length = *buflen; > + *buflen = length; > + if (max_length < length) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Encryption key buffer too small", > + ME_ERROR_LOG_ONLY | ME_NOTE); I don't think you need an error here, this is not an error from the caller point of view, but just a way to query key length. > + return ENCRYPTION_KEY_BUFFER_TOO_SMALL; > + } > + memcpy(data, info.data, length); > + return 0; > +} > + > +static unsigned int cache_get_version (unsigned int key_id) > +{ > + unsigned int ver; > + mtx.lock(); > + ver= latest_version_cache[key_id]; > + mtx.unlock(); > + if (ver) > + { > + return ver; > + } > + else > + { > + return ENCRYPTION_KEY_VERSION_INVALID; > + } > +} > + > +static char* vault_url; > +static char* token; > +static char* vault_ca; > +int timeout; > +int max_retries; > +char caching_enabled; > +char use_cache_on_timeout; why the other four aren't static? > + > +static MYSQL_SYSVAR_STR(vault_ca, vault_ca, > + PLUGIN_VAR_RQCMDARG | PLUGIN_VAR_READONLY, > + "Path to the Certificate Authority (CA) bundle (is a file " > + "that contains root and intermediate certificates)", > + NULL, NULL, ""); > + > +static MYSQL_SYSVAR_STR(vault_url, vault_url, > + PLUGIN_VAR_RQCMDARG | PLUGIN_VAR_READONLY, > + "HTTP[s] URL that is used to connect to the Hashicorp Vault server", > + NULL, NULL, "https://127.0.0.1:8200/v1/kv"); Is this a reasonable default for a vault url? I'd rather keep it empty. > + > +static MYSQL_SYSVAR_STR(token, token, > + PLUGIN_VAR_RQCMDARG | PLUGIN_VAR_READONLY | PLUGIN_VAR_NOSYSVAR, > + "Authentication token that passed to the Hashicorp Vault " > + "in the request header", > + NULL, NULL, ""); > + > +static MYSQL_SYSVAR_INT(timeout, timeout, > + PLUGIN_VAR_RQCMDARG, > + "Duration (in seconds) for the Hashicorp Vault server " > + "connection timeout", > + NULL, NULL, 15, 0, 86400, 1); > + > +static MYSQL_SYSVAR_INT(max_retries, max_retries, > + PLUGIN_VAR_RQCMDARG, > + "Number of server request retries in case of timeout", > + NULL, NULL, 3, 0, INT_MAX, 1); > + > +static MYSQL_SYSVAR_BOOL(caching_enabled, caching_enabled, > + PLUGIN_VAR_RQCMDARG, > + "Enable key caching (storing key values received from " > + "the Hashicorp Vault server in the local memory)", > + NULL, NULL, 1); > + > +static MYSQL_SYSVAR_BOOL(use_cache_on_timeout, use_cache_on_timeout, > + PLUGIN_VAR_RQCMDARG, > + "In case of timeout (when accessing the vault server) " > + "use the value taken from the cache", > + NULL, NULL, 0); > + > +static struct st_mysql_sys_var *settings[] = { > + MYSQL_SYSVAR(vault_url), > + MYSQL_SYSVAR(token), > + MYSQL_SYSVAR(vault_ca), > + MYSQL_SYSVAR(timeout), > + MYSQL_SYSVAR(max_retries), > + MYSQL_SYSVAR(caching_enabled), > + MYSQL_SYSVAR(use_cache_on_timeout), > + NULL > +}; > + > +static std::string get_error_from_curl (CURLcode curl_code, char *curl_errbuf) > +{ > + size_t len = strlen(curl_errbuf); > + std::ostringstream stream; > + if (curl_code != CURLE_OK) > + { > + stream << "CURL returned this error code: " << curl_code; > + stream << " with error message : "; > + if (len) > + stream << curl_errbuf; > + else > + stream << curl_easy_strerror(curl_code); > + } > + return stream.str(); > +} > + > +/* > + Reasonable length limit to protect against accidentally reading > + the wrong key or from trying to overload the server with unnecessary > + work to receive too long responses to requests: > +*/ > +#define max_response_size 131072 conventionally, it should be uppercase > + > +static size_t write_response_memory (void *contents, size_t size, size_t nmemb, > + void *userp) > +{ > + size_t realsize = size * nmemb; > + if (size != 0 && realsize / size != nmemb) > + return 0; // multiplication overflow detected this cannot happen, size is documented to be always 1, and the max data size is CURL_MAX_WRITE_SIZE anyway. > + std::ostringstream *read_data = static_cast<std::ostringstream *>(userp); > + size_t current_length = read_data->tellp(); > + if (current_length + realsize > max_response_size) > + return 0; // response size limit exceeded > + read_data->write(static_cast<char *>(contents), realsize); > + if (!read_data->good()) > + return 0; > + return realsize; > +} > + > +enum { > + OPERATION_OK, > + OPERATION_TIMEOUT, > + OPERATION_ERROR > +}; > + > +static CURLcode > + perform_with_retries (CURL *curl, std::ostringstream *read_data_stream) > +{ > + int retries= max_retries; > + CURLcode curl_res; > + do { > + curl_res= curl_easy_perform(curl); > + if (curl_res != CURLE_OPERATION_TIMEDOUT) > + { > + break; > + } > + read_data_stream->clear(); > + read_data_stream->str(""); I presume, this wasn't tested at all? > + } while (retries--); > + return curl_res; > +} > + > +#define max_token_size 32768 uppercase again > + > +static size_t token_len; > + > +static int curl_run (char *url, std::string *response, bool soft_timeout) > +{ > + const static char *x_vault_token = "X-Vault-Token:"; > + const static size_t x_vault_token_len = strlen(x_vault_token); > +/* > + Checking the maximum allowable length to protect > + against allocating too much memory on the stack: > +*/ > + if (token_len > max_token_size) 32K is a lot, tokens are much shorter and stack isn't big. You can safely reduce max_token_size to, say 8K. and why do you check it here in curl_run anyway? you can check it in hashicorp_key_management_plugin_init. once. > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Maximum allowed token length exceeded", > + ME_ERROR_LOG_ONLY); > + return OPERATION_ERROR; > + } > + char curl_errbuf[CURL_ERROR_SIZE]; > + std::ostringstream read_data_stream; > + long http_code = 0; > + struct curl_slist *list = NULL; > + CURLcode curl_res = CURLE_OK; > + CURL *curl = curl_easy_init(); > + if (curl == NULL) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Cannot initialize curl session", > + ME_ERROR_LOG_ONLY); > + return OPERATION_ERROR; > + } > + size_t buf_len = x_vault_token_len + token_len + 1; > + char *token_header = (char *) alloca(buf_len); > + snprintf(token_header, buf_len, "%s%s", x_vault_token, token); also there's no need to allocate the buffer on the stack and concatenate the header here every time, you can do it once in hashicorp_key_management_plugin_init too. > + curl_errbuf[0] = '\0'; > + if ((list= curl_slist_append(list, token_header)) == NULL || and you can prepare slist only once too (I think). > + (curl_res= curl_easy_setopt(curl, CURLOPT_ERRORBUFFER, curl_errbuf)) != > + CURLE_OK || > + (curl_res= curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, > + write_response_memory)) != CURLE_OK || > + (curl_res= curl_easy_setopt(curl, CURLOPT_WRITEDATA, > + &read_data_stream)) != > + CURLE_OK || > + (curl_res= curl_easy_setopt(curl, CURLOPT_HTTPHEADER, list)) != > + CURLE_OK || > + /* > + The options CURLOPT_SSL_VERIFYPEER and CURLOPT_SSL_VERIFYHOST are > + set explicitly to withstand possible future changes in curl defaults: > + */ > + (curl_res= curl_easy_setopt(curl, CURLOPT_SSL_VERIFYPEER, 1)) != > + CURLE_OK || > + (curl_res= curl_easy_setopt(curl, CURLOPT_SSL_VERIFYHOST, 2L)) != > + CURLE_OK || > + (strlen(vault_ca) != 0 && > + (curl_res= curl_easy_setopt(curl, CURLOPT_CAINFO, vault_ca)) != > + CURLE_OK) || > + (curl_res= curl_easy_setopt(curl, CURLOPT_USE_SSL, CURLUSESSL_ALL)) != > + CURLE_OK || > + (curl_res= curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L)) != > + CURLE_OK || > + (timeout && > + ((curl_res= curl_easy_setopt(curl, CURLOPT_CONNECTTIMEOUT, timeout)) != > + CURLE_OK || > + (curl_res= curl_easy_setopt(curl, CURLOPT_TIMEOUT, timeout)) != > + CURLE_OK)) || > + (curl_res = curl_easy_setopt(curl, CURLOPT_URL, url)) != CURLE_OK || > + (curl_res = perform_with_retries(curl, &read_data_stream)) != CURLE_OK || > + (curl_res = curl_easy_getinfo (curl, CURLINFO_RESPONSE_CODE, > + &http_code)) != CURLE_OK) > + { > + curl_slist_free_all(list); > + curl_easy_cleanup(curl); > + if (soft_timeout && curl_res == CURLE_OPERATION_TIMEDOUT) > + { > + return OPERATION_TIMEOUT; > + } > + my_printf_error(ER_UNKNOWN_ERROR, > + get_error_from_curl(curl_res, curl_errbuf).c_str(), really? strlen, ostringstream, append/append/append, then you return std::string and then print its c_str()? It's printf, you can do directly my_printf_error(ER_UNKNOWN_ERROR, "CURL returned this error code %u" " with error message: %s", curl_res, curl_errbuf); Which is much easier to read, doesn't do all these string operations and reallocs, and it's also safe, because your variant will crash if any of the curl error messages will include % sign. > + ME_ERROR_LOG_ONLY); > + return OPERATION_ERROR; > + } > + curl_slist_free_all(list); > + curl_easy_cleanup(curl); > + *response = read_data_stream.str(); > + bool is_error = http_code < 200 || http_code >= 300; > + if (is_error) > + { > + const char *res = response->c_str(); > + /* > + Error 404 requires special handling - in case the server > + returned an empty array of error strings (the value of the > + "error" object in JSON is equal to an empty array), we should > + ignore this error at this level, since this means the missing > + key (this problem is handled at a higher level), but if the > + error object contains anything other than empty array, then > + we need to print the error message to the log: > + */ > + if (http_code == 404) > + { > + const char *err; > + int err_len; > + if (json_get_object_key(res, res + strlen(res), > + "errors", &err, &err_len) == JSV_ARRAY) > + { > + const char *ev; > + int ev_len; > + if (json_get_array_item(err, err + err_len, 0, &ev, &ev_len) == > + JSV_NOTHING) > + { > + *response = std::string(""); > + is_error = false; > + } > + } > + } > + if (is_error) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Hashicorp server error: %d, response: %s", > + ME_ERROR_LOG_ONLY | ME_WARNING, http_code, res); > + } > + } > + return is_error ? OPERATION_ERROR : OPERATION_OK; > +} > + > +#define max_url_size 32768 uppercase > + > +static size_t vault_url_len; > + > +static unsigned int get_latest_version (unsigned int key_id) > +{ > + std::string response_str; > + const char *response; > + const char *js, *ver; > + int js_len, ver_len; > + size_t response_len; > +/* > + Checking the maximum allowable length to protect > + against allocating too much memory on the stack: > +*/ > + if (vault_url_len > max_url_size) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Maximum allowed vault URL length exceeded", > + 0); > + return true; > + } > +/* > + Maximum buffer length = url length plus 20 characters of > + a 64-bit unsigned integer, plus a slash character, plus > + a length of the "data/" string and plus a zero byte: > +*/ please, indent all comments properly (here and above). now they'll cause diffs to be rather unreadable. > + size_t buf_len = vault_url_len + (20 + 5 + 2); > + char *url = (char *) alloca(buf_len); > + if (vault_url_len && vault_url[vault_url_len - 1] != '/') > + { > + snprintf(url, buf_len, "%s/data/%u", vault_url, key_id); > + } > + else > + { > + snprintf(url, buf_len, "%sdata/%u", vault_url, key_id); > + } same comment as above about run-time checks. You can make sure that vault_url_len is within limits and vault_url doesn't end with a slash in hashicorp_key_management_plugin_init > + bool use_cache= caching_enabled && use_cache_on_timeout; > + int rc; > + if ((rc= curl_run(url, &response_str, use_cache)) != OPERATION_OK) would be good to have some cache expiration time and skip curl completely if the cached value is sufficiently new. e.g. instead of caching_enabled you can have caching_lifetime in seconds. Where 0 means "no caching". > + { > + if (rc == OPERATION_TIMEOUT) > + { > + unsigned int ver = cache_get_version(key_id); > + if (ver != ENCRYPTION_KEY_VERSION_INVALID) > + { > + return ver; > + } > + } > + my_printf_error(ER_UNKNOWN_ERROR, "Unable to get key data", 0); > + return ENCRYPTION_KEY_VERSION_INVALID; > + } > + response = response_str.c_str(); > + response_len = response_str.size(); > + /* > + If the key is not found, this is not considered a fatal error, > + but we need to add an informational message to the log: > + */ > + if (response_len == 0) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Key not found", always specify the plugin name in any my_printf_error. Having only 2020-06-29 1:41:23 28 [ERRROR] Key not found is not very helpful. > + ME_ERROR_LOG_ONLY | ME_NOTE); > + return ENCRYPTION_KEY_VERSION_INVALID; > + } > + if (json_get_object_key(response, response + response_len, "data", > + &js, &js_len) != JSV_OBJECT) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Unable to get data object (http response is: %s)", > + 0, response); > + return ENCRYPTION_KEY_VERSION_INVALID; > + } > + if (json_get_object_key(js, js + js_len, "metadata", > + &ver, &ver_len) != JSV_OBJECT) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Unable to get metadata object (http response is: %s)", > + 0, response); > + return ENCRYPTION_KEY_VERSION_INVALID; > + } > + if (json_get_object_key(ver, ver + ver_len, "version", > + &ver, &ver_len) != JSV_NUMBER) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Unable to get version number (http response is: %s)", > + 0, response); > + return ENCRYPTION_KEY_VERSION_INVALID; > + } > + /* > + Here we may use atoi, but it has undefined overflow behavior > + and does not work with unsigned integers, so use strtoul: > + */ No need to explain why you didn't use atoi, this is not helpful. Or, if you'd like to explain, explain why you didn't use atol, atof, strtod, and sscanf :) > + errno = 0; > + unsigned long version = strtoul(ver, NULL, 10); > + /* > + If the result of the conversion does not fit into an > + unsigned integer or if an error (for example, overflow) > + occurred during the conversion, then we need to return > + an error: > + */ thanks, but do not say in a comment what the code does. explain *why* it does it. But in this case both are very clear, so just remove the comment above. Thanks. > + if (version > UINT_MAX || (version == ULONG_MAX && errno)) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Integer conversion error (for version number)", no "http response is: %s" here? > + 0); > + return ENCRYPTION_KEY_VERSION_INVALID; > + } > + unsigned int int_ver= (unsigned int) version; > + if (!caching_enabled) > + { > + return int_ver; > + } > + if (json_get_object_key(js, js + js_len, "data", > + &js, &js_len) != JSV_OBJECT) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Unable to get second-level data object " > + "(http response is: %s)", > + 0, response); > + return ENCRYPTION_KEY_VERSION_INVALID; > + } > + const char* key; > + int key_len; > + if (json_get_object_key(js, js + js_len, "data", > + &key, &key_len) != JSV_STRING) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Unable to get data string (http response is: %s)", > + 0, response); > + return ENCRYPTION_KEY_VERSION_INVALID; > + } > + KEY_INFO info; > + if (((unsigned int) key_len >> 1) > sizeof(info.data)) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Encryption key data is too long", > + ME_ERROR_LOG_ONLY | ME_NOTE); > + return ENCRYPTION_KEY_VERSION_INVALID; > + } > + unsigned int length = 0; > + while (key_len >= 2) > + { > + int c1 = key[0]; > + int c2 = key[1]; > + if (! isxdigit(c1) || ! isxdigit(c2)) > + { > + break; > + } > + c1 -= '0'; > + c2 -= '0'; > + if (c1 > 9) > + { > + c1 -= 'A' - '0'; > + if (c1 > 15) > + { > + c1 -= 'a' - 'A'; > + } > + } > + if (c2 > 9) > + { > + c2 -= 'A' - '0'; > + if (c1 > 15) > + { > + c2 -= 'a' - 'A'; > + } > + } please write a macro or an inline function for that. don't repeat the same code twice. E.g. static inline int c2xdigit(char *c) { return c >= 'a' ? c-'a'+10 : c >= 'A' ? c-'A'+10 : c - '0'; } > + info.data[length++] = (c1 << 4) + c2; > + key += 2; > + key_len -= 2; > + } > + if (key_len) > + { > + if (key_len != 1) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Syntax error - the key data should contain only " > + "hexadecimal digits", > + 0); > + } > + else > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Syntax error - extra character in the key data", > + 0); > + } > + return ENCRYPTION_KEY_VERSION_INVALID; > + } > + info.key_id = key_id; > + info.key_version = int_ver; > + info.length = length; > + cache_add(key_id, int_ver, &info); > + return int_ver; > +} > + > +static unsigned int get_key_from_vault (unsigned int key_id, > + unsigned int key_version, > + unsigned char *dstbuf, > + unsigned int *buflen) > +{ > + if (caching_enabled && > + cache_get(key_id, key_version, dstbuf, buflen) != > + ENCRYPTION_KEY_VERSION_INVALID) > + { > + return 0; > + } > + std::string response_str; > + const char *response; > + const char *js, *key; > + int js_len, key_len; > + size_t response_len; > +/* > + Checking the maximum allowable length to protect > + against allocating too much memory on the stack: > +*/ > + if (vault_url_len > max_url_size) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Maximum allowed vault URL length exceeded", > + 0); > + return true; > + } > +/* > + Maximum buffer length = url length plus 40 characters of the > + two 64-bit unsigned integers, plus a slash character, plus a > + question mark, plus length of the "data/" and the "version=" > + strings and plus a zero byte: > +*/ > + size_t buf_len = vault_url_len + (40 + 5 + 8 + 3); > + char *url = (char *) alloca(buf_len); > + if (vault_url_len && vault_url[vault_url_len - 1] != '/') > + { > + if (key_version != ENCRYPTION_KEY_VERSION_INVALID) > + snprintf(url, buf_len, "%s/data/%u?version=%u", vault_url, key_id, > + key_version); > + else > + snprintf(url, buf_len, "%s/data/%u", vault_url, key_id); > + } > + else > + { > + if (key_version != ENCRYPTION_KEY_VERSION_INVALID) > + snprintf(url, buf_len, "%sdata/%u?version=%u", vault_url, key_id, > + key_version); > + else > + snprintf(url, buf_len, "%sdata/%u", vault_url, key_id); > + } > + if (curl_run(url, &response_str, false) != OPERATION_OK) > + { > + my_printf_error(ER_UNKNOWN_ERROR, "Unable to get key data", 0); > + return ENCRYPTION_KEY_VERSION_INVALID; > + } > + response = response_str.c_str(); > + response_len = response_str.size(); > + /* > + If the key is not found, this is not considered a fatal error, > + but we need to add an informational message to the log: > + */ > + if (response_len == 0) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Key not found", > + ME_ERROR_LOG_ONLY | ME_NOTE); > + return ENCRYPTION_KEY_VERSION_INVALID; > + } > + if (json_get_object_key(response, response + response_len, "data", > + &js, &js_len) != JSV_OBJECT) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Unable to get data object (http response is: %s)", > + 0, response); > + return ENCRYPTION_KEY_VERSION_INVALID; > + } > +#ifndef NDEBUG > + unsigned long version; > +#else > + unsigned long version= key_version; > + if (caching_enabled && > + key_version == ENCRYPTION_KEY_VERSION_INVALID) > +#endif > + { > + const char *ver; > + int ver_len; > + if (json_get_object_key(js, js + js_len, "metadata", > + &ver, &ver_len) != JSV_OBJECT) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Unable to get metadata object (http response is: %s)", > + 0, response); > + return ENCRYPTION_KEY_VERSION_INVALID; > + } > + if (json_get_object_key(ver, ver + ver_len, "version", > + &ver, &ver_len) != JSV_NUMBER) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Unable to get version number (http response is: %s)", > + 0, response); > + return ENCRYPTION_KEY_VERSION_INVALID; > + } > + /* > + Here we may use atoi, but it has undefined overflow behavior > + and does not work with unsigned integers, so use strtoul: > + */ > + errno = 0; > + version = strtoul(ver, NULL, 10); > + /* > + If the result of the conversion does not fit into an > + unsigned integer or if an error (for example, overflow) > + occurred during the conversion, then we need to return > + an error: > + */ > + if (version > UINT_MAX || (version == ULONG_MAX && errno)) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Integer conversion error (for version number)", > + 0); > + return ENCRYPTION_KEY_VERSION_INVALID; > + } > + } > +#ifndef NDEBUG > + /* > + An internal check that is needed only for debugging the plugin > + operation - in order to ensure that we get from the Hashicorp Vault > + server exactly the version of the key that is needed: > + */ > + if (key_version != ENCRYPTION_KEY_VERSION_INVALID && > + key_version != version) > + { > + my_printf_error(ER_UNKNOWN_ERROR, "Key version mismatch", 0); > + return ENCRYPTION_KEY_VERSION_INVALID; > + } > +#endif > + if (json_get_object_key(js, js + js_len, "data", > + &js, &js_len) != JSV_OBJECT) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Unable to get second-level data object " > + "(http response is: %s)", > + 0, response); > + return ENCRYPTION_KEY_VERSION_INVALID; > + } > + if (json_get_object_key(js, js + js_len, "data", > + &key, &key_len) != JSV_STRING) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Unable to get data string (http response is: %s)", > + 0, response); > + return ENCRYPTION_KEY_VERSION_INVALID; > + } > + unsigned int max_length = *buflen; > + if (((unsigned int) key_len >> 1) > max_length) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Encryption key buffer too small", > + ME_ERROR_LOG_ONLY | ME_NOTE); > + *buflen = key_len; > + return ENCRYPTION_KEY_BUFFER_TOO_SMALL; > + } > + unsigned int length = 0; > + while (key_len >= 2) > + { > + int c1 = key[0]; > + int c2 = key[1]; > + if (! isxdigit(c1) || ! isxdigit(c2)) > + { > + break; > + } > + c1 -= '0'; > + c2 -= '0'; > + if (c1 > 9) > + { > + c1 -= 'A' - '0'; > + if (c1 > 15) > + { > + c1 -= 'a' - 'A'; > + } > + } > + if (c2 > 9) > + { > + c2 -= 'A' - '0'; > + if (c1 > 15) > + { > + c2 -= 'a' - 'A'; > + } > + } > + dstbuf[length++] = (c1 << 4) + c2; > + key += 2; > + key_len -= 2; > + } > + *buflen = length; > + if (key_len) > + { > + if (key_len != 1) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Syntax error - the key data should contain only " > + "hexadecimal digits", > + 0); > + } > + else > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "Syntax error - extra character in the key data", > + 0); > + } > + return ENCRYPTION_KEY_VERSION_INVALID; > + } > + if (caching_enabled) > + { > + unsigned int int_ver= (unsigned int) version; > + KEY_INFO info; > + info.key_id = key_id; > + info.key_version = int_ver; > + info.length = length; > + memcpy(info.data, dstbuf, length); > + cache_add(key_id, int_ver, &info); > + } > + return 0; 1. you don't update the latest_version_cache here. 2. this function has lots of code essentially the same as in the get_latest_version. Please reuse, don't copy-paste. > +} > + > +struct st_mariadb_encryption hashicorp_key_management_plugin= { > + MariaDB_ENCRYPTION_INTERFACE_VERSION, > + get_latest_version, > + get_key_from_vault, > + 0, 0, 0, 0, 0 > +}; > + > +#ifdef _MSC_VER > + > +static int setenv(const char *name, const char *value, int overwrite) > +{ > + if (!overwrite) > + { > + size_t len= 0; > + int rc= getenv_s(&len, NULL, 0, name); > + if (rc) > + { > + return rc; > + } > + if (len) > + { > + errno = EINVAL; > + return EINVAL; > + } > + } > + return _putenv_s(name, value); > +} > + > +#endif > + > +static char *local_token= NULL; > + > +static int hashicorp_key_management_plugin_init(void *p) > +{ > + char *token_env= getenv("VAULT_TOKEN"); > + token_len = strlen(token); > + if (token_len == 0) { > + if (token_env) > + { > + token_len = strlen(token_env); > + if (token_len != 0) > + { > + token = (char *) malloc(token_len + 1); > + memcpy(token, token_env, token_len + 1); > + local_token = token; > + } > + } > + if (token_len == 0) { > + my_printf_error(ER_UNKNOWN_ERROR, > + "The --hashicorp-key-management-token option value " > + "or the value of the corresponding parameter in the " > + "configuration file must be specified, otherwise the " > + "VAULT_TOKEN environment variable must be set", > + 0); > + return 1; > + } > + } > + else > + { > + /* > + If the VAULT_TOKEN environment variable is not set or > + is not equal to the value of the token parameter, then > + we must set (overwrite) it for correct operation of > + the mariabackup: > + */ > + bool not_equal= token_env != NULL && strcmp(token_env, token) != 0; > + if (token_env == NULL || not_equal) > + { > + setenv("VAULT_TOKEN", token, 1); > + if (not_equal) > + { > + my_printf_error(ER_UNKNOWN_ERROR, > + "The --hashicorp-key-management-token option value " > + "or the value of the corresponding parameter is not " > + "equal to the value of the VAULT_TOKEN environment " > + "variable", > + ME_ERROR_LOG_ONLY | ME_WARNING); > + } what mariabackup child processes need it? > + } > + } > + curl_global_init(CURL_GLOBAL_ALL); > + vault_url_len = strlen(vault_url); > + return 0; > +} > + > +static int hashicorp_key_management_plugin_deinit(void *p) > +{ > + latest_version_cache.clear(); > + key_info_cache.clear(); > + curl_global_cleanup(); > + if (local_token) > + { > + free(local_token); > + } > + return 0; > +} > + > +/* > + Plugin library descriptor > +*/ > +maria_declare_plugin(hashicorp_key_management) > +{ > + MariaDB_ENCRYPTION_PLUGIN, > + &hashicorp_key_management_plugin, > + "hashicorp_key_management", > + "MariaDB Corporation", > + "HashiCorp Vault key management plugin", > + PLUGIN_LICENSE_GPL, > + hashicorp_key_management_plugin_init, > + hashicorp_key_management_plugin_deinit, > + 0x0102 /* 1.02 */, > + NULL, /* status variables */ > + settings, > + "1.02", > + MariaDB_PLUGIN_MATURITY_ALPHA > +} > +maria_declare_plugin_end; > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] Weekly Report
by Kartik Soneji 29 Jun '20

29 Jun '20

Hi, This week - I really struggled with the ABI check failing with the error: CMake Error at cmake/do_abi_check.cmake:82 (MESSAGE): ABI check found difference between /workspace/server/include/mysql/plugin_audit.h.pp and /workspace/server/abi_check.out Running make abi_update fixed the issue. - Using services means copying a large portion of the headers into the server. This may cause licensing issues. Overview: | Library | License | | AZlib | Zlib | | BZip2 | BSD-like | | LZ4 | Library files under BSD 2-Clause and other under GPL v2 | | LZMA | Public Domain | | LZO | GPL v2+ | | Snappy | Custom license | | Xpress | ??? | | Zlib | Zlib | | ZStandard | Dual: BSD + GPL v2 | - Tested the LZMA service by calling the function from a dynamic plugin as per step 2 of MDEV-22895. - Started working on making the service compatible with static plugins as per step 3 of MDEV-22895. Next week - Finish making the service compatible with static plugins. - Work on MDEV-22895. Thanks, Kartik

1 0

[Maria-developers] Progress report for MDEV-12326
by Mohammed Hammaad Mateen 28 Jun '20

28 Jun '20

Greetings, Hope you are safe and doing great, This post describes the things I've done during 22-28 June for Coding phase-1 under the mentor-ship of Sergei Golubchik and Oleksandr Byelkin for GSoC-20 to implement MDEV-12326. The tasks taken up for this week was to work towards validating the fields of the return list, requested by the client when it uses *multi-table DELETE....RETURNING *or* multi-table UPDATE....RETURNING*. If an invalid column name is requested in the return list then it should result in displaying: *ERROR 1054 (42S22): Unknown column '<invalid column>' in 'field list' *. Things I learnt while implementing this was, firstly that *setup_fields* and *setup_wild* functions validate the *returning_item_list* i.e. column names, expressions, etc. in *RETURNING* statement and understood that validation is done implicitly. Moreover, *setup_returning_fields* method could also be used interchangeably. Secondly, if we have to completely run a .test file that has many failing test cases then we should provide --force option twice, as in*:* *./mtr <file_name> --force --force. * I've also written and recorded the result for the above usecase, for *multi-table DELETE.... RETURNING *and will also do the same for *multi-table UPDATE.... RETURNING v*ery soon. All in all coding phase-1 was really informative and I would like to thank *MariaDB Foundation* for having me. All the code is uploaded up to date in these remote repositories, please feel free to comment. MULTI-TABLE UPDATE RETURNING: https://github.com/MariaDB/server/compare/10.5...MohammedHMateen:MDEV-12326… MULTI-TABLE DELETE RETURNING: https://github.com/MariaDB/server/compare/10.5...MohammedHMateen:MDEV-12326… Thank you Regards, Mohammed Hammaad Mateen

1 0

Re: [Maria-developers] 97f7d4a9b4d: MDEV-22726: Add check that one can't change general or slow log to a
by Sergei Golubchik 26 Jun '20

26 Jun '20

Hi, Vlad! On Jun 26, Vlad Lesin wrote: > >> > >> + if (create_info->db_type == maria_hton && > >> + create_info->transactional != HA_CHOICE_NO) > >> + { > >> + my_error(ER_TRANSACTIONAL_ARIA_LOG_ENGINE, MYF(0)); > >> + DBUG_RETURN(true); > >> + } > > > > What about InnoDB? Or RocksDB? Or any other transactional engine? > > There is flag HTON_SUPPORT_LOG_TABLES, which is checked earlier in > mysql_alter_table(). It's not checked on CREATE, only on ALTER, so one can easily bypass this check. And, anyway, why an engine would not support log tables? Only because it's transactional? Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 97f7d4a9b4d: MDEV-22726: Add check that one can't change general or slow log to a
by Sergei Golubchik 25 Jun '20

25 Jun '20

Hi, Vlad! On Jun 25, Vlad Lesin wrote: > revision-id: 97f7d4a9b4d (mariadb-10.2.31-290-g97f7d4a9b4d) > parent(s): e3104c4a8c6 > author: Vlad Lesin <vlad_lesin(a)mail.ru> > committer: Vlad Lesin <vlad_lesin(a)mail.ru> > timestamp: 2020-06-24 14:37:54 +0300 > message: > > MDEV-22726: Add check that one can't change general or slow log to a > transactional engine the commit subject should be in one line (and I personally only put one space between the MDEV number and the text, not a colon, because the subject length is quite limited to waste it on pretty delimiters that don't add any meaningful information) > diff --git a/sql/share/errmsg-utf8.txt b/sql/share/errmsg-utf8.txt > index ff495f69a4f..8db992c7501 100644 > --- a/sql/share/errmsg-utf8.txt > +++ b/sql/share/errmsg-utf8.txt > @@ -7731,3 +7731,5 @@ ER_GEOJSON_EMPTY_COORDINATES > > ER_MYROCKS_CANT_NOPAD_COLLATION > eng "MyRocks doesn't currently support collations with \"No pad\" attribute." > +ER_TRANSACTIONAL_ARIA_LOG_ENGINE > + eng "Only non-transactional Aria table can be used for logging" No, sorry, you cannot add new error messages in 10.2. Or 10.3 or 10.4. Because adding a new error message to anything below 10.5 would mean that 10.5 error numbers will shift and we aren't allowed to do it for compatibility reasons. The typical workaround is to use my_print_error(ER_something_else, "Only non-transactional Aria table can be used for logging"); (with the appropriate error number, MYF(0), etc) > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index 7cce3bcc323..b3a600eec36 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -8925,6 +8925,13 @@ bool mysql_alter_table(THD *thd,char *new_db, char *new_name, > DBUG_RETURN(true); > } > > + if (create_info->db_type == maria_hton && > + create_info->transactional != HA_CHOICE_NO) > + { > + my_error(ER_TRANSACTIONAL_ARIA_LOG_ENGINE, MYF(0)); > + DBUG_RETURN(true); > + } What about InnoDB? Or RocksDB? Or any other transactional engine? > #ifdef WITH_PARTITION_STORAGE_ENGINE > if (alter_info->flags & Alter_info::ALTER_PARTITION) > { > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 9bdf35e90f3: MDEV-18215: mariabackup does not report unknown command line options
by Sergei Golubchik 25 Jun '20

25 Jun '20

Hi, Vlad! Some comments about your command-line option handling code On Jun 25, Vlad Lesin wrote: > revision-id: 9bdf35e90f3 (mariadb-10.4.11-242-g9bdf35e90f3) > parent(s): ceaa8b647a1 > author: Vlad Lesin <vlad_lesin(a)mail.ru> > committer: Vlad Lesin <vlad_lesin(a)mail.ru> > timestamp: 2020-06-14 13:23:07 +0300 > message: > > MDEV-18215: mariabackup does not report unknown command line options > MDEV-21298: mariabackup doesn't read from the [mariadbd] and [mariadbd-X.Y] > server option groups from configuration files > MDEV-21301: mariabackup doesn't read [mariadb-backup] option group in > configuration file > > All three issues require to change the same code, that is why their > fixes are joined in one commit. > > The fix is in invoking load_defaults_or_exit() and handle_options() for > backup-specific groups separately from client-server groups to let the last > handle_options() call fail on unknown backup-specific options. > > The order of options procesing is the following: > 1) Load server groups and process server options, ignore unknown > options > 2) Load client groups and process client options, ignore unknown > options > 3) Load backup groups and process client-server options, exit on > unknown option > 4) Process --mysqld-args command line options, ignore unknown options > > New global flag my_handle_options_init_variables was added to have > ability to invoke handle_options() for the same allowed options set > several times without re-initialising previously set option values. > > --password value destroying is moved from option processing callback to > mariabackup's handle_options() function to have ability to invoke server's > handle_options() several times for the same possible allowed options > set. > > Galera invokes wsrep_sst_mariabackup.sh with mysqld command line > options to configure mariabackup as close to the server as possible. > It is not known what server options are supported by mariabackup when the > script is invoked. That is why new mariabackup option "--mysqld-args" is added, > all unknown options that follow this option will be silently ignored. > > wsrep_sst_mariabackup.sh was also changed to: > - use "--mysqld-args" mariabackup option to pass mysqld options, > - remove deprecated innobackupex mode, > - remove unsupported mariabackup options: > --encrypt > --encrypt-key > --rebuild-indexes > --rebuild-threads > > diff --git a/extra/mariabackup/xtrabackup.cc b/extra/mariabackup/xtrabackup.cc > index 852ef4efe56..19058398258 100644 > --- a/extra/mariabackup/xtrabackup.cc > +++ b/extra/mariabackup/xtrabackup.cc > @@ -5990,11 +5977,23 @@ void setup_error_messages() > die("could not initialize error messages"); > } > > -void > -handle_options(int argc, char **argv, char ***argv_client, char ***argv_server) > +/** Handle mariabackup options. The options are handled with the following > +order: > + > +1) Load server groups and process server options, ignore unknown options > +2) Load client groups and process client options, ignore unknown options > +3) Load backup groups and process client-server options, exit on unknown option > +4) Process --mysqld-args options, ignore unknown options > + > +@param[in] argc arguments count > +@param[in] argv arguments array > +@param[out] argv_server server options including loaded from server groups > +@param[out] argv_client client options including loaded from client groups > +@param[out] argv_backup backup options including loaded from backup groups */ > +void handle_options(int argc, char **argv, char ***argv_server, > + char ***argv_client, char ***argv_backup) could you please rename this function? handle_options() is part of my_getopt api, the main function to parse the argv array of options. this one has a different prototype, so C++ can distinguish between them, but it's very confusing, like having your own strcmp(int, char*, void*). > { > /* Setup some variables for Innodb.*/ > srv_operation = SRV_OPERATION_RESTORE; > > files_charset_info = &my_charset_utf8_general_ci; > @@ -6021,49 +6020,64 @@ handle_options(int argc, char **argv, char ***argv_client, char ***argv_server) > bool prepare = false; > > char conf_file[FN_REFLEN]; > - int argc_client = argc; > - int argc_server = argc; > - > - /* scan options for group and config file to load defaults from */ > - for (i = 1; i < argc; i++) { > - > - char *optend = strcend(argv[i], '='); > - > - if (strncmp(argv[i], "--defaults-group", > - optend - argv[i]) == 0) { > - defaults_group = optend + 1; > - append_defaults_group(defaults_group, > - xb_server_default_groups, > - array_elements(xb_server_default_groups)); > - } > > - if (strncmp(argv[i], "--login-path", > - optend - argv[i]) == 0) { > - append_defaults_group(optend + 1, > - xb_client_default_groups, > - array_elements(xb_client_default_groups)); > - } > + // array_elements() will not work for load_defaults, as it is defined > + // as external symbol, so let's use dynamic array to have ability to > + // add new server default groups > + std::vector<const char *> server_default_groups; > > - if (!strncmp(argv[i], "--prepare", > - optend - argv[i])) { > - prepare = true; > - } > + for (const char **default_group= load_default_groups; *default_group; > + ++default_group) > + server_default_groups.push_back(*default_group); > > - if (!strncmp(argv[i], "--apply-log", > - optend - argv[i])) { > - prepare = true; > - } > + std::vector<char *> mysqld_args; > + std::vector<char *> mariabackup_args; > + mysqld_args.push_back(argv[0]); > + mariabackup_args.push_back(argv[0]); > > - if (!strncmp(argv[i], "--target-dir", > - optend - argv[i]) && *optend) { > - target_dir = optend + 1; > - } > + /* scan options for group and config file to load defaults from */ > + for (i= 1; i < argc; i++) > + { > + char *optend= strcend(argv[i], '='); > + if (mysqld_args.size() > 1 || > + strncmp(argv[i], "--mysqld-args", optend - argv[i]) == 0) > + { > + mysqld_args.push_back(argv[i]); > + continue; > + } > + else > + mariabackup_args.push_back(argv[i]); > > - if (!*optend && argv[i][0] != '-') { > - target_dir = argv[i]; > - } > - } > + if (strncmp(argv[i], "--defaults-group", optend - argv[i]) == 0) > + { > + defaults_group= optend + 1; > + server_default_groups.push_back(defaults_group); > + } > + else if (strncmp(argv[i], "--login-path", optend - argv[i]) == 0) > + { > + append_defaults_group(optend + 1, xb_client_default_groups, > + array_elements(xb_client_default_groups)); > + } > + else if (!strncmp(argv[i], "--prepare", optend - argv[i])) > + { > + prepare= true; > + } > + else if (!strncmp(argv[i], "--apply-log", optend - argv[i])) > + { > + prepare= true; > + } > + else if (!strncmp(argv[i], "--target-dir", optend - argv[i]) && > + *optend) > + { > + target_dir= optend + 1; > + } I don't understand. Why is this manual parsing of options if you use my_getopt anyway? > + else if (!*optend && argv[i][0] != '-') > + { > + target_dir= argv[i]; > + } > + } > > + server_default_groups.push_back(NULL); > snprintf(conf_file, sizeof(conf_file), "my"); > > if (prepare && target_dir) { > @@ -6102,7 +6122,6 @@ handle_options(int argc, char **argv, char ***argv_client, char ***argv_server) > optp->u_max_value = (G_PTR *) &global_max_value; what is that??? > } > > - > /* Throw a descriptive error if --defaults-file or --defaults-extra-file > is not the first command line argument */ and what is that??? You know, if there are ~10 programs using load_defaults/my_getopt and not a single one of them is doing this, perhaps mariabackup shouldn't be doing it either? > for (int i = 2 ; i < argc ; i++) { > @@ -6143,18 +6163,76 @@ handle_options(int argc, char **argv, char ***argv_client, char ***argv_server) > xb_client_options, xb_get_one_option))) > exit(ho_error); > > + /* 3) Load backup groups and process client-server options, exit on > + unknown option */ > + > + load_defaults_or_exit(conf_file, backup_default_groups, &argc_backup, > + argv_backup); > + for (n= 0; (*argv_backup)[n]; n++) > + { > + }; > + argc_backup= n; > + > + my_handle_options_init_variables = FALSE; > + > + if (argc_backup > 0 && > + (ho_error= handle_options(&argc_backup, argv_backup, > + xb_server_options, xb_get_one_option))) > + exit(ho_error); I think this whole approach is wrong. And my_handle_options_init_variables highlights it. There should be only one invocation of handle_options(), otherwise you won't get the correct behavior of what options override what. You can call handle_options() separately for server and client options, because those are disjoint sets of options. But you cannot call it twice for xb_server_options. Now you call handle_options 5 times. I suggest to rewrite the code to do it only twice. As: read *all* groups with load_defaults_or_exit(), and read *all* options with one handle_options() call. This will be the actual parsing and setting values of variables, ignoring unknown options. But you still want to issue an error for unknown options, so truncate the list of command-line options before --mysqld_args, read only mariabackup groups, call handle_options again, erroring out on unknown options. this should not set any values, it's just for an error, so before that you reset all pointers to variables - just like you do now for --maximum. Because this approach parses all options at once, you'll get the same behavior regarding what options override what, as any other tool that uses my_getopt. > + > + /* Add back the program name handle_options removes */ > + ++argc_backup; > + --(*argv_backup); > + > + if (innobackupex_mode && argc_backup > 0 && > + !ibx_handle_options(&argc_backup, argv_backup)) > + exit(EXIT_FAILURE); > + > + my_getopt_skip_unknown = FALSE; > + > + if (argc_backup > 0 && > + (ho_error= handle_options(&argc_backup, argv_backup, > + xb_client_options, xb_get_one_option))) > + exit(ho_error); > + > + if (opt_password) > + { > + char *argument= opt_password; > + char *start= argument; > + opt_password= my_strdup(opt_password, MYF(MY_FAE)); > + while (*argument) > + *argument++= 'x'; // Destroy argument > + if (*start) > + start[1]= 0; > + } > + > + /* 4) Process --mysqld-args options, ignore unknown options */ > + > + my_getopt_skip_unknown = TRUE; > + > + int argc_mysqld = static_cast<int>(mysqld_args.size()); > + if (argc_mysqld > 1) > + { > + char **argv_mysqld= &mysqld_args[0]; > + if ((ho_error= handle_options(&argc_mysqld, &argv_mysqld, > + xb_server_options, xb_get_one_option))) > + exit(ho_error); > + } > + > + my_handle_options_init_variables = TRUE; > + > /* Reject command line arguments that don't look like options, i.e. are > not of the form '-X' (single-character options) or '--option' (long > options) */ > - for (int i = 0 ; i < argc_client ; i++) { > - const char * const opt = (*argv_client)[i]; > + for (int i = 0 ; i < argc_backup ; i++) { > + const char * const opt = (*argv_backup)[i]; > > if (strncmp(opt, "--", 2) && > !(strlen(opt) == 2 && opt[0] == '-')) { > bool server_option = true; > > - for (int j = 0; j < argc_server; j++) { > - if (opt == (*argv_server)[j]) { > + for (int j = 0; j < argc_backup; j++) { > + if (opt == (*argv_backup)[j]) { > server_option = false; > break; > } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] 258a0fb47b7: MDEV-22951: rpl.rpl_slave_alias_replica failed in buildbot with wrong result
by sujatha 24 Jun '20

24 Jun '20

revision-id: 258a0fb47b74363d0919d6a18b9b2e67d45d4cf7 (mariadb-10.5.2-490-g258a0fb47b7) parent(s): d1bb7f9143ceafe9cbdc1e53c4ffd4abe232cb97 author: Sujatha committer: Sujatha timestamp: 2020-06-24 15:33:45 +0530 message: MDEV-22951: rpl.rpl_slave_alias_replica failed in buildbot with wrong result Problem: ======== rpl.rpl_slave_alias_replica test fails sporadically on build bot as shown below. rpl.rpl_slave_alias_replica 'innodb,stmt' w4 [ fail ] +++ /usr/local/mariadb-10.5.4-linux-x86_64/mysql-test/suite/rpl/r/ rpl_slave_alias_replica.reject 2020-06-12 09:52:25.191267945 +0000 @@ -22,6 +22,7 @@ SHOW REPLICA HOSTS; Server_id Host Port Master_id 2 127.0.0.1 SLAVE_PORT 1 +2 127.0.0.1 SLAVE_PORT 1 Analysis: ======== The SHOW REPLICA HOSTS command reports an additional row for the same server occasionally. This happens because of zombie dump threads on the master. Error log from master also confirms that "A slave with the same server_uuid/server_id as this slave has connected to the master". Since SHOW REPLICA HOSTS goes through the list of dump threads and reports the corresponding hosts,the same host is reported twice. Fix: === Instead of capturing the SHOW REPLICA HOSTS command in result file use wait_show_condition.inc. This will ensure that SHOW REPLICA HOSTS command fetches slave host details. --- mysql-test/suite/rpl/r/rpl_slave_alias_replica.result | 5 ++--- mysql-test/suite/rpl/t/rpl_slave_alias_replica.test | 10 ++++++++-- 2 files changed, 10 insertions(+), 5 deletions(-) diff --git a/mysql-test/suite/rpl/r/rpl_slave_alias_replica.result b/mysql-test/suite/rpl/r/rpl_slave_alias_replica.result index 8262cf8212a..2a49e80075f 100644 --- a/mysql-test/suite/rpl/r/rpl_slave_alias_replica.result +++ b/mysql-test/suite/rpl/r/rpl_slave_alias_replica.result @@ -18,10 +18,9 @@ START REPLICA; include/wait_for_slave_io_to_start.inc include/wait_for_slave_sql_to_start.inc connection master; +connection slave; +connection master; "Command: SHOW SLAVE HOSTS --> SHOW REPLICA HOSTS" -SHOW REPLICA HOSTS; -Server_id Host Port Master_id -2 127.0.0.1 SLAVE_PORT 1 DROP USER 'repl_user'; connection slave; "Command: SHOW SLAVE IO/SQL THREAD --> SHOW REPLICA IO/SQL THREAD" diff --git a/mysql-test/suite/rpl/t/rpl_slave_alias_replica.test b/mysql-test/suite/rpl/t/rpl_slave_alias_replica.test index 8f2cd1a4eb3..324821a325f 100644 --- a/mysql-test/suite/rpl/t/rpl_slave_alias_replica.test +++ b/mysql-test/suite/rpl/t/rpl_slave_alias_replica.test @@ -45,10 +45,16 @@ START REPLICA; --source include/wait_for_slave_io_to_start.inc --source include/wait_for_slave_sql_to_start.inc +--connection master +--sync_slave_with_master + --connection master --echo "Command: SHOW SLAVE HOSTS --> SHOW REPLICA HOSTS" ---replace_result $SLAVE_MYPORT SLAVE_PORT -SHOW REPLICA HOSTS; +let $show_statement= SHOW REPLICA HOSTS; +let $field= Server_id; +# Slave's server_id 2 +let $condition= ='2'; +source include/wait_show_condition.inc; DROP USER 'repl_user'; --sync_slave_with_master

1 0

[Maria-developers] Progress Report GSoC 2020 (15-20 June)
by Anton Mikhaylenko 24 Jun '20

24 Jun '20

Good day, everybody. In this post, I want to describe what I did this week, June 15-20. I`m working with threadpool concurrency. Have made some fixes. Change type of adjustment interval. Used the existing time interval to adjust the throttling. Regards, Anton MIkhailenko

1 0

[Maria-developers] Review for: MDEV-17399 Add support for JSON_TABLE, part #6
by Sergey Petrunia 22 Jun '20

22 Jun '20

Hi Alexey, > revision-id: 0d342ab173cdbf5d35c5f2833c66b6004b9e908e (mariadb-10.5.2-330-g0d342ab173c) > parent(s): b1ab211dee599eabd9a5b886fafa3adea29ae041 > author: Alexey Botchkov > committer: Alexey Botchkov > timestamp: 2020-06-08 14:51:04 +0400 > message: > > MDEV-17399 Add support for JSON_TABLE. > > ha_json_table handler implemented. > JSON_TABLE() added to SQL syntax. > First, some general input: == Assertion with Prepared Statements == prepare s from 'select * from json_table(?, \'$[*]\' columns( color varchar(100) path \'$.color\', price text path \'$.price\', seq for ordinality ) ) as T order by color desc; ' mysqld: /home/psergey/dev-git2/10.5-hf-review/sql/item.h:1002: virtual bool Item::fix_fields(THD*, Item**): Assertion `basic_const_item()' failed. == Default value for TEXT == Default value for VARCHAR(100) is SQL NULL, while for TEXT it is empty string. Is this intentional? If yes, I would like to see the spec of the logic behind this. select * from json_table('[{"color": "blue", "price": 50}, {"color": "red", "price": 100}]', '$[*]' columns(color varchar(100) path '$.nonexistent', seq for ordinality ) ) as T; +-------+------+ | color | seq | +-------+------+ | NULL | 1 | | NULL | 2 | +-------+------+ select * from json_table('[{"color": "blue", "price": 50}, {"color": "red", "price": 100}]', '$[*]' columns(color text path '$.nonexistent', seq for ordinality ) ) as T; +-------+------+ | color | seq | +-------+------+ | | 1 | | | 2 | +-------+------+ == JSON_TABLE can have a non-unique alias? == create table t20(a varchar(100)); insert into t20 values (); create table t21(a varchar(100)); insert into t21 values (); select * from t20 as T, t21 as T; ERROR 1066 (42000): Not unique table/alias: 'T' Good. This is the expected behaviour. select * from t20 as T , json_table(T.a, '$[*]' columns(color varchar(100) path '$.nonexistent', seq for ordinality ) ) as T; Empty set (0.000 sec) No error. I think this is wrong. Why would JSON_TABLE be allowed to use an alias that conflicts with another table? == EXPLAIN doesn't show table function == I had requested this in previous review(s): Table function use should be shown in EXPLAIN [FORMAT=JSON]. Here's how MySQL does it: https://gist.github.com/spetrunia/257c929b0c04a866faf6428e733999ad Note that this can be extended to handle other table functions. if there are no other ideas, let's use the same syntax. == Comments == I still think the patch doesn't have good comments. Let me get back to you on this. > diff --git a/sql/share/errmsg-utf8.txt b/sql/share/errmsg-utf8.txt > index 49cabec9916..3db438ee470 100644 > --- a/sql/share/errmsg-utf8.txt > +++ b/sql/share/errmsg-utf8.txt > @@ -7965,3 +7965,7 @@ ER_NOT_ALLOWED_IN_THIS_CONTEXT > eng "'%-.128s' is not allowed in this context" > ER_DATA_WAS_COMMITED_UNDER_ROLLBACK > eng "Engine %s does not support rollback. Changes where commited during rollback call" > +ER_JSON_TABLE_ERROR_ON_FIELD > + eng "Field '%s' can't be set for JSON_TABLE '%s'." > +ER_JSON_TABLE_DOUBLE_ON_RESPONSE > + eng "ON %s response specified more than once in JSON_TABLE definition." I agree that using this error solves the issue I've complained about, but this doesn't look like the right way to do it: * Why not just emit a parser error with YYERROR in this case? * Better: why not just remove the recursive definition from the grammar? ... > diff --git a/sql/table_function.cc b/sql/table_function.cc ... > + > +/* > + handler::print_error has a case statement for error numbers. > + This value is (10000) is far out of range and will envoke the > + default: case.(Current error range is 120-159 from include/my_base.h) > + */ > +#define HA_ERR_INVALID_JSON 10000 So, is this needed anymore? On the other hand, the new solution with returning HA_ERR_TABLE_IN_FK_CHECK doesn't look very good either. A plugin could register its own error codes, but this engine is not a plugin. Let's discuss with Serg and get back to this. > +static int print_path(String *str, const json_path_t *p) > +{ > + return str->append('\"') || > + str->append((const char *) p->s.c_str, p->s.str_end - p->s.c_str) || > + str->append('\"'); > +} The above doesn't do proper quoting, which is wrong. Testcase: select * from json_table('[{"co\\\\lor": "blue", "price": 50}]', '$[*]' columns( color varchar(100) path '$.co\\\\lor') ) as T; +-------+ | color | +-------+ | blue | +-------+ create view v2 as select * from json_table('[{"co\\\\lor": "blue", "price": 50}]', '$[*]' columns( color varchar(100) path '$.co\\\\lor') ) as T; select * from v2; +-------+ | color | +-------+ | NULL | +-------+ BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Progress report for MDEV-12326
by Mohammed Hammaad Mateen 22 Jun '20

22 Jun '20

Greetings, Hope you are safe and doing great, This post describes the things I've done during 15-21 June under the mentor-ship of Sergei Golubchik and Oleksandr Byelkin for GSoC-20 to implement MDEV-12326. The tasks taken up for this week was to work towards extending the parser for multi-table DELETE....RETURNING and multi-table UPDATE....RETURNING, side-by-side, thereby changing the sql_yacc.yy file, for incorporating opt_returning rule in respective query format. I've also updated the test suite for multi-table UPDATE.... RETURNING and extended the parser by adding select_result parameter in multi-update and multi-delete functions, so as to enable returning a result set possible, whenever we query with RETURNING clause. All the code is uploaded up to date in these remote repositories, please feel free to comment. MULTI-TABLE UPDATE RETURNING: https://github.com/MariaDB/server/compare/10.5...MohammedHMateen:MDEV-12326… MULTI-TABLE DELETE RETURNING: https://github.com/MariaDB/server/compare/10.5...MohammedHMateen:MDEV-12326… Thank you Regards, Mohammed Hammaad Mateen

1 0

[Maria-developers] 97e932919bd: MDEV-22665: Print ranges in the optimizer trace created for non-indexed columns when optimizer_use_condition_selectivity >2
by Sergey Petrunia 17 Jun '20

17 Jun '20

Hi Varun, Please find some minor comments below. Ok to push when addressed. > revision-id: 97e932919bd53b854e0cea0d0001d185d7e63cc0 (mariadb-10.4.11-199-g97e932919bd) > parent(s): 0a5668f5128c731a346abf41afdc6fed33164ffc > author: Varun Gupta > committer: Varun Gupta > timestamp: 2020-05-26 22:36:03 +0530 > message: > > MDEV-22665: Print ranges in the optimizer trace created for non-indexed columns when optimizer_use_condition_selectivity >2 > > Now the optimizer trace shows the ranges constructed while getting estimates from EITS > > --- > mysql-test/main/opt_trace.result | 94 ++++++++++++++++++++++++++++++ > mysql-test/main/opt_trace.test | 18 ++++++ > sql/field.cc | 40 +++++++++++++ > sql/field.h | 1 + > sql/opt_range.cc | 123 ++++++++++++++++++++++++++++----------- > 5 files changed, 241 insertions(+), 35 deletions(-) ... > diff --git a/sql/opt_range.cc b/sql/opt_range.cc > index 66c870dd2ac..2cef483ef28 100644 > --- a/sql/opt_range.cc > +++ b/sql/opt_range.cc > @@ -15804,6 +15821,38 @@ void QUICK_GROUP_MIN_MAX_SELECT::dbug_dump(int indent, bool verbose) > > #endif /* !DBUG_OFF */ > > + > +/* > + @brief Print the comparison operator for the min range > +*/ > + > +static void print_min_range_operator(String *out, const key_range key) Do you really need to pass the whole struct key_range by value here? I think passing key.flag would be sufficient. > +{ > + if (key.flag == HA_READ_AFTER_KEY) > + out->append(STRING_WITH_LEN(" < ")); > + else if (key.flag == HA_READ_KEY_EXACT || > + key.flag == HA_READ_KEY_OR_NEXT) > + out->append(STRING_WITH_LEN(" <= ")); > + else > + out->append(STRING_WITH_LEN(" ? ")); > +} > + > + > +/* > + @brief Print the comparison operator for the max range > +*/ > + > +static void print_max_range_operator(String *out, const key_range key) > +{ Same as with print_min_range_operator: pass key.flag instead. > + if (key.flag == HA_READ_BEFORE_KEY) > + out->append(STRING_WITH_LEN(" < ")); > + else if (key.flag == HA_READ_AFTER_KEY) > + out->append(STRING_WITH_LEN(" <= ")); > + else > + out->append(STRING_WITH_LEN(" ? ")); > +} > + > + > static > void print_range(String *out, const KEY_PART_INFO *key_part, > KEY_MULTI_RANGE *range, uint n_key_parts) BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Progress Report GSoC 2020 (8-13 June)
by Anton Mikhaylenko 17 Jun '20

17 Jun '20

Good day, everybody. In this post, I want to describe what I did this week, June 8-13. Added counting № of completed tasks for during period. During this week I tried to understand why threads dying fast. https://github.com/sapiest/server/commit/7ffa698044092e346353e4754d50188ef8… Regards, Anton MIkhailenko

1 0

Re: [Maria-developers] 3c81ed2f564: Add mytop to Client component for rpm package
by Sergei Golubchik 16 Jun '20

16 Jun '20

Hi, Anel! On Jun 16, Anel Husakovic wrote: > revision-id: 3c81ed2f564 (mariadb-10.5.2-359-g3c81ed2f564) > parent(s): a793ae5bc12 > author: Anel Husakovic <anel(a)mariadb.org> > committer: Anel Husakovic <anel(a)mariadb.org> > timestamp: 2020-06-06 08:21:01 +0200 > message: > > Add mytop to Client component for rpm package > > Part of MDEV-22552 This is ok. But don't push it yet, let's have all commits ok to push them all at once Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 4cbbc796dcc: MDEV-22420 DDL on temporary object is prohibited when XA is in prepare state
by Sergei Golubchik 16 Jun '20

16 Jun '20

Hi, Andrei! Ok to push! Thanks! On Jun 16, Andrei Elkin wrote: > >> > What about RENAME? What other operations are there that don't > >> > open a table? > >> > >> So RENAME is in above and all that fits to > >> stmt_causes_implicit_commit() that is flagged CF_AUTO_COMMIT_TRANS > >> (that involves CF_IMPLICIT_COMMIT_BEGIN). That reads every DDL, > >> Admin, Account that are bin-logged. > > > > I mean RENAME for temporary tables. Does it also cause an implicit > > commit? > > Yes, it does, as any syntax whose lex > lex->tmp_table() > returns false here. > > RENAME specifically does not have 'temporary' qualifier. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 4cbbc796dcc: MDEV-22420 DDL on temporary object is prohibited when XA is in prepare state
by Sergei Golubchik 16 Jun '20

16 Jun '20

Hi, Andrei! On Jun 16, Andrei Elkin wrote: > > > What about RENAME? What other operations are there that don't open a table? > > So RENAME is in above and all that fits to stmt_causes_implicit_commit() > that is flagged CF_AUTO_COMMIT_TRANS (that involves > CF_IMPLICIT_COMMIT_BEGIN). > That reads every DDL, Admin, Account that are bin-logged. I mean RENAME for temporary tables. Does it also cause an implicit commit? Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] MariaDB support in WAL-G
by Кирилл Юлдашев 16 Jun '20

16 Jun '20

Hi! I implemented MariaDB support in the WAL-G backup system (Here is README https://github.com/wal-g/wal-g/blob/master/MySQL.md)) the implementation turned out to be compatible with MySQL. Can I ask someone from MariaDB to do a small design review and send it to me? I am defending my diploma on this topic on June 17 this year, and the developers ' opinion would be very useful. Integration tests: https://github.com/wal-g/wal-g/tree/master/docker/mariadb_tests/scripts/tes… Internal database stuff and command handlers: https://github.com/wal-g/wal-g/tree/master/internal/databases/mysql Commands: https://github.com/wal-g/wal-g/tree/master/cmd/mysql But PITR for MariaDB is not working yet, because of problems which occurred during the implementation stage due to the lack of documentation. I'll write about my problems later in another email. Thanks! Regards, Kirill

1 0

Re: [Maria-developers] 4cbbc796dcc: MDEV-22420 DDL on temporary object is prohibited when XA is in prepare state
by Sergei Golubchik 15 Jun '20

15 Jun '20

Hi, Andrei! On Jun 15, Andrei Elkin wrote: > revision-id: 4cbbc796dcc (mariadb-10.5.2-335-g4cbbc796dcc) > parent(s): a74e724e061 > author: Andrei Elkin <andrei.elkin(a)mariadb.com> > committer: Andrei Elkin <andrei.elkin(a)mariadb.com> > timestamp: 2020-06-11 17:24:15 +0300 > message: > > MDEV-22420 DDL on temporary object is prohibited when XA is in prepare state > > The parser must reject DDL operations on temporary objects when > they may modify or alter such object, including temporary tables and sequences. > The rejection is regardless (has been already in place for bin-loggable DML:s) > of the binlogging capability of the server or connection. > > The patch implements the requirement. A binlog test is added. > > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > index 86f81dd001a..6ac0cf96024 100644 > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -4890,6 +4890,11 @@ mysql_execute_command(THD *thd) > } > else > { > + if (thd->transaction->xid_state.check_has_uncommitted_xa()) > + { > + thd->transaction->xid_state.er_xaer_rmfail(); > + goto error; > + } > status_var_decrement(thd->status_var.com_stat[lex->sql_command]); > status_var_increment(thd->status_var.com_drop_tmp_table); Two comments. 1. Where is this checked for non-temporary tables? check_has_uncommitted_xa is tested in open_tables(), and it's not called for DROP. What about RENAME? What other operations are there that don't open a table? 2. check_has_uncommitted_xa() already does er_xaer_rmfail() internally, you don't need to repeat it. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] Progress report for MDEV-12326
by Mohammed Hammaad Mateen 15 Jun '20

15 Jun '20

Greetings, Hope you are safe and doing great, This post describes the things I've done during 8-14 June under the mentor-ship of Sergei Golubchik and Oleksandr Byelkin for GSoC-20 to implement MDEV-12326. The tasks taken up for this week was to work towards extending the parser for multi-table DELETE....RETURNING by changing the sql_yacc.yy file, for incorporating opt_returning rule in multi-delete query format. The RETURNING_SYM in opt_returning rule incorporates the RETURNING keyword and works completely fine with <column_names> but for some reason it fails and crashes the server for RETURNING * . I'll try to run it in a debugger and will try to fix it soon. Meanwhile, I'll also be pushing the code on this branch (https://github.com/MariaDB/server/pull/1590/commits) Thank you Regards, Mohammed Hammaad Mateen

1 0

Re: [Maria-developers] c48b190824a: Fixed error messages from DROP VIEW to align with DROP TABLE
by Sergei Golubchik 14 Jun '20

14 Jun '20

Hi, Michael! On Jun 13, Michael Widenius wrote: > revision-id: c48b190824a (mariadb-10.5.2-405-gc48b190824a) > parent(s): 50bad912977 > author: Michael Widenius <michael.widenius(a)gmail.com> > committer: Michael Widenius <michael.widenius(a)gmail.com> > timestamp: 2020-06-12 12:04:05 +0300 > message: > > Fixed error messages from DROP VIEW to align with DROP TABLE > > - Produce a "Note" for all wrongly dropped objects > (Like doing DROP VIEW on a table). > - IF EXISTS ends with a list of all not existing objects, instead of a > separate note for every not existing object. > > Other things: > - Fixed bug where one could do CREATE TEMPORARY SEQUENCE multiple times > and create multiple temporary sequences with the same name. > > diff --git a/mysql-test/main/create_drop_view.result b/mysql-test/main/create_drop_view.result > index 92fbf5ac9e5..7cfa818ccbf 100644 > --- a/mysql-test/main/create_drop_view.result > +++ b/mysql-test/main/create_drop_view.result > @@ -61,5 +61,6 @@ Warnings: > Note 4092 Unknown VIEW: 'test.v1' > DROP VIEW IF EXISTS t1; > Warnings: > +Warning 1347 'test.t1' is not of type 'VIEW' please, add a test for the case when a user has no privileges on test.t1, the object existence is not leaked out (if one has no privileges to know whether t1 exists or whether t1 is a view or a table, the error message should not divulge it). (the same for tables and for sequences in your next commit) > Note 4092 Unknown VIEW: 'test.t1' > DROP TABLE t1; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 250bc3b6d74: Ensure that table is truly dropped when using DROP TABLE
by Sergei Golubchik 14 Jun '20

14 Jun '20

Hi, Michael! On Jun 11, Michael Widenius wrote: > revision-id: 250bc3b6d74 (mariadb-10.5.2-390-g250bc3b6d74) > parent(s): ba2c2cfb20e > author: Michael Widenius <michael.widenus(a)gmail.com> > committer: Michael Widenius <michael.widenus(a)gmail.com> > timestamp: 2020-06-11 11:33:55 +0300 > message: > > Ensure that table is truly dropped when using DROP TABLE > > MDEV-11412 AliSQL: [Feature] Issue#34 Support force drop table please, put this line ^^^ first in the comment comment. > The used code is largely based on code from Tencent > > The problem is that in some rare cases there may be a conflict between .frm > files and the files in the storage engine. In this case the DROP TABLE > was not able to properly drop the table. > > Some MariaDB/MySQL forks has solved this by adding a FORCE option to TSQL? Or some other fork? > DROP TABLE. After some discussion among MariaDB developers, we concluded > that users expects that DROP TABLE should always work, even if the > table would not be consistent. There should not be a need to use a > separate keyword to ensure that the table is really deleted. > > The used solution is: > - If a .frm table doesn't exists, try dropping the table from all storage > engines. > - If the .frm table exists but the table does not exist in the engine > try dropping the table from all storage engines. I don't see why is that needed, if .frm exists, it tells exactly what storage engine to use, there is no point in trying other engines. > - Update storage engines using many table files (.CVS, MyISAM, Aria) to > succeed with the drop even if some of the files are missing. > - Add HTON_AUTOMATIC_DELETE_TABLE to handlerton's where delete_table() > is not needed and always succeed. This is used by ha_delete_table_force() > to know which handlers to ignore when trying to drop a table without > a .frm file. > > The disadvantage of this solution is that a DROP TABLE on a non existing > table will be a bit slower as we have to ask all active storage engines > if they know anything about the table. > > Other things: > - Added a new flag MY_IGNORE_ENOENT to my_delete() to not give an error > if the file doesn't exist. This simplifies some of the code. > - Don't clear thd->error in ha_delete_table() if there was an active > error. This is a bug fix. > - handler::delete_table() will not abort if first file doesn't exists. > This is bug fix to handle the case when a drop table was aborted in > the middle. > - Cleaned up mysql_rm_table_no_locks() to ensure that if_exists uses > same code path as when it's not used. > - Use non_existing_Table_error() to detect if table didn't exists. > Old code used different errors tests in different position. > - Table_triggers_list::drop_all_triggers() now drops trigger file if > it can't be parsed instead of leaving it hanging around (bug fix) > - InnoDB doesn't anymore print error about .frm file out of sync with > InnoDB directory if .frm file does not exists. This change was required > to be able to try to drop an InnoDB file when .frm doesn't exists. > - Fixed bug in mi_delete_table() where the .MYD file would not be dropped > if the .MYI file didn't exists. > - Fixed memory leak in Mroonga when deleting non existing table > - Fixed memory leak in Connect when deleting non existing table > > Bugs fixed introduced by the original version of this commit: > MDEV-22826 Presence of Spider prevents tables from being force-deleted from > other engines I don't think this part of the comment makes any sense, when one looks through git history there is no "original version of this commit". It's like a rolled back transaction, that value did not exist, unless you use READ UNCOMMITTED :) > diff --git a/mysql-test/main/drop_table_force.test b/mysql-test/main/drop_table_force.test > new file mode 100644 > index 00000000000..8fdd79465b7 > --- /dev/null > +++ b/mysql-test/main/drop_table_force.test > @@ -0,1 +1,197 @@ > +--source include/have_log_bin.inc > +--source include/have_innodb.inc > + > +# > +# This test is based on the orginal test from Tencent for DROP TABLE ... FORCE > +# In MariaDB we did reuse the code but MariaDB does not require the FORCE > +# keyword to drop a table even if the .frm file or some engine files are > +# missing. > +# To make it easy to see the differences between the orginal code and > +# the new one, we have left some references to the original test case > +# > + > +CALL mtr.add_suppression("Operating system error number"); > +CALL mtr.add_suppression("The error means the system cannot"); > +CALL mtr.add_suppression("returned OS error 71"); not sure it's a good idea, error numbers aren't very portable, this will blow up some day like a time bomb > + > +let $DATADIR= `select @@datadir`; > + > +--echo #Test1: table with missing .ibd can be dropped directly > +# drop table without ibd > +create table t1(a int)engine=innodb; > +--remove_file $DATADIR/test/t1.ibd > +drop table t1; > +--list_files $DATADIR/test/ > + > +# Original DROP TABLE .. FORCE required SUPER privilege. MariaDB doesn't > +--echo # Test droping table without frm without super privilege > + > +# create table t1 and rm frm > +create table t1(a int) engine=innodb; > +--remove_file $DATADIR/test/t1.frm > + > +# create test user > +create user test identified by '123456'; > +grant all privileges on test.t1 to 'test'@'%'identified by '123456' with grant option; why with grant option? better remove it if it's not essential for drop to work. if it is essential, why is it? > + > +# connect as test > +connect (con_test, localhost, test,'123456', ); > +--connection con_test > + > +# drop table with user test > +drop table t1; > +--error ER_BAD_TABLE_ERROR > +drop table t1; > + > +# connect as root > +--connection default > + > +--disconnect con_test > +drop user test; > + > +# check files in datadir about t1 > +--list_files $DATADIR/test/ > + > +--echo #Test4: drop table can drop consistent table as well > +create table t1(a int) engine=innodb; > +drop table t1; aha, so it works! :) > + > +# check files in datadir about t1 > +--list_files $DATADIR/test/ > + > +--echo #Test5: drop table with triger, and with missing frm > +# create table t1 with triger and rm frm > +create table t1(a int)engine=innodb; > +create trigger t1_trg before insert on t1 for each row begin end; > + > +let $DATADIR= `select @@datadir`; > +--remove_file $DATADIR/test/t1.frm > + > +drop table t1; > +--error ER_BAD_TABLE_ERROR > +drop table t1; > + > +# check files in datadir about t1 > +--list_files $DATADIR/test/ > + > +--echo #Test6: table with foreign key references can not be dropped > +# create table with foreign key reference and rm frm > +CREATE TABLE parent (id INT NOT NULL, PRIMARY KEY (id)) ENGINE=INNODB; > +CREATE TABLE child (id INT, parent_id INT, INDEX par_ind (parent_id), FOREIGN KEY (parent_id) REFERENCES parent(id) ON DELETE CASCADE) ENGINE=INNODB; > +--remove_file $DATADIR/test/parent.frm > + > +# parent can not be dropped when there are foreign key references > +--error ER_ROW_IS_REFERENCED_2 > +drop table parent; > + > +# parent can be dropped when there are no foreign key references > +drop table child; > +drop table parent; > + > +# check files in datadir about child and parent > +--list_files $DATADIR/test/ > + > +--echo #Test7: drop table twice > +create table t1(a int)engine=innodb; > +--remove_file $DATADIR/test/t1.frm > + > +# first drop table will success > +drop table t1; > + > +# check files in datadir about t1 > +--list_files $DATADIR/test/ > + > +# second drop with if exists will also ok > +drop table if exists t1; > + > +# check files in datadir about t1 > +--list_files $DATADIR/test/ > + > +--echo #Test8: check compatibility with if exists > +create table t1(a int)engine=innodb; > +--remove_file $DATADIR/test/t1.frm > + > +# first drop will success > +drop table t1; > + > +# check files in datadir about t1 > +--list_files $DATADIR/test/ > + > +# second drop with if exists will success > +drop table if exists t1; you've just tested it in Test7, haven't you? > + > +--echo #Test9: check compatibility with restrict/cascade > +# create table with foreign key reference and rm frm > +CREATE TABLE parent (id INT NOT NULL, PRIMARY KEY (id)) ENGINE=INNODB; > +CREATE TABLE child (id INT, parent_id INT, INDEX par_ind (parent_id), FOREIGN KEY (parent_id) REFERENCES parent(id) ON DELETE CASCADE) ENGINE=INNODB; > + > +# parent can not be dropped when there are froeign key references typo "foreign" > +--error ER_ROW_IS_REFERENCED_2 > +drop table parent; > +--error ER_ROW_IS_REFERENCED_2 > +drop table parent restrict; > +--error ER_ROW_IS_REFERENCED_2 > +drop table parent cascade; > +--error ER_ROW_IS_REFERENCED_2 > +drop table parent; > +--error ER_ROW_IS_REFERENCED_2 > +drop table parent restrict; > +--error ER_ROW_IS_REFERENCED_2 > +drop table parent cascade; > + > +# parent can be dropped when there are no foreign key references > +drop table child; > +drop table parent; > + > +--echo #Test10: drop non-innodb engine table returns ok > +# create myisam table t1 and rm .frm > +create table t1(a int) engine=myisam; > +--remove_file $DATADIR/test/t1.frm > +--replace_result \\ / > +drop table t1; > + > +# create myisam table t1 and rm .MYD > +create table t1(a int) engine=myisam; > +--remove_file $DATADIR/test/t1.MYD > +--replace_result \\ / > +drop table t1; > + > +# create myisam table t1 and rm .MYI > +create table t1(a int) engine=myisam; > +--remove_file $DATADIR/test/t1.MYI > +--replace_result \\ / > +drop table t1; > +--list_files $DATADIR/test/ > + > +# create Aria table t1 and rm .frm and .MAD > +create table t1(a int) engine=aria; > +--remove_file $DATADIR/test/t1.frm > +--remove_file $DATADIR/test/t1.MAD > +--list_files $DATADIR/test/ > +--error ER_BAD_TABLE_ERROR > +drop table t1; > +--replace_result \\ / > +show warnings; > +--list_files $DATADIR/test/ > + > +# create Aria table t2 and rm .frm and .MAI > +create table t2(a int) engine=aria; > +flush tables; > +--remove_file $DATADIR/test/t2.frm > +--remove_file $DATADIR/test/t2.MAI > +--list_files $DATADIR/test/ > +--error ER_BAD_TABLE_ERROR > +drop table t2; > +--replace_result \\ / > +show warnings; > +--list_files $DATADIR/test/ > + > +# create Aria table t2 and rm .MAI and .MAD > +create table t2(a int) engine=aria; > +flush tables; > +--remove_file $DATADIR/test/t2.MAD > +--remove_file $DATADIR/test/t2.MAI > +--list_files $DATADIR/test/ > +--replace_result \\ / > +drop table t2; may be csv and archive tests too? > diff --git a/sql/handler.cc b/sql/handler.cc > index 1af0157783e..5dd02d057c7 100644 > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -2674,26 +2712,34 @@ const char *get_canonical_filename(handler *file, const char *path, > } > > > -/** delete a table in the engine > +/** > + Delete a table in the engine > > + @return 0 Table was deleted > + @return -1 Table didn't exists, no error given > + @return # Error from table handler > + > @note > ENOENT and HA_ERR_NO_SUCH_TABLE are not considered errors. > - The .frm file will be deleted only if we return 0. > + The .frm file should be deleted by the caller only if we return <= 0. > */ > + > int ha_delete_table(THD *thd, handlerton *table_type, const char *path, > - const LEX_CSTRING *db, const LEX_CSTRING *alias, bool generate_warning) > + const LEX_CSTRING *db, const LEX_CSTRING *alias, > + bool generate_warning) > { > handler *file; > char tmp_path[FN_REFLEN]; > int error; > TABLE dummy_table; > TABLE_SHARE dummy_share; > + bool is_error= thd->is_error(); > DBUG_ENTER("ha_delete_table"); > > /* table_type is NULL in ALTER TABLE when renaming only .frm files */ > if (table_type == NULL || table_type == view_pseudo_hton || > ! (file=get_new_handler((TABLE_SHARE*)0, thd->mem_root, table_type))) > - DBUG_RETURN(0); > + DBUG_RETURN(-1); "Table didn't exists, no error given"? if get_new_handler() fails it's OOM. > > bzero((char*) &dummy_table, sizeof(dummy_table)); > bzero((char*) &dummy_share, sizeof(dummy_share)); > @@ -2723,8 +2769,10 @@ int ha_delete_table(THD *thd, handlerton *table_type, const char *path, > } > if (intercept) > { > + /* Clear error if we got it in this function */ > + if (!is_error) > thd->clear_error(); this is generally a wrong approach. One should use push_internal_handler to silence errors, not thd->is_error() or thd->clear_error(). In this particilar case there will be no errors if there was no error before ha_delete_table() - which is correct. But if there was an error before, this will leave spurious and confusing errors in `SHOW WARNINGS` output. > - error= 0; > + error= -1; > } > } > delete file; > @@ -4365,45 +4413,63 @@ uint handler::get_dup_key(int error) > > @note > We assume that the handler may return more extensions than > - was actually used for the file. > + was actually used for the file. We also assume that the first > + extension is the most important one. If this exist and we can't delete > + that one we will abort the delete. after "the most important one" you can add "(see the comment near handlerton::tablefile_extensions)" > + If the first one doesn't exists, we have to try to delete all other "doesn't exist" > + extension as there is chance that the server had crashed between > + the delete of the first file and the next > > @retval > 0 If we successfully deleted at least one file from base_ext and > didn't get any other errors than ENOENT > + > @retval > !0 Error > */ > + > int handler::delete_table(const char *name) > { > - int saved_error= 0; > - int error= 0; > - int enoent_or_zero; > + int saved_error= ENOENT; > + bool abort_if_first_file_error= 1; > + bool some_file_deleted= 0; > + DBUG_ENTER("handler::delete_table"); > > + // For discovery tables, it's ok if first file doesn't exists why would that be ok? > if (ht->discover_table) > - enoent_or_zero= 0; // the table may not exist in the engine, it's ok > - else > - enoent_or_zero= ENOENT; // the first file of bas_ext() *must* exist > + { > + abort_if_first_file_error= 0; > + saved_error= 0; > + if (!bas_ext()) > + { > + DBUG_ASSERT(ht->flags & HTON_AUTOMATIC_DELETE_TABLE); > + DBUG_RETURN(0); // Drop succeded > + } > + } > > for (const char **ext= bas_ext(); *ext ; ext++) > { > + int error; > - if (mysql_file_delete_with_symlink(key_file_misc, name, *ext, 0)) > + if ((error= mysql_file_delete_with_symlink(key_file_misc, name, *ext, > + MYF(0)))) why do you need this `error` variable, if you never use it? > { > if (my_errno != ENOENT) > { > + saved_error= my_errno; > /* > - If error on the first existing file, return the error. > + If error other than file not found on the first existing file, > + return the error. > Otherwise delete as much as possible. > */ > - if (enoent_or_zero) > - return my_errno; > - saved_error= my_errno; > + if (abort_if_first_file_error) > + DBUG_RETURN(saved_error); > } > } > else > - enoent_or_zero= 0; // No error for ENOENT > - error= enoent_or_zero; > + some_file_deleted= 1; > + abort_if_first_file_error= 0; > } > - return saved_error ? saved_error : error; > + DBUG_RETURN(some_file_deleted && saved_error == ENOENT ? 0 : saved_error); > } > > > @@ -4439,5 +4505,20 @@ void handler::drop_table(const char *name) > > > /** > + Return true if the error from drop table means that the > + table didn't exists > +*/ > + > +bool non_existing_table_error(int error) > +{ > + return (error == ENOENT || error == HA_ERR_NO_SUCH_TABLE || > + error == HA_ERR_UNSUPPORTED || > + error == ER_NO_SUCH_TABLE || > + error == ER_NO_SUCH_TABLE_IN_ENGINE || > + error == ER_WRONG_OBJECT); > +} this should, I believe, be inline in handler.h > + > + > +/** > Performs checks upon the table. > > @@ -4899,5 +4981,89 @@ handler::ha_drop_table(const char *name) > > > /** > + Structure used during force drop table. > +*/ > + > +struct st_force_drop_table_params > +{ > + const char *path; > + const LEX_CSTRING *db; > + const LEX_CSTRING *alias; > + bool generate_warning; > + int error; > +}; > + > + > +/** > + Try to delete table from a given plugin > + Table types with discovery is ignored as these .frm files would have > + been created during discovery and thus doesn't need to be found > + for drop table force > +*/ > + > +static my_bool delete_table_force(THD *thd, plugin_ref plugin, void *arg) > +{ > + handlerton *hton = plugin_hton(plugin); > + st_force_drop_table_params *param = (st_force_drop_table_params *)arg; > + > + /* > + We have to ignore HEAP tables as these may not have been created yet > + We also remove engines that is using discovery (as these will recrate > + any missing .frm if needed) and tables marked with > + HTON_AUTOMATIC_DELETE_TABLE as for these we can't check if the table > + ever existed. > + */ > + if (!hton->discover_table && hton->db_type != DB_TYPE_HEAP && > + !(hton->flags & HTON_AUTOMATIC_DELETE_TABLE)) > + { > + int error; > + error= ha_delete_table(thd, hton, param->path, param->db, > + param->alias, param->generate_warning); This is doing a a lot of extra work. dummy_table/dummy_share, if()s. Allocating and constructing new handlers. I suggest the following: 1. Introduce new handlerton method hton->drop_table(). By default it'll be hton->get_new_handler() get_canonical_filename() file->ha_delete_table() like now, so all engines will still work. But later we can start moving drop table functionality out of the handler. In this commit only engines like blackhole will have a dummy drop_table() method. 2. Here you can do, basically hton->drop_table(hton, param->path) without checking for discovery, without a new flag HTON_AUTOMATIC_DELETE_TABLE. I'm not sure why you check for DB_TYPE_HEAP, could you please explain that? > + if (error > 0 && !non_existing_table_error(error)) > + param->error= error; > + if (error == 0) > + { > + param->error= 0; > + return TRUE; // Table was deleted > + } > + } > + return FALSE; > +} > + > +/** > + @brief > + Traverse all plugins to delete table when .frm file is missing. > + > + @return -1 Table was not found in any engine > + @return 0 Table was found in some engine and delete succeded > + @return # Error from first engine that had a table but didn't succeed to > + delete the table > + @return HA_ERR_ROW_IS_REFERENCED if foreign key reference is encountered, > + > +*/ > + > +int ha_delete_table_force(THD *thd, const char *path, const LEX_CSTRING *db, > + const LEX_CSTRING *alias, bool generate_warning) > +{ > + st_force_drop_table_params param; > + Table_exists_error_handler no_such_table_handler; > + DBUG_ENTER("ha_delete_table_force"); > + > + param.path= path; > + param.db= db; > + param.alias= alias; > + param.generate_warning= generate_warning; this is never used, you always invoke ha_delete_table_force() with generate_warning=0. > + param.error= -1; // Table not found > + > + thd->push_internal_handler(&no_such_table_handler); > + if (plugin_foreach(thd, delete_table_force, MYSQL_STORAGE_ENGINE_PLUGIN, > + &param)) > + param.error= 0; // Delete succeded > + thd->pop_internal_handler(); > + DBUG_RETURN(param.error); > +} > + > + > +/** > Create a table in the engine: public interface. > > diff --git a/sql/handler.h b/sql/handler.h > index f592c635c5d..f5ccdbe1ee4 100644 > --- a/sql/handler.h > +++ b/sql/handler.h > @@ -1780,6 +1780,13 @@ handlerton *ha_default_tmp_handlerton(THD *thd); > */ > #define HTON_TRANSACTIONAL_AND_NON_TRANSACTIONAL (1 << 17) > > +/* > + The engine doesn't keep track of tables, delete_table() is not > + needed and delete_table() always returns 0 (table deleted). This flag > + mainly used to skip storage engines in case of ha_delete_table_force() > +*/ > +#define HTON_AUTOMATIC_DELETE_TABLE (1 << 18) > + > class Ha_trx_info; > > struct THD_TRANS > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index 61b1113f680..7c79acef3ea 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -2327,36 +2329,42 @@ int mysql_rm_table_no_locks(THD *thd, TABLE_LIST *tables, bool if_exists, > thd->find_temporary_table(table) && > table->mdl_request.ticket != NULL)); > > + /* First try to delete temporary tables and temporary sequences */ > - if (table->open_type == OT_BASE_ONLY || !is_temporary_table(table) || > - (drop_sequence && table->table->s->table_type != TABLE_TYPE_SEQUENCE)) > - error= 1; > - else > + if ((table->open_type != OT_BASE_ONLY && is_temporary_table(table)) && unnecessary parentheses > + (!drop_sequence || table->table->s->table_type == TABLE_TYPE_SEQUENCE)) > { > table_creation_was_logged= table->table->s->table_creation_was_logged; > if (thd->drop_temporary_table(table->table, &is_trans, true)) > { > + /* > + This is a very unlikely scenaro as dropping a temporary table > + should always work. Would be better if we tried to drop all > + temporary tables before giving the error. > + */ > error= 1; > goto err; > } > - error= 0; > table->table= 0; > + temporary_table_was_dropped= 1; > } > > - if ((drop_temporary && if_exists) || !error) > + if ((drop_temporary && if_exists) || temporary_table_was_dropped) > { > /* > This handles the case of temporary tables. We have the following cases: > > - . "DROP TEMPORARY" was executed and a temporary table was affected > - (i.e. drop_temporary && !error) or the if_exists was specified (i.e. > - drop_temporary && if_exists). > - > - . "DROP" was executed but a temporary table was affected (.i.e > - !error). > + - "DROP TEMPORARY" was executed and table was dropped > + temporary_table_was_dropped == 1 > + - "DROP TEMPORARY IF EXISTS" was specified but no temporary table > + existed > + temporary_table_was_dropped == 0 > */ > if (!dont_log_query && table_creation_was_logged) > { > /* > + DROP TEMPORARY succeded. For the moment when we only come > + here on success (error == 0) DBUG_ASSERT(error == 0) please > + > If there is an error, we don't know the type of the engine > at this point. So, we keep it in the trx-cache. > */ > @@ -2387,7 +2395,8 @@ int mysql_rm_table_no_locks(THD *thd, TABLE_LIST *tables, bool if_exists, > is no need to proceed with the code that tries to drop a regular > table. > */ > - if (!error) continue; > + if (temporary_table_was_dropped) can temporary_table_was_dropped be false here? > + continue; > } > else if (!drop_temporary) > { > @@ -2402,48 +2411,33 @@ int mysql_rm_table_no_locks(THD *thd, TABLE_LIST *tables, bool if_exists, > path_length= build_table_filename(path, sizeof(path) - 1, db.str, > alias.str, reg_ext, 0); > } > + > DEBUG_SYNC(thd, "rm_table_no_locks_before_delete_table"); > error= 0; > - if (drop_temporary || > - (ha_table_exists(thd, &db, &alias, &table_type, &is_sequence) == 0 && > + if (drop_temporary) > + { > + /* "DROP TEMPORARY" but a temporary table was not found */ > + error= ENOENT; > + } > + else if (((frm_exists= ha_table_exists(thd, &db, &alias, &table_type, > + &is_sequence)) == 0 && this is an overkill. ha_table_exists() will do a full table discovery, you don't need it here, because you'll try to delete anyway. So, basically you only need dd_frm_type() here. > table_type == 0) || > (!drop_view && (was_view= (table_type == view_pseudo_hton))) || > (drop_sequence && !is_sequence)) > { > /* > One of the following cases happened: > - . "DROP TEMPORARY" but a temporary table was not found. > . "DROP" but table was not found > . "DROP TABLE" statement, but it's a view. > . "DROP SEQUENCE", but it's not a sequence > */ > - was_table= drop_sequence && table_type; > - if (if_exists) > - { > - char buff[FN_REFLEN]; > - int err= (drop_sequence ? ER_UNKNOWN_SEQUENCES : > - ER_BAD_TABLE_ERROR); > - String tbl_name(buff, sizeof(buff), system_charset_info); > - tbl_name.length(0); > - tbl_name.append(&db); > - tbl_name.append('.'); > - tbl_name.append(&table->table_name); > - push_warning_printf(thd, Sql_condition::WARN_LEVEL_NOTE, > - err, ER_THD(thd, err), > - tbl_name.c_ptr_safe()); > - > - /* > - Our job is done here. This statement was added to avoid executing > - unnecessary code farther below which in some strange corner cases > - caused the server to crash (see MDEV-17896). > - */ > - goto log_query; > - } > - else > - { > - non_tmp_error = (drop_temporary ? non_tmp_error : TRUE); > - error= 1; > - } > + wrong_drop_sequence= drop_sequence && table_type; > + was_table|= wrong_drop_sequence; > + local_non_tmp_error= 1; > + error= -1; > + if ((!frm_exists && !table_type) || // no .frm > + if_exists) > + error= ENOENT; > } > else > { > @@ -2509,49 +2507,138 @@ int mysql_rm_table_no_locks(THD *thd, TABLE_LIST *tables, bool if_exists, > goto err; > } > } > - else > + > + /* > + Delete the .frm file if we managed to delete the table from the > + engine or the table didn't exists in the engine > + */ > + if (likely(!error) || non_existing_table_error(error)) > { > /* Delete the table definition file */ > strmov(end,reg_ext); > if (table_type && table_type != view_pseudo_hton && > - table_type->discover_table) > + (table_type->discover_table || error)) I don't think this new drop table logic needs to have any special handling of discovery. If the table didn't exist in engine - it's the normal case now, discovery or not, it's be handled normally. > { > /* > - Table type is using discovery and may not need a .frm file. > + Table type is using discovery and may not need a .frm file > + or the .frm file existed but no table in engine. > Delete it silently if it exists > */ > - (void) mysql_file_delete(key_file_frm, path, MYF(0)); > + if (mysql_file_delete(key_file_frm, path, > + MYF(MY_WME | MY_IGNORE_ENOENT))) > + error= my_errno; > } > else if (unlikely(mysql_file_delete(key_file_frm, path, > - MYF(MY_WME)))) > + !error ? MYF(MY_WME) : > + MYF(MY_WME | MY_IGNORE_ENOENT)))) > { > frm_delete_error= my_errno; > DBUG_ASSERT(frm_delete_error); > } > } > + frm_was_deleted= 1; > if (thd->replication_flags & OPTION_IF_EXISTS) > log_if_exists= 1; > > - if (likely(!error)) > + if (frm_delete_error) > { > - int trigger_drop_error= 0; > + /* > + Remember error if unexpected error from dropping the .frm file > + or we got an error from ha_delete_table() > + */ > + if (frm_delete_error != ENOENT) > + error= frm_delete_error; > + else if (if_exists && ! error) > + thd->clear_error(); no, please, don't do thd->clear_error(). I've seen and fixed so many bugs because of that. The last one - just an hour ago. Use push_internal_handler() instead. > + } > + if (likely(!error) || !frm_delete_error) > + non_tmp_table_deleted= TRUE; > > - if (likely(!frm_delete_error)) > + if (likely(!error) || non_existing_table_error(error)) > { > + trigger_drop_executed= 1; > + > + if (Table_triggers_list::drop_all_triggers(thd, &db, > + &table->table_name, > + MYF(MY_WME | > + MY_IGNORE_ENOENT))) > + error= error ? error : -1; > + } > + local_non_tmp_error|= MY_TEST(error); > + } > + > + /* > + If there was no .frm file and the table is not temporary, > + scan all engines try to drop the table from there. > + This is to ensure we don't have any partial table files left. > + > + We check for trigger_drop_executed to ensure we don't again try > + to drop triggers when it failed above (after sucecssfully dropping > + the table). > + */ > + if (non_existing_table_error(error) && !drop_temporary && > + table_type != view_pseudo_hton && !trigger_drop_executed && > + !wrong_drop_sequence) > + { > + char *end; > + int ferror= 0; > + > + /* Remove extension for delete */ > + *(end = path + path_length - reg_ext_length) = '\0'; > + ferror= ha_delete_table_force(thd, path, &db, > + &table->table_name, 0); > + if (!ferror) > + { > + /* Table existed and was deleted */ > non_tmp_table_deleted= TRUE; > - trigger_drop_error= > - Table_triggers_list::drop_all_triggers(thd, &db, > - &table->table_name); > + local_non_tmp_error= 0; > + error= 0; > } > + if (ferror <= 0) > + { > + ferror= 0; // Ignore table not found > > - if (unlikely(trigger_drop_error) || > - (frm_delete_error && frm_delete_error != ENOENT)) > - error= 1; > - else if (frm_delete_error && if_exists) > - thd->clear_error(); > + /* Delete the table definition file */ > + if (!frm_was_deleted) > + { > + strmov(end, reg_ext); > + if (mysql_file_delete(key_file_frm, path, > + MYF(MY_WME | MY_IGNORE_ENOENT))) > + ferror= my_errno; > } > - non_tmp_error|= MY_TEST(error); > + if (Table_triggers_list::drop_all_triggers(thd, &db, > + &table->table_name, > + MYF(MY_WME | > + MY_IGNORE_ENOENT))) > + ferror= -1; Why do you need two copies of code to drop triggers? Why cannot you do it once, on the first drop_all_triggers() place? > } > + if (!error) > + error= ferror; > + } > + > + /* > + Don't give an error if we are using IF EXISTS for a table that > + didn't exists > + */ > + > + if (if_exists && non_existing_table_error(error)) > + { > + char buff[FN_REFLEN]; > + int err= (drop_sequence ? ER_UNKNOWN_SEQUENCES : > + ER_BAD_TABLE_ERROR); > + String tbl_name(buff, sizeof(buff), system_charset_info); > + tbl_name.length(0); > + tbl_name.append(&db); > + tbl_name.append('.'); > + tbl_name.append(&table->table_name); > + push_warning_printf(thd, Sql_condition::WARN_LEVEL_NOTE, > + err, ER_THD(thd, err), > + tbl_name.c_ptr_safe()); > + error= 0; > + local_non_tmp_error= 0; > + drop_table_not_done= 1; > + } > + non_tmp_error|= local_non_tmp_error; > > if (error) > { > diff --git a/storage/blackhole/ha_blackhole.cc b/storage/blackhole/ha_blackhole.cc > index 15548350b20..98589f1d043 100644 > --- a/storage/blackhole/ha_blackhole.cc > +++ b/storage/blackhole/ha_blackhole.cc > @@ -399,7 +399,7 @@ static int blackhole_init(void *p) > blackhole_hton= (handlerton *)p; > blackhole_hton->db_type= DB_TYPE_BLACKHOLE_DB; > blackhole_hton->create= blackhole_create_handler; > - blackhole_hton->flags= HTON_CAN_RECREATE; > + blackhole_hton->flags= HTON_CAN_RECREATE | HTON_AUTOMATIC_DELETE_TABLE; > > mysql_mutex_init(bh_key_mutex_blackhole, > &blackhole_mutex, MY_MUTEX_INIT_FAST); > diff --git a/storage/blackhole/ha_blackhole.h b/storage/blackhole/ha_blackhole.h > index 6ee30877b64..646fba6da9f 100644 > --- a/storage/blackhole/ha_blackhole.h > +++ b/storage/blackhole/ha_blackhole.h > @@ -95,6 +95,10 @@ class ha_blackhole: public handler > THR_LOCK_DATA **store_lock(THD *thd, > THR_LOCK_DATA **to, > enum thr_lock_type lock_type); > + int delete_table(const char *name) > + { > + return 0; > + } > private: > virtual int write_row(const uchar *buf); > virtual int update_row(const uchar *old_data, const uchar *new_data); > diff --git a/storage/connect/ha_connect.cc b/storage/connect/ha_connect.cc > index 0194b08c792..b1b44085e53 100644 > --- a/storage/connect/ha_connect.cc > +++ b/storage/connect/ha_connect.cc > @@ -5192,7 +5192,8 @@ int ha_connect::delete_or_rename_table(const char *name, const char *to) > } // endif pos > > } // endif open_table_def > - > + else > + rc= ENOENT; > free_table_share(share); > } else // Temporary file > ok= true; > diff --git a/storage/connect/mysql-test/connect/r/drop-open-error.result b/storage/connect/mysql-test/connect/r/drop-open-error.result > index f0ad8553d8b..34f58a845dc 100644 > --- a/storage/connect/mysql-test/connect/r/drop-open-error.result > +++ b/storage/connect/mysql-test/connect/r/drop-open-error.result > @@ -2,6 +2,8 @@ create table t1 (c varchar(8)); > create table tcon engine=connect table_type=mysql CONNECTION='mysql://root@localhost/test/t1' SRCDEF='select c from t1 where c in ("foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar", "foo", "bar", "qux", "foobar")'; > ERROR HY000: Too long value for 'SRCDEF' > drop table mdev9949; > +Warnings: > +Warning 1017 Can't find file: './test/mdev9949.dos' (errno: 2 "No such file or directory") > drop table t1; > select @@secure_file_priv 'must be NULL'; > must be NULL > diff --git a/storage/federated/ha_federated.cc b/storage/federated/ha_federated.cc > index 00168791d35..b62b74d12bb 100644 > --- a/storage/federated/ha_federated.cc > +++ b/storage/federated/ha_federated.cc > @@ -484,7 +484,8 @@ int federated_db_init(void *p) > federated_hton->commit= federated_commit; > federated_hton->rollback= federated_rollback; > federated_hton->create= federated_create_handler; > - federated_hton->flags= HTON_ALTER_NOT_SUPPORTED | HTON_NO_PARTITION; > + federated_hton->flags= (HTON_ALTER_NOT_SUPPORTED | HTON_NO_PARTITION | > + HTON_AUTOMATIC_DELETE_TABLE); > > /* > Support for transactions disabled until WL#2952 fixes it. > diff --git a/storage/federated/ha_federated.h b/storage/federated/ha_federated.h > index 080d0ebd5f0..2e785f87858 100644 > --- a/storage/federated/ha_federated.h > +++ b/storage/federated/ha_federated.h > @@ -244,7 +244,10 @@ class ha_federated: public handler > void update_auto_increment(void); > int repair(THD* thd, HA_CHECK_OPT* check_opt); > int optimize(THD* thd, HA_CHECK_OPT* check_opt); > - > + int delete_table(const char *name) > + { > + return 0; > + } > int delete_all_rows(void); > int truncate(); > int create(const char *name, TABLE *form, > diff --git a/storage/federatedx/ha_federatedx.cc b/storage/federatedx/ha_federatedx.cc > index 6547964cc11..2370473236e 100644 > --- a/storage/federatedx/ha_federatedx.cc > +++ b/storage/federatedx/ha_federatedx.cc > @@ -438,7 +438,8 @@ int federatedx_db_init(void *p) > federatedx_hton->rollback= ha_federatedx::rollback; > federatedx_hton->discover_table_structure= ha_federatedx::discover_assisted; > federatedx_hton->create= federatedx_create_handler; > - federatedx_hton->flags= HTON_ALTER_NOT_SUPPORTED; > + federatedx_hton->flags= (HTON_ALTER_NOT_SUPPORTED | > + HTON_AUTOMATIC_DELETE_TABLE); > federatedx_hton->create_derived= create_federatedx_derived_handler; > federatedx_hton->create_select= create_federatedx_select_handler; > > diff --git a/storage/federatedx/ha_federatedx.h b/storage/federatedx/ha_federatedx.h > index a62456e1c33..67fe5f8cc22 100644 > --- a/storage/federatedx/ha_federatedx.h > +++ b/storage/federatedx/ha_federatedx.h > @@ -431,7 +431,10 @@ class ha_federatedx: public handler > void update_auto_increment(void); > int repair(THD* thd, HA_CHECK_OPT* check_opt); > int optimize(THD* thd, HA_CHECK_OPT* check_opt); > - > + int delete_table(const char *name) > + { > + return 0; > + } > int delete_all_rows(void); > int create(const char *name, TABLE *form, > HA_CREATE_INFO *create_info); //required > diff --git a/storage/innobase/handler/ha_innodb.cc b/storage/innobase/handler/ha_innodb.cc > index 6940f8ff3bd..97cfdb01d9f 100644 > --- a/storage/innobase/handler/ha_innodb.cc > +++ b/storage/innobase/handler/ha_innodb.cc > @@ -13347,5 +13347,18 @@ ha_innobase::discard_or_import_tablespace( > } > > /** > + @return 1 if frm file exists > + @return 0 if it doesn't exists > +*/ > + > +static bool frm_file_exists(const char *path) > +{ > + char buff[FN_REFLEN]; > + strxnmov(buff, FN_REFLEN, path, reg_ext, NullS); > + return !access(buff, F_OK); > +} > + > + > +/** > Drops a table from an InnoDB database. Before calling this function, > MySQL calls innobase_commit to commit the transaction of the current user. > @@ -13446,7 +13459,9 @@ inline int ha_innobase::delete_table(const char* name, enum_sql_command sqlcom) > } > } > > - if (err == DB_TABLE_NOT_FOUND) { > + if (err == DB_TABLE_NOT_FOUND && > + frm_file_exists(name)) > + { > /* Test to drop all tables which matches db/tablename + '#'. > Only partitions can have '#' as non-first character in > the table name! > diff --git a/storage/mroonga/mrn_table.cpp b/storage/mroonga/mrn_table.cpp > index b1b2db6161b..037a6a59487 100644 > --- a/storage/mroonga/mrn_table.cpp > +++ b/storage/mroonga/mrn_table.cpp > @@ -1080,6 +1080,7 @@ TABLE_SHARE *mrn_create_tmp_table_share(TABLE_LIST *table_list, const char *path > if (open_table_def(thd, share, GTS_TABLE)) > { > *error = ER_CANT_OPEN_FILE; > + mrn_free_tmp_table_share(share); > DBUG_RETURN(NULL); > } > DBUG_RETURN(share); > diff --git a/storage/myisam/mi_delete_table.c b/storage/myisam/mi_delete_table.c > index 7990c3e8a80..d318b44720a 100644 > --- a/storage/myisam/mi_delete_table.c > +++ b/storage/myisam/mi_delete_table.c > @@ -28,19 +28,23 @@ > > int mi_delete_table(const char *name) > { > + int error= 0; > DBUG_ENTER("mi_delete_table"); > > #ifdef EXTRA_DEBUG > check_table_is_closed(name,"delete"); > #endif > > - if (mysql_file_delete_with_symlink(mi_key_file_kfile, name, MI_NAME_IEXT, MYF(MY_WME)) || > - mysql_file_delete_with_symlink(mi_key_file_dfile, name, MI_NAME_DEXT, MYF(MY_WME))) > - DBUG_RETURN(my_errno); > + if (mysql_file_delete_with_symlink(mi_key_file_kfile, name, MI_NAME_IEXT, > + MYF(MY_WME))) > + error= my_errno; > + if (mysql_file_delete_with_symlink(mi_key_file_dfile, name, MI_NAME_DEXT, > + MYF(MY_WME))) > + error= my_errno; > > // optionally present: > mysql_file_delete_with_symlink(mi_key_file_dfile, name, ".OLD", MYF(0)); > mysql_file_delete_with_symlink(mi_key_file_dfile, name, ".TMD", MYF(0)); > > - DBUG_RETURN(0); > + DBUG_RETURN(error); > } > diff --git a/storage/perfschema/ha_perfschema.cc b/storage/perfschema/ha_perfschema.cc > index 33d4c854c8b..c4c59d109f5 100644 > --- a/storage/perfschema/ha_perfschema.cc > +++ b/storage/perfschema/ha_perfschema.cc > @@ -95,10 +95,11 @@ static int pfs_init_func(void *p) > > pfs_hton->create= pfs_create_handler; > pfs_hton->show_status= pfs_show_status; > - pfs_hton->flags= HTON_ALTER_NOT_SUPPORTED | > + pfs_hton->flags= (HTON_ALTER_NOT_SUPPORTED | > HTON_TEMPORARY_NOT_SUPPORTED | > HTON_NO_PARTITION | > - HTON_NO_BINLOG_ROW_OPT; > + HTON_NO_BINLOG_ROW_OPT | > + HTON_AUTOMATIC_DELETE_TABLE); > > /* > As long as the server implementation keeps using legacy_db_type, > diff --git a/storage/sequence/sequence.cc b/storage/sequence/sequence.cc > index 8684a5c60b9..31522b8f3b7 100644 > --- a/storage/sequence/sequence.cc > +++ b/storage/sequence/sequence.cc > @@ -69,10 +69,15 @@ class ha_seq: public handler > > /* open/close/locking */ > int create(const char *name, TABLE *table_arg, > - HA_CREATE_INFO *create_info) { return HA_ERR_WRONG_COMMAND; } > + HA_CREATE_INFO *create_info) > + { return HA_ERR_WRONG_COMMAND; } > > int open(const char *name, int mode, uint test_if_locked); > int close(void); > + int delete_table(const char *name) > + { > + return 0; > + } > THR_LOCK_DATA **store_lock(THD *, THR_LOCK_DATA **, enum thr_lock_type); > > /* table scan */ > @@ -503,6 +508,7 @@ static int init(void *p) > hton->savepoint_set= hton->savepoint_rollback= hton->savepoint_release= > dummy_savepoint; > hton->create_group_by= create_group_by_handler; > + hton->flags= HTON_AUTOMATIC_DELETE_TABLE; > return 0; > } > > @@ -526,4 +532,3 @@ maria_declare_plugin(sequence) > MariaDB_PLUGIN_MATURITY_STABLE > } > maria_declare_plugin_end; > - > diff --git a/storage/sphinx/ha_sphinx.cc b/storage/sphinx/ha_sphinx.cc > index 8403c767796..d60a4d229e6 100644 > --- a/storage/sphinx/ha_sphinx.cc > +++ b/storage/sphinx/ha_sphinx.cc > @@ -696,7 +696,7 @@ handlerton sphinx_hton = > NULL, // create_cursor_read_view > NULL, // set_cursor_read_view > NULL, // close_cursor_read_view > - HTON_CAN_RECREATE > + HTON_CAN_RECREATE | HTON_AUTOMATIC_DELETE_TABLE > }; > #else > static handlerton * sphinx_hton_ptr = NULL; > @@ -749,7 +749,7 @@ static int sphinx_init_func ( void * p ) > hton->close_connection = sphinx_close_connection; > hton->show_status = sphinx_show_status; > hton->panic = sphinx_panic; > - hton->flags = HTON_CAN_RECREATE; > + hton->flags = HTON_CAN_RECREATE | HTON_AUTOMATIC_DELETE_TABLE; > #endif > } > SPH_RET(0); > diff --git a/storage/spider/spd_table.cc b/storage/spider/spd_table.cc > index 1607dd07902..583bfebbed6 100644 > --- a/storage/spider/spd_table.cc > +++ b/storage/spider/spd_table.cc > @@ -7249,7 +7249,7 @@ int spider_db_init( > DBUG_ENTER("spider_db_init"); > spider_hton_ptr = spider_hton; > > - spider_hton->flags = HTON_NO_FLAGS; > + spider_hton->flags = HTON_NO_FLAGS | HTON_AUTOMATIC_DELETE_TABLE; > #ifdef HTON_CAN_READ_CONNECT_STRING_IN_PARTITION > spider_hton->flags |= HTON_CAN_READ_CONNECT_STRING_IN_PARTITION; > #endif Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 2

[Maria-developers] Adding proper expressions to mysqltest
by Игорь Балашенко 14 Jun '20

14 Jun '20

----------------------------------------------------------- Sorry, I had to duplicate this message again to get more attention. I didn`t forget about theme this time ------------------------------------------------------------ Hello mariaDB developers! I am a participant of GSOC 2020 mentored by Nikita Malyavin and Sergey Golubchik. Currently I am working on mysqltest parser improvements and I have a question to discuss with everyone: To be short: Currently there are expressions in if and while to be calculated i.e if(<expr>) code But it is written using switch and works only in condition block. Code like: let $a= 1 + 2 Makes $a = "1 + 2" Obviously mysqltest syntax needs proper expressions thats why I decided to make them. There are following questions: What syntax do you want to see? let $a = 1 + 2 $a = 3 Is a bad idea because we will break lots of previous test cases. let $a = $((1 + 2)) That is done in a bash style, so it seems to be more comfortable to everyone. Another question is what operators should be in expressions? To begin with I want to implement: +, -. *, /, ++, --, <, >, ==, != To implement expressions I want to use Byson to make it easier to rewrite parser if needed I would be glad to read feedback and critique. ------------------------------

1 0

[Maria-developers] (no subject)
by Игорь Балашенко 12 Jun '20

12 Jun '20

Hello mariaDB developers! I am a participant of GSOC 2020 mentored by Nikita Malyavin and Sergey Golubchik. Currently I am working on mysqltest parser improvements and I have a question to discuss with everyone: To be short: Currently there are expressions in if and while to be calculated i.e if(<expr>) code But it is written using switch and works only in condition block. Code like: let $a= 1 + 2 Makes $a = "1 + 2" Obviously mysqltest syntax needs proper expressions thats why I decided to make them. There are following questions: What syntax do you want to see? let $a = 1 + 2 $a = 3 Is a bad idea because we will break lots of previous taest cases. let $a = $((1 + 2)) That is done in a bash style, so it seems to be more comfortable to everyone. Another question is what operators should be in expressions? To begin with I want to implement: +, -. *, /, ++, --, <, >, ==, != To implement expressions I want to use Byson to make it easier to rewrite parser if needed I would be glad to read feedback and critique. ------------------------------ GSOC2020 participant Balashenko Igor

3 2

Re: [Maria-developers] 2849076672d: MDEV-16470: switch off user variables (and fixes of its support)
by Sergei Golubchik 12 Jun '20

12 Jun '20

Hi, Oleksandr! On Jun 12, Oleksandr Byelkin wrote: > revision-id: 2849076672d (mariadb-10.5.2-322-g2849076672d) > parent(s): bb47050e1fd > author: Oleksandr Byelkin <sanja(a)mariadb.com> > committer: Oleksandr Byelkin <sanja(a)mariadb.com> > timestamp: 2020-06-12 08:32:56 +0200 > message: > > MDEV-16470: switch off user variables (and fixes of its support) > > diff --git a/client/mysqltest.cc b/client/mysqltest.cc > index 55c4a66a89d..24e0cd25108 100644 > --- a/client/mysqltest.cc > +++ b/client/mysqltest.cc > &data, &data_length)) > { > dynstr_append(ds, "-- "); > - switch (type) > + if (type <= SESSION_TRACK_END) > { > - case SESSION_TRACK_SYSTEM_VARIABLES: > - dynstr_append(ds, "Tracker : SESSION_TRACK_SYSTEM_VARIABLES\n"); > - break; > - case SESSION_TRACK_SCHEMA: > - dynstr_append(ds, "Tracker : SESSION_TRACK_SCHEMA\n"); > - break; > - case SESSION_TRACK_STATE_CHANGE: > - dynstr_append(ds, "Tracker : SESSION_TRACK_STATE_CHANGE\n"); > - break; > - case SESSION_TRACK_GTIDS: > - dynstr_append(ds, "Tracker : SESSION_TRACK_GTIDS\n"); > - break; > - case SESSION_TRACK_TRANSACTION_CHARACTERISTICS: > - dynstr_append(ds, "Tracker : SESSION_TRACK_TRANSACTION_CHARACTERISTICS\n"); > - break; > - case SESSION_TRACK_TRANSACTION_TYPE: > - dynstr_append(ds, "Tracker : SESSION_TRACK_TRANSACTION_TYPE\n"); > - break; > - default: > - DBUG_ASSERT(0); > - dynstr_append(ds, "\n"); > + dynstr_append(ds, trking_info_desc[type]); > + } > + else > + { > + DBUG_ASSERT(0); > + dynstr_append(ds, "Trecker???\n"); "Tracker" > } > - > > dynstr_append(ds, "-- "); > dynstr_append_mem(ds, data, data_length); > diff --git a/libmariadb b/libmariadb > index ce74fd0c400..db7adf63fbf 160000 > --- a/libmariadb > +++ b/libmariadb > @@ -1 +1 @@ > -Subproject commit ce74fd0c4009ed9f4bcbdb4a01e96c823e961dc3 > +Subproject commit db7adf63fbf242a07c355f192a4d8c60323b5d89 No, I don't see any need for that now. C/C does not know anything about SESSION_TRACK_USER_VARIABLES, let's just keep it that way. > diff --git a/sql/item_func.cc b/sql/item_func.cc > index da185f9878b..448bfd380a2 100644 > --- a/sql/item_func.cc > +++ b/sql/item_func.cc > @@ -4702,10 +4702,12 @@ update_hash(user_var_entry *entry, bool set_null, void *ptr, size_t length, > entry->unsigned_flag= unsigned_arg; > } > entry->type=type; > +#ifdef USER_VAR_TACKING TRACKING > #ifndef EMBEDDED_LIBRARY > THD *thd= current_thd; > thd->session_tracker.user_variables.mark_as_changed(thd, entry); > #endif > +#endif // USER_VAR_TACKING > return 0; > } > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] 4f528bb514c: MDEV-22011: DISTINCT with JSON_ARRAYAGG gives wrong results
by Sergey Petrunia 11 Jun '20

11 Jun '20

Hi Varun, I only have input about comments/variable names. Please find it below. On Tue, Mar 24, 2020 at 10:14:38AM +0530, Varun wrote: > revision-id: 4f528bb514c607d848b30119cea135e37e0d9c69 (mariadb-10.5.0-425-g4f528bb514c) > parent(s): 1cd8e8213b6019794ca7f01e8276cfd1ba6408cf > author: Varun Gupta > committer: Varun Gupta > timestamp: 2020-03-24 10:13:28 +0530 > message: > > MDEV-22011: DISTINCT with JSON_ARRAYAGG gives wrong results > > For JSON_ARRAYAGG with DISTINCT also store the null bytes for the record > inside the Unique object > > --- > mysql-test/main/func_json.result | 4 +- > sql/item_jsonfunc.cc | 36 ++++++++++++-- > sql/item_jsonfunc.h | 9 ++-- > sql/item_sum.cc | 101 ++++++++++++++++++++++++++++++++++----- > sql/item_sum.h | 13 +++-- > 5 files changed, 133 insertions(+), 30 deletions(-) > > diff --git a/sql/item_jsonfunc.cc b/sql/item_jsonfunc.cc > index a3b5d3b7fac..ed496df8169 100644 > --- a/sql/item_jsonfunc.cc > +++ b/sql/item_jsonfunc.cc > @@ -1452,6 +1452,31 @@ static int append_json_value(String *str, Item *item, String *tmp_val) > } > > > +bool append_json_value(String *str, String *res, Item *item, bool null_value) There is another append_json_value right above this one. That function has no comments but let's not follow that bad pattern. - Please add a function comment describing this function: @brief should describe this function @detail should describe the difference from the one above. It is very counter-intuitive that the parameter named "res" does not hold the result of this function. Please rename it (to value?) (can it be changed to be const String* ?) > +{ > + if (null_value) > + return str->append("null", 4); > + > + if (item->type_handler()->is_bool_type()) > + { > + return ((*res)[0] == '1') ? > + str->append("true", 4) : > + str->append("false", 5); > + } > + > + if (item->is_json_type()) > + return str->append(res->ptr(), res->length()); > + > + if (item->result_type() == STRING_RESULT) > + { > + return str->append("\"", 1) || > + st_append_escaped(str, res) || > + str->append("\"", 1); > + } > + return st_append_escaped(str, res); This made me wonder why something is escaped if it is not enclosed in quotes, but now I see this is just copied from above. > +} > + > + > static int append_json_keyname(String *str, Item *item, String *tmp_val) > { > String *sv= item->val_str(tmp_val); > diff --git a/sql/item_sum.cc b/sql/item_sum.cc > index 5c732095573..a04dbdc19e2 100644 > --- a/sql/item_sum.cc > +++ b/sql/item_sum.cc > @@ -3552,6 +3552,54 @@ int group_concat_key_cmp_with_distinct(void* arg, const void* key1, > } > > > +int group_concat_key_cmp_with_distinct_with_nulls(void* arg, > + const void* key1_arg, > + const void* key2_arg) > +{ > + Item_func_group_concat *item_func= (Item_func_group_concat*)arg; > + > + uchar *key1= (uchar*)key1_arg + item_func->table->s->null_bytes; > + uchar *key2= (uchar*)key2_arg + item_func->table->s->null_bytes; > + > + for (uint i= 0; i < item_func->arg_count_field; i++) This loop is actually not needed, right? JSON_ARRAYAGG only accepts one argument. Please add a note. > + { > + Item *item= item_func->args[i]; > + /* > + If item is a const item then either get_tmp_table_field returns 0 > + or it is an item over a const table. > + */ > + if (item->const_item()) > + continue; > + /* > + We have to use get_tmp_table_field() instead of > + real_item()->get_tmp_table_field() because we want the field in > + the temporary table, not the original field > + */ > + Field *field= item->get_tmp_table_field(); > + > + if (!field) > + continue; > + > + if (field->is_null_in_record((uchar*)key1_arg) && > + field->is_null_in_record((uchar*)key2_arg)) > + continue; > + > + if (field->is_null_in_record((uchar*)key1_arg)) > + return -1; > + > + if (field->is_null_in_record((uchar*)key2_arg)) > + return 1; > + > + uint offset= (field->offset(field->table->record[0]) - > + field->table->s->null_bytes); > + int res= field->cmp(key1 + offset, key2 + offset); > + if (res) > + return res; > + } > + return 0; > +} > + > + > /** > function of sort for syntax: GROUP_CONCAT(expr,... ORDER BY col,... ) > */ > @@ -3650,9 +3699,11 @@ int dump_leaf_key(void* key_arg, element_count count __attribute__((unused)), > result->append(*item->separator); > > > + bool is_null= false; > for (; arg < arg_end; arg++) > { > String *res; > + is_null= false; (1) > /* > We have to use get_tmp_table_field() instead of > real_item()->get_tmp_table_field() because we want the field in > @@ -3661,7 +3712,11 @@ int dump_leaf_key(void* key_arg, element_count count __attribute__((unused)), > because it contains both order and arg list fields. > */ > if ((*arg)->const_item()) > + { > res= (*arg)->val_str(&tmp); > + if ((*arg)->null_value) > + is_null= TRUE; (2) Please either use true/false (preferred) or TRUE / FALSE. > + } > else > { > Field *field= (*arg)->get_tmp_table_field(); ... > diff --git a/sql/item_sum.h b/sql/item_sum.h > index b10d9f5974d..908dbe3cce9 100644 > --- a/sql/item_sum.h > +++ b/sql/item_sum.h > @@ -1897,14 +1899,19 @@ class Item_func_group_concat : public Item_sum > */ > Item_func_group_concat *original; > > + bool exclude_nulls; > + > /* > Used by Item_func_group_concat and Item_func_json_arrayagg. The latter > needs null values but the former doesn't. > */ Is this comment relevant here? I think it's better to remove this and instead add a comment for the exclude_nulls above. > - bool add(bool exclude_nulls); > + bool add(); > > friend int group_concat_key_cmp_with_distinct(void* arg, const void* key1, > const void* key2); > + friend int group_concat_key_cmp_with_distinct_with_nulls(void* arg, > + const void* key1, > + const void* key2); > friend int group_concat_key_cmp_with_order(void* arg, const void* key1, > const void* key2); > friend int dump_leaf_key(void* key_arg, BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Improving `source` in mysqltest
by Nikita Malyavin 11 Jun '20

11 Jun '20

Hello mariadb developers! This year we have a student on mysqltest parser improvements, mentored by me and @serg. The major improvement will be proper expressions support, which you'll soon receive an email about. Another a bit more minor-ish improvement we want our student Igor to make is parameters support for source command, which I want to ask you about, which way is better to implement. It's quite spread in the tests to parametrize source with variables, for example > --let SEARCH_FILE=$MYSQLTEST_VARDIR/log/mysqld.1.err > --let SEARCH_PATTERN= This connection closed normally without > authentication > --source include/search_pattern_in_file.inc Another example is period/create_triggers.inc. So, we want make it possible to write > --source do_something.inc A=1 B=foobar And the question is, how should it behave. One way is to make it a strict equivalent of > let A=1; > let B=foobar; > source do_something.inc; This allows to reuse previously set values, for example: > --source do_something.inc A=1 B=foobar > ... > --source do_something.inc A=2 Which literally means > let A=1; > let B=foobar; > --source do_something.inc; ... let A=2; > --source do_something.inc; Another way is to make it work like local arguments, so the arguments are put in the stack, and as a result > let A=qwe; > --source do_something.inc A=1 B=foobar > --echo $A; outputs qwe. What do you think of this, what's more suitable for us? This is an important question, since all of us are using mysqltest, and we probably wouldn't change the behavior ever later. -- Yours truly, Nikita Malyavin

1 0

Re: [Maria-developers] MDEV-21831: Assertion `length == pack_length()' failed in Field_inet6::sort_string upon INSERT into RocksDB table
by Sergey Petrunia 11 Jun '20

11 Jun '20

On Thu, Jun 11, 2020 at 03:41:24PM +0530, Varun Gupta wrote: > > /* > > pack_length() returns size (in bytes) used to store field data in > > memory > > (i.e. it returns the maximum size of the field in a row of the table, > > which is located in RAM). > > */ > > virtual uint32 pack_length() const { return (uint32) field_length; } > > > > so, it is size in bytes, while strnxfrmlen() accepts length *in > > characters*. > > That is, strnxfrmlen(field->pack_length()) is certainly not correct. > > > > > This is not correct, strnxfrmlen accepts length in bytes and there we > change the length to characters and then calculate the weights. You're right. I've missed this part. The patch is ok to push, then. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Progress report week 5
by Mohammed Hammaad Mateen 10 Jun '20

10 Jun '20

Greetings, Hope you are safe and doing great, This post describes the things I've done during 1-7 June under the mentor-ship of Sergei Golubchik and Oleksandr Byelkin for GSoC-20. The tasks taken up for this week Greetings mentors, I've realized that to extend the parser for multi delete, I should be changing the sql_yacc.yy file, for incorporating opt_returning rule in multi-delete. opt_returning rule works completely fine with single table delete, but to tweak it to adapt for multi-delete, required me to learn some bison grammar syntax and semantics and that took me a couple of days. Moreover everytime I made changes to the grammar, the server used to crash rendering all changes invalid. I'll try to push the code that was triggering crashes, shortly. Thank you Regards, Mohammed Hammaad Mateen

1 0

Re: [Maria-developers] 5ff9525ff86: MDEV-22722 Assertion "inited==NONE" failed in handler::ha_index_init on the slave during UPDATE
by Sergei Golubchik 10 Jun '20

10 Jun '20

Hi, Sachin! On Jun 10, Sachin wrote: > revision-id: 5ff9525ff86 (mariadb-10.5.2-290-g5ff9525ff86) > parent(s): ff881056b8d > author: Sachin <sachin.setiya(a)mariadb.com> > committer: Sachin <sachin.setiya(a)mariadb.com> > timestamp: 2020-06-07 16:45:01 +0530 > message: > > diff --git a/sql/log_event_server.cc b/sql/log_event_server.cc > index 5910ece12f0..ffda352c866 100644 > --- a/sql/log_event_server.cc > +++ b/sql/log_event_server.cc > @@ -5570,6 +5570,12 @@ int Rows_log_event::do_apply_event(rpl_group_info *rgi) > { > master_had_triggers= table->master_had_triggers; > bool transactional_table= table->file->has_transactions_and_rollback(); > + if (get_general_type_code() == WRITE_ROWS_EVENT) > + table->file->prepare_for_insert(0); > + else if (get_general_type_code() == UPDATE_ROWS_EVENT || > + get_general_type_code() == DELETE_ROWS_EVENT) > + table->file->prepare_for_insert(1); > + You can simplify it as table->file->prepare_for_insert(get_general_type_code() != WRITE_ROWS_EVENT); otherwise ok to push Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] MDEV-21831: Assertion `length == pack_length()' failed in Field_inet6::sort_string upon INSERT into RocksDB table
by Sergey Petrunia 09 Jun '20

09 Jun '20

Hi Varun, > MDEV-21831: Assertion `length == pack_length()' failed in Field_inet6::sort_string upon INSERT into RocksDB table > > For INET6 columns the values are stored as BINARY columns and returned to the client in TEXT format. > For rocksdb the indexes store mem-comparable images for columns, so use the pack_length() to store > the mem-comparable form for INET6 columns. This would also remain consistent with CHAR columns. > > diff --git a/storage/rocksdb/rdb_datadic.cc b/storage/rocksdb/rdb_datadic.cc > index 21a9b57b555e..6e73535d1de7 100644 > --- a/storage/rocksdb/rdb_datadic.cc > +++ b/storage/rocksdb/rdb_datadic.cc > @@ -3317,7 +3317,9 @@ bool Rdb_field_packing::setup(const Rdb_key_def *const key_descr, > field->field_length = field->char_length() * cs->mbmaxlen. > */ > const CHARSET_INFO *cs = field->charset(); > - m_max_image_len = cs->strnxfrmlen(field->field_length); > + m_max_image_len = cs->strnxfrmlen(type == MYSQL_TYPE_STRING ? > + field->pack_length() : > + field->field_length); Field::pack_length is defined as: /* pack_length() returns size (in bytes) used to store field data in memory (i.e. it returns the maximum size of the field in a row of the table, which is located in RAM). */ virtual uint32 pack_length() const { return (uint32) field_length; } so, it is size in bytes, while strnxfrmlen() accepts length *in characters*. That is, strnxfrmlen(field->pack_length()) is certainly not correct. Rdb_field_packing::m_max_image_len is defined as: /* Length of mem-comparable image of the field, in bytes */ int m_max_image_len; while pack_length() seems to describe table->record[0] format (or is it KeyTupleFormat) ? Field::sort_length() doesn't have a definition but it seems to be a closer match as it describes the length of the mem-comparable form? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] 06942363ebc: MDEV-14347 CREATE PROCEDURE returns no error when using an unknown variable
by Sergei Golubchik 09 Jun '20

09 Jun '20

Hi, Alexander! On Jun 07, Alexander Barkov wrote: > revision-id: 06942363ebc (mariadb-10.5.2-311-g06942363ebc) > parent(s): 0d6d63e1505 > author: Alexander Barkov <bar(a)mariadb.com> > committer: Alexander Barkov <bar(a)mariadb.com> > timestamp: 2020-06-03 12:11:07 +0400 > message: > > MDEV-14347 CREATE PROCEDURE returns no error when using an unknown variable > > diff --git a/sql/item.cc b/sql/item.cc > index 8ea6366e6c4..5800b9bc95e 100644 > --- a/sql/item.cc > +++ b/sql/item.cc > @@ -764,6 +764,29 @@ bool Item_field::collect_item_field_processor(void *arg) > } > > > +void Item_ident::undeclared_spvar_error() const Currently it's used only in Item_field::unknown_splocal_processor(). Do you plan to use this somewhere else? Why is it a separate method and in Item_ident? > +{ > + /* > + We assume this is an unknown SP variable, possibly a ROW variable. > + Print the leftmost name in the error: > + SET var=a; -> a > + SET var=a.b; -> a > + SET var=a.b.c; -> a > + */ > + my_error(ER_SP_UNDECLARED_VAR, MYF(0), db_name.str ? db_name.str : > + table_name.str ? table_name.str : > + field_name.str); > +} > + > +bool Item_field::unknown_splocal_processor(void *arg) > +{ > + DBUG_ENTER("Item_field::unknown_splocal_processor"); > + DBUG_ASSERT(type() == FIELD_ITEM); > + undeclared_spvar_error(); > + DBUG_RETURN(true); > +} > + > + > bool Item_field::add_field_to_set_processor(void *arg) > { > DBUG_ENTER("Item_field::add_field_to_set_processor"); > @@ -8012,6 +8035,27 @@ void Item_ref::cleanup() > } > > > +bool Item_ref::unknown_splocal_processor(void *arg) > +{ > + DBUG_ENTER("Item_ref::unknown_splocal_processor"); > + DBUG_ASSERT(type() == REF_ITEM); > + DBUG_ASSERT(ref_type() == REF); > + undeclared_spvar_error(); > + DBUG_RETURN(true); > +} > + > + > +bool Item_ref::walk(Item_processor processor, bool walk_subquery, void *arg) > +{ > + if (processor == &Item::unknown_splocal_processor) > + return unknown_splocal_processor(arg); Why is that? This is very unusual, normally walk() should not change its behavior based on the processor. > + if (ref && *ref) > + return (*ref)->walk(processor, walk_subquery, arg) || > + (this->*processor)(arg); > + return false; > +} > + > + > /** > Transform an Item_ref object with a transformer callback function. > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 2

Re: [Maria-developers] JSON aggregate functions working on JSON documents
by Sergey Petrunia 08 Jun '20

08 Jun '20

On Mon, Jun 08, 2020 at 04:05:48PM +0400, Alexey Botchkov wrote: > So the problem is that we (MariaDB) doesn't consider the JSON field content > as a JSON, but as a plain text. > That affects not only aggregate functions, but any other expecting JSON > arguments. > To me it's the bug, should be fixed. Will you report it, Sergei? Filed: https://jira.mariadb.org/browse/MDEV-22837 BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Progress Report GSoC 2020 (1-6 June)
by Anton Mikhaylenko 08 Jun '20

08 Jun '20

Good day, everybody. In this post, I want to describe what I did this week, June 1-6. I have added the initial implementation of the hill_climbing class. It includes an update function - the main function that changes the required number of threads. To implement this function, I added the Complex class - the base class for working with complex numbers. https://github.com/sapiest/server/commits/10.5-update_threadpool_concurrency The main work is not completed yet, as the algorithm requires adjustments and adaptation in MariaDB environment. Regards, Anton MIkhailenko

1 0

[Maria-developers] JSON aggregate functions working on JSON documents
by Sergey Petrunia 08 Jun '20

08 Jun '20

Hi Alexey, I'm looking at how JSON_ARRAYAGG and JSON_OBJECTAGG work at processing JSON documents. I'm trying these examples, first on MySQL-8: create table t30 (col1 json); insert into t30 values('{"color":"red", "size":1}' ); insert into t30 values('{"color":"blue", "size":2}' ); select JSON_ARRAYAGG(col1) from t30; +-------------------------------------------------------------+ | JSON_ARRAYAGG(col1) | +-------------------------------------------------------------+ | [{"size": 1, "color": "red"}, {"size": 2, "color": "blue"}] | +-------------------------------------------------------------+ Good. create table t31 (name varchar(100), contents json); insert into t31 values('obj1', '{"color":"blue", "size":2}' ); insert into t31 values('obj2', '{"color":"red", "size":1}' ); select JSON_OBJECTAGG(name, contents) from t31; +-----------------------------------------------------------------------------+ | JSON_OBJECTAGG(name, contents) | +-----------------------------------------------------------------------------+ | {"obj1": {"size": 2, "color": "blue"}, "obj2": {"size": 1, "color": "red"}} | +-----------------------------------------------------------------------------+ Good. Now, trying these on MariaDB, I get: MariaDB [test]> select JSON_ARRAYAGG(col1) from t30; +------------------------------------------------------------------------+ | JSON_ARRAYAGG(col1) | +------------------------------------------------------------------------+ | ["{\"color\":\"red\", \"size\":1}","{\"color\":\"blue\", \"size\":2}"] | +------------------------------------------------------------------------+ MariaDB [test]> select JSON_OBJECTAGG(name, contents) from t31; +---------------------------------------------------------------------------------------+ | JSON_OBJECTAGG(name, contents) | +---------------------------------------------------------------------------------------+ | {"obj1":"{\"color\":\"blue\", \"size\":2}", "obj2":"{\"color\":\"red\", \"size\":1}"} | +---------------------------------------------------------------------------------------+ The JSON documents are coverted into strings. I think, the users will prefer MySQL's behavior. But which behaivor is "correct"? (i.e. required by the standard, etc) Is MariaDBs's behaivor a bug? Is this expected? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] MDEV-11563: GROUP_CONCAT(DISTINCT ...) may produce a non-distinct list
by Sergey Petrunia 07 Jun '20

07 Jun '20

Hi Varun, > revision-id: c408c41230c4a9de0114d13aaf861d3ced65f4c7 (mariadb-10.5.0-424-gc408c41230c) > parent(s): 95da2113a050ad739fdaf60ee871329468a01554 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2020-03-31 11:57:37 +0530 > message: > > MDEV-11563: GROUP_CONCAT(DISTINCT ...) may produce a non-distinct list > The commit message should describe the error and the fix made. (especially important here as this patch fixes only a part of the issue with GROUP_CONCAT) > --- > mysql-test/main/func_gconcat.result | 53 ++++++++++++++++++++++--------------- > mysql-test/main/func_gconcat.test | 7 +++++ > sql/item_sum.cc | 40 +++++++++++++++++----------- > sql/item_sum.h | 3 ++- > 4 files changed, 65 insertions(+), 38 deletions(-) > > diff --git a/sql/item_sum.h b/sql/item_sum.h > index e221d04397d..b10d9f5974d 100644 > --- a/sql/item_sum.h > +++ b/sql/item_sum.h > @@ -1879,7 +1879,8 @@ class Item_func_group_concat : public Item_sum > bool warning_for_row; > bool always_null; > bool force_copy_fields; > - bool no_appended; > + /** True if result has been written to output buffer. */ > + bool m_result_finalized; Why does this variable use the m_ convention when others do not? I think this is confusing (I stop for a moment to wonder how is it different from all others every time I see the name). Please remove this. What's "output buffer"? this->result ? And what does "result" mean here - the entire return value of GROUP_CONCAT function or some part of it? Please improve the comment. > /** Limits the rows in the result */ > Item *row_limit; > /** Skips a particular number of rows in from the result*/ Ok to push after this is addressed. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Is it okay that JSON_OBJECTAGG creates objects with duplicate elements?
by Sergey Petrunia 07 Jun '20

07 Jun '20

Hi Alexey, Consider this example: create table t20 (a varchar(10), b int); insert into t20 values ('red', 1), ('blue', 2), ('red',1001), ('blue', 1002); select json_objectagg(a,b) from t20; +----------------------------------------------+ | json_objectagg(a,b) | +----------------------------------------------+ | {"red":1, "blue":2, "red":1001, "blue":1002} | +----------------------------------------------+ the created object has multiple members with the same name. Is this expected or should be reported as a bug? In MySQL-8, the above example produces this: +-----------------------------+ | json_objectagg(a,b) | +-----------------------------+ | {"red": 1001, "blue": 1002} | +-----------------------------+ BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] 0f91c62d332: MDEV-22722 Assertion "inited==NONE" failed in handler::ha_index_init on the slave during UPDATE
by Sergei Golubchik 07 Jun '20

07 Jun '20

Hi, Sachin! Two questions: 1. This looks like there was no replication tests for periods of long uniques at all? Please add them (create an MDEV, like "replication tests of long unique") and add various relevant replication tests in 10.4. 2. It feels a bit risky to backport my changes to 10.4. At least you definitely should take the latest code from 10.5. Not cherry-pick one commit, but copy handler::prepare_for_insert() and other relevant code from 10.5. May be it'd be safer not to backport? What would your fix look like if you wouldn't have prepare_for_insert()? Also, a couple of comments to the code, see below. On Jun 03, Sachin wrote: > revision-id: 0f91c62d332 (mariadb-10.4.11-225-g0f91c62d332) > parent(s): c01b3d91690 > author: Sachin <sachin.setiya(a)mariadb.com> > committer: Sachin <sachin.setiya(a)mariadb.com> > timestamp: 2020-06-02 08:24:24 +0530 > message: > > MDEV-22722 Assertion "inited==NONE" failed in handler::ha_index_init on the slave during UPDATE > > Add missing call for handler->prepare_for_insert() in rows event. > > diff --git a/mysql-test/main/long_unique_bugs_replication.test b/mysql-test/main/long_unique_bugs_replication.test > new file mode 100644 > index 00000000000..5348be3e7d3 > --- /dev/null > +++ b/mysql-test/main/long_unique_bugs_replication.test > @@ -0,0 +1,22 @@ > +# > +# Long unique bugs related to master slave replication > +# > + > +# > +# MDEV-22722 Assertion "inited==NONE" failed in handler::ha_index_init on the slave during UPDATE > +# > + > +--source include/master-slave.inc > +--source include/have_binlog_format_row.inc master-slave.inc should always be included last, after all have_xxx.inc files that may cause the test to be skipped. > + > +create table t1 (i1 int, a1 text, unique key i1 (a1)) engine=myisam; > +insert into t1 values (1,1); > +update t1 set a1 = 'd' limit 1; > + > +sync_slave_with_master; > +connection slave; > + > +connection master; > +drop table t1; > + > +--source include/rpl_end.inc > diff --git a/sql/log_event.cc b/sql/log_event.cc > index 3707f73a716..f4c0c546a38 100644 > --- a/sql/log_event.cc > +++ b/sql/log_event.cc > @@ -11568,6 +11568,7 @@ int Rows_log_event::do_apply_event(rpl_group_info *rgi) > { > master_had_triggers= table->master_had_triggers; > bool transactional_table= table->file->has_transactions(); > + table->file->prepare_for_insert(); > /* > table == NULL means that this table should not be replicated > (this was set up by Table_map_log_event::do_apply_event() > @@ -14344,6 +14345,7 @@ int Rows_log_event::find_row(rpl_group_info *rgi) > issue_long_find_row_warning(get_general_type_code(), m_table->alias.c_ptr(), > is_index_scan, rgi); > table->default_column_bitmaps(); > + table->file->prepare_for_insert(); Why do you need it here, I'd think that do_apply_event() should be enough? > DBUG_RETURN(error); > } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] MDEV-16230: Server crashes when Analyze format=json is run with a window function with empty PARTITION BY and ORDER BY clauses
by Sergey Petrunia 03 Jun '20

03 Jun '20

Hi Varun, * Please limit the commit comment to ~80 chars wide. * The .result file has r_total_time_ms. Use analyze-format.inc to avoid this. Ok to push after this is addressed. > commit 9e755e21799695b9a837a240a86ba5ef863ae7b3 > Author: Varun Gupta <varun.gupta(a)mariadb.com> > Date: Tue Jun 2 16:31:53 2020 +0530 > > MDEV-16230: Server crashes when Analyze format=json is run with a window function with empty PARTITION BY and ORDER BY clauses > > Currently when both PARTITION BY and ORDER BY clauses are empty then we create a Item with the first field > in the select list and sort with that field. It should be created as an Item_temptable_field instead of > Item_field because the print() function continues to work even if the table has been dropped. > ... BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] e14dcfe0bc7: MDEV-22715: SIGSEGV in radixsort_for_str_ptr and in native_compare/my_qsort2 (optimized builds)
by Sergey Petrunia 03 Jun '20

03 Jun '20

Hi Varun, Please add a comment above the declaration of Field::sort_string() that some implementations actually may write up to 8 bytes regardless of what size was requested. Ok to push after that. > revision-id: e14dcfe0bc743aa6b4ed2b3c4f7e9314aa01d6eb (mariadb-10.1.43-158-ge14dcfe0bc7) > parent(s): db1f73b26edaace41345f19bca507a7848aa39a8 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2020-05-29 21:13:38 +0530 > message: > > MDEV-22715: SIGSEGV in radixsort_for_str_ptr and in native_compare/my_qsort2 (optimized builds) > > For DECIMAL[(M[,D])] datatype max_sort_length was not being honoured which was leading to buffer > overflow while making the sort key. The fix to this problem would be to create sort keys for decimals > with atmost max_sort_key bytes > > Important: > The minimum value of max_sort_length has been raised to 8 (previously was 4), > so fixed size datatypes like DOUBLE and BIGINIT are not truncated for > lower values of max_sort_length. > ... BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] 7df57cc8206: MDEV-22719 Long unique keys are not created when individual key_part->length < max_key_length but SUM(key_parts->length) > max_key_length
by Sergei Golubchik 03 Jun '20

03 Jun '20

Hi, Sachin! ok to push On Jun 03, Sachin wrote: > revision-id: 7df57cc8206 (mariadb-10.4.11-226-g7df57cc8206) > parent(s): 0f91c62d332 > author: Sachin <sachin.setiya(a)mariadb.com> > committer: Sachin <sachin.setiya(a)mariadb.com> > timestamp: 2020-06-03 13:47:39 +0530 > message: > > MDEV-22719 Long unique keys are not created when individual key_part->length < max_key_length but SUM(key_parts->length) > max_key_length > > Make UNIQUE HASH key in case when key_info->key_length > max_key_length > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 1e9ea675a18: MDEV-21282 Assertion 'mariadb_table' failed in gcol.innodb_virtual_debug_purge
by Sergei Golubchik 03 Jun '20

03 Jun '20

Hi, Thirunarayanan! Sure, it's purely InnoDB functionality. If you say that purge thread already hold MDL for the table, then open_purge_table() doesn't need to. Ok to push as you see fit. On Jun 03, Thirunarayanan Balathandayuthapani wrote: > revision-id: 1e9ea675a18 (mariadb-10.5.2-304-g1e9ea675a18) > parent(s): f22093ad392 > author: Thirunarayanan Balathandayuthapani <thiru(a)mariadb.com> > committer: Thirunarayanan Balathandayuthapani <thiru(a)mariadb.com> > timestamp: 2020-06-01 18:02:17 +0530 > message: > > MDEV-21282 Assertion 'mariadb_table' failed in gcol.innodb_virtual_debug_purge > > - commit ea37b144094a0c2ebfc6774047fd473c1b2a8658 (MDEV-16678) caused > a regression. when purge thread tries to open the table for virtual > column computation, there is no need to acquire MDL for the table. > Because purge thread already hold MDL for the table Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] c01b3d91690: MDEV-22551 Server crashes in skip_trailing_space upon ANALYZE on table with long unique, or Assertion `marked_for_read()' fails
by Sergei Golubchik 03 Jun '20

03 Jun '20

Hi, Sachin! On Jun 03, Sachin wrote: > revision-id: c01b3d91690 (mariadb-10.4.11-224-gc01b3d91690) > parent(s): 0552ed86782 > author: Sachin <sachin.setiya(a)mariadb.com> > committer: Sachin <sachin.setiya(a)mariadb.com> > timestamp: 2020-06-02 08:24:13 +0530 > message: > > MDEV-22551 Server crashes in skip_trailing_space upon ANALYZE on table with long unique, or Assertion `marked_for_read()' fails > > If Blob(or parent) field read_set is not set then Do not calc hash in > that case. No, I think this is wrong. The problem, again, is that read_set bits are set incorrectly, and the fix is, again, to set them correctly, not to perform more read_set checks at run time. Think about what's happening here. The server tries to calculate a generated column value (because gcol is present in read_set) when not all argument columns were read (because they are not in read_set). This is an impossible situation, a gcol can _only_ be in read_set if all its dependencies are also there, you've seen in TABLE::mark_virtual_columns_for_write() that for every marked gcol all dependencies are also recursively marked. So in this case ANALYZE breaks this rule and should be fixed not to do it. There are two possible fixes - either ANALYZE should mark dependencies for all gcols (marking blobs in this bug) or ANALYZE should skip and not mark gcols. Better ask somebody who's more famiiar with EITS about what fix is preferred. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] e2017fc3a5f: MDEV-22441 implement a generic way to change a value of a variable in scope
by Sergei Golubchik 03 Jun '20

03 Jun '20

Hi, Eugene! On Jun 03, Eugene Kosov wrote: > revision-id: e2017fc3a5f (mariadb-10.2.31-219-ge2017fc3a5f) This should go into 10.5, not into old-old GA 10.2. Besides, 10.2 cannot use C++11 features, 10.5 can. > parent(s): 1b3adaab25c > author: Eugene Kosov <claprix(a)yandex.ru> > committer: Eugene Kosov <eugene.kosov(a)mariadb.com> > timestamp: 2020-05-27 12:45:41 +0300 > message: > > MDEV-22441 implement a generic way to change a value of a variable in scope > > Scope_value<T> class changes a variable value to some temporary value while > it exits and restores an old value in it's destructor. > > make_scope_value() is a function targeted for C++11 which allows to omit > a template parameter which is required by Scope_value<T> I think I'd name it backup_and_change_scoped() or something like that. But it's just a suggestion, feel free to keep make_scope_value() name if you like it better. > Sql_mode_save removed and the only usage is replaced with Scope_value<T> > > Sql_mode_save removal should not be merged in newer branches! > > --- > include/scope.h | 46 ++++++++++++++++++++++++++++++++++++++++++++++ > sql/sql_acl.cc | 7 ++++--- > sql/sql_class.h | 16 ---------------- > 3 files changed, 50 insertions(+), 19 deletions(-) > > diff --git a/include/scope.h b/include/scope.h > new file mode 100644 > index 00000000000..81061ceed37 > --- /dev/null > +++ b/include/scope.h > @@ -0,0 +1,46 @@ > +/* > + Copyright (c) 2020, MariaDB Corporation. > + > + This program is free software; you can redistribute it and/or modify > + it under the terms of the GNU General Public License as published by > + the Free Software Foundation; version 2 of the License. > + > + This program is distributed in the hope that it will be useful, > + but WITHOUT ANY WARRANTY; without even the implied warranty of > + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + GNU General Public License for more details. > + > + You should have received a copy of the GNU General Public License > + along with this program; if not, write to the Free Software > + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1335 USA */ > + > +#pragma once use include guards, please, not a pragma > + > +namespace maria and, as you remember, we didn't quite decide on a global server-wide namespace, so it's not the MDEV that should introduce it. > +{ > + > +template <typename T> class Scope_value > +{ > +public: > + Scope_value(T &variable, const T &scope_value) > + : variable_(variable), saved_value_(variable) > + { > + variable= scope_value; > + } > + > + ~Scope_value() { variable_= saved_value_; } > + > +private: > + T &variable_; > + T saved_value_; > +}; > + > +// Starting from C++11 this allows to omit template argument like this: > +// auto _ = make_scope_value(var, tmp_value); > +template <typename T> > +Scope_value<T> make_scope_value(T &variable, const T &scope_value) > +{ > + return Scope_value<T>(variable, scope_value); > +} as you've seen from Sql_mode_save usage, sometimes the value is assigned later, not when it's saved. Please, add another helper that doesn't require to provide a value. Like template <typename T> Scope_value<T> backup_scoped(T &variable) { return Scope_value<T>(variable, variable); } > + > +} // namespace maria Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 0552ed86782: MDEV-21804 Assertion `marked_for_read()' failed upon INSERT into table with long unique blob under binlog_row_image=NOBLOB
by Sergei Golubchik 03 Jun '20

03 Jun '20

Hi, Sachin! ok to push with one change, see below. On Jun 03, Sachin wrote: > revision-id: 0552ed86782 (mariadb-10.4.11-223-g0552ed86782) > parent(s): 61b2cd38d48 > author: Sachin <sachin.setiya(a)mariadb.com> > committer: Sachin <sachin.setiya(a)mariadb.com> > timestamp: 2020-06-02 08:24:02 +0530 > message: > > MDEV-21804 Assertion `marked_for_read()' failed upon INSERT into table with long unique blob under binlog_row_image=NOBLOB > > Problem:- Calling mark_columns_per_binlog_row_image() earlier may change the > result of mark_virtual_columns_for_write() , Since it can set the bitmap on > for virtual column, and henceforth mark_virtual_column_deps(field) will > never be called in mark_virtual_column_with_deps. > > This bug is not specific for long unique, It also fails for this case > create table t2(id int primary key, a blob, b varchar(20) as (LEFT(a,2))); > > diff --git a/sql/table.cc b/sql/table.cc > index bc8ddbc0f9d..8edbbe8d599 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -7040,7 +7040,6 @@ void TABLE::mark_columns_needed_for_update() > DBUG_ENTER("TABLE::mark_columns_needed_for_update"); > bool need_signal= false; > > - mark_columns_per_binlog_row_image(); > > if (triggers) > triggers->mark_fields_used(TRG_EVENT_UPDATE); > @@ -7048,6 +7047,7 @@ void TABLE::mark_columns_needed_for_update() > mark_default_fields_for_write(FALSE); > if (vfield) > need_signal|= mark_virtual_columns_for_write(FALSE); > + mark_columns_per_binlog_row_image(); I think that's too early. In that if() below you can see > if (file->ha_table_flags() & HA_REQUIRES_KEY_COLUMNS_FOR_DELETE) > { > KEY *end= key_info + s->keys; ... if (bitmap_fast_test_and_set(read_set, idx)) continue; if (field[idx]->vcol_info) mark_virtual_col(field[idx]); and mark_columns_per_binlog_row_image() might break that. Better to put mark_columns_per_binlog_row_image() at the very end, just before signalling. > @@ -7132,7 +7132,6 @@ void TABLE::mark_columns_needed_for_update() > void TABLE::mark_columns_needed_for_insert() > { > DBUG_ENTER("mark_columns_needed_for_insert"); > - mark_columns_per_binlog_row_image(); > > if (triggers) > { > @@ -7152,6 +7151,7 @@ void TABLE::mark_columns_needed_for_insert() > /* Mark virtual columns for insert */ > if (vfield) > mark_virtual_columns_for_write(TRUE); > + mark_columns_per_binlog_row_image(); this looks ok. > if (check_constraints) > mark_check_constraint_columns_for_read(); > DBUG_VOID_RETURN; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] Progress report - Week 4
by Mohammed Hammaad Mateen 02 Jun '20

02 Jun '20

Greetings, Hope you are safe and doing great, This post describes the things I've done in my fourth week [25-31 May] of Community Bonding Period under the mentor-ship of Sergei Golubchik and Oleksandr Byelkin for GSoC-20. The tasks taken up for this week was to study and analyze *INSERT SELECT RETURNING* and developing a test suite for *MULTIPLE TABLE DELETE RETURNING.* The following attachment contains the test cases for evaluating Multiple Delete Returning, as they are developed before actual implementation of code, please do notify me of any improvement needed, I'd be glad to resolve. Regards, Mohammed Hammaad Mateen

2 1

Re: [Maria-developers] fab8349af86: MDEV-21804 Assertion `marked_for_read()' failed upon INSERT into table with long unique blob under binlog_row_image=NOBLOB
by Sergei Golubchik 01 Jun '20

01 Jun '20

Hi, Sachin! On May 31, Sachin wrote: > revision-id: fab8349af86 (mariadb-10.4.11-223-gfab8349af86) > parent(s): 61b2cd38d48 > author: Sachin <sachin.setiya(a)mariadb.com> > committer: Sachin <sachin.setiya(a)mariadb.com> > timestamp: 2020-05-29 11:56:45 +0530 > message: > > MDEV-21804 Assertion `marked_for_read()' failed upon INSERT into table with long unique blob under binlog_row_image=NOBLOB > > Problem:- > In the case of BINLOG_ROW_IMAGE_NOBLOB table->readset is PKE + non blob fields > But while updating virtual fields we need to read the blob field, which will > generate assert failure. > Solution:- > If binlog_row_image == NOBLOB, then set read_bit for the blob field. > This bug is not specific for long unique, It also fails for this case > create table t2(id int primary key, a blob, b varchar(20) as (LEFT(a,2))); > > diff --git a/sql/field.cc b/sql/field.cc > index 0a8fdc3d3f5..bcad056e76a 100644 > --- a/sql/field.cc > +++ b/sql/field.cc > @@ -8467,6 +8467,10 @@ longlong Field_blob::val_int(void) > String *Field_blob::val_str(String *val_buffer __attribute__((unused)), > String *val_ptr) > { > + ulong binlog_row_image= table->in_use->variables.binlog_row_image; > + if (table->read_set && !bitmap_is_set(table->read_set, field_index) && > + binlog_row_image == BINLOG_ROW_IMAGE_NOBLOB) > + bitmap_set_bit(table->read_set, field_index); > DBUG_ASSERT(marked_for_read()); > char *blob; > memcpy(&blob, ptr+packlength, sizeof(char*)); A couple of red flags here. First, you don't fix BINLOG_ROW_IMAGE_MINIMAL. And generally you should not modify read/write sets from val_str, if should be done earlier before the execution phase. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] Review for: MDEV-17399 Add support for JSON_TABLE, part #5
by Sergey Petrunia 26 May '20

26 May '20

Hi Alexey, On Fri, May 15, 2020 at 04:26:02PM +0400, Alexey Botchkov wrote: > See the next iteration here > https://github.com/MariaDB/server/commit/692cb566096d61b240ec26e84fc7d3c7d1… > So the ha_json_table;:position() was implemented. > The size of the reference depends on the nested path depth - each level > adds some 9 bytes to the initial 5. > That can be decreased but i decided to keep it simple initially and i doubt > we're going to have really deep > nesting in realistic scenarios. This is ok. I'm not sure if the server has a limit on the rowid size. Perhaps there is, and in that case we just need to limit the depth we allow. > And i'd like to ask you for some testing > queries here. Or just the model how > to produce queries that will be using the ::position() extensively. I've provided a testcase for the position() call in my previous email. https://gist.github.com/spetrunia/a905d51731c58f5439bd9f70c64cdc43 As for rnd_pos(), one case that I'm aware of is when the query does a filesort, and the row being sorted either includes a blob, or has the total length exceeding certain limit. I've tried constructing an example with blobs, and it fails with an assertion before the execution reaches rnd_pos() calls: select * from json_table('[{"color": "blue", "price": 50}, {"color": "red", "price": 100}, {"color": "rojo", "price": 10.0}, {"color": "blanco", "price": 11.0}]', '$[*]' columns( color varchar(100) path '$.color', price text path '$.price', seq for ordinality ) ) as T order by color desc; fails an assertion: mysqld: /home/psergey/dev-git/10.5-json/sql/field.cc:8309: virtual int Field_blob::store(const char*, size_t, CHARSET_INFO*): Assertion `marked_for_write_or_computed()' failed. Once that is fixed, this should use position() and rnd_pos() calls. This will likely expose more issues with rnd_pos(), see my comments to the Json_table_nested_path::set_position below. == A problem with VIEWs == mysql> select * -> from -> json_table( -> '[ {"color": "red", "price": 1}, {"color": "black", "price": 2}]', -> '$[*]' columns( color varchar(100) path '$.color')) as `T_A` where T_A.color<>'azul' ; +-------+ | color | +-------+ | red | | black | +-------+ 2 rows in set (6.34 sec) Good so far. Now, let's try creating a VIEW from this: create view v1 as select * from json_table( '[ {"color": "red", "price": 1}, {"color": "black", "price": 2}]', '$[*]' columns( color varchar(100) path '$.color')) as `T_A` where T_A.color<>'azul' ; select * from v1; ERROR 4041 (HY000): Unexpected end of JSON path in argument 2 to function 'JSON_TABLE' Examining the .frm file, I see something that looks like garbage data: query=select `T_A`.`color` AS `color` from JSON_TABLE(\'[ {"color": "red", "price": 1}, {"color": "black", "price": 2}]\', \'\'\0<D8>7^A<9C><FF>^?\0\0\0\0\0\0\0\0\0\0\0\0^A<A5><A5><A5><A5><A5>^H<D0>kWUU\0\0[ {"color": "red", "price": 1}, {"color": "black", "price": 2}]\0$[*]\'\' COLUMNS (`color` PATH \'^A<9C><FF>^?\0\0\0\0\0\0^A\0\0\0\0\0\0\0<A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5>\0\0\0\0<A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5><A5> <A5><A5><A5><A5><A5><A5> D^A<9C><FF>^?\0\0H:^A<9C><FF>^?\0\0<A5><A5><A5><A5><A5><A5><A5><A5>100\0<8F><8F><8F><8F>$.color\')) T_A where `T_A`.`color` <> \'azul\' EXPLAIN EXTENDED ...; SHOW WARNINGS; - also print something odd. == EXPLAIN [FORMAT=JSON] == I think EXPLAIN output should provide an indication that a table function is used. MySQL does it like so: <TODO> == Unneeded recursive rules in grammar == I've already complained about this: the grammar has recursive ON EMPTY, ON ERROR rules, which cause the following to be accepted (note the two ON EMPTY clauses) : select * from json_table('[{"color": "blue", "price": 50},{"color": "red"}]', '$[*]' columns( price varchar(255) path '$.price' default 'xyz' on empty default 'abc' on empty ) ) as T; > commit 692cb566096d61b240ec26e84fc7d3c7d13f024c > Author: Alexey Botchkov <holyfoot(a)askmonty.org> > Date: Fri May 15 15:25:28 2020 +0400 > > MDEV-17399 Add support for JSON_TABLE. > > Syntax for JSON_TABLE added. > The ha_json_table handler added. Executing the JSON_TABLE we > create the temporary table of the ha_json_table, add dependencies > of other tables and sometimes conditions to WHERE. > and sometimes conditions to WHERE. I think this is not true anymore? ... > diff --git a/include/my_base.h b/include/my_base.h > index 7efa5eb9673..89ef3e8e7c1 100644 > --- a/include/my_base.h > +++ b/include/my_base.h > @@ -523,6 +523,13 @@ enum ha_base_keytype { > #define HA_ERR_TABLESPACE_MISSING 194 /* Missing Tablespace */ > #define HA_ERR_SEQUENCE_INVALID_DATA 195 > #define HA_ERR_SEQUENCE_RUN_OUT 196 > + > +/* > + Share the error code to not increment the HA_ERR_LAST for now, > + as it disturbs some storage engine's tests. > + Probably should be fixed later. > +*/ > +#define HA_ERR_INVALID_JSON HA_ERR_TABLE_IN_FK_CHECK We definitely can't have this in the final patch. We need to either: A. Use an engine-specific error code. Check out MyRocks and ha_rocksdb::get_error_message() for an example of how this is done B. Indeed introduce a generic "Invalid JSON" error code. I'm hesitant about B, let's discuss this with other developers. ... > diff --git a/sql/table_function.cc b/sql/table_function.cc > new file mode 100644 > index 00000000000..71f2378ce7d > --- /dev/null > +++ b/sql/table_function.cc ... > + > +class ha_json_table: public handler > +{ Please add comments about these > +protected: > + Table_function_json_table *m_jt; > + String m_tmps; > + String *m_js; > + uchar *m_cur_pos; > +public: > + ha_json_table(TABLE_SHARE *share_arg, Table_function_json_table *jt): > + handler(&table_function_hton.m_hton, share_arg), m_jt(jt) > + { > + /* > + set the mark_trx_read_write_done to avoid the > + handler::mark_trx_read_write_internal() call. > + It relies on &ha_thd()->ha_data[ht->slot].ha_info[0] to be set. > + But we don't set the ha_data for the ha_json_table, and > + that call makes no sence for ha_json_table. > + */ > + mark_trx_read_write_done= 1; > + ref_length= (jt->m_depth+1) * (4 + 1) + > + jt->m_depth * (sizeof(Json_table_nested_path *)); > + } > + ~ha_json_table() {} > + handler *clone(const char *name, MEM_ROOT *mem_root) { return NULL; } > + const char *index_type(uint inx) { return "NONE"; } > + /* Rows also use a fixed-size format */ > + enum row_type get_row_type() const { return ROW_TYPE_FIXED; } > + ulonglong table_flags() const > + { > + return (HA_FAST_KEY_READ | /*HA_NO_BLOBS |*/ HA_NULL_IN_KEY | > + HA_CAN_SQL_HANDLER | > + HA_REC_NOT_IN_SEQ | HA_NO_TRANSACTIONS | > + HA_HAS_RECORDS | HA_CAN_HASH_KEYS); > + } > + ulong index_flags(uint inx, uint part, bool all_parts) const > + { > + return HA_ONLY_WHOLE_INDEX | HA_KEY_SCAN_NOT_ROR; > + } > + uint max_supported_keys() const { return 1; } > + uint max_supported_key_part_length() const { return MAX_KEY_LENGTH; } > + double scan_time() { return 1000000.0; } > + double read_time(uint index, uint ranges, ha_rows rows) { return 0.0; } The above two functions are never called. Please * add a comment saying that. * add a DBUG_ASSERT() into them to back the point made by the comment :-) > + int open(const char *name, int mode, uint test_if_locked); > + int close(void) { return 0; } > + int rnd_init(bool scan); > + int rnd_next(uchar *buf); > + int rnd_pos(uchar * buf, uchar *pos); > + void position(const uchar *record); > + int can_continue_handler_scan() { return 1; } > + int info(uint); > + int extra(enum ha_extra_function operation); > + THR_LOCK_DATA **store_lock(THD *thd, THR_LOCK_DATA **to, > + enum thr_lock_type lock_type) > + { return NULL; } > + int create(const char *name, TABLE *form, HA_CREATE_INFO *create_info) > + { return 1; } > +private: > + void update_key_stats(); > +}; > + ... > +void Json_table_nested_path::set_position(const char *j_start, const uchar *pos) > +{ * This needs a comment. * I don't see where the value of m_ordinality_counter is restored? * The same about m_null - its value is not restored either? > + if (m_nested) > + { > + memcpy(&m_cur_nested, pos, sizeof(m_cur_nested)); > + pos+= sizeof(m_cur_nested); > + } > + > + m_engine.s.c_str= (uchar *) j_start + sint4korr(pos); > + m_engine.state= (int) pos[4]; > + if (m_cur_nested) > + m_cur_nested->set_position(j_start, pos); > +} > + ... > +int ha_json_table::info(uint) > +{ > + /* We don't want 0 or 1 in stats.records. */ > + stats.records= 4; > + return 0; Does this value matter? As far as I understand it doesn't, as the optimizer will use the estimates obtained from Table_function_json_table::get_estimates. Please add a comment about this. > +} > + ... Please document this function. What's last_column? Why does print() method get it and return it? > +int Json_table_nested_path::print(THD *thd, TABLE_LIST *sql_table, String *str, > + List_iterator_fast<Json_table_column> &it, > + Json_table_column **last_column) > +{ > + Json_table_nested_path *c_path= this; ... > diff --git a/sql/table_function.h b/sql/table_function.h > new file mode 100644 > index 00000000000..49d650cb7b2 > --- /dev/null > +++ b/sql/table_function.h > @@ -0,0 +1,168 @@ ... > +#include <json_lib.h> > + > +/* > + The Json_table_nested_path represents the 'current nesting' level > + for a set of JSON_TABLE columns. > + Each column (Json_table_column instance) is linked with corresponding > + 'nested path' object and gets it's piece of JSON to parse during the computation > + phase. > + The root 'nested_path' is always present as a part of Table_function_json_table, > + then other 'nested_paths' can be created and linked into a tree structure when new > + 'NESTED PATH' is met. The nested 'nested_paths' are linked with 'm_nested', the same-level > + 'nested_paths' are linked with 'm_next_nested'. > + So for instance > + JSON_TABLE( '...', '$[*]' > + COLUMNS( a INT PATH '$.a' , > + NESTED PATH '$.b[*]' COLUMNS (b INT PATH '$', > + NESTED PATH '$.c[*]' COLUMNS(x INT PATH '$')), > + NESTED PATH '$.n[*]' COLUMNS (z INT PAHT '$')) > + results in 4 'nested_path' created: > + root nested_b nested_c nested_n > + m_path '$[*]' '$.b[*]' '$.c[*]' '$.n[*] > + m_nested &nested_b &nested_c NULL NULL > + n_next_nested NULL &nested_n NULL NULL > + > +and 4 columns created: > + a b x z > + m_nest &root &nested_b &nested_c &nested_n > +*/ > + > + > +class Json_table_column; > + > +class Json_table_nested_path : public Sql_alloc > +{ > +public: > + bool m_null; > + json_path_t m_path; > + json_engine_t m_engine; > + json_path_t m_cur_path; > + > + /* Counts the rows produced. Value is set to the FOR ORDINALITY coluns */ > + longlong m_ordinality_counter; > + > + Json_table_nested_path *m_parent; > + Json_table_nested_path *m_nested, *m_next_nested; > + Json_table_nested_path **m_nested_hook; > + Json_table_nested_path *m_cur_nested; Please add documentation about the above members. I see the diagram above, but I think text descriptions are also needed for each member. What's m_nested_hook? > + Json_table_nested_path(Json_table_nested_path *parent_nest): > + m_parent(parent_nest), m_nested(0), m_next_nested(0), > + m_nested_hook(&m_nested) {} > + int set_path(THD *thd, const LEX_CSTRING &path); > + void scan_start(CHARSET_INFO *i_cs, const uchar *str, const uchar *end); > + int scan_next(); > + int print(THD *thd, TABLE_LIST *sql_table, String *str, > + List_iterator_fast<Json_table_column> &it, > + Json_table_column **last_column); > + void get_current_position(const char *j_start, uchar *pos) const; > + void set_position(const char *j_start, const uchar *pos); > +}; ... > +class Table_function_json_table : public Sql_alloc Please document the class and the data members. > +{ > +public: > + Item *m_json; > + Json_table_nested_path m_nested_path; > + List<Json_table_column> m_columns; > + table_map m_dep_tables; > + uint m_depth, m_cur_depth; > + > + Table_function_json_table(Item *json): m_json(json), m_nested_path(0), > + m_depth(0), m_cur_depth(0) {} > + > + /* > + Used in sql_yacc.yy. > + Represents the current NESTED PATH level being parsed. > + */ > + Json_table_nested_path *m_sql_nest; > + void add_nested(Json_table_nested_path *np); > + void leave_nested(); BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] 070df171c1e: MDEV-16937 Strict SQL with system versioned tables causes issues
by Sergei Golubchik 26 May '20

26 May '20

Hi, Aleksey! ok to push On May 26, Aleksey Midenkov wrote: > revision-id: 070df171c1e (mariadb-10.3.21-91-g070df171c1e) > parent(s): ad41da5c934 > author: Aleksey Midenkov <midenok(a)gmail.com> > committer: Aleksey Midenkov <midenok(a)gmail.com> > timestamp: 2020-05-25 14:47:26 +0300 > message: > > MDEV-16937 Strict SQL with system versioned tables causes issues > > Respect system fields in NO_ZERO_DATE mode. > > This is the subject for refactoring in MDEV-19597 > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] cd9cab54aac: MDEV-20015 Assertion `!in_use->is_error()' failed in TABLE::update_virtual_field
by Sergei Golubchik 26 May '20

26 May '20

Hi, Aleksey! Ok to push, thanks! On May 26, Aleksey Midenkov wrote: > revision-id: cd9cab54aac (mariadb-10.2.31-123-gcd9cab54aac) > parent(s): d275ecbd208 > author: Aleksey Midenkov <midenok(a)gmail.com> > committer: Aleksey Midenkov <midenok(a)gmail.com> > timestamp: 2020-05-25 20:56:31 +0300 > message: > > MDEV-20015 Assertion `!in_use->is_error()' failed in TABLE::update_virtual_field > > update_virtual_field() is called as part of index rebuild in > ha_myisam::repair() (MDEV-5800) which is done on bulk INSERT finish. > > Assertion in update_virtual_field() was put as part of MDEV-16222 > because update_virtual_field() returns in_use->is_error(). The idea: > wrongly mixed semantics of error status before update_virtual_field() > and the status returned by update_virtual_field(). The former can > falsely influence the latter. > > diff --git a/sql/table.cc b/sql/table.cc > index d6d86d96016..2429bb12abe 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -7707,15 +7707,17 @@ int TABLE::update_virtual_fields(handler *h, enum_vcol_update_mode update_mode) > > int TABLE::update_virtual_field(Field *vf) > { > - DBUG_ASSERT(!in_use->is_error()); > - Query_arena backup_arena; > DBUG_ENTER("TABLE::update_virtual_field"); > + Query_arena backup_arena; > + Counting_error_handler count_errors; > + in_use->push_internal_handler(&count_errors); > in_use->set_n_backup_active_arena(expr_arena, &backup_arena); > bitmap_clear_all(&tmp_set); > vf->vcol_info->expr->walk(&Item::update_vcol_processor, 0, &tmp_set); > vf->vcol_info->expr->save_in_field(vf, 0); > in_use->restore_active_arena(expr_arena, &backup_arena); > - DBUG_RETURN(in_use->is_error()); > + in_use->pop_internal_handler(); > + DBUG_RETURN(count_errors.errors); > } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 7d593466a22: MDEV-20015 Assertion `!in_use->is_error()' failed in TABLE::update_virtual_field
by Sergei Golubchik 25 May '20

25 May '20

Hi, Aleksey! I don't see what you've changed. We've discussed that fix and that one isn't supposed to swap Diagnostics_area's like that. And in your new patch you do exactly the same. Possible correct approaches: * don't return in_use->is_error(), return the return value of vf->vcol_info->expr->walk() || vf->vcol_info->expr->save_in_field() This means that Item_field::update_vcol_processor() should also do the same, I suspect * Use thd->push_internal_handler() and Counting_error_handler. Or, better, Turn_errors_to_warnings_handler with counting. This is the simplest one. there's a third option: * always return 0, because, looking how it's used, I don't really see how update_virtual_field() can ever get an error. But it's not a particularly future-proof approach. And I just might be wrong about errors. On Apr 23, Aleksey Midenkov wrote: > revision-id: 7d593466a22 (mariadb-10.2.28-4-g7d593466a22) > parent(s): 7bc26de591c > author: Aleksey Midenkov <midenok(a)gmail.com> > committer: Aleksey Midenkov <midenok(a)gmail.com> > timestamp: 2019-11-07 10:45:21 +0300 > message: > > MDEV-20015 Assertion `!in_use->is_error()' failed in TABLE::update_virtual_field > > Preserve and restore statement DA. > > update_virtual_field() is called as part of index rebuild in > ha_myisam::repair() (MDEV-5800) which is done on bulk INSERT finish. > > Assertion in update_virtual_field() was put as part of MDEV-16222 > because update_virtual_field() returns in_use->is_error(). The idea: > wrongly mixed semantics of error status before update_virtual_field() > and the status returned by update_virtual_field(). The former can > falsely influence the latter. > > Preserve global error status and run update_virtual_field() with clear > DA since no matter how SQL command is finished it must update the > index after bulk INSERT. > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

[Maria-developers] Progress Report - Week 3
by Mohammed Hammaad Mateen 25 May '20

25 May '20

Greetings, Hope you are safe and doing great, This post describes the things I've done in my third week [ 18-24 May ] of Community Bonding Period under the mentor-ship of Sergei Golubchik and Oleksandr Byelkin for GSoC-20. The tasks taken up for this week was to study and analyze: *INSERT.... RETURNING:* • *TableList* consists of 1 table instance to use, where each instance is a structure consisting of db argument, table name argument, alias argument and lock type argument. *Referenced this from the table.h file.* • In *sql_parse.cc* <http://sql_parse.cc> under *case SQLCOM_INSERT*: we do a DBUG_ASSERT where first_table == all_table && first_table!=0 // we check if only 1 table is used and the table used is already created. • if lex-> has_returning() we increment the system status var by 1 and perform analyze.. insert.. returning. • we compute result by mysql_insert function by passing the following arguments. - Thread Handler thd - all table instances - list of all the fields used - the values we want to insert - the list of update feilds - and update value list - duplicate flag - ignore - result of select • if Inserting fails due to some reason we equate result to send_explain(thd) • we also update the MYSQL_INSERT_DONE with result of insert and the the row count. *INSERT.. SELECT.. RETURNING:* • We fix the lock on first table. • lock other tables until command is written to the binary log. • the procedure is same as discussed earlier with respect to insert.... returning. • To switch to the second table we traverse from first_table to (->) next_local and we compute select result with the help of select insert function. • we can now unlock the tables and we also need to check if something changed after unlocking, of that happens we should invalidate the table from the query cache using query_cache_invalidate3 function. • Manual cleaning of select result obtained from select insert must be done. Regards, Mohammed Hammaad Mateen

1 0

Re: [Maria-developers] MDEV-22461: JOIN::make_aggr_tables_info(): Assertion `select_options & (1ULL << 17)' failed.
by Sergey Petrunia 24 May '20

24 May '20

Hi Varun, - Please add a testcase to the testsuite - Please address a few cosmetic comments below - Ok to push after addressed. > commit 5e448b77a3812e65623c0a1214049322ace3aacf > Author: Varun Gupta <varun.gupta(a)mariadb.com> > Date: Tue May 5 20:44:43 2020 +0530 > > MDEV-22461: JOIN::make_aggr_tables_info(): Assertion `select_options & (1ULL << 17)' failed. > > A temporary table is needed for window function computation but if only a NAMED WINDOW SPEC > is used and there is no window function, then there is no need to create a temporary > table as there is no stage to compute WINDOW FUNCTION > > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index c601946cfa0..0a1d0c2dbcc 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -2014,11 +2014,16 @@ JOIN::optimize_inner() > } > > need_tmp= test_if_need_tmp_table(); > - //TODO this could probably go in test_if_need_tmp_table. > - if (this->select_lex->window_specs.elements > 0) { > - need_tmp= TRUE; > + > + /* > + If window functions are present then we can't have simple_order set to > + TRUE as the window function needs a temp table for compuatation. typo: compuatation. > + ORDER BY is computed after the window function computation is done, so > + the sort would be done on the temp table. > + */ > + if (this->select_lex->have_window_funcs()) Is there a need to have "this->select_lex"? I think not, please remove, as it only confuses the reader. > simple_order= FALSE; > - } > + > > /* > If the hint FORCE INDEX FOR ORDER BY/GROUP BY is used for the table > diff --git a/sql/sql_select.h b/sql/sql_select.h > index 0e011c9267a..7a892c1af89 100644 > --- a/sql/sql_select.h > +++ b/sql/sql_select.h > @@ -1645,7 +1645,8 @@ class JOIN :public Sql_alloc > ((select_distinct || !simple_order || !simple_group) || > (group_list && order) || > MY_TEST(select_options & OPTION_BUFFER_RESULT))) || > - (rollup.state != ROLLUP::STATE_NONE && select_distinct)); > + (rollup.state != ROLLUP::STATE_NONE && select_distinct) || > + select_lex->have_window_funcs()); Please amend the function comment above this code to cover the new condition as well. > } > bool choose_subquery_plan(table_map join_tables); > void get_partial_cost_and_fanout(int end_tab_idx, > BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] b22a28c2295: fixup! 3fe5cd5e1785e3e8de7add9977a1c2ddd403538b
by Michael Widenius 22 May '20

22 May '20

Hi! On Fri, May 22, 2020 at 3:27 PM Andrei Elkin <andrei.elkin(a)mariadb.com> wrote: <cut> CORNER CASES: read-only, pure myisam, binlog-*, @@skip_log_bin, etc > > Aria just makes yet another previously unknown use case of an engine > that produces THD::ha_info but does not support 2pc, which the assert > implied. > > To explain more, the original block > > #ifndef DBUG_OFF > for (ha_info= thd->transaction.all.ha_list; rw_count > 1 && ha_info; > ha_info= ha_info->next()) > DBUG_ASSERT(ha_info->ht() != binlog_hton); > #endif > > claims there most be no binlog hton in a transaction consisting of more > than 1 hton:s, *when* (at this point) this transaction has not been > binlogged yet. > So combination of binlog + Innodb hton would be raise the assert, to > question "why the heck binlogging has not been done yet?!". > > Aria is different from Innodb in this context in that binlogging > was done at the end of the statement, so to miss > `cache_mngr->need_unlog' flagging (which is at xa prepare time logging). Thanks for the explanation, we should have had that in the code. > I think we should fix the assert rather than to remove. This way: > > cat > assert.diff <<. > diff --git a/sql/log.cc b/sql/log.cc > index 792c6bb1a99..aaf1fae1cd6 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -10124,6 +10124,16 @@ int TC_LOG_BINLOG::unlog_xa_prepare(THD *thd, bool all) > Ha_trx_info *ha_info; > uint rw_count= ha_count_rw_all(thd, &ha_info); > bool rc= false; > +#ifndef DBUG_OFF > + bool no_binlog= true, exist_no_2pc= false; > + for (ha_info= thd->transaction->all.ha_list; rw_count > 1 && ha_info; > + ha_info= ha_info->next()) > + { > + no_binlog= no_binlog && ha_info->ht() != binlog_hton; > + exist_no_2pc= exist_no_2pc || !ha_info->ht()->prepare; > + } > + DBUG_ASSERT(no_binlog || exist_no_2pc); > +#endif > > if (rw_count > 0) > { > . > > I tested it briefly with running XA:s on combination of engines > including. On which tree did you test? bb-10.5-aria? In the end, after discussions on slack, we ended with: #ifndef DBUG_OFF if (rw_count > 1) { /* There must be no binlog_hton used in a transaction consisting of more than 1 engine, *when* (at this point) this transaction has not been binlogged. The one exception is if there is an engine without a prepare method, as in this case the engine doesn't support XA and we have to ignore this check. */ bool binlog= false, exist_hton_without_prepare= false; for (ha_info= thd->transaction->all.ha_list; ha_info; ha_info= ha_info->next()) { if (ha_info->ht() == binlog_hton) binlog= true; if (!ha_info->ht()->prepare) exist_hton_without_prepare= true; } DBUG_ASSERT(!binlog || exist_hton_without_prepare); } #endif > > The reason it was not hit > > is that before Aria was not treated as transactional we could only > > come here in case of errors > > and that code was was apparently not tested, at least with binary logging on. > > > > > With Aria we could come here in case of rollback and we got assert for > > cases that was perfectly ok. > > Well, what 'rollback' do you mean? The function is invoked only > for ha_prepare(). As far as I remember, I come into this code in some edge case when something failed that normally never fails. I think the issue was that Aria doesn't have a prepare handler, which caused issues in ha_prepare(), but not sure. Anyway, we now have a solution for this. Regards, Monty

1 0

Re: [Maria-developers] acbe14b122c: Aria will now register it's transactions
by Sergei Golubchik 22 May '20

22 May '20

Hi, Michael! See below On May 20, Michael Widenius wrote: > revision-id: acbe14b122c (mariadb-10.5.2-255-gacbe14b122c) > parent(s): 48a758696bf > author: Michael Widenius <monty(a)mariadb.com> > committer: Michael Widenius <monty(a)mariadb.com> > timestamp: 2020-05-19 17:52:17 +0300 > message: > > Aria will now register it's transactions > > MDEV-22531 Remove maria::implicit_commit() > diff --git a/sql/sql_class.h b/sql/sql_class.h > index d4a95fa3fd8..a7a071f7cdb 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -5149,6 +5151,40 @@ class THD: public THD_count, /* this must be first */ > > }; > > + > +/* > + Start a new independent transaction for the THD. > + The old one is stored in this object and restored when calling > + restore_old_transaction() or when the object is freed > +*/ > + > +class start_new_trans > +{ > + /* container for handler's private per-connection data */ > + Ha_data old_ha_data[MAX_HA]; > + struct THD::st_transactions *old_transaction, new_transaction; > + Open_tables_backup open_tables_state_backup; > + MDL_savepoint mdl_savepoint; > + PSI_transaction_locker *m_transaction_psi; > + THD *org_thd; > + uint in_sub_stmt; > + uint server_status; > + > +public: > + start_new_trans(THD *thd); > + ~start_new_trans() > + { > + destroy(); > + } > + void destroy() > + { > + if (org_thd) // Safety > + restore_old_transaction(); > + new_transaction.free(); > + } > + void restore_old_transaction(); interesting. You made restore_old_transaction() public and you use it in many places. Why not to use destroy() instead? When one would want to use restore_old_transaction() instead of destroy()? > +}; > + > /** A short cut for thd->get_stmt_da()->set_ok_status(). */ > > inline void > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index dda8e00f6bf..51d7380f622 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -5742,6 +5744,90 @@ void THD::mark_transaction_to_rollback(bool all) > } > > > +/** > + Commit the whole transaction (both statment and all) > + > + This is used mainly to commit an independent transaction, > + like reading system tables. > + > + @return 0 0k > + @return <>0 error code. my_error() has been called() > +*/ > + > +int THD::commit_whole_transaction_and_close_tables() > +{ > + int error, error2; > + DBUG_ENTER("THD::commit_whole_transaction_and_close_tables"); > + > + /* > + This can only happened if we failed to open any table in the > + new transaction > + */ > + if (!open_tables) > + DBUG_RETURN(0); Generally, I think, it should still end the transaction here. Or add an assert that there is no active transaction at the moment. > + > + /* > + Ensure table was locked (opened with open_and_lock_tables()). If not > + the THD can't be part of any transactions and doesn't have to call > + this function. > + */ > + DBUG_ASSERT(lock); > + > + error= ha_commit_trans(this, FALSE); > + /* This will call external_lock to unlock all tables */ > + if ((error2= mysql_unlock_tables(this, lock))) > + { > + my_error(ER_ERROR_DURING_COMMIT, MYF(0), error2); > + error= error2; > + } > + lock= 0; > + if ((error2= ha_commit_trans(this, TRUE))) > + error= error2; > + close_thread_tables(this); I wonder why you're doing it in that specific order. commit(stmt)-unlock-commit(all)-close > + DBUG_RETURN(error); > +} > + > +/** > + Start a new independent transaction > +*/ > + > +start_new_trans::start_new_trans(THD *thd) > +{ > + org_thd= thd; > + mdl_savepoint= thd->mdl_context.mdl_savepoint(); > + memcpy(old_ha_data, thd->ha_data, sizeof(old_ha_data)); > + thd->reset_n_backup_open_tables_state(&open_tables_state_backup); > + bzero(thd->ha_data, sizeof(thd->ha_data)); > + old_transaction= thd->transaction; > + thd->transaction= &new_transaction; > + new_transaction.on= 1; > + in_sub_stmt= thd->in_sub_stmt; > + thd->in_sub_stmt= 0; > + server_status= thd->server_status; > + m_transaction_psi= thd->m_transaction_psi; > + thd->m_transaction_psi= 0; > + thd->server_status&= ~(SERVER_STATUS_IN_TRANS | > + SERVER_STATUS_IN_TRANS_READONLY); > + thd->server_status|= SERVER_STATUS_AUTOCOMMIT; > +} Few thoughts: 1. If you need to save and restore _all that_ then, perhaps, all that should be inside st_transactions ? 2. strictly speaking, ha_data is _per connection_. If you just bzero it, the engine will think it's a new connection, and you cannot just overwrite it on restore without hton->close_connection. A strictly "proper" solution would be to introduce ha_data per transaction in addition to what we have now. But it looks like an overkill. So I'd just add close_system_tables() now into restore_old_transaction() A strictly "proper" solution would be to introduce ha_data per transaction in addition to what we have now. But it looks like an overkill. So I'd just add ha_close_connection() now into restore_old_transaction() and that's all. I see that you've added free_transaction() call, but isn't it redundant? ha_close_connection() does that now. Otherwise very good, thanks, I cannot wait to start using it more for other features. > + > + > +void start_new_trans::restore_old_transaction() > +{ > + org_thd->transaction= old_transaction; > + org_thd->restore_backup_open_tables_state(&open_tables_state_backup); > + ha_free_transactions(org_thd); > + memcpy(org_thd->ha_data, old_ha_data, sizeof(old_ha_data)); > + org_thd->mdl_context.rollback_to_savepoint(mdl_savepoint); > + org_thd->in_sub_stmt= in_sub_stmt; > + org_thd->server_status= server_status; > + if (org_thd->m_transaction_psi) > + MYSQL_COMMIT_TRANSACTION(org_thd->m_transaction_psi); > + org_thd->m_transaction_psi= m_transaction_psi; > + org_thd= 0; > +} > + > + > /** > Decide on logging format to use for the statement and issue errors > or warnings as needed. The decision depends on the following > diff --git a/sql/event_db_repository.cc b/sql/event_db_repository.cc > index af43d92dea7..82b3968de85 100644 > --- a/sql/event_db_repository.cc > +++ b/sql/event_db_repository.cc > @@ -742,7 +748,8 @@ Event_db_repository::create_event(THD *thd, Event_parse_data *parse_data, > ret= 0; > > end: > - close_thread_tables(thd); > + if (table) > + thd->commit_whole_transaction_and_close_tables(); You're checking twice. Here in the caller, and the first thing inside commit_whole_transaction_and_close_tables(). Here and in many places. If you want to check in the caller, there's no need to check inside? You can add an assert instead, can't you? > thd->mdl_context.rollback_to_savepoint(mdl_savepoint); > > thd->variables.sql_mode= saved_mode; > @@ -1117,22 +1124,20 @@ update_timing_fields_for_event(THD *thd, > TABLE *table= NULL; > Field **fields; > int ret= 1; > - enum_binlog_format save_binlog_format; > MYSQL_TIME time; > DBUG_ENTER("Event_db_repository::update_timing_fields_for_event"); > > - /* > - Turn off row binlogging of event timing updates. These are not used > - for RBR of events replicated to the slave. > - */ > - save_binlog_format= thd->set_current_stmt_binlog_format_stmt(); > - > DBUG_ASSERT(thd->security_ctx->master_access & PRIV_IGNORE_READ_ONLY); > > if (open_event_table(thd, TL_WRITE, &table)) > goto end; Why do you reuse a current transaction, instead of creating a new one as everywhere above? > > fields= table->field; > + /* > + Turn off row binlogging of event timing updates. These are not used > + for RBR of events replicated to the slave. > + */ > + table->file->row_logging= 0; > > if (find_named_event(event_db_name, event_name, table)) > goto end; > diff --git a/sql/handler.h b/sql/handler.h > index e4903172c33..ebe23b97062 100644 > --- a/sql/handler.h > +++ b/sql/handler.h > @@ -1765,6 +1766,12 @@ handlerton *ha_default_tmp_handlerton(THD *thd); > */ > #define HTON_TABLE_MAY_NOT_EXIST_ON_SLAVE (1 << 15) > > +/* > + True if handler cannot roolback transactions. If not true, the transaction > + will be put in the transactional binlog cache. > +*/ > +#define HTON_NO_ROLLBACK (1 << 16) How is it different from HA_PERSISTENT_TABLE? > + > class Ha_trx_info; > > struct THD_TRANS > diff --git a/sql/ha_partition.cc b/sql/ha_partition.cc > index 582c9bb110b..0c53bdc5bbe 100644 > --- a/sql/ha_partition.cc > +++ b/sql/ha_partition.cc > @@ -11016,7 +11016,7 @@ int ha_partition::check_misplaced_rows(uint read_part_id, bool do_repair) > If the engine supports transactions, the failure will be > rollbacked. if you're changing it anyway, can you also change rollbacked -> rolled back ? thanks. > */ > - if (!m_file[correct_part_id]->has_transactions()) > + if (!m_file[correct_part_id]->has_transactions_and_rollback()) > { > /* Log this error, so the DBA can notice it and fix it! */ > sql_print_error("Table '%-192s' failed to move/insert a row" > diff --git a/sql/handler.cc b/sql/handler.cc > index 39841cc28d7..e3f5773717d 100644 > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -4514,7 +4515,6 @@ void handler::mark_trx_read_write_internal() > */ > if (ha_info->is_started()) > { > - DBUG_ASSERT(has_transaction_manager()); there should be *some* assert here, I think > /* > table_share can be NULL in ha_delete_table(). See implementation > of standalone function ha_delete_table() in sql_base.cc. > diff --git a/sql/share/errmsg-utf8.txt b/sql/share/errmsg-utf8.txt > index 250da9948a0..1f3bf212d3f 100644 > --- a/sql/share/errmsg-utf8.txt > +++ b/sql/share/errmsg-utf8.txt > @@ -7961,3 +7961,5 @@ ER_KEY_CONTAINS_PERIOD_FIELDS > eng "Key %`s cannot explicitly include column %`s" > ER_KEY_CANT_HAVE_WITHOUT_OVERLAPS > eng "Key %`s cannot have WITHOUT OVERLAPS" > +ER_DATA_WAS_COMMITED_UNDER_ROLLBACK > + eng "Engine %s does not support rollback. Changes where commited during rollback call" A confusing error message. I'd just say "table is not transactional". This is the user point of view difference between a "transactional engine" like InnoDB and "crash safe engine" like Aria. It's very confusing to invent a new category of "transactional engines that cannot roll back" even if internally Aria is treated like one. > diff --git a/sql/sp.cc b/sql/sp.cc > index 51bbeeef368..1629290eb73 100644 > --- a/sql/sp.cc > +++ b/sql/sp.cc > @@ -470,27 +471,29 @@ static Proc_table_intact proc_table_intact; > currently open tables will be saved, and from which will be > restored when we will end work with mysql.proc. > > + NOTES > + On must have a start_new_trans object active when calling this function I think this: DBUG_ASSERT(thd->transcation != &thd->default_transaction); could be a bit safer than a NOTE :) > + > @retval > 0 Error > @retval > \# Pointer to TABLE object of mysql.proc > */ > > -TABLE *open_proc_table_for_read(THD *thd, Open_tables_backup *backup) > +TABLE *open_proc_table_for_read(THD *thd) > { > TABLE_LIST table; > - > DBUG_ENTER("open_proc_table_for_read"); > > table.init_one_table(&MYSQL_SCHEMA_NAME, &MYSQL_PROC_NAME, NULL, TL_READ); > > - if (open_system_tables_for_read(thd, &table, backup)) > + if (open_system_tables_for_read(thd, &table)) > DBUG_RETURN(NULL); > > if (!proc_table_intact.check(table.table, &proc_table_def)) > DBUG_RETURN(table.table); > > - close_system_tables(thd, backup); > + thd->commit_whole_transaction_and_close_tables(); > > DBUG_RETURN(NULL); > } > @@ -517,7 +520,8 @@ static TABLE *open_proc_table_for_update(THD *thd) > MDL_savepoint mdl_savepoint= thd->mdl_context.mdl_savepoint(); > DBUG_ENTER("open_proc_table_for_update"); > same? also need start_new_trans? > > - table_list.init_one_table(&MYSQL_SCHEMA_NAME, &MYSQL_PROC_NAME, NULL, TL_WRITE); > + table_list.init_one_table(&MYSQL_SCHEMA_NAME, &MYSQL_PROC_NAME, NULL, > + TL_WRITE); > > if (!(table= open_system_table_for_update(thd, &table_list))) > DBUG_RETURN(NULL); > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > index 6c938670fdc..325a0f1d41d 100644 > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -4259,7 +4259,7 @@ bool open_tables(THD *thd, const DDL_options_st &options, > list, we still need to call open_and_process_routine() to take > MDL locks on the routines. > */ > - if (thd->locked_tables_mode <= LTM_LOCK_TABLES) > + if (thd->locked_tables_mode <= LTM_LOCK_TABLES && *sroutine_to_open) why? > { > /* > Process elements of the prelocking set which are present there > @@ -8887,17 +8887,16 @@ bool is_equal(const LEX_CSTRING *a, const LEX_CSTRING *b) > open_system_tables_for_read() > thd Thread context. > table_list List of tables to open. > - backup Pointer to Open_tables_state instance where > - information about currently open tables will be > - saved, and from which will be restored when we will > - end work with system tables. > > NOTES > + Caller should have used start_new_trans object to start a new > + transcation when reading system tables. assert, may be? > + > Thanks to restrictions which we put on opening and locking of > system tables for writing, we can open and lock them for reading > - even when we already have some other tables open and locked. One > - must call close_system_tables() to close systems tables opened > - with this call. > + even when we already have some other tables open and locked. > + One should call thd->commit_whole_transaction_and_close_tables() > + to close systems tables opened with this call. > > NOTES > In some situations we use this function to open system tables for > diff --git a/sql/sql_help.cc b/sql/sql_help.cc > index c9307b578fc..3ccab553bfe 100644 > --- a/sql/sql_help.cc > +++ b/sql/sql_help.cc > @@ -709,8 +709,9 @@ static bool mysqld_help_internal(THD *thd, const char *mask) > Reset and backup the current open tables state to > make it possible. > */ > - Open_tables_backup open_tables_state_backup; > - if (open_system_tables_for_read(thd, tables, &open_tables_state_backup)) > + start_new_trans new_trans(thd); I'm starting to think that this (and in some other places) is rather redundant. Transactions can be expensive, the overhead being quite big. It seems like a overkill to wrap read-only system table accesses in a separate transaction, they could as well be performed as a part of the parent transaction. Write accesses has to be wrapped in a dedicated transaction, of course. > + > + if (open_system_tables_for_read(thd, tables)) > goto error2; > > /* > diff --git a/sql/sql_show.cc b/sql/sql_show.cc > index 2528134f4ee..db5b4d1c5fd 100644 > --- a/sql/sql_show.cc > +++ b/sql/sql_show.cc > @@ -6105,7 +6105,7 @@ static my_bool iter_schema_engines(THD *thd, plugin_ref plugin, > table->field[1]->store(option_name, strlen(option_name), scs); > table->field[2]->store(plugin_decl(plugin)->descr, > strlen(plugin_decl(plugin)->descr), scs); > - tmp= &yesno[MY_TEST(hton->commit)]; > + tmp= &yesno[MY_TEST(hton->commit && !(hton->flags & HTON_NO_ROLLBACK))]; Why not to say that hton->rollback==NULL means no rollback? Then you won't need a new flag for that. And you'll remove redundancy, what would it mean if HTON_NO_ROLLBACK is present, but hton->rollback!=NULL? Or the other way around, HTON_NO_ROLLBACK is not present, but hton->rollback==NULL? > table->field[3]->store(tmp->str, tmp->length, scs); > table->field[3]->set_notnull(); > tmp= &yesno[MY_TEST(hton->prepare)]; > diff --git a/sql/sql_statistics.cc b/sql/sql_statistics.cc > index 55e8e52c052..7a817aea97e 100644 > --- a/sql/sql_statistics.cc > +++ b/sql/sql_statistics.cc > @@ -230,17 +230,17 @@ index_stat_def= {INDEX_STAT_N_FIELDS, index_stat_fields, 4, index_stat_pk_col}; > Open all statistical tables and lock them > */ > > -static int open_stat_tables(THD *thd, TABLE_LIST *tables, > - Open_tables_backup *backup, bool for_write) > +static int open_stat_tables(THD *thd, TABLE_LIST *tables, bool for_write) > { > int rc; > - > Dummy_error_handler deh; // suppress errors > + DBUG_ASSERT(thd->transaction != &thd->default_transaction); I see, you already use such an assert :) > + > thd->push_internal_handler(&deh); > init_table_list_for_stat_tables(tables, for_write); > init_mdl_requests(tables); > thd->in_sub_stmt|= SUB_STMT_STAT_TABLES; > - rc= open_system_tables_for_read(thd, tables, backup); > + rc= open_system_tables_for_read(thd, tables); > thd->in_sub_stmt&= ~SUB_STMT_STAT_TABLES; > thd->pop_internal_handler(); > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] b22a28c2295: fixup! 3fe5cd5e1785e3e8de7add9977a1c2ddd403538b
by Sergei Golubchik 22 May '20

22 May '20

Hi, Michael! On May 20, Michael Widenius wrote: > revision-id: b22a28c2295 (mariadb-10.5.2-256-gb22a28c2295) > parent(s): acbe14b122c > author: Michael Widenius <monty(a)mariadb.com> > committer: Michael Widenius <monty(a)mariadb.com> > timestamp: 2020-05-19 17:52:18 +0300 > message: > > fixup! 3fe5cd5e1785e3e8de7add9977a1c2ddd403538b > > MDEV-22607 Assertion `ha_info->ht() != binlog_hton' failed in > MYSQL_BIN_LOG::unlog_xa_prepare > > diff --git a/sql/log.cc b/sql/log.cc > index 7e9e231358a..792c6bb1a99 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -10128,11 +10128,6 @@ int TC_LOG_BINLOG::unlog_xa_prepare(THD *thd, bool all) > if (rw_count > 0) > { > /* an empty XA-prepare event group is logged */ > -#ifndef DBUG_OFF > - for (ha_info= thd->transaction->all.ha_list; rw_count > 1 && ha_info; > - ha_info= ha_info->next()) > - DBUG_ASSERT(ha_info->ht() != binlog_hton); > -#endif > rc= write_empty_xa_prepare(thd, cache_mngr); // normally gains need_unlog > trans_register_ha(thd, true, binlog_hton, 0); // do it for future commmit > } What does this assertion mean and what does it mean that it no longer holds? Regards, Sergei

2 1

Re: [Maria-developers] 20d54b09830: Update galera to work with independent sub transactions
by Sergei Golubchik 22 May '20

22 May '20

Hi, Michael! I see, you basically disable wsrep for these out-of-band transactions. I'm sure it works fine now, with Aria. But will it work with InnoDB? On May 20, Michael Widenius wrote: > revision-id: 20d54b09830 (mariadb-10.5.2-257-g20d54b09830) > parent(s): b22a28c2295 > author: Michael Widenius <monty(a)mariadb.com> > committer: Michael Widenius <monty(a)mariadb.com> > timestamp: 2020-05-19 17:55:00 +0300 > message: > > Update galera to work with independent sub transactions > > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index 51d7380f622..4c87cbeee59 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -5806,6 +5806,8 @@ start_new_trans::start_new_trans(THD *thd) > server_status= thd->server_status; > m_transaction_psi= thd->m_transaction_psi; > thd->m_transaction_psi= 0; > + wsrep_on= thd->variables.wsrep_on; > + thd->variables.wsrep_on= 0; > thd->server_status&= ~(SERVER_STATUS_IN_TRANS | > SERVER_STATUS_IN_TRANS_READONLY); > thd->server_status|= SERVER_STATUS_AUTOCOMMIT; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] c5238d7b3b7: MDEV-22545: my_vsnprintf behaves not as in C standard
by Sergei Golubchik 20 May '20

20 May '20

Hi, Oleksandr! On May 20, Oleksandr Byelkin wrote: > revision-id: c5238d7b3b7 (mariadb-10.5.3-200-gc5238d7b3b7) > parent(s): 8f9c53586bf > author: Oleksandr Byelkin <sanja(a)mariadb.com> > committer: Oleksandr Byelkin <sanja(a)mariadb.com> > timestamp: 2020-05-20 08:35:12 +0200 > message: > > MDEV-22545: my_vsnprintf behaves not as in C standard > > Added parameter %T for string which should be visibly truncated. > > diff --git a/include/mysql/service_my_snprintf.h b/include/mysql/service_my_snprintf.h > index bd1f069c527..94a9e07dcbb 100644 > --- a/include/mysql/service_my_snprintf.h > +++ b/include/mysql/service_my_snprintf.h > @@ -69,6 +70,9 @@ > Format 'M': takes one integer, prints this integer, space, double quote > error message, double quote. In other words > printf("%M", n) === printf("%d \"%s\"", n, strerror(n)) > + > + Format 'T': takes string and print it like s but if the strints shoud be should > + truncated puts "..." at the end. > */ > > #ifdef __cplusplus > diff --git a/mysql-test/main/frm-debug.result b/mysql-test/main/frm-debug.result > index caf344e241d..332d7e00a8f 100644 > --- a/mysql-test/main/frm-debug.result > +++ b/mysql-test/main/frm-debug.result > @@ -13,8 +13,8 @@ CREATE TABLE t1 (c01 INT, c02 CHAR(20), c03 TEXT, c04 DOUBLE); > Warnings: > Note 1105 build_frm_image: Field data type info length: 14 > Note 1105 DBUG: [0] name='c01' type_info='' > -Note 1105 DBUG: [1] name='c02' type_info='xc...' > -Note 1105 DBUG: [2] name='c03' type_info='xb...' > +Note 1105 DBUG: [1] name='c02' type_info='xchar' > +Note 1105 DBUG: [2] name='c03' type_info='xblob' good > Note 1105 DBUG: [3] name='c04' type_info='' > SET SESSION debug_dbug="-d,frm_data_type_info_emulate"; > SET SESSION debug_dbug="-d,frm_data_type_info"; > diff --git a/mysql-test/main/xml.result b/mysql-test/main/xml.result > index efaca961b4a..84dbd68438f 100644 > --- a/mysql-test/main/xml.result > +++ b/mysql-test/main/xml.result > @@ -1007,7 +1007,7 @@ Warning 1292 Truncated incorrect INTEGER value: 'string ' > Warning 1292 Truncated incorrect INTEGER value: 'string ' > DROP PROCEDURE spxml; > select UpdateXML('<a>a</a>',repeat('a b ',1000),''); > -ERROR HY000: XPATH syntax error: 'b a b a b a b a b a b a b a b...' > +ERROR HY000: XPATH syntax error: 'b a b a b a b a b a b a b a b a ' this doesn't look right, the string is truncated, isn't it? > select ExtractValue('<a>a</a>', '/a[@x=@y0123456789_0123456789_0123456789_0123456789]'); > ERROR HY000: XPATH error: comparison of two nodesets is not supported: '=@y0123456789_0123456789_0123...' > select ExtractValue('<a>a</a>', '/a[@x=$y0123456789_0123456789_0123456789_0123456789]'); > @@ -1110,9 +1110,9 @@ NULL > # Bug#57279 updatexml dies with: Assertion failed: str_arg[length] == 0 > # > SELECT UPDATEXML(NULL, (LPAD(0.1111E-15, '2011', 1)), 1); > -ERROR 22007: Illegal double '111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111...' value found during parsing > +ERROR 22007: Illegal double '111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111111' value found during parsing same here, lpad appends 2011 characters, so this "double" is clearly truncated. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] b97e45651d1: MDEV-16937 Strict SQL with system versioned tables causes issues
by Sergei Golubchik 19 May '20

19 May '20

Hi, Aleksey! First - is commit b97e45651d1 the one I was supposed to review? On May 17, Aleksey Midenkov wrote: > revision-id: b97e45651d1 (mariadb-10.4.7-33-gb97e45651d1) > parent(s): 7587975bf06 > author: Aleksey Midenkov <midenok(a)gmail.com> > committer: Aleksey Midenkov <midenok(a)gmail.com> > timestamp: 2019-08-19 11:58:07 +0300 > message: > > MDEV-16937 Strict SQL with system versioned tables causes issues > > Vers SQL: respect system fields in NO_ZERO_DATE mode. > > This is the subject for refactoring in tempesta-tech/mariadb#379 > > diff --git a/mysql-test/suite/versioning/engines.combinations b/mysql-test/suite/versioning/engines.combinations > index 26b5bab23f1..57e2af6cd06 100644 > --- a/mysql-test/suite/versioning/engines.combinations > +++ b/mysql-test/suite/versioning/engines.combinations > @@ -7,5 +7,10 @@ default-storage-engine=innodb > [myisam] > default-storage-engine=myisam > > +[traditional] > +default-storage-engine=myisam > +sql-mode=traditional I'm not sure about it. It's, of course, safe to run all tests with every possible settings, but it really blows up testing times. May be it'd make sense to have a more targeted test here? > + > [heap] > default-storage-engine=memory > + > diff --git a/mysql-test/suite/versioning/r/select.result b/mysql-test/suite/versioning/r/select.result > index 3569268ce1d..68df246af6b 100644 > --- a/mysql-test/suite/versioning/r/select.result > +++ b/mysql-test/suite/versioning/r/select.result > @@ -45,7 +45,7 @@ ASOF_x y > 7 107 > 8 108 > 9 109 > -select x as FROMTO_x, y from t1 for system_time from timestamp '0-0-0 0:0:0' to timestamp @t1; > +select x as FROMTO_x, y from t1 for system_time from timestamp '1970-01-01 00:00:00' to timestamp @t1; Does this test use a fixed time zone? If not, is 1970-01-01 00:00:00 a valid timestamp in all time zones? If unsure, better use 1971, that will definitely be fine everywhere. > FROMTO_x y > 0 100 > 1 101 > diff --git a/sql/field.cc b/sql/field.cc > index 969c32a5180..f5580239a9f 100644 > --- a/sql/field.cc > +++ b/sql/field.cc > @@ -11139,6 +11139,13 @@ bool Field::save_in_field_default_value(bool view_error_processing) > { > THD *thd= table->in_use; > > + /* > + TODO: refactor setting the system fields via default_value mechanism. > + This condition will go away as well as other conditions with VERS_SYSTEM_FIELD. > + */ MDEV? There's a pull request, but it is marked as obsolete > + if (flags & VERS_SYSTEM_FIELD) > + return false; > + > if (unlikely(flags & NO_DEFAULT_VALUE_FLAG && > real_type() != MYSQL_TYPE_ENUM)) > { > diff --git a/sql/unireg.cc b/sql/unireg.cc > index d019b5f8a75..484ad0334e4 100644 > --- a/sql/unireg.cc > +++ b/sql/unireg.cc > @@ -1060,7 +1060,8 @@ static bool make_empty_rec(THD *thd, uchar *buff, uint table_options, > !f_bit_as_char(field->pack_flag)) > null_count+= field->length & 7; > > - error= make_empty_rec_store_default(thd, regfield, field->default_value); > + if (!(field->flags & VERS_SYSTEM_FIELD)) > + error= make_empty_rec_store_default(thd, regfield, field->default_value); will the field be initialized at all? It should have something, even bzero will do, but we shouldn't dump uninited memory to frm. > delete regfield; // Avoid memory leaks > if (error) > goto err; > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 3

[Maria-developers] Progress report - Week 2
by Mohammed Hammaad Mateen 18 May '20

18 May '20

Greetings, Hope you are safe and doing great, This post describes the things I've done in my second week [ 11-17 May ] of Community Bonding Period under the mentor-ship of Sergei Golubchik and Oleksandr Byelkin for GSoC-20. Tasks done this week were mostly the continuation of the leftover tasks from the previous week and some related study. Things I've done include: 1:* I was caught up in analysing how was DELETE .... RETURNING implemented, which led me to read from:* • *sp_head.cc* : this made me understand that sp actually meant stored procedures but there is another called PS which I'm yet to figure out. • *sql_parse.cc* : from here I understood that delete return as actually { Analyse->Delete->Return } and all the logic for returning is executed if(lex->has_returning()). 2: Though I've raised the pull request, *MDEV-14558* is not yet completely tested as it does not include tests for *mysql \s*, moreover, I'm facing problems to escape the backslash inorder to write a test-case for *mysql \s* . Regards, Mohammed Hammaad Mateen

1 0

Re: [Maria-developers] 82b6a5cb85f: MDEV-22556: Incorrect result for window function when using encrypt-tmp-files=ON
by Sergei Golubchik 17 May '20

17 May '20

Hi, Varun! Ok to push. One comment below On May 17, Varun Gupta wrote: > revision-id: 82b6a5cb85f (mariadb-10.4.11-192-g82b6a5cb85f) > parent(s): 3bfe305c5cd > author: Varun Gupta <varun.gupta(a)mariadb.com> > committer: Varun Gupta <varun.gupta(a)mariadb.com> > timestamp: 2020-05-15 02:37:16 +0530 > message: > > MDEV-22556: Incorrect result for window function when using encrypt-tmp-files=ON > > diff --git a/mysql-test/suite/encryption/t/tempfiles_encrypted.opt b/mysql-test/suite/encryption/t/tempfiles_encrypted.opt > new file mode 100644 > index 00000000000..f515f7b8689 > --- /dev/null > +++ b/mysql-test/suite/encryption/t/tempfiles_encrypted.opt > @@ -0,0 +1,5 @@ > +--encrypt-tmp_files=ON > +--loose-file-key-management-encryption-algorithm=aes_cbc > +--file-key-management=FORCE > +--loose-file-key-management-filename=$MTR_SUITE_DIR/t/filekeys-data.enc > +--loose-file-key-management-filekey=FILE:$MTR_SUITE_DIR/t/filekeys-data.key this is very redundant, you include have_file_key_management_plugin.inc and it already sets all options for the file_key_management plugin to work. so you only need --encrypt-tmp_files=ON here. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] Progress Report - Week 1-2
by Anton Mikhaylenko 17 May '20

17 May '20

Good day, everyone. In this post, I want to describe the things I've done in the 1-2 weeks of GSoC-20. For begin, I messaged with my mentor Vladislav Vaintroub and I got some tips on where to start. The first step was to run the server and tests for MariaDB on Linux and Windows. At this stage, a little bug was found, which was documented by my mentor https://jira.mariadb.org/browse/MDEV-22479. The next step was to look at the theoretical part of my work. Specifically, what algorithm to use for dynamic concurrency. This is the algorithm "hill-climbing", which is used in .NET ThreadPool https://www.researchgate.net/publication/228977836_Optimizing_concurrency_l… . In the last step, I started to get into the work of the current ThreadPool in MariaDB. At the moment this step is not completed yet. What needs to be done next: Fully understand the theoretic part and how the current thread pool works. Regards, Anton MIkhailenko

1 0

Re: [Maria-developers] Review for MDEV-19751.
by Aleksey Midenkov 14 May '20

14 May '20

Hi, Alexey! Actually the bug is applicable only to primary key and only when default field list is set (i.e. empty KEY() clause). So updated the patch accordingly. Please peek again. On Wed, May 13, 2020 at 2:40 PM Alexey Botchkov <holyfoot(a)mariadb.com> wrote: > > Hello, Aleksey! > > > commit 5ef168e00f7b8c821463e46535a4beccb47ec8ea > + if (key_info->flags & HA_INVISIBLE_KEY) > + continue; > > I'd add || !(key_info->flags & HA_NOSAME). > Can the non-unique index participate in the KEY partitioning? > > + for (uint kp= 0; kp < key_info->user_defined_key_parts; ++kp) > + { > + const KEY_PART_INFO &key_part= key_info->key_part[kp]; > + for (Field **part_field= tab_part_info->part_field_array; > + *part_field; ++part_field) > + { > + if (*part_field == key_part.field) > + { > + *partition_changed= TRUE; > + goto search_finished; > + } > + } // for (part_field) > + } // for (key_part) > > So you check if the key has any of the partitioning fields in it. > In this case we are going to disallow INPLACE with no reason if the > unrelated key is removed just containing the primary key column. > I'd say we should check for the exact match, when the key has > all the part_field_array in that exact order. Disagree? > > And for the test - please add the case for a table with no PRIMARY KEY, bug with the UNIQUE instead. > > Best regards! > HF > -- All the best, Aleksey Midenkov @midenok

1 0

[Maria-developers] Progress Report - Week 1
by Mohammed Hammaad Mateen 12 May '20

12 May '20

Greetings, Hope you are safe and doing great, This post describes the things I've done in my first week [ 4-10 May ] of Community Bonding Period under the mentor-ship of Sergei Golubchik and Oleksandr Byelkin for GSoC-20 and I must acknowledge that it was really an interactive experience and at the same time informative as well. Things I've done include: 1: *Raising my first ever pull request for MariaDB under MDEV-14558 *: Learnt about *strnxmov*, a function of C++ which I had never used earlier. I also came across the procedure of testing my implemented code. Wrote some test cases for the same and generated their corresponding results. 2:* Understand the working of the functions under multi_delete*: Ran it in the debugger and tried to familiarize myself with the internals of how Multiple Table Delete actually works. Things to-do: 1: Though I've raised the pull request, MDEV-14558 is not completed tested as it does not include tests for *mysql \s*, moreover, test cases have to be refactored. 2: Learn more about the working of *multi_delete* so as to develop a parser for Multiple Table Delete Returning. 3: Write some test cases for Multiple Table Delete Returning before hand. Regards, Mohammed Hammaad Mateen

2 1

[Maria-developers] Review for: MDEV-17399 Add support for JSON_TABLE, part #3
by Sergey Petrunia 10 May '20

10 May '20

Hi Alexey, Please find the next batch of input below. (I haven't finished the review but I'm sending it now so you can start addressing it) > commit 68ca18518337443090bdeddf6d612129b6843c21 > Author: Alexey Botchkov <holyfoot(a)askmonty.org> > Date: Fri Apr 17 14:42:40 2020 +0400 > > MDEV-17399 Add support for JSON_TABLE. > > Syntax for JSON_TABLE added. > The ha_json_table handler added. Executing the JSON_TABLE we > create the temporary table of the ha_json_table, add dependencies > of other tables and sometimes conditions to WHERE. > ... > +public: > + ha_json_table(TABLE_SHARE *share_arg, Table_function_json_table *jt): > + handler(&table_function_hton.m_hton, share_arg), m_jt(jt) > + { > + mark_trx_read_write_done= 1; This definitely needs a comment. > + ~ha_json_table() {} > + handler *clone(const char *name, MEM_ROOT *mem_root) { return NULL; } > + const char *index_type(uint inx) { return "HASH"; } Why is ha_json_table still trying to pretend it supports indexes? Is this a left-over from the old way of interfacing with the optimizer that can now be removed? > +int Table_function_json_table::setup(THD *thd, TABLE_LIST *sql_table, > + COND **cond) > +{ > + if (m_json->fix_fields(thd, &m_json)) > + return TRUE; > + > + m_dep_tables= m_json->used_tables(); Why does this function have a 'cond' argument? Please remove it if it's not necessary. == Duplicate column names are allowed == select * from JSON_TABLE('{"color": "black", "price": 1000}', "$" COLUMNS (rowSeq FOR ORDINALITY, color VARCHAR(100) PATH '$.color', color VARCHAR(100) PATH '$.price') ) as T; +--------+-------+-------+ | rowSeq | color | color | +--------+-------+-------+ | 0 | black | 1000 | +--------+-------+-------+ Is it really okay that duplicate column names are allowed? == RIGHT JOIN is allowed == create table t1 (item_name varchar(32), item_props varchar(1024)); insert into t1 values ('Laptop', '{"color": "black", "price": 1000}'); insert into t1 values ('Jeans', '{"color": "blue", "price": 50}'); This query should not be allowed, because JSON_TABLE has a dependency on an inner side of an outer join: select * from t1 right join json_table(t1.item_props, '$' columns(color varchar(100) path '$.color', f2 for ordinality) ) as T on 1; === NATURAL JOIN crashes == create table t10 (color varchar(100), f2 int); select * from t10 natural join json_table(t1.item_props, '$' columns(color varchar(100) path '$.color', f2 for ordinality) ) as JT ; Thread 31 "mysqld" received signal SIGSEGV, Segmentation fault. 0x0000555555d8ac12 in Select_limit_counters::get_select_limit (this=0xa5a5a5a5a5a5ad2d) at /home/psergey/dev-git/10.5-json/sql/sql_limit.h:67 (gdb) wher #0 0x0000555555d8ac12 in Select_limit_counters::get_select_limit (this=0xa5a5a5a5a5a5ad2d) at /home/psergey/dev-git/10.5-json/sql/sql_limit.h:67 #1 0x0000555555e2a4a0 in JOIN::optimize_inner (this=0x7fffac018520) at /home/psergey/dev-git/10.5-json/sql/sql_select.cc:1805 #2 0x0000555555e29b12 in JOIN::optimize (this=0x7fffac018520) at /home/psergey/dev-git/10.5-json/sql/sql_select.cc:1612 BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 1

[Maria-developers] OSX fix need on MariaDB-10.5
by Daniel Black 09 May '20

09 May '20

Mac lovers (who can use C/C++ and a debugger), As your MariaDB developers are hard at work getting 10.5 ready for you, unfortunately a bug has slipped into the 10.5 branch making it unusable on OSX (at least that's what it looks like on the CI infrastructure). The bug report is here: https://jira.mariadb.org/browse/MDEV-22173 As the time of developers is a little short as is the availability of OSX hardware, I'm hoping some of you are able to spend a few hours tackling this task to understand it better, and if possible, create patch. Steps: 1. build environment https://mariadb.com/kb/en/Build_Environment_Setup_for_Mac/ 2. clone the source git clone --branch 10.5 https://github.com/MariaDB/server.git mariadb-server cd mariadb-server git submodule update --recursive --init 3. build instructions https://mariadb.com/kb/en/generic-build-instructions/ (you don't need to install) 4. test (from build directory) mysql-test/mysql-test-run main.select might be useful here: --debugger=NAME or --gdb client-debugger=NAME or client-gdb To collaborative share your findings and to ask for further help MariaDB's community chat stream has been created for you. https://mariadb.zulipchat.com/#narrow/stream/118759-general/topic/Get.20OSX… To paraphrase J.F.Kennedy ask not what your database server community can do for you — ask what you can do for your database server community. Keep the community spirit going, Daniel Black -- Sent as a community member who noticed OSX failures and that everyone is busy

2 3

Re: [Maria-developers] 4312f382b6f: MDEV-22312: Bad error message for SET DEFAULT ROLE when user account is not granted the role
by Sergei Golubchik 08 May '20

08 May '20

Hi, Anel! On May 08, Anel Husakovic wrote: > revision-id: 4312f382b6f (mariadb-10.1.43-138-g4312f382b6f) > parent(s): 8c4b5261210 > author: Anel Husakovic <anel(a)mariadb.org> > committer: Anel Husakovic <anel(a)mariadb.org> > timestamp: 2020-05-07 16:49:33 +0200 > message: > > MDEV-22312: Bad error message for SET DEFAULT ROLE when user account is not granted the role > > diff --git a/mysql-test/suite/roles/set_role-recursive.test b/mysql-test/suite/roles/set_role-recursive.test > index 23d623e1966..dcf3dd5b6fc 100644 > --- a/mysql-test/suite/roles/set_role-recursive.test > +++ b/mysql-test/suite/roles/set_role-recursive.test > @@ -44,7 +44,7 @@ select * from mysql.roles_mapping; > > --sorted_result > show grants; > ---error ER_INVALID_ROLE > +--error ER_SET_DEFAULT_ROLE_FOR_USER > set role test_role2; > select current_user(), current_role(); > --sorted_result > diff --git a/sql/share/errmsg-utf8.txt b/sql/share/errmsg-utf8.txt > index c5e83062f0f..83ecd4735e8 100644 > --- a/sql/share/errmsg-utf8.txt > +++ b/sql/share/errmsg-utf8.txt > @@ -7144,3 +7144,5 @@ ER_WARN_AGGFUNC_DEPENDENCE > ukr "Агрегатна функція '%-.192s)' з SELECTу #%d належить до SELECTу #%d" > WARN_INNODB_PARTITION_OPTION_IGNORED > eng "<%-.64s> option ignored for InnoDB partition" > +ER_SET_DEFAULT_ROLE_FOR_USER > + eng "User '%s' has not been granted a role '%s'." Sorry, this is an absolute no-no. We cannot add new error messages to the old GA versions. Only to the very last RC/GA or to any alpha/beta. When you add an error message, all error messages added after it in newer versions will shift, change their numbers. And error numbers in RC/GA versions cannot change. Instead of a new error message and my_error(ER_SET_DEFAULT_ROLE_FOR_USER, MYF(0), c_usr.ptr(), rolename); you have to write my_printf_error(ER_INVALID_ROLE, "User %`s has not been granted a role %`s", MYF(0), c_usr.c_ptr(), rolename); note other changes: * %`s instead of '%s' for correct identifier quoting * no dot at the end * c_ptr() because printf expects a zero-terminated C string, and ptr() doesn't guarantee that. > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index b6a6f806e50..37141dd2390 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -2041,6 +2041,9 @@ static int check_user_can_set_role(const char *user, const char *host, > ACL_USER_BASE *acl_user_base; > ACL_USER *UNINIT_VAR(acl_user); > bool is_granted= FALSE; > + bool user_can_see_role= FALSE; > + char buf[1024]; > + String c_usr(buf, sizeof(buf), &my_charset_bin); use StringBuffer<1024> c_str; > int result= 0; > > /* clear role privileges */ > @@ -2083,14 +2086,39 @@ static int check_user_can_set_role(const char *user, const char *host, > is_granted= TRUE; > break; > } > + > + for (uint i=0; i < acl_user->role_grants.elements; i++) > + { > + ACL_ROLE *r= *(dynamic_element(&acl_user->role_grants, i, ACL_ROLE**)); > + if (!strcmp(safe_str(r->user.str), safe_str(rolename))) > + { > + size_t len= acl_user->user.length + acl_user->hostname_length + 2; > + c_usr.alloc(len); > + const char c= '@'; > + strmov(strmov(strmov(const_cast<char*>(c_usr.ptr()), > + safe_str(user)), &c), safe_str(host)); > + user_can_see_role= TRUE; > + } > + } No, you repeat this for() block for evert role grant of every role's grantee. And you only use it when there was an error, an uncommon case. Do not do the work that you aren't going to use. And the check doesn't seem correct. You want to use the same rule as for applicable_roles, you need to traverse the graph. Perhaps you can simply use check_role_is_granted() though. And also you need to check whether the current user has read access on mysql.user table - in that case it can see all roles, granted or not. > } > > /* According to SQL standard, the same error message must be presented */ this comment now belongs inside the if(), above ER_INVALID_ROLE > if (!is_granted) > { > - my_error(ER_INVALID_ROLE, MYF(0), rolename); > - result= 1; > - goto end; > + /* Handling of the bad error message - MDEV-22312 */ don't put MDEV numbers into the code, please. > + /* If the current user can see the role generate different type of error. */ > + if(user_can_see_role) > + { > + my_error(ER_SET_DEFAULT_ROLE_FOR_USER, MYF(0), c_usr.ptr(), rolename); > + result= 1; > + goto end; > + } > + else > + { > + my_error(ER_INVALID_ROLE, MYF(0), rolename); > + result= 1; > + goto end; > + } > } > > if (access) Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] Welcome to Google Summer of Code 2020 with MariaDB
by Vicențiu Ciorbaru 04 May '20

04 May '20

Hi! GSoC 2020 accepted proposals have just been announced and you are part of the successful applicants. This year we had a lot of very good proposals and so we have a one of the highest number of students (7) to mentor for GSoC. Congratulations on the effort to learn about the program, get involved and submit successful proposals! The fun part begins now! If you have not done so already, please reach out to your mentor and seek further advice about your project. Each one of your projects can be implemented in multiple ways, some more complex than others. I strongly advise you to create a concrete plan based off of your proposal with your mentor and aim to stick to it. The MariaDB Server codebase is a complex one with lots of different parts, however your mentor can guide you to ensure you have a successful GSoC. Remember that you are here to learn about Open Source, what it means to be part of an Open Source community, to interact with the community, all while becoming better programmers. Please take advantage of this opportunity and keep your technical discussions public, on Zulip https://mariadb.zulipchat.com, and on our developer mailing list https://launchpad.net/~maria-developers. By keeping your discussions public, you get the benefit of not only having your mentor see your questions but all of the MariaDB community. You never know when someone might lend a hand in case your mentor is not yet online. :) The mantra of our community should be that we are always welcoming to new people. Do not be afraid to ask questions. There are no bad or stupid or silly questions. The only key requirement is that you first make an effort to find the answer yourself and only afterwards seek outside help. If you can show a bit of involvement, the community will show you the same. I recommend that you agree with your mentor on some form of weekly progress reporting, via either a blog-post, an email, or something similar that can be publicly shared. I can not stress this enough: we are working on an open source project, our work must be done in the open and it is important to *show the world the good we are doing*. Hope all of you will have a great GSoC! Vicențiu MariaDB Foundation GSoC Admin

1 0

Re: [Maria-developers] d08860b28f3: MDEV-22374: VIEW with security definer require FILE privilege from definer not invoker in case of INTO OUTFILE
by Sergei Golubchik 04 May '20

04 May '20

Hi, Oleksandr! On May 04, Oleksandr Byelkin wrote: > revision-id: d08860b28f3 (mariadb-5.5.67-15-gd08860b28f3) > parent(s): ac2604f923f > author: Oleksandr Byelkin <sanja(a)mariadb.com> > committer: Oleksandr Byelkin <sanja(a)mariadb.com> > timestamp: 2020-04-28 09:16:33 +0200 > message: > > MDEV-22374: VIEW with security definer require FILE privilege from definer not invoker in case of INTO OUTFILE > > Check INTO OUTFILE clause always from invoker. > > --- > mysql-test/r/view_grant.result | 21 +++++++++++++++++++++ > mysql-test/t/view_grant.test | 33 +++++++++++++++++++++++++++++++++ > sql/sql_parse.cc | 18 +++++++++--------- > 3 files changed, 63 insertions(+), 9 deletions(-) > > diff --git a/mysql-test/r/view_grant.result b/mysql-test/r/view_grant.result > index b2d3a0b8ca4..37dc6adc978 100644 > --- a/mysql-test/r/view_grant.result > +++ b/mysql-test/r/view_grant.result > @@ -1587,3 +1587,24 @@ USE test; > DROP DATABASE mysqltest1; > DROP USER 'mysqluser1'@'%'; > DROP USER 'mysqluser2'@'%'; > +# > +# MDEV-22374: VIEW with security definer require FILE privilege from definer > +# not invoker in case of INTO OUTFILE > +# > +create user test@localhost; > +grant select on test.* to test@localhost; > +create table t1 (a int); > +create definer=test@localhost sql security definer view v1 as select * from t1; > +# check that ot works without view typo "it" > +select * INTO OUTFILE '../..//tmp/test_out_txt' from t1; > +# check that ot works without file same why "//" ? > +select * from v1; > +a > +# rights for file should be taken from current user not view > +select * INTO OUTFILE '../../tmp/test_out_txt' from (select count(*) from v1) as dv1; > +select * INTO OUTFILE '../..//tmp/test_out_txt' from (select * from v1) as dv1; > +select * INTO OUTFILE '../..//tmp/test_out_txt' from v1; same > +drop view v1; please add reverse tests too: +create sql security definer view v1 as select * from t1; +connect(con1, localhost, test); +error 1045; +select * INTO OUTFILE '../..//tmp/test_out_txt' from t1; +select * from v1; +error 1045; +select * INTO OUTFILE '../../tmp/test_out_txt' from (select count(*) from v1) as dv1; +error 1045; +select * INTO OUTFILE '../..//tmp/test_out_txt' from (select * from v1) as dv1; +error 1045; +select * INTO OUTFILE '../..//tmp/test_out_txt' from v1; +connection default; > +drop table t1; > +drop user test@localhost; > +# End of 5.5 tests > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > index ae5a6b4cd35..515990c879f 100644 > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -2206,15 +2206,15 @@ mysql_execute_command(THD *thd) > lex->exchange != NULL implies SELECT .. INTO OUTFILE and this > requires FILE_ACL access. > */ > - ulong privileges_requested= lex->exchange ? SELECT_ACL | FILE_ACL : > - SELECT_ACL; > - > - if (all_tables) > - res= check_table_access(thd, > - privileges_requested, > - all_tables, FALSE, UINT_MAX, FALSE); > - else > - res= check_access(thd, privileges_requested, any_db, NULL, NULL, 0, 0); > + if (lex->exchange) > + res= check_access(thd, FILE_ACL, any_db, NULL, NULL, 0, 0); > + if (!res) > + { > + if (all_tables) > + res= check_table_access(thd, SELECT_ACL, all_tables, FALSE, UINT_MAX, FALSE); > + else > + res= check_access(thd, SELECT_ACL, any_db, NULL, NULL, 0, 0); > + } No, I think this special check for a special SELECT ... INTO OUTFILE case is wrong. Why does the server even checks global privileges via views? It should not do it, I think, not for FILE not for any other global or db level privilege. I'd try something like diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc --- a/sql/sql_acl.cc +++ b/sql/sql_acl.cc @@ -4699,7 +4699,7 @@ bool check_grant(THD *thd, ulong want_access, TABLE_LIST > Save a copy of the privileges without the SHOW_VIEW_ACL attribute. It will be checked during making view. */ - tl->grant.orig_want_privilege= (want_access & ~SHOW_VIEW_ACL); + tl->grant.orig_want_privilege= want_access & ~SHOW_VIEW_ACL & TABLE_ACLS; } mysql_rwlock_rdlock(&LOCK_grant); diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc --- a/sql/sql_parse.cc +++ b/sql/sql_parse.cc @@ -5204,7 +5204,8 @@ check_table_access(THD *thd, ulong requirements,TABLE_LIS> Register access for view underlying table. Remove SHOW_VIEW_ACL, because it will be checked during making view */ - table_ref->grant.orig_want_privilege= (want_access & ~SHOW_VIEW_ACL); + table_ref->grant.orig_want_privilege= + want_access & ~SHOW_VIEW_ACL & TABLE_ACLS; if (table_ref->schema_table_reformed) { Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] 3ead2cea95c: MDEV-13266: Race condition in ANALYZE TABLE / statistics collection
by Sergey Petrunia 04 May '20

04 May '20

Hi Varun, On Sun, Apr 12, 2020 at 09:08:37PM +0530, Varun wrote: > revision-id: 3ead2cea95c32b7ceaf6e6ec81f7afbd9137cfe9 (mariadb-10.2.31-103-g3ead2cea95c) > parent(s): d565895bbd8e2a163be48b9bac51fccbf3949c80 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2020-04-12 21:05:36 +0530 > message: > > MDEV-13266: Race condition in ANALYZE TABLE / statistics collection > > Fixing a race condition while collecting the engine independent statistics. > The issue here was when the statistics was collected on specific indexes > then because of some race condition the statistics for indexes was not > collected. The TABLE::keys_in_use_for_query was set to 0 in such cases. > > This happens when the table is opened from TABLE_SHARE instead of the > table share stored in table_cache. It would be nice to write down the race condition that we've caught (in case we need this info later) Is my understanding correct: the error scenario is as follows: Thread1> start running "ANALYZE TABLE t PERISTENT FOR COLUMNS (..) INDEXES ($list)" Thread1> Walk through $list and save it in TABLE::keys_in_use_for_query Thread1> Close/re-open tables Thread2> Make some use of table t. This involves taking table t from Thread2> the table cache, and putting it back (with TABLE::keys_in_use_for_query reset to 0) Thread1> continue collecting EITS stats. Since TABLE::keys_in_use_for_query we will not collect statistics for indexes in $list. Please confirm (and if not, describe the race condition). The patch also introduces this behaviour: analyze table ... persistent for ... indexes(no_such_index); will now cause engine statistics to be still collected. Before the patch it exited with an error. But then, this is consistent with what happens for analyze table ... persistent for columns(no_such_column) ... So I guess this is ok. > --- > sql/sql_admin.cc | 46 +++++++++++++++++++++------------------------- > 1 file changed, 21 insertions(+), 25 deletions(-) > > diff --git a/sql/sql_admin.cc b/sql/sql_admin.cc > index ab95fdc340c..0613495f202 100644 > --- a/sql/sql_admin.cc > +++ b/sql/sql_admin.cc > @@ -769,31 +769,6 @@ static bool mysql_admin_table(THD* thd, TABLE_LIST* tables, > (table->table->s->table_category == TABLE_CATEGORY_USER && > (get_use_stat_tables_mode(thd) > NEVER || > lex->with_persistent_for_clause)); > - > - > - if (!lex->index_list) > - { > - tab->keys_in_use_for_query.init(tab->s->keys); > - } > - else > - { > - int pos; > - LEX_STRING *index_name; > - List_iterator_fast<LEX_STRING> it(*lex->index_list); > - > - tab->keys_in_use_for_query.clear_all(); > - while ((index_name= it++)) > - { > - if (tab->s->keynames.type_names == 0 || > - (pos= find_type(&tab->s->keynames, index_name->str, > - index_name->length, 1)) <= 0) > - { > - compl_result_code= result_code= HA_ADMIN_INVALID; > - break; > - } > - tab->keys_in_use_for_query.set_bit(--pos); > - } > - } > } > > if (result_code == HA_ADMIN_OK) > @@ -878,6 +853,27 @@ static bool mysql_admin_table(THD* thd, TABLE_LIST* tables, > } > tab->file->column_bitmaps_signal(); > } > + if (!lex->index_list) > + tab->keys_in_use_for_query.init(tab->s->keys); > + else > + { > + int pos; > + LEX_STRING *index_name; > + List_iterator_fast<LEX_STRING> it(*lex->index_list); > + > + tab->keys_in_use_for_query.clear_all(); > + while ((index_name= it++)) > + { > + if (tab->s->keynames.type_names == 0 || > + (pos= find_type(&tab->s->keynames, index_name->str, > + index_name->length, 1)) <= 0) > + { > + compl_result_code= result_code= HA_ADMIN_INVALID; > + break; > + } > + tab->keys_in_use_for_query.set_bit(--pos); > + } > + } > if (!(compl_result_code= > alloc_statistics_for_table(thd, table->table)) && > !(compl_result_code= > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

2 2

[Maria-developers] Igor please review: MDEV-22377: Subquery in an [UPDATE] query uses full scan instead of range
by Sergey Petrunia 30 Apr '20

30 Apr '20

Hello Igor, Could you please review the below: commit 793b100cc01228f01ead841d6c45f9e2b07786f9 Author: Sergei Petrunia <psergey(a)askmonty.org> Date: Thu Apr 30 01:25:11 2020 +0300 MDEV-22377: Subquery in an [UPDATE] query uses full scan instead of range When doing IN->EXISTS rewrite, Item_in_subselect::inject_in_to_exists_cond injects equalities into subquery's WHERE condition. The problem is that build_equal_items() has already been called for the subquery's WHERE, and tampering with the WHERE condition can prevent equality propagation from working. If the subquery's WHERE is an Item_cond_and with multiple equalities: - Item_equal objects form a suffix sub-list of the list in WHERE's Item_cond_and::list. The suffix is stored in JOIN::cond_equal->current_level. - Item_cond_and::m_cond_equal must also be preserved. This patch makes inject_in_to_exists_cond() not to break these properties diff --git a/mysql-test/main/subselect2.result b/mysql-test/main/subselect2.result index a3d7fda7abc..c54d635230f 100644 --- a/mysql-test/main/subselect2.result +++ b/mysql-test/main/subselect2.result @@ -262,7 +262,7 @@ id select_type table type possible_keys key key_len ref rows filtered Extra 2 DEPENDENT SUBQUERY t2c ALL NULL NULL NULL NULL 2 100.00 Using where; Using join buffer (incremental, BNL join) Warnings: Note 1276 Field or reference 'test.t1.a' of SELECT #2 was resolved in SELECT #1 -Note 1003 /* select#1 */ select `test`.`t1`.`a` AS `a`,`test`.`t1`.`c1` AS `c1` from `test`.`t1` where !<expr_cache><`test`.`t1`.`c1`,`test`.`t1`.`a`>(<in_optimizer>(`test`.`t1`.`c1`,<exists>(/* select#2 */ select `test`.`t2a`.`c2` from `test`.`t2` `t2a` join `test`.`t2` `t2b` join `test`.`t2` `t2c` where (`test`.`t2b`.`m` <> `test`.`t1`.`a` or `test`.`t2b`.`m` = `test`.`t2a`.`m`) and trigcond(<cache>(`test`.`t1`.`c1`) = `test`.`t2a`.`c2` or `test`.`t2a`.`c2` is null) and `test`.`t2c`.`c2` = `test`.`t2b`.`c2` and `test`.`t2b`.`n` = `test`.`t2a`.`m` having trigcond(`test`.`t2a`.`c2` is null)))) +Note 1003 /* select#1 */ select `test`.`t1`.`a` AS `a`,`test`.`t1`.`c1` AS `c1` from `test`.`t1` where !<expr_cache><`test`.`t1`.`c1`,`test`.`t1`.`a`>(<in_optimizer>(`test`.`t1`.`c1`,<exists>(/* select#2 */ select `test`.`t2a`.`c2` from `test`.`t2` `t2a` join `test`.`t2` `t2b` join `test`.`t2` `t2c` where `test`.`t2c`.`c2` = `test`.`t2b`.`c2` and `test`.`t2b`.`n` = `test`.`t2a`.`m` and (`test`.`t2b`.`m` <> `test`.`t1`.`a` or `test`.`t2b`.`m` = `test`.`t2a`.`m`) and trigcond(<cache>(`test`.`t1`.`c1`) = `test`.`t2a`.`c2` or `test`.`t2a`.`c2` is null) having trigcond(`test`.`t2a`.`c2` is null)))) DROP TABLE t1,t2; # # MDEV-614, also MDEV-536, also LP:1050806: diff --git a/mysql-test/main/subselect3.result b/mysql-test/main/subselect3.result index 5c4544a1b05..0034f61ac23 100644 --- a/mysql-test/main/subselect3.result +++ b/mysql-test/main/subselect3.result @@ -169,7 +169,7 @@ id select_type table type possible_keys key key_len ref rows filtered Extra 2 DEPENDENT SUBQUERY t2 ref a a 5 test.t1.b 1 100.00 Using where Warnings: Note 1276 Field or reference 'test.t3.oref' of SELECT #2 was resolved in SELECT #1 -Note 1003 /* select#1 */ select `test`.`t3`.`a` AS `a`,`test`.`t3`.`oref` AS `oref`,<expr_cache><`test`.`t3`.`a`,`test`.`t3`.`oref`>(<in_optimizer>(`test`.`t3`.`a`,<exists>(/* select#2 */ select `test`.`t1`.`a` from `test`.`t1` join `test`.`t2` where `test`.`t2`.`b` = `test`.`t3`.`oref` and trigcond(<cache>(`test`.`t3`.`a`) = `test`.`t1`.`a` or `test`.`t1`.`a` is null) and `test`.`t2`.`a` = `test`.`t1`.`b` having trigcond(`test`.`t1`.`a` is null)))) AS `Z` from `test`.`t3` +Note 1003 /* select#1 */ select `test`.`t3`.`a` AS `a`,`test`.`t3`.`oref` AS `oref`,<expr_cache><`test`.`t3`.`a`,`test`.`t3`.`oref`>(<in_optimizer>(`test`.`t3`.`a`,<exists>(/* select#2 */ select `test`.`t1`.`a` from `test`.`t1` join `test`.`t2` where `test`.`t2`.`a` = `test`.`t1`.`b` and `test`.`t2`.`b` = `test`.`t3`.`oref` and trigcond(<cache>(`test`.`t3`.`a`) = `test`.`t1`.`a` or `test`.`t1`.`a` is null) having trigcond(`test`.`t1`.`a` is null)))) AS `Z` from `test`.`t3` drop table t1, t2, t3; create table t1 (a int NOT NULL, b int NOT NULL, key(a)); insert into t1 values @@ -197,7 +197,7 @@ id select_type table type possible_keys key key_len ref rows filtered Extra 2 DEPENDENT SUBQUERY t2 ref a a 4 test.t1.b 1 100.00 Using where Warnings: Note 1276 Field or reference 'test.t3.oref' of SELECT #2 was resolved in SELECT #1 -Note 1003 /* select#1 */ select `test`.`t3`.`a` AS `a`,`test`.`t3`.`oref` AS `oref`,<expr_cache><`test`.`t3`.`a`,`test`.`t3`.`oref`>(<in_optimizer>(`test`.`t3`.`a`,<exists>(/* select#2 */ select `test`.`t1`.`a` from `test`.`t1` join `test`.`t2` where `test`.`t2`.`b` = `test`.`t3`.`oref` and trigcond(<cache>(`test`.`t3`.`a`) = `test`.`t1`.`a`) and `test`.`t2`.`a` = `test`.`t1`.`b`))) AS `Z` from `test`.`t3` +Note 1003 /* select#1 */ select `test`.`t3`.`a` AS `a`,`test`.`t3`.`oref` AS `oref`,<expr_cache><`test`.`t3`.`a`,`test`.`t3`.`oref`>(<in_optimizer>(`test`.`t3`.`a`,<exists>(/* select#2 */ select `test`.`t1`.`a` from `test`.`t1` join `test`.`t2` where `test`.`t2`.`a` = `test`.`t1`.`b` and `test`.`t2`.`b` = `test`.`t3`.`oref` and trigcond(<cache>(`test`.`t3`.`a`) = `test`.`t1`.`a`)))) AS `Z` from `test`.`t3` drop table t1,t2,t3; create table t1 (oref int, grp int); insert into t1 (oref, grp) values diff --git a/mysql-test/main/subselect3_jcl6.result b/mysql-test/main/subselect3_jcl6.result index 4260676cc37..b7b18bf80e0 100644 --- a/mysql-test/main/subselect3_jcl6.result +++ b/mysql-test/main/subselect3_jcl6.result @@ -172,7 +172,7 @@ id select_type table type possible_keys key key_len ref rows filtered Extra 2 DEPENDENT SUBQUERY t2 ref a a 5 test.t1.b 1 100.00 Using where; Using join buffer (flat, BKA join); Key-ordered Rowid-ordered scan Warnings: Note 1276 Field or reference 'test.t3.oref' of SELECT #2 was resolved in SELECT #1 -Note 1003 /* select#1 */ select `test`.`t3`.`a` AS `a`,`test`.`t3`.`oref` AS `oref`,<expr_cache><`test`.`t3`.`a`,`test`.`t3`.`oref`>(<in_optimizer>(`test`.`t3`.`a`,<exists>(/* select#2 */ select `test`.`t1`.`a` from `test`.`t1` join `test`.`t2` where `test`.`t2`.`b` = `test`.`t3`.`oref` and trigcond(<cache>(`test`.`t3`.`a`) = `test`.`t1`.`a` or `test`.`t1`.`a` is null) and `test`.`t2`.`a` = `test`.`t1`.`b` having trigcond(`test`.`t1`.`a` is null)))) AS `Z` from `test`.`t3` +Note 1003 /* select#1 */ select `test`.`t3`.`a` AS `a`,`test`.`t3`.`oref` AS `oref`,<expr_cache><`test`.`t3`.`a`,`test`.`t3`.`oref`>(<in_optimizer>(`test`.`t3`.`a`,<exists>(/* select#2 */ select `test`.`t1`.`a` from `test`.`t1` join `test`.`t2` where `test`.`t2`.`a` = `test`.`t1`.`b` and `test`.`t2`.`b` = `test`.`t3`.`oref` and trigcond(<cache>(`test`.`t3`.`a`) = `test`.`t1`.`a` or `test`.`t1`.`a` is null) having trigcond(`test`.`t1`.`a` is null)))) AS `Z` from `test`.`t3` drop table t1, t2, t3; create table t1 (a int NOT NULL, b int NOT NULL, key(a)); insert into t1 values @@ -200,7 +200,7 @@ id select_type table type possible_keys key key_len ref rows filtered Extra 2 DEPENDENT SUBQUERY t2 ref a a 4 test.t1.b 1 100.00 Using where; Using join buffer (flat, BKA join); Key-ordered Rowid-ordered scan Warnings: Note 1276 Field or reference 'test.t3.oref' of SELECT #2 was resolved in SELECT #1 -Note 1003 /* select#1 */ select `test`.`t3`.`a` AS `a`,`test`.`t3`.`oref` AS `oref`,<expr_cache><`test`.`t3`.`a`,`test`.`t3`.`oref`>(<in_optimizer>(`test`.`t3`.`a`,<exists>(/* select#2 */ select `test`.`t1`.`a` from `test`.`t1` join `test`.`t2` where `test`.`t2`.`b` = `test`.`t3`.`oref` and trigcond(<cache>(`test`.`t3`.`a`) = `test`.`t1`.`a`) and `test`.`t2`.`a` = `test`.`t1`.`b`))) AS `Z` from `test`.`t3` +Note 1003 /* select#1 */ select `test`.`t3`.`a` AS `a`,`test`.`t3`.`oref` AS `oref`,<expr_cache><`test`.`t3`.`a`,`test`.`t3`.`oref`>(<in_optimizer>(`test`.`t3`.`a`,<exists>(/* select#2 */ select `test`.`t1`.`a` from `test`.`t1` join `test`.`t2` where `test`.`t2`.`a` = `test`.`t1`.`b` and `test`.`t2`.`b` = `test`.`t3`.`oref` and trigcond(<cache>(`test`.`t3`.`a`) = `test`.`t1`.`a`)))) AS `Z` from `test`.`t3` drop table t1,t2,t3; create table t1 (oref int, grp int); insert into t1 (oref, grp) values diff --git a/mysql-test/main/subselect4.result b/mysql-test/main/subselect4.result index 3dc019cc6c1..e355cfc60f0 100644 --- a/mysql-test/main/subselect4.result +++ b/mysql-test/main/subselect4.result @@ -2349,7 +2349,7 @@ id select_type table type possible_keys key key_len ref rows filtered Extra 1 PRIMARY t2 system NULL NULL NULL NULL 1 100.00 2 DEPENDENT SUBQUERY t1 ALL NULL NULL NULL NULL 2 100.00 Using where Warnings: -Note 1003 /* select#1 */ select 3 AS `f` from dual where !<expr_cache><3>(<in_optimizer>(3,<exists>(/* select#2 */ select `test`.`t1`.`b` from `test`.`t1` where (`test`.`t1`.`c` = 'USA' or `test`.`t1`.`c` <> 'USA') and trigcond(<cache>(3) = `test`.`t1`.`b` or `test`.`t1`.`b` is null) and `test`.`t1`.`b` = `test`.`t1`.`a` having trigcond(`test`.`t1`.`b` is null)))) +Note 1003 /* select#1 */ select 3 AS `f` from dual where !<expr_cache><3>(<in_optimizer>(3,<exists>(/* select#2 */ select `test`.`t1`.`b` from `test`.`t1` where `test`.`t1`.`b` = `test`.`t1`.`a` and (`test`.`t1`.`c` = 'USA' or `test`.`t1`.`c` <> 'USA') and trigcond(<cache>(3) = `test`.`t1`.`b` or `test`.`t1`.`b` is null) having trigcond(`test`.`t1`.`b` is null)))) SELECT * FROM t2 WHERE f NOT IN (SELECT b FROM t1 WHERE 0 OR (c IN ('USA') OR c NOT IN ('USA')) AND a = b); @@ -2582,3 +2582,29 @@ id select_type table type possible_keys key key_len ref rows Extra 1 PRIMARY <subquery2> eq_ref distinct_key distinct_key 4 func 1 2 MATERIALIZED t2 ALL NULL NULL NULL NULL 100 drop table t0, t1, t2; +# +# MDEV-22377: Subquery in an UPDATE query uses full scan instead of range +# +create table t0(a int); +insert into t0 values (0),(1),(2),(3),(4),(5),(6),(7),(8),(9); +create table t1(nkey1 int, col1 int); +insert into t1 select A.a+10*B.a+100*C.a, A.a+10*B.a+100*C.a from t0 A, t0 B, t0 C; +create table t2(key1 int, col1 int, key(key1)); +insert into t2 select A.a+10*B.a, A.a+10*B.a from t0 A, t0 B; +alter table t0 add b int; +# This must use the join order of t0,t2,t1 +# And table t2 must use range access, not full scan: +explain +select * +from t0 +where +a not in (SELECT t1.col1 +FROM t1 JOIN t2 ON t2.key1 = t1.nkey1 +WHERE +t1.nkey1 IN (1,2,3,4,5) and t2.col1>t0.b +); +id select_type table type possible_keys key key_len ref rows Extra +1 PRIMARY t0 ALL NULL NULL NULL NULL 10 Using where +2 DEPENDENT SUBQUERY t2 range key1 key1 5 NULL 5 Using index condition; Using where +2 DEPENDENT SUBQUERY t1 ALL NULL NULL NULL NULL 1000 Using where; Using join buffer (flat, BNL join) +drop table t0, t1, t2; diff --git a/mysql-test/main/subselect4.test b/mysql-test/main/subselect4.test index bc8db41c505..e3d8b5be832 100644 --- a/mysql-test/main/subselect4.test +++ b/mysql-test/main/subselect4.test @@ -2115,6 +2115,32 @@ where exists (select * from t2 where t2.a=t1.a order by t2.b limit 2,3); explain select * from t1 where t1.a in (select t2.a from t2 order by t2.b); +drop table t0, t1, t2; + +--echo # +--echo # MDEV-22377: Subquery in an UPDATE query uses full scan instead of range +--echo # +create table t0(a int); +insert into t0 values (0),(1),(2),(3),(4),(5),(6),(7),(8),(9); +create table t1(nkey1 int, col1 int); +insert into t1 select A.a+10*B.a+100*C.a, A.a+10*B.a+100*C.a from t0 A, t0 B, t0 C; + +create table t2(key1 int, col1 int, key(key1)); +insert into t2 select A.a+10*B.a, A.a+10*B.a from t0 A, t0 B; + +alter table t0 add b int; +--echo # This must use the join order of t0,t2,t1 +--echo # And table t2 must use range access, not full scan: +explain +select * +from t0 +where + a not in (SELECT t1.col1 + FROM t1 JOIN t2 ON t2.key1 = t1.nkey1 + WHERE + t1.nkey1 IN (1,2,3,4,5) and t2.col1>t0.b + ); drop table t0, t1, t2; + diff --git a/mysql-test/main/subselect_mat_cost_bugs.result b/mysql-test/main/subselect_mat_cost_bugs.result index a18c5e608f1..755f1e2ad13 100644 --- a/mysql-test/main/subselect_mat_cost_bugs.result +++ b/mysql-test/main/subselect_mat_cost_bugs.result @@ -100,7 +100,7 @@ id select_type table type possible_keys key key_len ref rows filtered Extra 2 DEPENDENT SUBQUERY t2 index c3 c3 9 NULL 2 100.00 Using where; Using index; Using join buffer (flat, BNL join) Warnings: Note 1276 Field or reference 'test.t1.pk' of SELECT #2 was resolved in SELECT #1 -Note 1003 /* select#1 */ select `test`.`t1`.`pk` AS `pk` from `test`.`t1` where <expr_cache><`test`.`t1`.`c1`,`test`.`t1`.`pk`>(<in_optimizer>(`test`.`t1`.`c1`,<exists>(/* select#2 */ select `test`.`t1a`.`c1` from `test`.`t1b` join `test`.`t2` left join `test`.`t1a` on(`test`.`t1a`.`c2` = `test`.`t1b`.`pk` and 2) where `test`.`t1`.`pk` <> 0 and <cache>(`test`.`t1`.`c1`) = `test`.`t1a`.`c1` and `test`.`t2`.`c3` = `test`.`t1b`.`c4`))) +Note 1003 /* select#1 */ select `test`.`t1`.`pk` AS `pk` from `test`.`t1` where <expr_cache><`test`.`t1`.`c1`,`test`.`t1`.`pk`>(<in_optimizer>(`test`.`t1`.`c1`,<exists>(/* select#2 */ select `test`.`t1a`.`c1` from `test`.`t1b` join `test`.`t2` left join `test`.`t1a` on(`test`.`t1a`.`c2` = `test`.`t1b`.`pk` and 2) where `test`.`t2`.`c3` = `test`.`t1b`.`c4` and `test`.`t1`.`pk` <> 0 and <cache>(`test`.`t1`.`c1`) = `test`.`t1a`.`c1`))) SELECT pk FROM t1 WHERE c1 IN @@ -364,7 +364,7 @@ AND a = SOME (SELECT b FROM t5)); id select_type table type possible_keys key key_len ref rows Extra 1 PRIMARY t3 ALL NULL NULL NULL NULL 2 Using where 2 DEPENDENT SUBQUERY t5 index c c 10 NULL 2 Using where; Using index; Start temporary -2 DEPENDENT SUBQUERY t4 eq_ref PRIMARY PRIMARY 4 test.t5.b 1 Using index condition; Using where; End temporary +2 DEPENDENT SUBQUERY t4 eq_ref PRIMARY PRIMARY 4 test.t5.b 1 Using where; End temporary SELECT * FROM t3 WHERE t3.b > ALL ( diff --git a/sql/item_subselect.cc b/sql/item_subselect.cc index 736dfbd33b4..726c58e49a2 100644 --- a/sql/item_subselect.cc +++ b/sql/item_subselect.cc @@ -2704,9 +2704,28 @@ bool Item_in_subselect::inject_in_to_exists_cond(JOIN *join_arg) and_args= ((Item_cond*) join_arg->conds)->argument_list(); if (join_arg->cond_equal) and_args->disjoin((List<Item> *) &join_arg->cond_equal->current_level); + + /* + If subquery's WHERE is an Item_cond_and, add the injected conditions + into it manually. If we just called and_items(...) that could + potentially replace the top-level Item_cond_and and current top-level + Item_cond_and::m_cond_equal will be lost. + */ + if (where_item->type() == Item::COND_ITEM && + ((Item_cond*)where_item)->functype() == Item_func::COND_AND_FUNC) + { + List_iterator<Item> it(*((Item_cond*)where_item)->argument_list()); + while (Item *item=it++) + and_args->push_back(item); + } + else + and_args->push_back(where_item); + + where_item= join_arg->conds; } + else + where_item= and_items(thd, join_arg->conds, where_item); - where_item= and_items(thd, join_arg->conds, where_item); if (where_item->fix_fields_if_needed(thd, 0)) DBUG_RETURN(true); // TIMOUR TODO: call optimize_cond() for the new where clause @@ -2719,12 +2738,15 @@ bool Item_in_subselect::inject_in_to_exists_cond(JOIN *join_arg) { /* The argument list of the top-level AND may change after fix fields. */ and_args= ((Item_cond*) join_arg->conds)->argument_list(); - List_iterator<Item_equal> li(join_arg->cond_equal->current_level); - Item_equal *elem; - while ((elem= li++)) - { - and_args->push_back(elem, thd->mem_root); - } + + /* + Note that Item_equal objects must be present in this form: + Item_cond::conds list must have the end of the list to be the + same linked list as join_arg->cond_equal->current_level. + The code in substitute_for_best_equal_field depends on it (see the + disjoin() call). + */ + and_args->append((List<Item> *)&join_arg->cond_equal->current_level); } } BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] 96aee3229a2: MENT-703: Disable rpl.rpl_skip_replication, rpl.rpl_set_statement_default_master, rpl.rpl_extra_col_master_myisam, rpl.rpl_slave_load_tmpdir_not_exist tests in ES 10.2 and above
by sujatha 29 Apr '20

29 Apr '20

revision-id: 96aee3229a288dae52c457ca808966092da6d821 (mariadb-10.4.5-669-g96aee3229a2) parent(s): a284ea1909d66e4284a964a4ca7d45b75c450ed7 author: Sujatha committer: Sujatha timestamp: 2020-04-29 13:28:23 +0530 message: MENT-703: Disable rpl.rpl_skip_replication, rpl.rpl_set_statement_default_master, rpl.rpl_extra_col_master_myisam, rpl.rpl_slave_load_tmpdir_not_exist tests in ES 10.2 and above Problem: ======= Disable following tests in ES 10.2 and above, till MDEV-13258 and MDEV-14203 are fixed. rpl.rpl_skip_replication, rpl.rpl_set_statement_default_master, rpl.rpl_extra_col_master_myisam, rpl.rpl_slave_load_tmpdir_not_exist --- mysql-test/suite/rpl/disabled.def | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/mysql-test/suite/rpl/disabled.def b/mysql-test/suite/rpl/disabled.def index 77b01bd8d1d..5a04f4ccd15 100644 --- a/mysql-test/suite/rpl/disabled.def +++ b/mysql-test/suite/rpl/disabled.def @@ -19,3 +19,7 @@ rpl_parallel2 : fails after MDEV-16172 rpl_semi_sync_after_sync : fails after MDEV-16172 rpl_slave_grp_exec: MDEV-10514 rpl_auto_increment_update_failure : disabled for now +rpl_skip_replication : MDEV-13258 +rpl_set_statement_default_master : MDEV-13258 +rpl_extra_col_master_myisam : MDEV-14203 +rpl_slave_load_tmpdir_not_exist : MDEV-14203

1 0

[Maria-developers] 7f8dd7b2044: MENT-703: Disable rpl.rpl_skip_replication, rpl.rpl_set_statement_default_master, rpl.rpl_extra_col_master_myisam, rpl.rpl_slave_load_tmpdir_not_exist tests in ES 10.2 and above
by sujatha 29 Apr '20

29 Apr '20

revision-id: 7f8dd7b20443e0247ca478cbfd98c43749acdae0 (mariadb-10.2.24-722-g7f8dd7b2044) parent(s): 52cb9e87e1d23aeb0958966807050a6a843c181c author: Sujatha committer: Sujatha timestamp: 2020-04-29 11:33:41 +0530 message: MENT-703: Disable rpl.rpl_skip_replication, rpl.rpl_set_statement_default_master, rpl.rpl_extra_col_master_myisam, rpl.rpl_slave_load_tmpdir_not_exist tests in ES 10.2 and above Problem: ======= Disable following tests in ES 10.2 and above, till MDEV-13258 and MDEV-14203 are fixed. rpl.rpl_skip_replication, rpl.rpl_set_statement_default_master, rpl.rpl_extra_col_master_myisam, rpl.rpl_slave_load_tmpdir_not_exist --- mysql-test/suite/rpl/disabled.def | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/mysql-test/suite/rpl/disabled.def b/mysql-test/suite/rpl/disabled.def index bdefb1660bd..951f8a6aa0f 100644 --- a/mysql-test/suite/rpl/disabled.def +++ b/mysql-test/suite/rpl/disabled.def @@ -16,3 +16,7 @@ rpl_partition_archive : MDEV-5077 2013-09-27 svoj Cannot exchange partition rpl_row_binlog_max_cache_size : MDEV-11092 rpl_row_index_choice : MDEV-11666 rpl_slave_grp_exec: MDEV-10514 +rpl_skip_replication : MDEV-13258 +rpl_set_statement_default_master : MDEV-13258 +rpl_extra_col_master_myisam : MDEV-14203 +rpl_slave_load_tmpdir_not_exist : MDEV-14203

1 0

Re: [Maria-developers] [Commits] 85b116d1ab9: MDEV-22317: SIGSEGV in my_free/delete_dynamic in optimized builds (ARIA)
by sujatha 28 Apr '20

28 Apr '20

Hello Andrei, Good morning. Thank you for the review comments. Please find my replies inline. On 27/04/20 11:26 pm, Andrei Elkin wrote: > Sujatha, howdy. > > Thanks for a prompt checking the case! > The patch looks good, only > >> - if (!wild_do_table.elements) >> + if (status && wild_do_table_inited) > I would suggest to reorder the && arg:s to be then > in temporal wrt their evaluation order. Sure Andrei. I have refined changes as shown below. diff --git a/sql/rpl_filter.cc b/sql/rpl_filter.cc index 635a0f4e2d6..8035763bf35 100644 --- a/sql/rpl_filter.cc +++ b/sql/rpl_filter.cc @@ -416,10 +416,13 @@ Rpl_filter::set_wild_do_table(const char* table_spec) status= parse_filter_rule(table_spec, &Rpl_filter::add_wild_do_table); - if (!wild_do_table.elements) + if (wild_do_table_inited && status) { - delete_dynamic(&wild_do_table); - wild_do_table_inited= 0; + if (!wild_do_table.elements) + { + delete_dynamic(&wild_do_table); + wild_do_table_inited= 0; + } } > >> revision-id: 85b116d1ab9ea85dcef63d259b8f6366466e2750 (mariadb-10.5.2-185-g85b116d1ab9) >> parent(s): fbe2712705d464bf8488df249c36115e2c1f63f7 >> author: Sujatha >> committer: Sujatha >> timestamp: 2020-04-27 17:43:51 +0530 >> message: >> >> MDEV-22317: SIGSEGV in my_free/delete_dynamic in optimized builds (ARIA) >> >> Problem: >> ======= >> SET @@GLOBAL.replicate_wild_ignore_table=''; >> SET @@GLOBAL.replicate_wild_do_table=''; >> >> Reports following valgrind error. >> >> Conditional jump or move depends on uninitialised value(s) >> Rpl_filter::set_wild_ignore_table(char const*) (rpl_filter.cc:439) >> >> Conditional jump or move depends on uninitialised value(s) >> at 0xF60390: delete_dynamic (array.c:304) >> by 0x74F3F2: Rpl_filter::set_wild_do_table(char const*) (rpl_filter.cc:421) >> >> Analysis: >> ======== >> List of values provided for options "wild_do_table" and "wild_ignore_table" are >> stored in DYNAMIC_ARRAYS. When an empty list is provided these dynamic arrays >> are not initialized. > Correct. And that's 'cos > > Rpl_filter::Rpl_filter() : > parallel_mode(SLAVE_PARALLEL_CONSERVATIVE), > table_rules_on(0), > do_table_inited(0), ignore_table_inited(0), > wild_do_table_inited(0), wild_ignore_table_inited(0) > { > do_db.empty(); > ignore_db.empty(); > rewrite_db.empty(); > } > > does not initialize the two members of your patch's interest with > empty()-ing. > I wonder if > > + wild_do_table.empty() > + wild_ignore_table.empty() > ? > > > that could work out as well which looks as more consistent solution to > me. [It feels like binlog_filter object might object this idea though]. 'do_db' and 'ignore_db' are of type list iterators. Hence they have 'empty' defined for them. 'empty' is not a member of DYNAMIC_ARRAY. I_List<i_string> do_db; I_List<i_string> ignore_db; /home/sujatha/bug_repo/MDEV-22317-10.5/sql/rpl_filter.cc:36:17: error: ‘DYNAMIC_ARRAY {aka struct st_dynamic_array}’ has no member named ‘empty’ wild_do_table.empty(); ^~~~~ /home/sujatha/bug_repo/MDEV-22317-10.5/sql/rpl_filter.cc:37:21: error: ‘DYNAMIC_ARRAY {aka struct st_dynamic_array}’ has no member named ‘empty’ wild_ignore_table.empty(); ^~~~~ "wild_do_table" and "wild_ignore_table" arrays get initialized as part of "add_wild_do_table" and "add_wild_ignore_table". The initialization is done through following calls. status= parse_filter_rule(table_spec, &Rpl_filter::add_wild_do_table); status= parse_filter_rule(table_spec, &Rpl_filter::add_wild_ignore_table); When default empty string is provided for replicate_wild_ignore_table='' and replicate_wild_do_table='' the dynamic array is not initialized. 'wild_do_table.elements' will be '0'. No cleanup is required in this case. Hence we should do the cleanup only when 'wild_do_table_inited' is true. Please let me know your thoughts. Thank you S.Sujatha > > Cheers, > > Andrei > > >> Existing code treats empty element list as an error and >> tries to clean the uninitialized list. This results in above valgrind issue. >> >> Fix: >> === >> The clean up should be initiated only when there is an error while parsing the >> 'wild_do_table' or 'wild_ignore_table' list and the dynamic_array is in >> initialized state. Otherwise for empty list it should simply return success. >> >> --- >> mysql-test/suite/rpl/r/rpl_filter_wild_tables_dynamic.result | 2 ++ >> mysql-test/suite/rpl/t/rpl_filter_wild_tables_dynamic.test | 2 ++ >> sql/rpl_filter.cc | 4 ++-- >> 3 files changed, 6 insertions(+), 2 deletions(-) >> >> diff --git a/mysql-test/suite/rpl/r/rpl_filter_wild_tables_dynamic.result b/mysql-test/suite/rpl/r/rpl_filter_wild_tables_dynamic.result >> index 47cd362a549..fe0b283fc7c 100644 >> --- a/mysql-test/suite/rpl/r/rpl_filter_wild_tables_dynamic.result >> +++ b/mysql-test/suite/rpl/r/rpl_filter_wild_tables_dynamic.result >> @@ -7,6 +7,8 @@ SET @@GLOBAL.replicate_wild_ignore_table="test.b%"; >> ERROR HY000: This operation cannot be performed as you have a running slave ''; run STOP SLAVE '' first >> connection slave; >> include/stop_slave.inc >> +SET @@GLOBAL.replicate_wild_do_table=""; >> +SET @@GLOBAL.replicate_wild_ignore_table=""; >> SET @@GLOBAL.replicate_wild_do_table="test.a%"; >> SET @@GLOBAL.replicate_wild_ignore_table="test.b%"; >> include/start_slave.inc >> diff --git a/mysql-test/suite/rpl/t/rpl_filter_wild_tables_dynamic.test b/mysql-test/suite/rpl/t/rpl_filter_wild_tables_dynamic.test >> index 6db61927eec..657a95cec15 100644 >> --- a/mysql-test/suite/rpl/t/rpl_filter_wild_tables_dynamic.test >> +++ b/mysql-test/suite/rpl/t/rpl_filter_wild_tables_dynamic.test >> @@ -13,6 +13,8 @@ SET @@GLOBAL.replicate_wild_ignore_table="test.b%"; >> >> connection slave; >> source include/stop_slave.inc; >> +SET @@GLOBAL.replicate_wild_do_table=""; >> +SET @@GLOBAL.replicate_wild_ignore_table=""; >> SET @@GLOBAL.replicate_wild_do_table="test.a%"; >> SET @@GLOBAL.replicate_wild_ignore_table="test.b%"; >> source include/start_slave.inc; >> diff --git a/sql/rpl_filter.cc b/sql/rpl_filter.cc >> index 635a0f4e2d6..49b498d3568 100644 >> --- a/sql/rpl_filter.cc >> +++ b/sql/rpl_filter.cc >> @@ -416,7 +416,7 @@ Rpl_filter::set_wild_do_table(const char* table_spec) >> >> status= parse_filter_rule(table_spec, &Rpl_filter::add_wild_do_table); >> >> - if (!wild_do_table.elements) >> + if (status && wild_do_table_inited) >> { >> delete_dynamic(&wild_do_table); >> wild_do_table_inited= 0; >> @@ -436,7 +436,7 @@ Rpl_filter::set_wild_ignore_table(const char* table_spec) >> >> status= parse_filter_rule(table_spec, &Rpl_filter::add_wild_ignore_table); >> >> - if (!wild_ignore_table.elements) >> + if (status && wild_ignore_table_inited) >> { >> delete_dynamic(&wild_ignore_table); >> wild_ignore_table_inited= 0; >> _______________________________________________ >> commits mailing list >> commits(a)mariadb.org >> https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

[Maria-developers] 85b116d1ab9: MDEV-22317: SIGSEGV in my_free/delete_dynamic in optimized builds (ARIA)
by sujatha 27 Apr '20

27 Apr '20

revision-id: 85b116d1ab9ea85dcef63d259b8f6366466e2750 (mariadb-10.5.2-185-g85b116d1ab9) parent(s): fbe2712705d464bf8488df249c36115e2c1f63f7 author: Sujatha committer: Sujatha timestamp: 2020-04-27 17:43:51 +0530 message: MDEV-22317: SIGSEGV in my_free/delete_dynamic in optimized builds (ARIA) Problem: ======= SET @@GLOBAL.replicate_wild_ignore_table=''; SET @@GLOBAL.replicate_wild_do_table=''; Reports following valgrind error. Conditional jump or move depends on uninitialised value(s) Rpl_filter::set_wild_ignore_table(char const*) (rpl_filter.cc:439) Conditional jump or move depends on uninitialised value(s) at 0xF60390: delete_dynamic (array.c:304) by 0x74F3F2: Rpl_filter::set_wild_do_table(char const*) (rpl_filter.cc:421) Analysis: ======== List of values provided for options "wild_do_table" and "wild_ignore_table" are stored in DYNAMIC_ARRAYS. When an empty list is provided these dynamic arrays are not initialized. Existing code treats empty element list as an error and tries to clean the uninitialized list. This results in above valgrind issue. Fix: === The clean up should be initiated only when there is an error while parsing the 'wild_do_table' or 'wild_ignore_table' list and the dynamic_array is in initialized state. Otherwise for empty list it should simply return success. --- mysql-test/suite/rpl/r/rpl_filter_wild_tables_dynamic.result | 2 ++ mysql-test/suite/rpl/t/rpl_filter_wild_tables_dynamic.test | 2 ++ sql/rpl_filter.cc | 4 ++-- 3 files changed, 6 insertions(+), 2 deletions(-) diff --git a/mysql-test/suite/rpl/r/rpl_filter_wild_tables_dynamic.result b/mysql-test/suite/rpl/r/rpl_filter_wild_tables_dynamic.result index 47cd362a549..fe0b283fc7c 100644 --- a/mysql-test/suite/rpl/r/rpl_filter_wild_tables_dynamic.result +++ b/mysql-test/suite/rpl/r/rpl_filter_wild_tables_dynamic.result @@ -7,6 +7,8 @@ SET @@GLOBAL.replicate_wild_ignore_table="test.b%"; ERROR HY000: This operation cannot be performed as you have a running slave ''; run STOP SLAVE '' first connection slave; include/stop_slave.inc +SET @@GLOBAL.replicate_wild_do_table=""; +SET @@GLOBAL.replicate_wild_ignore_table=""; SET @@GLOBAL.replicate_wild_do_table="test.a%"; SET @@GLOBAL.replicate_wild_ignore_table="test.b%"; include/start_slave.inc diff --git a/mysql-test/suite/rpl/t/rpl_filter_wild_tables_dynamic.test b/mysql-test/suite/rpl/t/rpl_filter_wild_tables_dynamic.test index 6db61927eec..657a95cec15 100644 --- a/mysql-test/suite/rpl/t/rpl_filter_wild_tables_dynamic.test +++ b/mysql-test/suite/rpl/t/rpl_filter_wild_tables_dynamic.test @@ -13,6 +13,8 @@ SET @@GLOBAL.replicate_wild_ignore_table="test.b%"; connection slave; source include/stop_slave.inc; +SET @@GLOBAL.replicate_wild_do_table=""; +SET @@GLOBAL.replicate_wild_ignore_table=""; SET @@GLOBAL.replicate_wild_do_table="test.a%"; SET @@GLOBAL.replicate_wild_ignore_table="test.b%"; source include/start_slave.inc; diff --git a/sql/rpl_filter.cc b/sql/rpl_filter.cc index 635a0f4e2d6..49b498d3568 100644 --- a/sql/rpl_filter.cc +++ b/sql/rpl_filter.cc @@ -416,7 +416,7 @@ Rpl_filter::set_wild_do_table(const char* table_spec) status= parse_filter_rule(table_spec, &Rpl_filter::add_wild_do_table); - if (!wild_do_table.elements) + if (status && wild_do_table_inited) { delete_dynamic(&wild_do_table); wild_do_table_inited= 0; @@ -436,7 +436,7 @@ Rpl_filter::set_wild_ignore_table(const char* table_spec) status= parse_filter_rule(table_spec, &Rpl_filter::add_wild_ignore_table); - if (!wild_ignore_table.elements) + if (status && wild_ignore_table_inited) { delete_dynamic(&wild_ignore_table); wild_ignore_table_inited= 0;

1 0

Re: [Maria-developers] 8a990ad1774: MDEV-18319 BIGINT UNSIGNED Performance issue
by Sergei Golubchik 27 Apr '20

27 Apr '20

Hi, Alexander! On Apr 26, Alexander Barkov wrote: > revision-id: 8a990ad1774 > author: Alexander Barkov <bar(a)mariadb.com> > committer: Alexander Barkov <bar(a)mariadb.com> > timestamp: 2019-03-25 19:19:48 +0400 > message: MDEV-18319 BIGINT UNSIGNED Performance issue > > diff --git a/mysql-test/main/errors.result b/mysql-test/main/errors.result > index ba05a2b37d45..96ad96395aab 100644 > --- a/mysql-test/main/errors.result > +++ b/mysql-test/main/errors.result > @@ -32,16 +32,21 @@ set sql_mode=default; > CREATE TABLE t1 (a INT); > SELECT a FROM t1 WHERE a IN(1, (SELECT IF(1=0,1,2/0))); > a > +Warnings: > +Warning 1365 Division by 0 Interesting. There was no warning before. > INSERT INTO t1 VALUES(1); > SELECT a FROM t1 WHERE a IN(1, (SELECT IF(1=0,1,2/0))); > a > 1 > diff --git a/mysql-test/main/func_in.result b/mysql-test/main/func_in.result > index 65313148bf80..59822ae1049c 100644 > --- a/mysql-test/main/func_in.result > +++ b/mysql-test/main/func_in.result > @@ -489,6 +489,7 @@ SELECT id FROM t1 WHERE id IN(4564, (SELECT IF(1=0,1,1/0)) ); > id > Warnings: > Warning 1365 Division by 0 > +Warning 1365 Division by 0 This is not very nice. May be we should use Item_cache in these cases? > DROP TABLE t1; > End of 5.0 tests > create table t1(f1 char(1)); > @@ -909,3 +910,71 @@ Warnings: > Warning 1292 Truncated incorrect time value: '' > Warning 1292 Truncated incorrect time value: '' > Warning 1292 Truncated incorrect time value: '' > +# > +# MDEV-18319 BIGINT UNSIGNED Performance issue > +# > +CREATE TABLE t1 ( > +id bigint(20) unsigned NOT NULL AUTO_INCREMENT PRIMARY KEY > +); > +FOR i IN 0..255 > +DO > +INSERT INTO t1 VALUES (); > +END FOR > +$$ > +SELECT MIN(id), MAX(id), COUNT(*) FROM t1; > +MIN(id) MAX(id) COUNT(*) > +1 256 256 interesting. why it's min=1 and max=255, if your loop is in 0..255? > +EXPLAIN SELECT id FROM t1 WHERE id IN (1,2); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 range PRIMARY PRIMARY 8 NULL 2 Using where; Using index > +EXPLAIN SELECT id FROM t1 WHERE id IN (9223372036854775806, 9223372036854775807); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 range PRIMARY PRIMARY 8 NULL 2 Using where; Using index > +EXPLAIN SELECT id FROM t1 WHERE id IN (9223372036854775807, 9223372036854775808); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 range PRIMARY PRIMARY 8 NULL 2 Using where; Using index > +EXPLAIN SELECT id FROM t1 WHERE id IN (1.0,2.0); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 range PRIMARY PRIMARY 8 NULL 2 Using where; Using index > +EXPLAIN SELECT id FROM t1 WHERE id IN (9223372036854775806.0, 9223372036854775807.0); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 range PRIMARY PRIMARY 8 NULL 2 Using where; Using index > +# Cannot compare this as INT (yet) > +EXPLAIN SELECT id FROM t1 WHERE id IN (9223372036854775807.0, 9223372036854775808.0); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 index PRIMARY PRIMARY 8 NULL 256 Using where; Using index > +DROP TABLE t1; > +CREATE TABLE t1 ( > +id bigint(20) NOT NULL AUTO_INCREMENT PRIMARY KEY > +); > +FOR i IN 0..255 > +DO > +INSERT INTO t1 VALUES (); > +END FOR > +$$ > +SELECT MIN(id), MAX(id), COUNT(*) FROM t1; > +MIN(id) MAX(id) COUNT(*) > +1 256 256 > +EXPLAIN SELECT id FROM t1 WHERE id IN (-1,-2); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 range PRIMARY PRIMARY 8 NULL 2 Using where; Using index > +EXPLAIN SELECT id FROM t1 WHERE id IN (-9223372036854775806, -9223372036854775807); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 range PRIMARY PRIMARY 8 NULL 2 Using where; Using index > +EXPLAIN SELECT id FROM t1 WHERE id IN (-9223372036854775807, -9223372036854775808); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 range PRIMARY PRIMARY 8 NULL 2 Using where; Using index > +EXPLAIN SELECT id FROM t1 WHERE id IN (-1.0,-2.0); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 range PRIMARY PRIMARY 8 NULL 2 Using where; Using index > +EXPLAIN SELECT id FROM t1 WHERE id IN (-9223372036854775806.0, -9223372036854775807.0); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 range PRIMARY PRIMARY 8 NULL 2 Using where; Using index > +EXPLAIN SELECT id FROM t1 WHERE id IN (-9223372036854775807.0, -9223372036854775808.0); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 range PRIMARY PRIMARY 8 NULL 2 Using where; Using index > +# Cannot compare this as INT (yet) > +EXPLAIN SELECT id FROM t1 WHERE id IN (-9223372036854775808.0, -9223372036854775809.0); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 index PRIMARY PRIMARY 8 NULL 256 Using where; Using index > +DROP TABLE t1; > diff --git a/mysql-test/main/func_in.test b/mysql-test/main/func_in.test > index b99fad159c22..04b0b328c27b 100644 > --- a/mysql-test/main/func_in.test > +++ b/mysql-test/main/func_in.test > @@ -690,3 +690,51 @@ SELECT > TIME'00:00:00'='' AS c1_true, > TIME'00:00:00' IN ('', TIME'10:20:30') AS c2_true, > TIME'00:00:00' NOT IN ('', TIME'10:20:30') AS c3_false; > + > +--echo # > +--echo # MDEV-18319 BIGINT UNSIGNED Performance issue > +--echo # > + > +CREATE TABLE t1 ( > + id bigint(20) unsigned NOT NULL AUTO_INCREMENT PRIMARY KEY > +); > +DELIMITER $$; > +FOR i IN 0..255 > +DO > + INSERT INTO t1 VALUES (); > +END FOR > +$$ > +DELIMITER ;$$ A couple of one-liners instead: insert t1 select seq from seq_1_to_256; or, in pure SQL insert t1 with recursive g(i) as (select 1 union select i+1 from g where i < 257) select * from g; > +SELECT MIN(id), MAX(id), COUNT(*) FROM t1; > +EXPLAIN SELECT id FROM t1 WHERE id IN (1,2); > +EXPLAIN SELECT id FROM t1 WHERE id IN (9223372036854775806, 9223372036854775807); > +EXPLAIN SELECT id FROM t1 WHERE id IN (9223372036854775807, 9223372036854775808); > + > +EXPLAIN SELECT id FROM t1 WHERE id IN (1.0,2.0); > +EXPLAIN SELECT id FROM t1 WHERE id IN (9223372036854775806.0, 9223372036854775807.0); > +--echo # Cannot compare this as INT (yet) > +EXPLAIN SELECT id FROM t1 WHERE id IN (9223372036854775807.0, 9223372036854775808.0); > +DROP TABLE t1; > + > + > +CREATE TABLE t1 ( > + id bigint(20) NOT NULL AUTO_INCREMENT PRIMARY KEY > +); > +DELIMITER $$; > +FOR i IN 0..255 > +DO > + INSERT INTO t1 VALUES (); > +END FOR > +$$ > +DELIMITER ;$$ > +SELECT MIN(id), MAX(id), COUNT(*) FROM t1; > +EXPLAIN SELECT id FROM t1 WHERE id IN (-1,-2); > +EXPLAIN SELECT id FROM t1 WHERE id IN (-9223372036854775806, -9223372036854775807); > +EXPLAIN SELECT id FROM t1 WHERE id IN (-9223372036854775807, -9223372036854775808); > + > +EXPLAIN SELECT id FROM t1 WHERE id IN (-1.0,-2.0); > +EXPLAIN SELECT id FROM t1 WHERE id IN (-9223372036854775806.0, -9223372036854775807.0); > +EXPLAIN SELECT id FROM t1 WHERE id IN (-9223372036854775807.0, -9223372036854775808.0); > +--echo # Cannot compare this as INT (yet) > +EXPLAIN SELECT id FROM t1 WHERE id IN (-9223372036854775808.0, -9223372036854775809.0); > +DROP TABLE t1; > diff --git a/sql/item.h b/sql/item.h > index 883cc791f384..9f215c74864b 100644 > --- a/sql/item.h > +++ b/sql/item.h > @@ -70,6 +70,11 @@ class Value: public st_value > bool is_temporal() const { return m_type == DYN_COL_DATETIME; } > bool is_string() const { return m_type == DYN_COL_STRING; } > bool is_decimal() const { return m_type == DYN_COL_DECIMAL; } > + Longlong_hybrid to_longlong_hybrid_native() const > + { > + DBUG_ASSERT(is_longlong()); > + return Longlong_hybrid(value.m_longlong, m_type == DYN_COL_UINT); > + } > }; > > > diff --git a/sql/item_cmpfunc.cc b/sql/item_cmpfunc.cc > index b2fa753f2bd2..c7176df43d6a 100644 > --- a/sql/item_cmpfunc.cc > +++ b/sql/item_cmpfunc.cc > @@ -3747,11 +3748,45 @@ bool Predicant_to_list_comparator::add_value(const char *funcname, > Item *tmpargs[2]; > tmpargs[0]= args->arguments()[m_predicant_index]; > tmpargs[1]= args->arguments()[value_index]; > - if (tmp.aggregate_for_comparison(funcname, tmpargs, 2, true)) > + if (tmp.aggregate_for_comparison(funcname, tmpargs, 2, false)) How would that work? Are you going to compare signed to unsigned as longlongs? > { > DBUG_ASSERT(current_thd->is_error()); > return true; > } > + /* > + Try to compare using type handler of the predicant when possible, > + as this can use indexes for conditions like: > + WHERE field IN (const1, .. constN) > + */ > + if (prefer_predicant_type_handler && > + args->arguments()[value_index]->const_item() && > + !args->arguments()[value_index]->is_expensive() && > + tmp.type_handler()->cmp_type() == DECIMAL_RESULT && > + args->arguments()[m_predicant_index]->cmp_type() == INT_RESULT) > + { > + /* > + For now we catch only one special case: > + an INT predicant can be compared to a DECIMAL constant > + using Longlong_hybrid comparison, when the DECIMAL constant: > + a. has no significant fractional digits, and > + b. is within the signed longlong range > + For DECIMAL constants in the range LONGLONG_MAX..ULONGLONG_MAX > + we cannot call to_longlong_hybrid() safely, because DECIMAL type > + Items usually set their "unsigned_flag" to "false", so val_int() > + will truncate such constants to LONGLONG_MAX (instead of ULONGLONG_MAX). > + TODO: fix to_longlong_hybrid() for DECIMAL LONGLONG_MAX..ULONGLONG_MAX > + TODO: move this code to Type_handler (will be done by MDEV-18898) > + TODO: skip constants outside of LONGLONG_MIN..ULONGLONG_MAX, as > + such conditon can never be true, e.g.: > + WHERE int_expr IN (.. -9223372036854775809 ..) > + WHERE int_expr IN (.. 18446744073709551616 ..) > + */ > + my_decimal buf, *dec= args->arguments()[value_index]->val_decimal(&buf); > + longlong res; > + if (dec && decimal_actual_fraction(dec) == 0 && > + my_decimal2int(0, dec, false /*SIGNED*/, &res) == 0) > + tmp.set_handler(&type_handler_longlong); why not to replace the item with Item_int or Item_cache_int here? Then you'll be able to handle decimals up to ULONGLONG_MAX, that's your first TODO. For the last TODO, you can simply remove the always-false element from the arguments[] array. > + } > m_comparators[m_comparator_count].m_handler= tmp.type_handler(); > m_comparators[m_comparator_count].m_arg_index= value_index; > m_comparator_count++; > diff --git a/sql/item_cmpfunc.h b/sql/item_cmpfunc.h > index 06f15503258a..4e84c3008578 100644 > --- a/sql/item_cmpfunc.h > +++ b/sql/item_cmpfunc.h > @@ -1588,29 +1588,28 @@ class cmp_item_sort_string :public cmp_item_string > > class cmp_item_int : public cmp_item_scalar > { > - longlong value; > + Longlong_hybrid m_value; > public: > - cmp_item_int() {} /* Remove gcc warning */ > + cmp_item_int(): m_value(0, false) {} > void store_value(Item *item) > { > - value= item->val_int(); > + m_value= item->to_longlong_hybrid(); I wonder if to_longlong_hybrid() returns a Longlong_hybrid like that, where the compiler will allocate it? Could you check it, please? > m_null_value= item->null_value; > } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 00f9df29224: MDEV-20787: Script dgcov.pl does not work
by Sergei Golubchik 23 Apr '20

23 Apr '20

Hi, Anel! On Apr 23, Anel Husakovic wrote: > revision-id: 00f9df29224 (mariadb-10.2.29-23-g00f9df29224) > parent(s): 6718d3bc324 > author: Anel Husakovic <anel(a)mariadb.org> > committer: Anel Husakovic <anel(a)mariadb.org> > timestamp: 2019-11-19 15:00:48 +0100 > message: > > MDEV-20787: Script dgcov.pl does not work > > Let's change CMakeList with `--coverage` flag as an alias for > `-fprofile-arcs -ftest-coverage -lgcov` in addition. > When the server is compiled with `-DENABLE_GCOV=ON`, from object files are generated > `.gcno` and `.gcda` files. > `./mtr --gcov is_check_constraint` is invoking the script calls > `./dgcov.pl --purge`, `./mtr is_check_constraint`, > `./dgcov.pl --generate>/var/last_changes.dgcov`. > The `purge` flag is clearing `.gcda` files (and others extensions), > while running the test new `.gcda` files are obtained. > With `generate` flag, `gcov -i` (`intermediate format`) is called > on obtained `<object-files-name>.gcda` files (`dbug.c.gcda` e.g.). > The patch is tested on `gcov 6.3` and `gcov 7.4` versions > and can be seen that resulting `.gcov` file for `6.3` creates > `<full path>.gcov` (`dbug.c.gcda.gcov` e.g.) file, > where `gcov 7.4` is still creating `object-file-names.gcov`(`dbug.c.gcov`) files > as `gcov` in general is doing. > > diff --git a/CMakeLists.txt b/CMakeLists.txt > index 0dcc2a75587..af025a0312f 100644 > --- a/CMakeLists.txt > +++ b/CMakeLists.txt > @@ -249,7 +249,7 @@ SET(CMAKE_C_FLAGS_DEBUG "${CMAKE_C_FLAGS_DEBUG} -DENABLED_DEBUG_SYNC") > > OPTION(ENABLE_GCOV "Enable gcov (debug, Linux builds only)" OFF) > IF (ENABLE_GCOV) > - MY_CHECK_AND_SET_COMPILER_FLAG("-fprofile-arcs -ftest-coverage -lgcov" DEBUG) > + MY_CHECK_AND_SET_COMPILER_FLAG("--coverage" DEBUG) Why? Since what version does gcc support it? > ENDIF() > > MY_CHECK_AND_SET_COMPILER_FLAG(-ggdb3 DEBUG) > diff --git a/mysql-test/dgcov.pl b/mysql-test/dgcov.pl > index fbc5540e697..47ffaca04ef 100755 > --- a/mysql-test/dgcov.pl > +++ b/mysql-test/dgcov.pl > @@ -68,8 +68,11 @@ if ($opt_purge) > system($cmd)==0 or die "system($cmd): $? $!"; > exit 0; > } > - > +my $gcov_vers= `gcov -v`; > +$gcov_vers=~ s/\D//g; > +$gcov_vers= substr($gcov_vers, 0, 1); my gcov is 9.3.0. gcc version 10 is already out. You get just the first digit, 10 will become 1. Better to something like $gcov_vers ~= s/^.*(\d+\.\d+\.\d+).*$/\1/s; > find(\&gcov_one_file, $root); > +undef $gcov_vers; why? > find(\&write_coverage, $root) if $opt_generate; > exit 0 if $opt_only_gcov; > > @@ -162,7 +165,16 @@ sub gcov_one_file { > } > > # now, read the generated file > - open FH, '<', "$_.gcov" or die "open(<$_.gcov): $!"; > + if($gcov_vers<7) > + { > + open FH, '<', "$_.gcov" or die "open(<$_.gcov): $!"; > + } > + else > + { > + my $f=substr $_, 0, -5; > + open FH, '<', "$f.gcov" or die "open(<$f.gcov): $!"; > + undef $f; > + } I'd rather write it more compact, in two lines like my $f = $gcov_vers < 7 ? $_ : substr $_, 0, -5; open FH, '<', "$f.gcov" or die "open(<$f.gcov): $!"; > my $fname; > while (<FH>) { > chomp; > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] fb963a7fd3b: MDEV-21117: --tc-heuristic-recover=rollback is not replication safe
by sujatha 21 Apr '20

21 Apr '20

revision-id: fb963a7fd3b3e0733fb7ad9ece6256aebd8c176c (mariadb-10.1.43-112-gfb963a7fd3b) parent(s): ad4b70562bb94dd063eebde5189c6e730d3120a2 author: Sujatha committer: Sujatha timestamp: 2020-04-21 18:19:34 +0530 message: MDEV-21117: --tc-heuristic-recover=rollback is not replication safe Problem: ======= When run after master server crash --tc-heuristic-recover=rollback produces inconsistent server state with binlog still containing transactions that were rolled back by the option. Such way recovered server may not be used for replication. Fix: === During "--tc-heuristic-recover=ROLLBACK", query the storage engine to get binlog file name and position corresponding to the last committed transaction. This marks the consistent binlog state. If last_commit_file is not set then checkpoint binary log file is considered as starting point. Look for first transactional event beyond the consistent binlog state. This will be the starting point for heuristic rollback. Consider this event specific starting point as binlog truncation position. Now traverse the rest of binlogs beyond this point. During this traversal check for the presence of DDL or non transactional operations, as they cannot be safely rolled back. If such events are found the truncation will not happen it will return an error. If only transactional events are found beyond the binlog truncation position it is safe to truncate binlog. The log gets truncated to the identified position and the GTID state is adjusted accordingly. If there are more binary logs beyond the being truncated file they are all removed. If recovery is initiated based on last committed transaction specific binary log name and position, and there is failure during recovery, then recovery may be retried by remove the source of the failure. i.e --tc-heuristic-recover=rollback may be retried. Retry will not be successful if recovery is based on checkpoint file. As at the end of tc-heuristic-recover a new binary log file is created which will change the checkpoint file. Appropriate error message is written to error log for each case. --- .../r/binlog_heuristic_rollback_active_log.result | 18 + .../binlog/r/binlog_truncate_multi_log.result | 33 ++ .../r/binlog_truncate_multi_log_unsafe.result | 30 ++ .../suite/binlog/r/binlog_truncate_none.result | 42 ++ .../binlog/r/binlog_truncate_retry_success.result | 28 + .../t/binlog_heuristic_rollback_active_log.test | 75 +++ .../suite/binlog/t/binlog_truncate_multi_log.test | 87 +++ .../binlog/t/binlog_truncate_multi_log_unsafe.test | 106 ++++ .../suite/binlog/t/binlog_truncate_none.test | 130 +++++ .../binlog/t/binlog_truncate_retry_success.test | 84 +++ .../suite/rpl/r/rpl_heuristic_fail_over.result | 53 ++ .../suite/rpl/t/rpl_heuristic_fail_over.test | 160 ++++++ sql/log.cc | 589 ++++++++++++++++++++- sql/log.h | 7 +- sql/mysqld.h | 3 +- storage/innobase/handler/ha_innodb.cc | 7 + storage/innobase/log/log0recv.cc | 5 +- storage/xtradb/handler/ha_innodb.cc | 7 + storage/xtradb/log/log0recv.cc | 5 +- 19 files changed, 1457 insertions(+), 12 deletions(-) diff --git a/mysql-test/suite/binlog/r/binlog_heuristic_rollback_active_log.result b/mysql-test/suite/binlog/r/binlog_heuristic_rollback_active_log.result new file mode 100644 index 00000000000..eff95d05aac --- /dev/null +++ b/mysql-test/suite/binlog/r/binlog_heuristic_rollback_active_log.result @@ -0,0 +1,18 @@ +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); +RESET MASTER; +CREATE TABLE t ( f INT ) ENGINE=INNODB; +INSERT INTO t VALUES (10); +SET DEBUG_SYNC= "commit_before_update_end_pos SIGNAL con1_ready WAIT_FOR con1_go"; +INSERT INTO t VALUES (20); +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; +# Kill the server +"One row should be present in table 't'" +SELECT COUNT(*) FROM t; +COUNT(*) +1 +"gtid_binlog_state should be 0-1-2 +SELECT @@GLOBAL.gtid_binlog_state; +@@GLOBAL.gtid_binlog_state +0-1-2 +DROP TABLE t; diff --git a/mysql-test/suite/binlog/r/binlog_truncate_multi_log.result b/mysql-test/suite/binlog/r/binlog_truncate_multi_log.result new file mode 100644 index 00000000000..1a3d49b5463 --- /dev/null +++ b/mysql-test/suite/binlog/r/binlog_truncate_multi_log.result @@ -0,0 +1,33 @@ +SET @old_max_binlog_size= @@GLOBAL.max_binlog_size; +SET GLOBAL max_binlog_size= 4096; +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); +RESET MASTER; +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +FLUSH LOGS; +"List of binary logs before rotation" +show binary logs; +Log_name File_size +master-bin.000001 # +master-bin.000002 # +SET DEBUG_SYNC= "commit_after_release_LOCK_log SIGNAL con1_ready WAIT_FOR con1_go"; +INSERT INTO t1 VALUES (2, REPEAT("x", 4100)); +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; +"List of binary logs after rotation" +show binary logs; +Log_name File_size +master-bin.000001 # +master-bin.000002 # +master-bin.000003 # +# Kill the server +"Zero rows shoule be present in table" +SELECT COUNT(*) FROM t1; +COUNT(*) +0 +SELECT @@GLOBAL.gtid_current_pos; +@@GLOBAL.gtid_current_pos +0-1-1 +DROP TABLE t1; +SELECT @@GLOBAL.gtid_binlog_state; +@@GLOBAL.gtid_binlog_state +0-1-2 diff --git a/mysql-test/suite/binlog/r/binlog_truncate_multi_log_unsafe.result b/mysql-test/suite/binlog/r/binlog_truncate_multi_log_unsafe.result new file mode 100644 index 00000000000..5b7d45d81de --- /dev/null +++ b/mysql-test/suite/binlog/r/binlog_truncate_multi_log_unsafe.result @@ -0,0 +1,30 @@ +SET @old_max_binlog_size= @@global.max_binlog_size; +SET GLOBAL max_binlog_size= 4096; +call mtr.add_suppression("Table '.*tm' is marked as crashed and should be repaired"); +call mtr.add_suppression("Got an error from unknown thread"); +call mtr.add_suppression("Checking table: '.*tm'"); +call mtr.add_suppression("Recovering table: '.*tm'"); +call mtr.add_suppression("tc-heuristic-recover cannot trim the binary log to"); +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); +RESET MASTER; +CREATE TABLE ti (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +CREATE TABLE tm (f INT) ENGINE=MYISAM; +SET DEBUG_SYNC= "commit_before_get_LOCK_commit_ordered SIGNAL con1_ready WAIT_FOR con1_go"; +INSERT INTO ti VALUES (2, REPEAT("x", 4100)); +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; +INSERT INTO tm VALUES (30);; +# Kill the server +FOUND /tc-heuristic-recover cannot trim the binary log to File/ in mysqld.1.err +"Zero rows shoule be present in 'ti' table." +SELECT COUNT(*) FROM ti; +COUNT(*) +0 +"One row must be present in 'tm' table." +SELECT COUNT(*) FROM tm; +COUNT(*) +1 +SELECT @@GLOBAL.gtid_current_pos; +@@GLOBAL.gtid_current_pos +0-1-4 +DROP TABLE ti, tm; diff --git a/mysql-test/suite/binlog/r/binlog_truncate_none.result b/mysql-test/suite/binlog/r/binlog_truncate_none.result new file mode 100644 index 00000000000..bae87985208 --- /dev/null +++ b/mysql-test/suite/binlog/r/binlog_truncate_none.result @@ -0,0 +1,42 @@ +SET @old_max_binlog_size= @@global.max_binlog_size; +SET GLOBAL max_binlog_size= 4096; +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); +RESET MASTER; +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +INSERT INTO t1 VALUES (1, REPEAT("x", 4100)); +CREATE TABLE t2 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +show binary logs; +Log_name File_size +master-bin.000001 # +master-bin.000002 # +# Kill the server +"Zero records should be there." +SELECT COUNT(*) FROM t1; +COUNT(*) +1 +show binary logs; +Log_name File_size +master-bin.000001 # +master-bin.000002 # +master-bin.000003 # +master-bin.000004 # +DROP TABLE t1,t2; +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); +RESET MASTER; +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +CREATE TABLE t2 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +show binary logs; +Log_name File_size +master-bin.000001 # +# Kill the server +"Zero records should be there." +SELECT COUNT(*) FROM t1; +COUNT(*) +0 +SHOW TABLES; +Tables_in_test +t1 +t2 +DROP TABLE t1,t2; diff --git a/mysql-test/suite/binlog/r/binlog_truncate_retry_success.result b/mysql-test/suite/binlog/r/binlog_truncate_retry_success.result new file mode 100644 index 00000000000..cd0c4c23d05 --- /dev/null +++ b/mysql-test/suite/binlog/r/binlog_truncate_retry_success.result @@ -0,0 +1,28 @@ +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); +RESET MASTER; +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +INSERT INTO t1 VALUES (1,'dummy'); +SET DEBUG_SYNC= "commit_after_release_LOCK_log SIGNAL con1_ready WAIT_FOR con1_go"; +INSERT INTO t1 VALUES (2,'dummy'); +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; +"List of binary logs" +show binary logs; +Log_name File_size +master-bin.000001 # +# Kill the server +TEST 1: Recovery with error +TEST 2: Recovery without error +"One row shoule be present in table" +SELECT COUNT(*) FROM t1; +COUNT(*) +1 +"Two gtids should be present 0-1-2" +SELECT @@GLOBAL.gtid_current_pos; +@@GLOBAL.gtid_current_pos +0-1-2 +"Two gtids should be present 0-1-3" +DROP TABLE t1; +SELECT @@GLOBAL.gtid_binlog_state; +@@GLOBAL.gtid_binlog_state +0-1-3 diff --git a/mysql-test/suite/binlog/t/binlog_heuristic_rollback_active_log.test b/mysql-test/suite/binlog/t/binlog_heuristic_rollback_active_log.test new file mode 100644 index 00000000000..87af83a7a92 --- /dev/null +++ b/mysql-test/suite/binlog/t/binlog_heuristic_rollback_active_log.test @@ -0,0 +1,75 @@ +# ==== Purpose ==== +# +# Test verifies the truncation of single binary log file. +# +# ==== Implementation ==== +# +# Steps: +# 0 - Create table t1 and insert a row. +# 1 - Insert an another row such that it gets written to binlog but commit +# in engine fails as server crashed at this point. +# 2 - Restart server with --tc-heuristic-recover=ROLLBACK +# 3 - Upon server start 'master-bin.000001' will be truncated to contain +# only the first insert +# +# ==== References ==== +# +# MDEV-21117: --tc-heuristic-recover=rollback is not replication safe + + +--source include/have_innodb.inc +--source include/have_log_bin.inc +--source include/have_debug.inc +--source include/have_binlog_format_statement.inc + +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); + +connect(master,localhost,root,,); +connect(master1,localhost,root,,); + +--connection master +RESET MASTER; +CREATE TABLE t ( f INT ) ENGINE=INNODB; +INSERT INTO t VALUES (10); + +--connection master1 +# Hold insert after write to binlog and before "run_commit_ordered" in engine +SET DEBUG_SYNC= "commit_before_update_end_pos SIGNAL con1_ready WAIT_FOR con1_go"; +send INSERT INTO t VALUES (20); + +--connection master +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; + +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +wait +EOF + +--source include/kill_mysqld.inc +--source include/wait_until_disconnected.inc +# +# Server restart +# +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: --tc-heuristic-recover=ROLLBACK +EOF +--source include/wait_until_disconnected.inc + +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: +EOF + +connection default; +--enable_reconnect +--source include/wait_until_connected_again.inc + +--connection master +--enable_reconnect +--source include/wait_until_connected_again.inc + +--echo "One row should be present in table 't'" +SELECT COUNT(*) FROM t; + +--echo "gtid_binlog_state should be 0-1-2 +SELECT @@GLOBAL.gtid_binlog_state; +DROP TABLE t; diff --git a/mysql-test/suite/binlog/t/binlog_truncate_multi_log.test b/mysql-test/suite/binlog/t/binlog_truncate_multi_log.test new file mode 100644 index 00000000000..0a6e011627e --- /dev/null +++ b/mysql-test/suite/binlog/t/binlog_truncate_multi_log.test @@ -0,0 +1,87 @@ +# ==== Purpose ==== +# +# Test verifies truncation of multiple binary logs. +# +# ==== Implementation ==== +# +# Steps: +# 0 - Create a table in innodb engine and execute FLUSH LOGS command to +# generate a new binary log. +# 1 - Set max_binlog_size= 4096. Insert a row such that the max_binlog_size +# is reached and binary log gets rotated. +# 2 - Using debug simulation make the server crash at a point where the DML +# transaction is written to binary log but not committed in engine. +# 3 - At the time of crash three binary logs will be there +# master-bin.0000001, master-bin.000002 and master-bin.000003. +# 4 - Restart server with --tc-heuristic-recover=ROLLBACK +# 5 - Since the prepared DML in master-bin.000002 the binary log will be +# truncated and master-bin.000003 will be removed. +# +# ==== References ==== +# +# MDEV-21117: --tc-heuristic-recover=rollback is not replication safe + + +--source include/have_innodb.inc +--source include/have_log_bin.inc +--source include/have_debug.inc +--source include/have_debug_sync.inc +--source include/have_binlog_format_row.inc + +SET @old_max_binlog_size= @@GLOBAL.max_binlog_size; +SET GLOBAL max_binlog_size= 4096; +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); + +RESET MASTER; + +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +FLUSH LOGS; +connect(master1,localhost,root,,); +connect(master2,localhost,root,,); + +--connection master1 +# Hold insert after write to binlog and before "run_commit_ordered" in engine. +# Use "commit_after_release_LOCK_log" sync point. This point is reached after +# the binary log end position is updated which actually triggers binlog to be +# rotated. +--echo "List of binary logs before rotation" +--source include/show_binary_logs.inc +SET DEBUG_SYNC= "commit_after_release_LOCK_log SIGNAL con1_ready WAIT_FOR con1_go"; +send INSERT INTO t1 VALUES (2, REPEAT("x", 4100)); + +--connection master2 +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; +--echo "List of binary logs after rotation" +--source include/show_binary_logs.inc +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +wait +EOF + +--source include/kill_mysqld.inc +--source include/wait_until_disconnected.inc + +# +# Server restart +# +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: --tc-heuristic-recover=ROLLBACK --debug-dbug=d,simulate_innodb_forget_commit_pos +EOF +--source include/wait_until_disconnected.inc + +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: +EOF + +connection default; +--enable_reconnect +--source include/wait_until_connected_again.inc + +--echo "Zero rows shoule be present in table" +SELECT COUNT(*) FROM t1; + +SELECT @@GLOBAL.gtid_current_pos; + +DROP TABLE t1; +SELECT @@GLOBAL.gtid_binlog_state; + diff --git a/mysql-test/suite/binlog/t/binlog_truncate_multi_log_unsafe.test b/mysql-test/suite/binlog/t/binlog_truncate_multi_log_unsafe.test new file mode 100644 index 00000000000..cf4f3d34a24 --- /dev/null +++ b/mysql-test/suite/binlog/t/binlog_truncate_multi_log_unsafe.test @@ -0,0 +1,106 @@ +# ==== Purpose ==== +# +# Test verifies truncation of multiple binary logs will report an error on +# unsafe scenario. +# +# ==== Implementation ==== +# +# Steps: +# 0 - Set max_binlog_size= 4096. Create a table 'ti' using transactional +# storage engine. Do an insert such that the max_binlog_size is reached +# and binary log gets rotated. Hold this thread at a position where +# transaction is written to binary log but not committed in engine. +# 1 - Create a table named 'tm' using non transactional storage engine. +# 2 - Insert a row in 'tm' table. The DML will reach the engine and it is +# also written to binary log. +# 3 - Kill and restart the server with --tc-heuristic-recover=ROLLBACK +# 4 - Check for binary log truncation unsafe message in error log. +# 5 - No rows should be present in 'ti' table. One row should be present in +# 'tm' table. +# +# ==== References ==== +# +# MDEV-21117: --tc-heuristic-recover=rollback is not replication safe + + +--source include/have_innodb.inc +--source include/have_log_bin.inc +--source include/have_debug.inc +--source include/have_debug_sync.inc +--source include/have_binlog_format_row.inc + +SET @old_max_binlog_size= @@global.max_binlog_size; +SET GLOBAL max_binlog_size= 4096; + +call mtr.add_suppression("Table '.*tm' is marked as crashed and should be repaired"); +call mtr.add_suppression("Got an error from unknown thread"); +call mtr.add_suppression("Checking table: '.*tm'"); +call mtr.add_suppression("Recovering table: '.*tm'"); +call mtr.add_suppression("tc-heuristic-recover cannot trim the binary log to"); +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); + +RESET MASTER; + +CREATE TABLE ti (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +CREATE TABLE tm (f INT) ENGINE=MYISAM; + +connect(master1,localhost,root,,); +connect(master2,localhost,root,,); + +--connection master1 +# Hold insert after write to binlog and before "run_commit_ordered" in engine +SET DEBUG_SYNC= "commit_before_get_LOCK_commit_ordered SIGNAL con1_ready WAIT_FOR con1_go"; +--send INSERT INTO ti VALUES (2, REPEAT("x", 4100)) + +--connection master2 +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; +--send INSERT INTO tm VALUES (30); + +--connection default +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +wait +EOF + +--source include/kill_mysqld.inc +--source include/wait_until_disconnected.inc + +# +# Server restart +# +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: --tc-heuristic-recover=ROLLBACK --debug-dbug=d,simulate_innodb_forget_commit_pos +EOF +--source include/wait_until_disconnected.inc + +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: +EOF + +connection default; +--enable_reconnect +--source include/wait_until_connected_again.inc + +# Check error log for correct messages. +let $log_error_= `SELECT @@GLOBAL.log_error`; +if(!$log_error_) +{ + # MySQL Server on windows is started with --console and thus + # does not know the location of its .err log, use default location + let $log_error_ = $MYSQLTEST_VARDIR/log/mysqld.1.err; +} +--let SEARCH_FILE=$log_error_ +--let SEARCH_RANGE=-50000 +--let SEARCH_PATTERN=tc-heuristic-recover cannot trim the binary log to File +--source include/search_pattern_in_file.inc + +--echo "Zero rows shoule be present in 'ti' table." +SELECT COUNT(*) FROM ti; +--echo "One row must be present in 'tm' table." +--replace_regex /Table '.*tm/Table 'tm/ +--disable_warnings +SELECT COUNT(*) FROM tm; +--enable_warnings +SELECT @@GLOBAL.gtid_current_pos; + +DROP TABLE ti, tm; diff --git a/mysql-test/suite/binlog/t/binlog_truncate_none.test b/mysql-test/suite/binlog/t/binlog_truncate_none.test new file mode 100644 index 00000000000..cc3fd4f0b37 --- /dev/null +++ b/mysql-test/suite/binlog/t/binlog_truncate_none.test @@ -0,0 +1,130 @@ +# ==== Purpose ==== +# +# Test case verifies no binlog truncation happens when non transactional +# events are found in binlog after the last committed transaction. +# +# ==== Implementation ==== +# +# Steps: +# 0 - Set max_binlog_size= 4096. Create a table and do an insert such that +# the max_binlog_size is reached and binary log gets rotated. +# 1 - Create a table in newly created binary log and crash the server +# 2 - Restart server with --tc-heuristic-recover=ROLLBACK +# 3 - Recovery code will get the last committed DML specific postion and +# will try to check if binlog can be truncated upto this position. +# Since a DDL is present beyond this no truncation will happen. +# ==== References ==== +# +# MDEV-21117: --tc-heuristic-recover=rollback is not replication safe + + +--source include/have_innodb.inc +--source include/have_log_bin.inc +--source include/have_debug.inc +--source include/have_binlog_format_row.inc + +SET @old_max_binlog_size= @@global.max_binlog_size; +SET GLOBAL max_binlog_size= 4096; + +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); + +RESET MASTER; + +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +INSERT INTO t1 VALUES (1, REPEAT("x", 4100)); +CREATE TABLE t2 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +--source include/show_binary_logs.inc + +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +wait +EOF + +--source include/kill_mysqld.inc +--source include/wait_until_disconnected.inc + +# +# Server restart +# +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: --tc-heuristic-recover=ROLLBACK +EOF +--source include/wait_until_disconnected.inc + +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: +EOF + +connection default; +--enable_reconnect +--source include/wait_until_connected_again.inc + +--echo "Zero records should be there." +SELECT COUNT(*) FROM t1; +--source include/show_binary_logs.inc +DROP TABLE t1,t2; + +# ==== Purpose ==== +# +# Test case verifies no binlog truncation happens when only DDLs are present in +# the binary log. Since none of the DMLs are performed in storage engine, +# Engine will not have last committed transaction file name or position. +# Truncation code should return success. +# +# ==== Implementation ==== +# +# Steps: +# 0 - Create two table t1, t2 +# 1 - Kill and restart server with --tc-heuristic-recover=ROLLBACK +# 2 - Only DDL statements are present in the binary log. Since +# no DML was performed engine will not have last commited transaction +# specific binary log name and position. Since no transactional events +# are found, truncation code should simply return. +# +# ==== References ==== +# +# MDEV-21117: --tc-heuristic-recover=rollback is not replication safe + + +--source include/have_innodb.inc +--source include/have_log_bin.inc +--source include/have_debug.inc +--source include/have_binlog_format_row.inc + +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); + +RESET MASTER; + +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +CREATE TABLE t2 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +--source include/show_binary_logs.inc + +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +wait +EOF + +--source include/kill_mysqld.inc +--source include/wait_until_disconnected.inc + +# +# Server restart +# +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: --tc-heuristic-recover=ROLLBACK --debug-dbug=d,simulate_innodb_forget_commit_pos +EOF +--source include/wait_until_disconnected.inc + +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: +EOF + +connection default; +--enable_reconnect +--source include/wait_until_connected_again.inc + +--echo "Zero records should be there." +SELECT COUNT(*) FROM t1; +SHOW TABLES; +DROP TABLE t1,t2; + diff --git a/mysql-test/suite/binlog/t/binlog_truncate_retry_success.test b/mysql-test/suite/binlog/t/binlog_truncate_retry_success.test new file mode 100644 index 00000000000..6e92667c9b4 --- /dev/null +++ b/mysql-test/suite/binlog/t/binlog_truncate_retry_success.test @@ -0,0 +1,84 @@ +# ==== Purpose ==== +# +# Test verifies that tc-heuristic-recover=ROLLBACK can be retried. +# +# ==== Implementation ==== +# +# Steps: +# 0 - Create a table in Innodb storage engine. Insert a row into the table. +# During the commit of this DML engine will persist last committed +# transaction specific binary log file name and position. +# 1 - Do an another DML into the table, and simulate a crash in the middle +# of DML commit, so that DML is present in binary log but not committed in +# the storage engine. +# 2 - Restart the server using --tc-heuristic-recover=ROLLBACK. Simulate an +# error during the binary log rollback operation. Verify appropriate +# error is reported in the error log. +# 3 - Retry the --tc-heuristic-recver=ROLLBACK operation. Verify that +# binary log gets truncated as expected. +# 4 - Verify that global gtid state is according to the rolled back +# transaction. +# ==== References ==== +# +# MDEV-21117: --tc-heuristic-recover=rollback is not replication safe + + +--source include/have_innodb.inc +--source include/not_embedded.inc +--source include/have_log_bin.inc +--source include/have_debug.inc +--source include/have_debug_sync.inc +--source include/have_binlog_format_row.inc + +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); + +RESET MASTER; + +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +INSERT INTO t1 VALUES (1,'dummy'); +connect(master1,localhost,root,,); +connect(master2,localhost,root,,); + +--connection master1 +SET DEBUG_SYNC= "commit_after_release_LOCK_log SIGNAL con1_ready WAIT_FOR con1_go"; +send INSERT INTO t1 VALUES (2,'dummy'); + +--connection master2 +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; +--echo "List of binary logs" +--source include/show_binary_logs.inc +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +wait +EOF + +--source include/kill_mysqld.inc +--source include/wait_until_disconnected.inc + +# +# Server restart +# +--echo TEST 1: Recovery with error +--error 1 +--exec $MYSQLD_LAST_CMD --tc-heuristic-recover=ROLLBACK --debug-dbug=d,fault_injection_opening_binlog > $MYSQLTEST_VARDIR/log/mysqld.1.err 2>&1 +--source include/wait_until_disconnected.inc + +--echo TEST 2: Recovery without error +--error 1 +--exec $MYSQLD_LAST_CMD --tc-heuristic-recover=ROLLBACK > $MYSQLTEST_VARDIR/log/mysqld.1.err 2>&1 +--source include/wait_until_disconnected.inc + +--source include/start_mysqld.inc + +--echo "One row shoule be present in table" +SELECT COUNT(*) FROM t1; + +--echo "Two gtids should be present 0-1-2" +SELECT @@GLOBAL.gtid_current_pos; + +--echo "Two gtids should be present 0-1-3" +DROP TABLE t1; +SELECT @@GLOBAL.gtid_binlog_state; + +--disconnect master1 +--disconnect master2 diff --git a/mysql-test/suite/rpl/r/rpl_heuristic_fail_over.result b/mysql-test/suite/rpl/r/rpl_heuristic_fail_over.result new file mode 100644 index 00000000000..02fc47def41 --- /dev/null +++ b/mysql-test/suite/rpl/r/rpl_heuristic_fail_over.result @@ -0,0 +1,53 @@ +include/rpl_init.inc [topology=1->2] +include/stop_slave.inc +set global rpl_semi_sync_slave_enabled = 1; +CHANGE MASTER TO master_use_gtid= current_pos; +include/start_slave.inc +ALTER TABLE mysql.gtid_slave_pos ENGINE=InnoDB; +set global rpl_semi_sync_master_enabled = 1; +set global rpl_semi_sync_master_wait_point=AFTER_SYNC; +SET @old_max_binlog_size= @@global.max_binlog_size; +SET GLOBAL max_binlog_size= 4096; +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +INSERT INTO t1 VALUES (1, 'dummy1'); +SET DEBUG_SYNC= "commit_before_update_end_pos SIGNAL con1_ready WAIT_FOR con1_go"; +INSERT INTO t1 VALUES (2, REPEAT("x", 4100)); +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; +# Kill the server +include/stop_slave.inc +SELECT @@GLOBAL.gtid_current_pos; +@@GLOBAL.gtid_current_pos +0-1-5 +FOUND /tc-heuristic-recover: Truncated binlog File: \.\/master\-bin\.000001 of Size:[0-9]*, to Position */ in mysqld.1.err +set global rpl_semi_sync_master_enabled = 1; +set global rpl_semi_sync_master_wait_point=AFTER_SYNC; +CHANGE MASTER TO master_host='127.0.0.1', master_port=SERVER_MYPORT_2, master_user='root', master_use_gtid=CURRENT_POS; +set global rpl_semi_sync_slave_enabled = 1; +include/start_slave.inc +INSERT INTO t1 VALUES (3, 'dummy3'); +SELECT COUNT(*) FROM t1; +COUNT(*) +2 +SHOW VARIABLES LIKE 'gtid_current_pos'; +Variable_name Value +gtid_current_pos 0-2-6 +SHOW VARIABLES LIKE 'gtid_current_pos'; +Variable_name Value +gtid_current_pos 0-2-6 +DROP TABLE t1; +set global rpl_semi_sync_master_enabled = 0; +set global rpl_semi_sync_slave_enabled = 0; +set global rpl_semi_sync_master_wait_point=default; +set global rpl_semi_sync_master_enabled = 0; +set global rpl_semi_sync_slave_enabled = 0; +set global rpl_semi_sync_master_wait_point=default; +include/stop_slave.inc +RESET MASTER; +RESET SLAVE; +RESET MASTER; +RESET SLAVE; +CHANGE MASTER TO master_host='127.0.0.1', master_port=SERVER_MYPORT_1, master_user='root', master_use_gtid=no; +include/start_slave.inc +include/rpl_end.inc diff --git a/mysql-test/suite/rpl/t/rpl_heuristic_fail_over.test b/mysql-test/suite/rpl/t/rpl_heuristic_fail_over.test new file mode 100644 index 00000000000..39b1f7545d0 --- /dev/null +++ b/mysql-test/suite/rpl/t/rpl_heuristic_fail_over.test @@ -0,0 +1,160 @@ +# ==== Purpose ==== +# +# Test verifies replication failover scenario. +# +# ==== Implementation ==== +# +# Steps: +# 0 - Have two servers with id's 1 and 2. Enable semi-sync based +# replication. Have semi sync master wait point as 'after_sync'. +# 1 - Create a table and insert a row. While inserting second row simulate +# a server crash at once the transaction is written to binlog, flushed +# and synced but the binlog position is not updated. +# 2 - Restart the server using tc-heuristic-recover=ROLLBACK +# 3 - Post restart switch the crashed master to slave. Execute CHANGE MASTER +# TO command to connect to server id 2. +# 4 - Slave should be able to connect to master. +# 5 - Execute some DML on new master with server id 2. Ensure that it gets +# replicated to server id 1. +# 6 - Verify the "gtid_current_pos" for correctness. +# 7 - Clean up +# +# ==== References ==== +# +# MDEV-21117: --tc-heuristic-recover=rollback is not replication safe + + +--source include/have_semisync.inc +--source include/have_innodb.inc +--source include/have_log_bin.inc +--source include/have_debug.inc +--source include/have_debug_sync.inc +--source include/have_binlog_format_row.inc +--let $rpl_topology=1->2 +--source include/rpl_init.inc + +--connection server_2 +--source include/stop_slave.inc +set global rpl_semi_sync_slave_enabled = 1; +CHANGE MASTER TO master_use_gtid= current_pos; +--source include/start_slave.inc + + +--connection server_1 +ALTER TABLE mysql.gtid_slave_pos ENGINE=InnoDB; +set global rpl_semi_sync_master_enabled = 1; +set global rpl_semi_sync_master_wait_point=AFTER_SYNC; +SET @old_max_binlog_size= @@global.max_binlog_size; +SET GLOBAL max_binlog_size= 4096; + +call mtr.add_suppression("Can't init tc log"); +call mtr.add_suppression("Aborting"); + +CREATE TABLE t1 (a INT PRIMARY KEY, b MEDIUMTEXT) ENGINE=Innodb; +INSERT INTO t1 VALUES (1, 'dummy1'); +--save_master_pos + +--connection server_2 +--sync_with_master + +--connection server_1 +connect(master1,localhost,root,,); +connect(master2,localhost,root,,); + +--connection master1 +# Hold insert after write to binlog and before "run_commit_ordered" in engine +SET DEBUG_SYNC= "commit_before_update_end_pos SIGNAL con1_ready WAIT_FOR con1_go"; +send INSERT INTO t1 VALUES (2, REPEAT("x", 4100)); + +--connection master2 +SET DEBUG_SYNC= "now WAIT_FOR con1_ready"; +--write_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +wait +EOF + +--source include/kill_mysqld.inc +--source include/wait_until_disconnected.inc + +--connection server_2 +--error 2003 +--source include/stop_slave.inc +SELECT @@GLOBAL.gtid_current_pos; + +--connection server_1 +# +# Server restart +# +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: --tc-heuristic-recover=ROLLBACK +EOF +--source include/wait_until_disconnected.inc + +--append_file $MYSQLTEST_VARDIR/tmp/mysqld.1.expect +restart: +EOF + +connection default; +--enable_reconnect +--source include/wait_until_connected_again.inc + +--connection server_1 +--enable_reconnect +--source include/wait_until_connected_again.inc + +# Check error log for correct messages. +let $log_error_= `SELECT @@GLOBAL.log_error`; +if(!$log_error_) +{ + # MySQL Server on windows is started with --console and thus + # does not know the location of its .err log, use default location + let $log_error_ = $MYSQLTEST_VARDIR/log/mysqld.1.err; +} +--let SEARCH_FILE=$log_error_ +--let SEARCH_RANGE=-50000 +--let SEARCH_PATTERN=tc-heuristic-recover: Truncated binlog File: \.\/master\-bin\.000001 of Size:[0-9]*, to Position * +--source include/search_pattern_in_file.inc + +--connection server_2 +set global rpl_semi_sync_master_enabled = 1; +set global rpl_semi_sync_master_wait_point=AFTER_SYNC; + +--connection server_1 +--replace_result $SERVER_MYPORT_2 SERVER_MYPORT_2 +eval CHANGE MASTER TO master_host='127.0.0.1', master_port=$SERVER_MYPORT_2, master_user='root', master_use_gtid=CURRENT_POS; +set global rpl_semi_sync_slave_enabled = 1; +--source include/start_slave.inc + +--connection server_2 +INSERT INTO t1 VALUES (3, 'dummy3'); +--save_master_pos + +--connection server_1 +--sync_with_master +SELECT COUNT(*) FROM t1; +SHOW VARIABLES LIKE 'gtid_current_pos'; + +--connection server_2 +SHOW VARIABLES LIKE 'gtid_current_pos'; +DROP TABLE t1; +set global rpl_semi_sync_master_enabled = 0; +set global rpl_semi_sync_slave_enabled = 0; +set global rpl_semi_sync_master_wait_point=default; +--save_master_pos + +--connection server_1 +--sync_with_master +set global rpl_semi_sync_master_enabled = 0; +set global rpl_semi_sync_slave_enabled = 0; +set global rpl_semi_sync_master_wait_point=default; +--source include/stop_slave.inc +RESET MASTER; +RESET SLAVE; + +--connection server_2 +RESET MASTER; +RESET SLAVE; +--replace_result $SERVER_MYPORT_1 SERVER_MYPORT_1 +eval CHANGE MASTER TO master_host='127.0.0.1', master_port=$SERVER_MYPORT_1, master_user='root', master_use_gtid=no; +--source include/start_slave.inc + +--source include/rpl_end.inc diff --git a/sql/log.cc b/sql/log.cc index 0efef6d1e29..66445944b40 100644 --- a/sql/log.cc +++ b/sql/log.cc @@ -3164,6 +3164,7 @@ MYSQL_BIN_LOG::MYSQL_BIN_LOG(uint *sync_period) checksum_alg_reset(BINLOG_CHECKSUM_ALG_UNDEF), relay_log_checksum_alg(BINLOG_CHECKSUM_ALG_UNDEF), description_event_for_exec(0), description_event_for_queue(0), + last_commit_pos_offset(0), current_binlog_id(0) { /* @@ -3173,6 +3174,7 @@ MYSQL_BIN_LOG::MYSQL_BIN_LOG(uint *sync_period) before main(). */ index_file_name[0] = 0; + last_commit_pos_file[0]= 0; bzero((char*) &index_file, sizeof(index_file)); bzero((char*) &purge_index_file, sizeof(purge_index_file)); } @@ -4576,8 +4578,8 @@ int MYSQL_BIN_LOG::open_purge_index_file(bool destroy) 0, 0, MYF(MY_WME | MY_NABP | MY_WAIT_IF_FULL))) { error= 1; - sql_print_error("MYSQL_BIN_LOG::open_purge_index_file failed to open register " - " file."); + sql_print_error("MYSQL_BIN_LOG::open_purge_index_file failed to open " + "register file."); } } DBUG_RETURN(error); @@ -4742,7 +4744,7 @@ int MYSQL_BIN_LOG::purge_index_entry(THD *thd, ulonglong *reclaimed_space, } goto err; } - + error= 0; DBUG_PRINT("info",("purging %s",log_info.log_file_name)); @@ -7877,6 +7879,7 @@ MYSQL_BIN_LOG::trx_group_commit_leader(group_commit_entry *leader) first= false; } + DEBUG_SYNC(leader->thd, "commit_before_update_end_pos"); /* update binlog_end_pos so it can be read by dump thread * * note: must be _after_ the RUN_HOOK(after_flush) or else @@ -8964,7 +8967,7 @@ int TC_LOG_MMAP::open(const char *opt_name) { if (my_errno != ENOENT) goto err; - if (using_heuristic_recover()) + if (using_heuristic_recover(opt_name)) return 1; if ((fd= mysql_file_create(key_file_tclog, logname, CREATE_MODE, O_RDWR | O_CLOEXEC, MYF(MY_WME))) < 0) @@ -9497,14 +9500,40 @@ TC_LOG_MMAP tc_log_mmap; 1 heuristic recovery was performed */ -int TC_LOG::using_heuristic_recover() +int TC_LOG::using_heuristic_recover(const char* opt_name) { + LOG_INFO log_info; + int error; + if (!tc_heuristic_recover) return 0; sql_print_information("Heuristic crash recovery mode"); + if (ha_recover(0)) + { sql_print_error("Heuristic crash recovery failed"); + } + + if (!strcmp(opt_name, opt_bin_logname) && + tc_heuristic_recover == TC_HEURISTIC_RECOVER_ROLLBACK) + { + if ((error= mysql_bin_log.find_log_pos(&log_info, NullS, 1))) + { + if (error != LOG_INFO_EOF) + sql_print_error("tc-heuristic-recover failed in find_log_pos(). " + "Error: %d", error); + } + else + { + if ((error= heuristic_binlog_rollback())) + { + sql_print_error("Heuristic crash recovery of binary log failed."); + return 1; + } + } + } + sql_print_information("Please restart mysqld without --tc-heuristic-recover"); return 1; } @@ -9512,6 +9541,554 @@ int TC_LOG::using_heuristic_recover() /****** transaction coordinator log for 2pc - binlog() based solution ******/ #define TC_LOG_BINLOG MYSQL_BIN_LOG +/** + Truncates the current binlog to specified position. Removes the rest of binlogs + which are present after this binlog file. + + @param truncate_file Holds the binlog name to be truncated + @param truncate_pos Position within binlog from where it needs to + truncated. + + @retval true ok + @retval false error + +*/ +bool MYSQL_BIN_LOG::truncate_and_remove_binlogs(const char *truncate_file, + my_off_t truncate_pos) +{ + int error= 0; +#ifdef HAVE_REPLICATION + LOG_INFO log_info; + THD *thd= current_thd; + my_off_t index_file_offset= 0; + File file= -1; + IO_CACHE cache; + MY_STAT s; + my_off_t binlog_size; + + if ((error= find_log_pos(&log_info, truncate_file, 1))) + { + sql_print_error("tc-heuristic-recover: Failed to locate binary log file:%s." + "Error:%d", truncate_file, error); + goto end; + } + + while (!(error= find_next_log(&log_info, 1))) + { + if (!index_file_offset) + { + index_file_offset= log_info.index_file_start_offset; + if ((error= open_purge_index_file(TRUE))) + { + sql_print_error("tc-heuristic-recover: Failed to open purge index " + "file:%s. Error:%d", purge_index_file_name, error); + goto end; + } + } + if ((error= register_purge_index_entry(log_info.log_file_name))) + { + sql_print_error("tc-heuristic-recover: Failed to copy %s to purge index" + " file. Error:%d", log_info.log_file_name, error); + goto end; + } + } + + if (error != LOG_INFO_EOF) + { + sql_print_error("tc-heuristic-recover: Failed to find the next binlog to " + "add to purge index register. Error:%d", error); + goto end; + } + + if (is_inited_purge_index_file()) + { + if (!index_file_offset) + index_file_offset= log_info.index_file_start_offset; + + if ((error= sync_purge_index_file())) + { + sql_print_error("tc-heuristic-recover: Failed to flush purge index " + "file. Error:%d", error); + goto end; + } + + // Trim index file + if ((error= + mysql_file_chsize(index_file.file, index_file_offset, '\n', + MYF(MY_WME))) || + (error= + mysql_file_sync(index_file.file, MYF(MY_WME|MY_SYNC_FILESIZE)))) + { + sql_print_error("tc-heuristic-recover: Failed to trim binlog index " + "file:%s to offset:%llu. Error:%d", index_file_name, + index_file_offset); + mysql_file_close(index_file.file, MYF(MY_WME)); + goto end; + } + + /* Reset data in old index cache */ + if ((error= reinit_io_cache(&index_file, READ_CACHE, (my_off_t) 0, 0, 1))) + { + sql_print_error("tc-heuristic-recover: Failed to reinit binlog index " + "file. Error:%d", error); + mysql_file_close(index_file.file, MYF(MY_WME)); + goto end; + } + + /* Read each entry from purge_index_file and delete the file. */ + if ((error= purge_index_entry(thd, NULL, TRUE))) + { + sql_print_error("tc-heuristic-recover: Failed to process registered " + "files that would be purged."); + goto end; + } + } + + DBUG_ASSERT(truncate_pos); + + if ((file= mysql_file_open(key_file_binlog, truncate_file, + O_RDWR | O_BINARY, MYF(MY_WME))) < 0) + { + error= 1; + sql_print_error("tc-heuristic-recover: Failed to open binlog file:%s for " + "truncation.", truncate_file); + goto end; + } + my_stat(truncate_file, &s, MYF(0)); + binlog_size= s.st_size; + + /* Change binlog file size to truncate_pos */ + if ((error= + mysql_file_chsize(file, truncate_pos, 0, MYF(MY_WME))) || + (error= mysql_file_sync(file, MYF(MY_WME|MY_SYNC_FILESIZE)))) + { + sql_print_error("tc-heuristic-recover: Failed to trim the " + "binlog file:%s to size:%llu. Error:%d", + truncate_file, truncate_pos, error); + goto end; + } + else + { + sql_print_information("tc-heuristic-recover: Truncated binlog " + "File: %s of Size:%llu, to Position:%llu.", + truncate_file, binlog_size, truncate_pos); + } + if (!(error= init_io_cache(&cache, file, IO_SIZE, WRITE_CACHE, + (my_off_t) truncate_pos, 0, MYF(MY_WME|MY_NABP)))) + { + /* + Write Stop_log_event to ensure clean end point for the binary log being + truncated. + */ + Stop_log_event se; + se.checksum_alg= (enum_binlog_checksum_alg) binlog_checksum_options; + if ((error= write_event(&se, &cache))) + { + sql_print_error("tc-heuristic-recover: Failed to write stop event to " + "binary log. Errno:%d", + (cache.error == -1) ? my_errno : error); + goto end; + } + if ((error= flush_io_cache(&cache)) || + (error= mysql_file_sync(file, MYF(MY_WME|MY_SYNC_FILESIZE)))) + { + sql_print_error("tc-heuristic-recover: Faild to write stop event to " + "binary log. Errno:%d", + (cache.error == -1) ? my_errno : error); + } + } + else + sql_print_error("tc-heuristic-recover: Failed to initialize binary log " + "cache for writing stop event. Errno:%d", + (cache.error == -1) ? my_errno : error); + +end: + if (file >= 0) + { + end_io_cache(&cache); + mysql_file_close(file, MYF(MY_WME)); + } + error= error || close_purge_index_file(); +#endif + return error > 0; +} + +/** + Returns the checkpoint binlog file name found in the lastest binlog file. + + @param checkpoint_file Holds the binlog checkpoint file name. + + @retval 0 ok + @retval 1 error + +*/ +int TC_LOG_BINLOG::get_binlog_checkpoint_file(char* checkpoint_file) +{ + Log_event *ev= NULL; + bool binlog_checkpoint_found= false; + LOG_INFO log_info; + const char *errmsg; + IO_CACHE log; + File file; + Format_description_log_event fdle(BINLOG_VERSION); + char log_name[FN_REFLEN]; + int error=1; + + if (!fdle.is_valid()) + return 1; + + if ((error= find_log_pos(&log_info, NullS, 1))) + { + sql_print_error("tc-heuristic-recover: find_log_pos() failed to read first " + "binary log entry from index file.(error: %d)", error); + return error; + } + + // Move to the last binary log. + do + { + strmake_buf(log_name, log_info.log_file_name); + } while (!(error= find_next_log(&log_info, 1))); + + if (error != LOG_INFO_EOF) + { + sql_print_error("tc-heuristic-recover: find_next_log() failed " + "(error: %d)", error); + return error; + } + if ((file= open_binlog(&log, log_name, &errmsg)) < 0) + { + sql_print_error("tc-heuristic-recover failed to open the binlog:%s for " + "reading checkpoint file name. Error: %s", + log_info.log_file_name, errmsg); + return error; + } + while (!binlog_checkpoint_found && + (ev= + Log_event::read_log_event(&log, 0, &fdle, opt_master_verify_checksum)) + && ev->is_valid()) + { + enum Log_event_type typ= ev->get_type_code(); + if (typ == BINLOG_CHECKPOINT_EVENT) + { + size_t dir_len; + Binlog_checkpoint_log_event *cev= (Binlog_checkpoint_log_event *)ev; + if (cev->binlog_file_len >= FN_REFLEN) + { + sql_print_error("tc-heuristic-recover: Incorrect binlog checkpoint " + "event with too long file name found."); + delete ev; + ev= NULL; + end_io_cache(&log); + mysql_file_close(file, MYF(MY_WME)); + return 1; + } + else + { + dir_len= dirname_length(log_name); + strmake(strnmov(checkpoint_file, log_name, dir_len), + cev->binlog_file_name, FN_REFLEN - 1 - dir_len); + binlog_checkpoint_found= true; + } + } + delete ev; + ev= NULL; + } // End of while + end_io_cache(&log); + mysql_file_close(file, MYF(MY_WME)); + file= -1; + /* + Old binary log without checkpoint found, binlog truncation is not + possible. Hence return error. + */ + if (!binlog_checkpoint_found) + return 1; + + return 0; +} + + +/** + Truncates the binary log, according to the transactions that got rolled + back from engine, during heuristic-recover=ROLLBACK. Global GTID state is + adjusted as per the truncated binlog. + + Called from @c TC_LOG::using_heuristic_recover(const char* opt_name) + + @param opt_name The base name of binary log. + + @return indicates success or failure of binlog rollback + @retval 0 success + @retval 1 failure + +*/ +int TC_LOG_BINLOG::heuristic_binlog_rollback() +{ + int error=0; +#ifdef HAVE_REPLICATION + Log_event *ev= NULL; + char engine_commit_file[FN_REFLEN]; + char binlog_truncate_file_name[FN_REFLEN]; + char checkpoint_file[FN_REFLEN]; + my_off_t binlog_truncate_pos= 0; + my_off_t engine_commit_pos= 0; + LOG_INFO log_info; + const char *errmsg; + IO_CACHE log; + File file=-1; + Format_description_log_event fdle(BINLOG_VERSION); + bool found_engine_commit_pos= false; + bool found_truncate_pos= false; + bool is_safe= true; + my_off_t tmp_truncate_pos=0; + rpl_gtid last_gtid; + bool last_gtid_standalone= false; + bool last_gtid_valid= false; + bool is_checkpoint_based_recovery= false; + + + DBUG_EXECUTE_IF("simulate_innodb_forget_commit_pos", + { + last_commit_pos_file[0]= 0; + };); + + // Initialize engine_commit_file/pos + if (last_commit_pos_file[0] != 0) + { + strmake_buf(engine_commit_file, last_commit_pos_file); + engine_commit_pos= last_commit_pos_offset; + sql_print_information("tc-heuristic-recover: Initialising heuristic " + "rollback of binary log using last committed " + "transaction specific binary log name:%s and " + "position:%llu", last_commit_pos_file, + last_commit_pos_offset); + } + else + { + if ((error= get_binlog_checkpoint_file(checkpoint_file))) + { + is_checkpoint_based_recovery= true; + sql_print_error("tc-heuristic-recover: Failed to read latest checkpoint " + "binary log name."); + goto end; + } + strmake_buf(engine_commit_file, checkpoint_file); + sql_print_information("tc-heuristic-recover: Initialising heuristic " + "rollback of binary log using last checkpoint " + "file:%s.", engine_commit_file); + /* + If there is no engine specific commit file we are doing checkpoint file + based recovery. Hence we mark "found_engine_commit_pos" true, and start + looking for the first transactional event group with is the candidate for + rollback. + */ + found_engine_commit_pos= true; + } + + if ((error= find_log_pos(&log_info, engine_commit_file, 1))) + { + sql_print_error("tc-heuristic-recover: Failed to locate binary log file:%s " + "in index file. Error:%d", engine_commit_file, error); + goto end; + } + if ((file= open_binlog(&log, log_info.log_file_name, &errmsg)) < 0 || + DBUG_EVALUATE_IF("fault_injection_opening_binlog", (errmsg="Unknown"), + FALSE)) + { + error= 1; + sql_print_error("tc-heuristic-recover: Failed to open the binlog:%s for " + "recovery. Error:%s", log_info.log_file_name, errmsg); + goto end; + } + + + error= read_state_from_file(); + if (error && error != 2) + { + sql_print_error("tc-heuristic-recover: Failed to load global gtid binlog " + "state from file"); + goto end; + } + if (!fdle.is_valid()) + { + error= 1; + sql_print_error("tc-heuristic-recover: Failed due to invalid format " + "description log event."); + goto end; + } + + for(;;) + { + while (is_safe && + (ev= Log_event::read_log_event(&log, 0, &fdle, + opt_master_verify_checksum)) && ev->is_valid()) + { + enum Log_event_type typ= ev->get_type_code(); + switch (typ) + { + case XID_EVENT: + if (ev->log_pos == engine_commit_pos) + { + found_engine_commit_pos= true; + } + break; + case GTID_LIST_EVENT: + { + Gtid_list_log_event *glev= (Gtid_list_log_event *)ev; + /* Initialise the binlog state from the Gtid_list event. */ + if (!found_truncate_pos && glev->count > 0 && + rpl_global_gtid_binlog_state.load(glev->list, glev->count)) + { + error= 1; + sql_print_error("tc-heuristic-recover: Failed to read GTID List " + "event."); + goto end; + } + } + break; + case GTID_EVENT: + { + Gtid_log_event *gev= (Gtid_log_event *)ev; + + /* Update the binlog state with any GTID logged after Gtid_list. */ + last_gtid.domain_id= gev->domain_id; + last_gtid.server_id= gev->server_id; + last_gtid.seq_no= gev->seq_no; + last_gtid_standalone= + ((gev->flags2 & Gtid_log_event::FL_STANDALONE) ? true : false); + last_gtid_valid= true; + if (gev->flags2 & Gtid_log_event::FL_TRANSACTIONAL && + !found_truncate_pos) + { + if ((engine_commit_pos == 0 || found_engine_commit_pos)) + { + found_truncate_pos= true; + strmake_buf(binlog_truncate_file_name, log_info.log_file_name); + binlog_truncate_pos= tmp_truncate_pos; + } + } + else + { + if (found_truncate_pos) + is_safe= false; + } + } + break; + default: + /* Nothing. */ + break; + }// End switch + if (!found_truncate_pos && last_gtid_valid && + ((last_gtid_standalone && !ev->is_part_of_group(typ)) || + (!last_gtid_standalone && + (typ == XID_EVENT || + (typ == QUERY_EVENT && + (((Query_log_event *)ev)->is_commit() || + ((Query_log_event *)ev)->is_rollback())))))) + { + if ((error= rpl_global_gtid_binlog_state.update_nolock(&last_gtid, + false))) + { + sql_print_error("tc-heuristic-recover: Failed to update GTID within " + "global gtid state."); + goto end; + } + last_gtid_valid= false; + } + // Used to identify the last group specific end position. + tmp_truncate_pos= ev->log_pos; + delete ev; + ev= NULL; + }// End While + if (file >= 0) + { + end_io_cache(&log); + mysql_file_close(file, MYF(MY_WME)); + file= -1; + } + if (is_safe) + { + if ((error= find_next_log(&log_info, 1))) + { + if (error != LOG_INFO_EOF) + { + sql_print_error("tc-heuristic-recover: Failed to read next binary " + "log during recovery."); + goto end; + } + else + { + error= 0; // LOG_INFO_EOF= -1 is not an error. + break; + } + } + if ((file= open_binlog(&log, log_info.log_file_name, &errmsg)) < 0) + { + error= 1; + sql_print_error("tc-heuristic-recover: Failed to open the binlog:%s for " + "recovery. Error:%s", log_info.log_file_name, errmsg); + goto end; + } + } + else + break; + } //end of for(;;) + sql_print_information("tc-heuristic-recover: Binary log to be truncated " + "File:%s Pos:%llu.", binlog_truncate_file_name, + binlog_truncate_pos); + if (!found_truncate_pos) + goto end; // Nothing to truncate + else + DBUG_ASSERT(binlog_truncate_pos > 0); + + if (is_safe) + { + if ((error= truncate_and_remove_binlogs(binlog_truncate_file_name, + binlog_truncate_pos))) + { + sql_print_error("tc-heuristic-recover: Failed to trim the binary log to " + "File:%s Pos:%llu.", binlog_truncate_file_name, + binlog_truncate_pos); + goto end; + } + } + else + { + sql_print_error("tc-heuristic-recover cannot trim the binary log to " + "File:%s Pos:%llu as unsafe statements (non-trans/DDL) " + "statements are found beyond the truncation position.", + binlog_truncate_file_name, binlog_truncate_pos); + } + if ((error= write_state_to_file())) + { + sql_print_error("tc-heuristic-recover: Failed to write global gtid state " + "to file"); + goto end; + } + +end: + if (file >= 0) + { + end_io_cache(&log); + mysql_file_close(file, MYF(MY_WME)); + } + if (error) + { + if (is_checkpoint_based_recovery) + { + sql_print_error("tc-heuristic-recover failed during binary log rollback." + "Further retry attemps won't succeed."); + } + else + sql_print_error("tc-heuristic-recover failed during binary log rollback. " + "After removing the source of the failure " + "--tc-heuristic-recover=rollback may be retried."); + } + +#endif + return error; +} + int TC_LOG_BINLOG::open(const char *opt_name) { int error= 1; @@ -9526,7 +10103,7 @@ int TC_LOG_BINLOG::open(const char *opt_name) return 1; } - if (using_heuristic_recover()) + if (using_heuristic_recover(opt_name)) { mysql_mutex_lock(&LOCK_log); /* generate a new binlog to mask a corrupted one */ diff --git a/sql/log.h b/sql/log.h index 277e5c6f69c..84c0a457b52 100644 --- a/sql/log.h +++ b/sql/log.h @@ -41,7 +41,8 @@ bool stmt_has_updated_non_trans_table(const THD* thd); class TC_LOG { public: - int using_heuristic_recover(); + int using_heuristic_recover(const char* opt_name); + virtual int heuristic_binlog_rollback() { return 0; }; TC_LOG() {} virtual ~TC_LOG() {} @@ -694,6 +695,7 @@ class MYSQL_BIN_LOG: public TC_LOG, private MYSQL_LOG void commit_checkpoint_notify(void *cookie); int recover(LOG_INFO *linfo, const char *last_log_name, IO_CACHE *first_log, Format_description_log_event *fdle, bool do_xa); + int heuristic_binlog_rollback(); int do_binlog_recovery(const char *opt_name, bool do_xa_recovery); #if !defined(MYSQL_CLIENT) @@ -794,6 +796,9 @@ class MYSQL_BIN_LOG: public TC_LOG, private MYSQL_LOG int purge_first_log(Relay_log_info* rli, bool included); int set_purge_index_file_name(const char *base_file_name); int open_purge_index_file(bool destroy); + bool truncate_and_remove_binlogs(const char *truncate_file, + my_off_t truncate_pos); + int get_binlog_checkpoint_file(char* checkpoint_file); bool is_inited_purge_index_file(); int close_purge_index_file(); int clean_purge_index_file(); diff --git a/sql/mysqld.h b/sql/mysqld.h index e939524dbff..515d5f39c3d 100644 --- a/sql/mysqld.h +++ b/sql/mysqld.h @@ -94,6 +94,7 @@ extern "C" MYSQL_PLUGIN_IMPORT CHARSET_INFO *system_charset_info; extern MYSQL_PLUGIN_IMPORT CHARSET_INFO *files_charset_info ; extern MYSQL_PLUGIN_IMPORT CHARSET_INFO *national_charset_info; extern MYSQL_PLUGIN_IMPORT CHARSET_INFO *table_alias_charset; +extern MYSQL_PLUGIN_IMPORT bool opt_bin_log; /** Character set of the buildin error messages loaded from errmsg.sys. @@ -104,7 +105,7 @@ extern CHARSET_INFO *character_set_filesystem; extern MY_BITMAP temp_pool; extern bool opt_large_files, server_id_supplied; -extern bool opt_update_log, opt_bin_log, opt_error_log; +extern bool opt_update_log, opt_error_log; extern my_bool opt_log, opt_bootstrap; extern my_bool opt_backup_history_log; extern my_bool opt_backup_progress_log; diff --git a/storage/innobase/handler/ha_innodb.cc b/storage/innobase/handler/ha_innodb.cc index 4de2cdbeaec..552348ee2c5 100644 --- a/storage/innobase/handler/ha_innodb.cc +++ b/storage/innobase/handler/ha_innodb.cc @@ -16493,6 +16493,13 @@ innobase_xa_recover( { DBUG_ASSERT(hton == innodb_hton_ptr); + if (opt_bin_log) + { + mysql_bin_log.last_commit_pos_offset= trx_sys_mysql_bin_log_pos; + strmake_buf(mysql_bin_log.last_commit_pos_file, + trx_sys_mysql_bin_log_name); + } + if (len == 0 || xid_list == NULL) { return(0); diff --git a/storage/innobase/log/log0recv.cc b/storage/innobase/log/log0recv.cc index 3b3c7c23224..c8a742d95e9 100644 --- a/storage/innobase/log/log0recv.cc +++ b/storage/innobase/log/log0recv.cc @@ -2,7 +2,7 @@ Copyright (c) 1997, 2017, Oracle and/or its affiliates. All Rights Reserved. Copyright (c) 2012, Facebook Inc. -Copyright (c) 2013, 2019, MariaDB Corporation. +Copyright (c) 2013, 2020, MariaDB Corporation. This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software @@ -3301,7 +3301,8 @@ void recv_recovery_from_checkpoint_finish(void) /*======================================*/ { - if (recv_needed_recovery) { + extern MYSQL_PLUGIN_IMPORT bool opt_bin_log; + if (opt_bin_log) { trx_sys_print_mysql_master_log_pos(); trx_sys_print_mysql_binlog_offset(); } diff --git a/storage/xtradb/handler/ha_innodb.cc b/storage/xtradb/handler/ha_innodb.cc index 2aafb1a44ee..a966f726a03 100644 --- a/storage/xtradb/handler/ha_innodb.cc +++ b/storage/xtradb/handler/ha_innodb.cc @@ -17165,6 +17165,13 @@ innobase_xa_recover( { DBUG_ASSERT(hton == innodb_hton_ptr); + if (opt_bin_log) + { + mysql_bin_log.last_commit_pos_offset= trx_sys_mysql_bin_log_pos; + strmake_buf(mysql_bin_log.last_commit_pos_file, + trx_sys_mysql_bin_log_name); + } + if (len == 0 || xid_list == NULL) { return(0); diff --git a/storage/xtradb/log/log0recv.cc b/storage/xtradb/log/log0recv.cc index dd55d31218a..a4844507328 100644 --- a/storage/xtradb/log/log0recv.cc +++ b/storage/xtradb/log/log0recv.cc @@ -2,7 +2,7 @@ Copyright (c) 1997, 2017, Oracle and/or its affiliates. All Rights Reserved. Copyright (c) 2012, Facebook Inc. -Copyright (c) 2013, 2019, MariaDB Corporation. +Copyright (c) 2013, 2020, MariaDB Corporation. This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software @@ -3397,7 +3397,8 @@ void recv_recovery_from_checkpoint_finish(void) /*======================================*/ { - if (recv_needed_recovery) { + extern MYSQL_PLUGIN_IMPORT bool opt_bin_log; + if (opt_bin_log) { trx_sys_print_mysql_master_log_pos(); trx_sys_print_mysql_binlog_offset(); }

1 0

Re: [Maria-developers] 784cc5970dd: MDEV-19650: Privilege bug on MariaDB 10.4
by Sergei Golubchik 21 Apr '20

21 Apr '20

Hi, Oleksandr! On Apr 21, Oleksandr Byelkin wrote: > > > OK, INSERT probably is not needed, but we have tons (more than 40 I think) > of DELETE in our tests (and so UPDATE should work). Yes, the view is delete-able and some columns are update-able. So, let's keep these privileges. INSERT and LOAD never worked. > Also we have not LOAD but this: > > rpl.rpl_current_user 'stmt' w7 [ fail ] > Test ended at 2020-04-21 09:34:44 > > CURRENT_TEST: rpl.rpl_current_user > mysqltest: In included file "./include/diff_tables.inc": > included from > /home/sanja/maria/git/10.4/mysql-test/suite/rpl/t/rpl_current_user.test at > line 65: > At line 170: query 'SELECT * INTO OUTFILE '$_dt_outfile' FROM > $_dt_database.$_dt_table ORDER BY `$_dt_column_list`' failed: 1356: View > 'test.v_user' references invalid table(s) or column(s) or function(s) or > definer/invoker of view lack rights to use them This must be a bug. If one does SELECT * INTO OUTFILE FROM some.view than the current user should have FILE, not the view definer. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 784cc5970dd: MDEV-19650: Privilege bug on MariaDB 10.4
by Sergei Golubchik 20 Apr '20

20 Apr '20

Hi, Oleksandr! On Apr 20, Oleksandr Byelkin wrote: > > > > > + > > > > > +CREATE TEMPORARY TABLE tmp_user_sys LIKE global_priv; > > > > > +INSERT INTO tmp_user_sys (Host,User,Priv) VALUES ('localhost','mariadb.sys','{"access":512,"plugin":"mysql_native_password","authentication_string":"","account_locked":true,"password_last_changed":0}'); > > > > > +INSERT INTO global_priv SELECT * FROM tmp_user_sys WHERE NOT @had_sys_user; > > > > > +DROP TABLE tmp_user_sys; > > > > > > > > 1. This could've been simply INSERT IGNORE, I suspect > > > > > > Nope, the idea is do not insert more than needed. > > > > Why would INSERT IGNORE insert more than needed? > > I thought you propose insert ignore instead of where clause, in any case I > do not see why it should help. I mean, instead of four lines above, I think, one can simply do INSERT IGNORE global_priv (Host,User,Priv) VALUES ('localhost','mariadb.sys','{"access":512,"plugin":"mysql_native_password","authentication_string":"","account_locked":true,"password_last_changed":0}'); > > > 2. why access:512 ? It's FILE_ACL, iirc. > > > > > > Because LOAD used in tests and so in reality probably, so we need > > FILE_ACL > > > > We need FILE_ACL, but why mariadb.sys account needs it? > > mysql.user is a read-only view of the mysql.global_priv table, > > its owner doesn't need FILE or INSERT/UPDATE/DELETE. > > As I told we somewhere in test suite (not main, but I don't remember > exactly) test which require the permission otherwise I would not return and > add more rights. Where? The view is clearly not insertable-into, so INSERT privilege is meaningless. Some fields are updateable, and one can delele from it. This was more accidental than intentional, but ok, let's not change it in 10.4. And why does it need FILE privilege? The view owner cannot possibly do any LOAD DATA. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 784cc5970dd: MDEV-19650: Privilege bug on MariaDB 10.4
by Sergei Golubchik 20 Apr '20

20 Apr '20

Hi, Oleksandr! On Apr 20, Oleksandr Byelkin wrote: > > Hi, Oleksandr! > > > > On Apr 20, Oleksandr Byelkin wrote: > > > revision-id: 784cc5970dd (mariadb-10.4.11-68-g784cc5970dd) > > > parent(s): c5e00fea102 > > > author: Oleksandr Byelkin <sanja(a)mariadb.com> > > > committer: Oleksandr Byelkin <sanja(a)mariadb.com> > > > timestamp: 2020-02-20 14:06:09 +0100 > > > message: > > > > > > MDEV-19650: Privilege bug on MariaDB 10.4 > > > > > > diff --git a/scripts/mysql_system_tables.sql > > > index 29f2a4c1ef6..af852444d0c 100644 > > > --- a/scripts/mysql_system_tables.sql > > > +++ b/scripts/mysql_system_tables.sql > > > @@ -33,9 +33,17 @@ CREATE TABLE IF NOT EXISTS db ( Host char(60) > > binary DEFAULT '' NOT NULL, Db c > > > -- Remember for later if db table already existed > > > set @had_db_table= @@warning_count != 0; > > > > > > -CREATE TABLE IF NOT EXISTS global_priv (Host char(60) binary DEFAULT > > '', User char(80) binary DEFAULT '', Priv JSON NOT NULL DEFAULT '{}' > > CHECK(JSON_VALID(Priv)), PRIMARY KEY Host (Host,User)) engine=Aria > > transactional=1 CHARACTER SET utf8 COLLATE utf8_bin comment='Users and > > global privileges'; > > > +CREATE TABLE IF NOT EXISTS global_priv (Host char(60) binary DEFAULT > > '', User char(80) binary DEFAULT '', Priv JSON NOT NULL DEFAULT '{}' > > CHECK(JSON_VALID(Priv)), PRIMARY KEY (Host,User)) engine=Aria > > transactional=1 CHARACTER SET utf8 COLLATE utf8_bin comment='Users and > > global privileges'; > > > > > > -CREATE DEFINER=root@localhost SQL SECURITY DEFINER VIEW IF NOT EXISTS > > user AS SELECT > > > +set @had_sys_user= @@warning_count != 0 OR 0 <> (select count(*) from > > mysql.global_priv where Host="localhost" and User="mariadb.sys"); > > > + > > > +CREATE TEMPORARY TABLE tmp_user_sys LIKE global_priv; > > > +INSERT INTO tmp_user_sys (Host,User,Priv) VALUES > > ('localhost','mariadb.sys','{"access":512,"plugin":"mysql_native_password","authentication_string":"","account_locked":true,"password_last_changed":0}'); > > > +INSERT INTO global_priv SELECT * FROM tmp_user_sys WHERE NOT > > @had_sys_user; > > > +DROP TABLE tmp_user_sys; > > > > 1. This could've been simply INSERT IGNORE, I suspect > > Nope, the idea is do not insert more than needed. Why would INSERT IGNORE insert more than needed? > 2. why access:512 ? It's FILE_ACL, iirc. > > Because LOAD used in tests and so in reality probably, so we need FILE_ACL We need FILE_ACL, but why mariadb.sys account needs it? mysql.user is a read-only view of the mysql.global_priv table, its owner doesn't need FILE or INSERT/UPDATE/DELETE. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 784cc5970dd: MDEV-19650: Privilege bug on MariaDB 10.4
by Sergei Golubchik 20 Apr '20

20 Apr '20

Hi, Oleksandr! On Apr 20, Oleksandr Byelkin wrote: > revision-id: 784cc5970dd (mariadb-10.4.11-68-g784cc5970dd) > parent(s): c5e00fea102 > author: Oleksandr Byelkin <sanja(a)mariadb.com> > committer: Oleksandr Byelkin <sanja(a)mariadb.com> > timestamp: 2020-02-20 14:06:09 +0100 > message: > > MDEV-19650: Privilege bug on MariaDB 10.4 > > Also fixes: > MDEV-21487: Implement option for mysql_upgrade that allows root@localhost to be replaced > MDEV-21486: Implement option for mysql_install_db that allows root@localhost to be replaced > > Add user mariadb.sys to be definer of user view > (and has right on underlying table global_priv for > required operation over global_priv > (SELECT,UPDATE,DELETE,INSERT,FILE)) > > Also changed definer of gis functions in case of creation, > but they work with any definer so upgrade script do not try > to push this change. > > diff --git a/scripts/mysql_system_tables.sql b/scripts/mysql_system_tables.sql > index 29f2a4c1ef6..af852444d0c 100644 > --- a/scripts/mysql_system_tables.sql > +++ b/scripts/mysql_system_tables.sql > @@ -33,9 +33,17 @@ CREATE TABLE IF NOT EXISTS db ( Host char(60) binary DEFAULT '' NOT NULL, Db c > -- Remember for later if db table already existed > set @had_db_table= @@warning_count != 0; > > -CREATE TABLE IF NOT EXISTS global_priv (Host char(60) binary DEFAULT '', User char(80) binary DEFAULT '', Priv JSON NOT NULL DEFAULT '{}' CHECK(JSON_VALID(Priv)), PRIMARY KEY Host (Host,User)) engine=Aria transactional=1 CHARACTER SET utf8 COLLATE utf8_bin comment='Users and global privileges'; > +CREATE TABLE IF NOT EXISTS global_priv (Host char(60) binary DEFAULT '', User char(80) binary DEFAULT '', Priv JSON NOT NULL DEFAULT '{}' CHECK(JSON_VALID(Priv)), PRIMARY KEY (Host,User)) engine=Aria transactional=1 CHARACTER SET utf8 COLLATE utf8_bin comment='Users and global privileges'; > > -CREATE DEFINER=root@localhost SQL SECURITY DEFINER VIEW IF NOT EXISTS user AS SELECT > +set @had_sys_user= @@warning_count != 0 OR 0 <> (select count(*) from mysql.global_priv where Host="localhost" and User="mariadb.sys"); > + > +CREATE TEMPORARY TABLE tmp_user_sys LIKE global_priv; > +INSERT INTO tmp_user_sys (Host,User,Priv) VALUES ('localhost','mariadb.sys','{"access":512,"plugin":"mysql_native_password","authentication_string":"","account_locked":true,"password_last_changed":0}'); > +INSERT INTO global_priv SELECT * FROM tmp_user_sys WHERE NOT @had_sys_user; > +DROP TABLE tmp_user_sys; 1. This could've been simply INSERT IGNORE, I suspect 2. why access:512 ? It's FILE_ACL, iirc. > + > +CREATE DEFINER='mariadb.sys'@'localhost' SQL SECURITY DEFINER VIEW IF NOT EXISTS user AS SELECT > Host, > User, > IF(JSON_VALUE(Priv, '$.plugin') IN ('mysql_native_password', 'mysql_old_password'), IFNULL(JSON_VALUE(Priv, '$.authentication_string'), ''), '') AS Password, > @@ -101,6 +109,11 @@ CREATE TABLE IF NOT EXISTS servers ( Server_name char(64) NOT NULL DEFAULT '', H > > CREATE TABLE IF NOT EXISTS tables_priv ( Host char(60) binary DEFAULT '' NOT NULL, Db char(64) binary DEFAULT '' NOT NULL, User char(80) binary DEFAULT '' NOT NULL, Table_name char(64) binary DEFAULT '' NOT NULL, Grantor char(141) DEFAULT '' NOT NULL, Timestamp timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, Table_priv set('Select','Insert','Update','Delete','Create','Drop','Grant','References','Index','Alter','Create View','Show view','Trigger','Delete versioning rows') COLLATE utf8_general_ci DEFAULT '' NOT NULL, Column_priv set('Select','Insert','Update','References') COLLATE utf8_general_ci DEFAULT '' NOT NULL, PRIMARY KEY (Host,Db,User,Table_name), KEY Grantor (Grantor) ) engine=Aria transactional=1 CHARACTER SET utf8 COLLATE utf8_bin comment='Table privileges'; > > +CREATE TEMPORARY TABLE tmp_user_sys LIKE tables_priv; > +INSERT INTO tmp_user_sys (Host,Db,User,Table_name,Grantor,Timestamp,Table_priv) VALUES ('localhost','mysql','mariadb.sys','global_priv','root@localhost','0','Select,Insert,Update,Delete'); why Insert,Update,Delete ? > +INSERT INTO tables_priv SELECT * FROM tmp_user_sys WHERE NOT @had_sys_user; > +DROP TABLE tmp_user_sys; > + > CREATE TABLE IF NOT EXISTS columns_priv ( Host char(60) binary DEFAULT '' NOT NULL, Db char(64) binary DEFAULT '' NOT NULL, User char(80) binary DEFAULT '' NOT NULL, Table_name char(64) binary DEFAULT '' NOT NULL, Column_name char(64) binary DEFAULT '' NOT NULL, Timestamp timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, Column_priv set('Select','Insert','Update','References') COLLATE utf8_general_ci DEFAULT '' NOT NULL, PRIMARY KEY (Host,Db,User,Table_name,Column_name) ) engine=Aria transactional=1 CHARACTER SET utf8 COLLATE utf8_bin comment='Column privileges'; > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] MDEV-17399 Add support for JSON_TABLE: for ORDINALITY
by Sergey Petrunia 19 Apr '20

19 Apr '20

Hi Alexey, The standard says "An ordinality column provides a sequential numbering of rows. Row numbering is 1-based." Consider this example from the standard: https://gist.github.com/spetrunia/d4d8564a3ed26148ae92035b24e1f294 SELECT jt.rowseq, jt.name, jt.zip FROM bookclub, JSON_TABLE(bookclub.jcol, "lax $" COLUMNS ( rowSeq FOR ORDINALITY, name VARCHAR(30) PATH 'lax $.Name', zip CHAR(5) PATH 'lax $.address.postalCode') ) AS jt +--------+---------+------+ | rowseq | name | zip | +--------+---------+------+ | 0 | John Sm | 1 | | 1 | Peter W | 9 | | 2 | James L | NULL | +--------+---------+------+ The numbering is 0-based, instead of 1-based. What is not entirely clear for me is when the numbering should be reset. For the above example, the SQL standard shows the resultset with rowseq 1-2-3 which hints that the numbering is "global". MySQL-8 produces "1-1-1", which looks like they reset the numbering for every scan of the JSON_TABLE output. What are you thoughts on this? If the numbering should be "global", should it be query-level global, or subselect-level global? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 1