- developers - lists.mariadb.org

Re: [Maria-developers] 06ce67c644b: MDEV-27653 long uniques don't work with unicode collations
by Sergei Golubchik 16 Jan '23

16 Jan '23

Hi, Alexander, On Jan 15, Alexander Barkov wrote: > revision-id: 06ce67c644b (mariadb-10.4.27-41-g06ce67c644b) > parent(s): 6cb84346e1b > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2023-01-10 18:27:16 +0400 > message: > > MDEV-27653 long uniques don't work with unicode collations > > diff --git a/sql/sql_admin.cc b/sql/sql_admin.cc > index 213d77f8237..5dd19c877af 100644 > --- a/sql/sql_admin.cc > +++ b/sql/sql_admin.cc > @@ -772,11 +772,14 @@ static bool mysql_admin_table(THD* thd, TABLE_LIST* tables, > int check_for_upgrade= file->ha_check_for_upgrade(check_opt); > > if (check_old_types == HA_ADMIN_NEEDS_ALTER || > - check_for_upgrade == HA_ADMIN_NEEDS_ALTER) > + check_old_types == HA_ADMIN_NEEDS_UPGRADE || > + check_for_upgrade == HA_ADMIN_NEEDS_ALTER || > + check_for_upgrade == HA_ADMIN_NEEDS_UPGRADE) eh. So old code was returning HA_ADMIN_NEEDS_ALTER actually quite intentionally. REPAIR TABLE was automatically switching to ALTER if these checks were returning HA_ADMIN_NEEDS_ALTER. So it didn't matter if the message confusingly said "Please use REPAIR", because it would've been ALTER internally anyway. Meaning, I suspect, that the old code was fine and you didn't need to change `case HA_ADMIN_NEEDS_ALTER` and didn't need to replace `return HA_ADMIN_NEEDS_UPGRADE` with HA_ADMIN_NEEDS_ALTER. All you needed to do was to return HA_ADMIN_NEEDS_ALTER from the check_long_hash_compatibility() method and the rest would've likely worked automatically. Or at least with much smaller changes. > { > /* We use extra_open_options to be able to open crashed tables */ > thd->open_options|= extra_open_options; > - result_code= admin_recreate_table(thd, table); > + result_code= admin_recreate_table(thd, table) ? HA_ADMIN_FAILED : > + HA_ADMIN_OK; good catch > thd->open_options&= ~extra_open_options; > goto send_result; > } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 0bef50e50b5: MDEV-20501: Assertion `maybe_null || !null_value' failed in Item_func_round::date_op
by Sergei Golubchik 14 Jan '23

14 Jan '23

Hi, Sergei, On Jan 13, Sergei Petrunia wrote: > revision-id: 0bef50e50b5 (mariadb-10.4.27-33-g0bef50e50b5) > parent(s): 5db970fc760 > author: Sergei Petrunia > committer: Sergei Petrunia > timestamp: 2023-01-04 16:50:12 +0300 > message: > > MDEV-20501: Assertion `maybe_null || !null_value' failed in Item_func_round::date_op > > When the optimizer finds a constant (or system) table $TBL which is empty > or has no matching row, it would set table->null_row=true. This is done > even for tables with table->maybe_null==0. > > Then, it would proceed to perform query optimization phases (what for?) > which will attempt to evaluate Item expressions referring to $TBL. > > Eventually some Item expression will get the value of $TBL.not_null_field, > get SQL NULL and fail an assertion. > > Fixed by not performing any query optimization steps after we've got > constant/empty tables with no matching rows. > > Test result changes contain a lot of changes like > - ... Impossible WHERE noticed after reading const tables > + ... no matching row in const table > > as well as other changes caused by slightly-different processing of > the special case of empty constant tables. > > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index 6a441c5047b..57ffd58f8b4 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -2382,6 +2382,9 @@ int JOIN::optimize_stage2() > if (subq_exit_fl) > goto setup_subq_exit; > > + if (zero_result_cause) > + goto setup_subq_exit; In many other cases it's, like zero_result_cause= "No matching min/max row"; subq_exit_fl= true; that is, subq_exit_fl is set whenever zero_result_cause is set. Meaning, perhaps you should do the same in your new code below and then you wouldn't need those lines you've added above? > if (unlikely(thd->check_killed())) > DBUG_RETURN(1); > > @@ -5392,7 +5398,32 @@ make_join_statistics(JOIN *join, List<TABLE_LIST> &tables_list, > } > } > } while (ref_changed); > - > + > + /* > + ConstRowNotFoundShortcut-1: > + Some constant/system tables have mo matching rows. This means that > + the join operation output will be empty. > + Short-cut further optimization steps. > + Note that some query plan steps will still be performed to handle > + implicit grouping, join result setup, etc. > + See also: ConstRowNotFoundShortcut-{2,3}. > + > + Do not do this if we're optimizing for some UNION's fake_select_lex. > + We might be running UglySubqueryReoptimization (grep for name) and > + in this case constant tables are not reliable. > + */ > + if ((join->const_table_map & ~found_const_table_map) && > + !(join->select_lex->master_unit() && > + join->select_lex->master_unit()->fake_select_lex == join->select_lex)) why wouldn't you check for `describe` here? "reoptimization" hack is only needed for EXPLAIN. > + { > + join->zero_result_cause= "no matching row in const table"; > + join->table_count=0; > + join->const_tables= 0; > + join->join_tab= NULL; > + > + DBUG_RETURN(0); > + } > + > join->sort_by_table= get_sort_by_table(join->order, join->group_list, > join->select_lex->leaf_tables, > join->const_table_map); Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 5dcde8f6523: MDEV-27653 long uniques don't work with unicode collations
by Sergei Golubchik 06 Jan '23

06 Jan '23

Hi, Alexander, Looks good. A couple of comments, see below On Jan 06, Alexander Barkov wrote: > revision-id: 5dcde8f6523 (mariadb-10.4.26-64-g5dcde8f6523) > parent(s): ce443c85547 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2022-10-28 15:37:44 +0400 > message: > > MDEV-27653 long uniques don't work with unicode collations > > diff --git a/mysql-test/main/ctype_utf8.test b/mysql-test/main/ctype_utf8.test > index cc61c2ae0fe..e2d4e4ab906 100644 > --- a/mysql-test/main/ctype_utf8.test > +++ b/mysql-test/main/ctype_utf8.test > @@ -2310,3 +2310,133 @@ VALUES (_latin1 0xDF) UNION VALUES(_utf8'a' COLLATE utf8_bin); > --echo # > --echo # End of 10.3 tests > --echo # > + > + > +--echo # > +--echo # Start of 10.4 tests > +--echo # > + > +--echo # > +--echo # MDEV-27653 long uniques don't work with unicode collations > +--echo # > + > +SET NAMES utf8mb3; > + > +# CHAR > + > +CREATE TABLE t1 ( > + a CHAR(30) COLLATE utf8mb3_general_ci, > + UNIQUE KEY(a) USING HASH > +); > +SHOW CREATE TABLE t1; > +INSERT INTO t1 VALUES ('a'); > +--error ER_DUP_ENTRY > +INSERT INTO t1 VALUES ('ä'); > +SELECT * FROM t1; > +DROP TABLE t1; > + > +CREATE TABLE t1 ( > + a CHAR(30) COLLATE utf8mb3_general_ci, > + UNIQUE KEY(a(10)) USING HASH > +); > +SHOW CREATE TABLE t1; > +INSERT INTO t1 VALUES ('a'); > +--error ER_DUP_ENTRY > +INSERT INTO t1 VALUES ('ä'); > +SELECT * FROM t1; > +DROP TABLE t1; > + > + > +# VARCHAR > + > +CREATE TABLE t1 ( > + a VARCHAR(30) COLLATE utf8mb3_general_ci, > + UNIQUE KEY(a) USING HASH > +); > +SHOW CREATE TABLE t1; > +INSERT INTO t1 VALUES ('a'); > +--error ER_DUP_ENTRY > +INSERT INTO t1 VALUES ('ä'); > +SELECT * FROM t1; > +DROP TABLE t1; > + > +CREATE TABLE t1 ( > + a VARCHAR(30) COLLATE utf8mb3_general_ci, > + UNIQUE KEY(a(10)) USING HASH > +); > +SHOW CREATE TABLE t1; > +INSERT INTO t1 VALUES ('a'); > +--error ER_DUP_ENTRY > +INSERT INTO t1 VALUES ('ä'); > +SELECT * FROM t1; > +DROP TABLE t1; > + > + > +# TEXT > + > +CREATE TABLE t1 (a TEXT COLLATE utf8mb3_general_ci UNIQUE); > +SHOW CREATE TABLE t1; > +INSERT INTO t1 VALUES ('a'); > +--error ER_DUP_ENTRY > +INSERT INTO t1 VALUES ('ä'); > +SELECT * FROM t1; > +DROP TABLE t1; > + > +CREATE TABLE t1 ( > + a LONGTEXT COLLATE utf8mb3_general_ci, > + UNIQUE KEY(a(10)) USING HASH > +); > +SHOW CREATE TABLE t1; > +INSERT INTO t1 VALUES ('a'); > +--error ER_DUP_ENTRY > +INSERT INTO t1 VALUES ('ä'); > +SELECT * FROM t1; > +DROP TABLE t1; > + > + > +# Testing upgrade: > +# Prior to MDEV-27653, the UNIQUE HASH function errorneously > +# took into account string octet length. > +# Old tables should still open and work, but with a wrong results. > + > +copy_file std_data/mysql_upgrade/mdev27653_100422_text.frm $MYSQLD_DATADIR/test/t1.frm; > +copy_file std_data/mysql_upgrade/mdev27653_100422_text.MYD $MYSQLD_DATADIR/test/t1.MYD; > +copy_file std_data/mysql_upgrade/mdev27653_100422_text.MYI $MYSQLD_DATADIR/test/t1.MYI; > +SHOW CREATE TABLE t1; > +SELECT a, OCTET_LENGTH(a) FROM t1 ORDER BY BINARY a; > +CHECK TABLE t1; > + > +# There is already a one byte value 'a' in the table > +--error ER_DUP_ENTRY > +INSERT INTO t1 VALUES ('A'); > + > +# There is already a two-byte value 'ä' in the table > +--error ER_DUP_ENTRY > +INSERT INTO t1 VALUES ('Ä'); > + > +# There were no three-byte values in the table so far. > +# The below value violates UNIQUE, but it gets inserted. > +# This is wrong but expected for a pre-MDEV-27653 table. > +INSERT INTO t1 VALUES ('Ấ'); > +SELECT a, OCTET_LENGTH(a) FROM t1 ORDER BY BINARY a; > +CHECK TABLE t1; > + > +# ALTER FORCE fails: it tries to rebuild the table > +# with a correct UNIQUE HASH function, but there are duplicates! > +--error ER_DUP_ENTRY > +ALTER TABLE t1 FORCE; please, try ALTER IGNORE TABLE too > + > +# Let's remove all duplicate values, so only the one-byte 'a' stays. > +# ALTER..FORCE should work after that. > +DELETE FROM t1 WHERE OCTET_LENGTH(a)>1; > +ALTER TABLE t1 FORCE; > + > +# Make sure that 'a' and 'ä' cannot co-exists any more, > +# because the table was recreated with a correct UNIQUE HASH function. > +--error ER_DUP_ENTRY > +INSERT INTO t1 VALUES ('ä'); > +DROP TABLE t1; > + > +--echo # > +--echo # End of 10.4 tests > +--echo # > diff --git a/sql/table.cc b/sql/table.cc > index b9260853381..97f71284005 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -1267,7 +1286,11 @@ bool parse_vcol_defs(THD *thd, MEM_ROOT *mem_root, TABLE *table, > list_item= new (mem_root) Item_field(thd, keypart->field); > field_list->push_back(list_item, mem_root); > } > - Item_func_hash *hash_item= new(mem_root)Item_func_hash(thd, *field_list); > + > + Item_func_hash *hash_item= make_unique_hash_func(thd, mem_root, > + table->s->mysql_version, > + field_list); would be good to fix CHECK ... FOR UPGRADE too. > + > Virtual_column_info *v= new (mem_root) Virtual_column_info(); > field->vcol_info= v; > field->vcol_info->expr= hash_item; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 92ff948d021: MDEV-29231 View returns wrong value with SQL_MODE 'NO_BACKSLASH_ESCAPES'
by Sergei Golubchik 06 Jan '23

06 Jan '23

Hi, Oleksandr, On Jan 06, Oleksandr Byelkin wrote: > revision-id: 92ff948d021 (mariadb-10.3.37-53-g92ff948d021) > parent(s): 180b2bcd538 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2022-12-05 20:42:03 +0100 > message: > > MDEV-29231 View returns wrong value with SQL_MODE 'NO_BACKSLASH_ESCAPES' > > 1. make view printing sql_mode independent > 1.1. Print always current escape character > 1.2. Allow escape character usage independent of sql_mode > 2. Add workaround for parsing "like 3 in (0,1) escape 3" problem > (alwaysuse parances in case of printing ESCAPE) > > diff --git a/mysql-test/main/ctype_cp1250_ch.result b/mysql-test/main/ctype_cp1250_ch.result > index b0aa4cff382..88175ae5013 100644 > --- a/mysql-test/main/ctype_cp1250_ch.result > +++ b/mysql-test/main/ctype_cp1250_ch.result > @@ -129,7 +129,7 @@ EXPLAIN EXTENDED SELECT * FROM t1 WHERE CONCAT(c1)='a' AND CONCAT(c1) LIKE 'a '; > id select_type table type possible_keys key key_len ref rows filtered Extra > 1 SIMPLE t1 ALL NULL NULL NULL NULL 2 100.00 Using where > Warnings: > -Note 1003 select `test`.`t1`.`c1` AS `c1` from `test`.`t1` where concat(`test`.`t1`.`c1`) = 'a' and concat(`test`.`t1`.`c1`) like 'a ' > +Note 1003 select `test`.`t1`.`c1` AS `c1` from `test`.`t1` where concat(`test`.`t1`.`c1`) = 'a' and concat(`test`.`t1`.`c1`) like ('a ') escape '\\' I don't think there's a need to print the escape clause here, it only makes the output less readable. > diff --git a/mysql-test/main/default.result b/mysql-test/main/default.result > index c9ac96367aa..5c72b894bf6 100644 > --- a/mysql-test/main/default.result > +++ b/mysql-test/main/default.result > @@ -1936,7 +1936,7 @@ CREATE OR REPLACE TABLE t1 ( col INT DEFAULT ( 1 LIKE ( NOW() BETWEEN '2000-01-0 > SHOW CREATE TABLE t1; > Table Create Table > t1 CREATE TABLE `t1` ( > - `col` int(11) DEFAULT (1 like (current_timestamp() between '2000-01-01' and '2012-12-12')) > + `col` int(11) DEFAULT (1 like (current_timestamp() between '2000-01-01' and '2012-12-12') escape '\\') So, there're basically two approaches to a fix and we're inconsistently using both for different items. 1. one can always print expressions in the environment-independent way - create longer less readable text - not all items support it, new syntax might be needed (cannot emulate NO_BACKSLASH_ESCAPES with the ESCAPE clause) 2. one can set the environment (sql_mode) to the same hard-coded value before parsing views and vcols - one cannot dump the view/table definition in one sql_mode and load in another + this isn't normally a problem, as mysqldump also uses a fixed sql_mode - some item properties (what are they?) might be only set at fix_fields, fixing sql_mode during parsing won't help there Could we try to agree to one approach and use it consistently for all items from now on? > diff --git a/mysql-test/main/parser.result b/mysql-test/main/parser.result > index 0bb4e82c8b8..79ec9f539eb 100644 > --- a/mysql-test/main/parser.result > +++ b/mysql-test/main/parser.result > @@ -1325,11 +1325,11 @@ select 1 between 2 in (3,4) and 5 AS `1 between (2 in (3,4)) and 5` > create or replace view v1 as select 1 between (2 like 3) and 4; > Select view_definition from information_schema.views where table_schema='test' and table_name='v1'; > view_definition > -select 1 between 2 like 3 and 4 AS `1 between (2 like 3) and 4` > +select 1 between 2 like (3) escape '\\' and 4 AS `1 between (2 like 3) and 4` > create or replace view v1 as select 1 not between (2 like 3) and 4; > Select view_definition from information_schema.views where table_schema='test' and table_name='v1'; > view_definition > -select 1 not between 2 like 3 and 4 AS `1 not between (2 like 3) and 4` > +select 1 not between 2 like (3) escape '\\' and 4 AS `1 not between (2 like 3) and 4` > drop view v1; > # > # Start of 10.2 tests > diff --git a/mysql-test/main/precedence.result b/mysql-test/main/precedence.result > index fc6579651b4..ce26ca2acf2 100644 > --- a/mysql-test/main/precedence.result > +++ b/mysql-test/main/precedence.result > @@ -113,7 +113,7 @@ NOT 2 != 3 NOT (2 != 3) (NOT 2) != 3 > create or replace view v1 as select NOT 2 LIKE 3, NOT (2 LIKE 3), (NOT 2) LIKE 3; > Select view_definition from information_schema.views where table_schema='test' and table_name='v1'; > view_definition > -select 2 not like 3 AS `NOT 2 LIKE 3`,2 not like 3 AS `NOT (2 LIKE 3)`,!2 like 3 AS `(NOT 2) LIKE 3` > +select 2 not like (3) escape '\\' AS `NOT 2 LIKE 3`,2 not like (3) escape '\\' AS `NOT (2 LIKE 3)`,!2 like (3) escape '\\' AS `(NOT 2) LIKE 3` > select NOT 2 LIKE 3, NOT (2 LIKE 3), (NOT 2) LIKE 3 union select * from v1; > NOT 2 LIKE 3 NOT (2 LIKE 3) (NOT 2) LIKE 3 > 1 1 0 > diff --git a/mysql-test/main/precedence_bugs.result b/mysql-test/main/precedence_bugs.result > index 723ab823b48..9654e3c31db 100644 > --- a/mysql-test/main/precedence_bugs.result > +++ b/mysql-test/main/precedence_bugs.result > @@ -54,7 +54,7 @@ drop table t1; > create view v1 as select 1 like (now() between '2000-01-01' and '2012-12-12' ); > show create view v1; > View v1 > -Create View CREATE ALGORITHM=UNDEFINED DEFINER=`root`@`localhost` SQL SECURITY DEFINER VIEW `v1` AS select 1 like (current_timestamp() between '2000-01-01' and '2012-12-12') AS `1 like (now() between '2000-01-01' and '2012-12-12' )` > +Create View CREATE ALGORITHM=UNDEFINED DEFINER=`root`@`localhost` SQL SECURITY DEFINER VIEW `v1` AS select 1 like (current_timestamp() between '2000-01-01' and '2012-12-12') escape '\\' AS `1 like (now() between '2000-01-01' and '2012-12-12' )` > character_set_client latin1 > collation_connection latin1_swedish_ci > drop view v1; if you'll always print ESCAPE clause, it'll make many tests in parser.test, precedence.test and precedence_bugs.test meaningless. Don't just update result files, please, remove tests that no longer make sense. Or rewrite them, when possible (but in precedence.test it's not). > diff --git a/sql/item_cmpfunc.cc b/sql/item_cmpfunc.cc > index b7b0c981c2d..702689a7496 100644 > --- a/sql/item_cmpfunc.cc > +++ b/sql/item_cmpfunc.cc > @@ -5262,10 +5262,15 @@ void Item_func_like::print(String *str, enum_query_type query_type) > str->append(STRING_WITH_LEN(" not ")); > str->append(func_name()); > str->append(' '); > - if (escape_used_in_parsing) > + if (escape_used_in_parsing || (query_type | QT_VIEW_INTERNAL)) > { > - args[1]->print_parenthesised(str, query_type, precedence()); > - str->append(STRING_WITH_LEN(" escape ")); > + /* > + Explicit parencies is workaround against parser bug which üprevent > + patsing "like 3 in (0,1) escape 3" correctly > + */ > + str->append('('); > + args[1]->print(str, query_type); > + str->append(STRING_WITH_LEN(") escape ")); no, this is wrong. Don't just force parentheses there. Instead, fix the precedence, so that the parentheses would be printed automatically when needed. > escape_item->print_parenthesised(str, query_type, higher_precedence()); > } > else > @@ -5390,10 +5395,7 @@ bool fix_escape_item(THD *thd, Item *escape_item, String *tmp_str, > if (escape_str) > { > const char *escape_str_ptr= escape_str->ptr(); > - if (escape_used_in_parsing && ( > - (((thd->variables.sql_mode & MODE_NO_BACKSLASH_ESCAPES) && > - escape_str->numchars() != 1) || > - escape_str->numchars() > 1))) > + if (escape_used_in_parsing && escape_str->numchars() > 1) I still think that ESCAPE '' doesn't mean "no escape character". I tried and in my tests it didn't. > { > my_error(ER_WRONG_ARGUMENTS,MYF(0),"ESCAPE"); > return TRUE; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 6ff49e48e4f: MDEV-26161: fix of view protocol
by Sergei Golubchik 06 Jan '23

06 Jan '23

Hi, Oleksandr, On Jan 06, Oleksandr Byelkin wrote: > revision-id: 6ff49e48e4f (mariadb-10.3.37-54-g6ff49e48e4f) > parent(s): 92ff948d021 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2022-12-06 10:22:06 +0100 > message: > > MDEV-26161: fix of view protocol could you please amend the commit comment to explain why the statement produces a different error in a view protocol? > diff --git a/mysql-test/main/gis.result b/mysql-test/main/gis.result > index 358be520b06..37613bc71a0 100644 > --- a/mysql-test/main/gis.result > +++ b/mysql-test/main/gis.result > @@ -4980,11 +4980,11 @@ ERROR HY000: Illegal parameter data type geometry for operation 'is_used_lock' > # MDEV-26161 crash in Gis_point::calculate_haversine > # > select st_distance_sphere(x'01030000000400000004000000000000', multipoint(point(124,204)), 10); > -ERROR 22003: Cannot get geometry object from data you send to the GEOMETRY field > +Got one of the listed errors > select st_distance_sphere(x'010300000004000000040000', multipoint(point(124,204)), 10); > -ERROR 22003: Cannot get geometry object from data you send to the GEOMETRY field > +Got one of the listed errors > select st_distance_sphere(x'010300000001000000040000', multipoint(point(124,204)), 10); > -ERROR 22003: Cannot get geometry object from data you send to the GEOMETRY field > +Got one of the listed errors > # > # End of 10.3 tests > # > diff --git a/mysql-test/main/gis.test b/mysql-test/main/gis.test > index 716fab9bfeb..07b207dcdc9 100644 > --- a/mysql-test/main/gis.test > +++ b/mysql-test/main/gis.test > @@ -3093,11 +3093,11 @@ SELECT IS_USED_LOCK(POINT(1,1)); > --echo # > --echo # MDEV-26161 crash in Gis_point::calculate_haversine > --echo # > ---error ER_CANT_CREATE_GEOMETRY_OBJECT > +--error ER_CANT_CREATE_GEOMETRY_OBJECT,ER_INTERNAL_ERROR > select st_distance_sphere(x'01030000000400000004000000000000', multipoint(point(124,204)), 10); > ---error ER_CANT_CREATE_GEOMETRY_OBJECT > +--error ER_CANT_CREATE_GEOMETRY_OBJECT,ER_INTERNAL_ERROR > select st_distance_sphere(x'010300000004000000040000', multipoint(point(124,204)), 10); > ---error ER_CANT_CREATE_GEOMETRY_OBJECT > +--error ER_CANT_CREATE_GEOMETRY_OBJECT,ER_INTERNAL_ERROR > select st_distance_sphere(x'010300000001000000040000', multipoint(point(124,204)), 10); > > --echo # > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] MDEV-28602 Wrong result with outer join, merged derived table and view
by Sergey Petrunia 04 Jan '23

04 Jan '23

Hi Rex, Please find below input for the commit. (I've actually committed an alternative fix now, but before doing that I wrote all the trivial comments below so I'm sending these anyway in order to show what is expected of a patch. I'm also including the rationale for creating an alternative patch) > commit 75af3195482d3da277825663d1150c0dfc55420a > Author: Rex <rex.johnston(a)mariadb.com> > Date: Fri Dec 23 05:28:59 2022 +1200 > > MDEV-28602 Wrong result with outer join, merged derived table and view > > const item reference being mishandled in outer to inner join on > filling in null values. > First, fixVersion. It is currently set to 10.11, but affectsVersion starts from 10.3 I think the bug and the fix are applicable for all versions. Second: commit comment. This is not descriptive enough: > const item reference being mishandled in outer to inner join on > filling in null values. Something like this would be better: <commit-comment> A LEFT JOIN with a constant as a column of the inner table produced wrong query result if the optimizer had to write the inner table column into a temp table: SELECT ... FROM (SELECT /*non-mergeable select*/ FROM t1 LEFT JOIN (SELECT 'Y' as Val) t2 ON ...) Fixed this by: 1. Making Item_ref::save_in_field() call check_null_ref() to see if the referred table has a NULL-complemented row. 2. In order to do #1, moved null_ref_table() and check_null_ref() from Item_direct_view_ref to Item_ref. </commit-comment> Note: it starts with a siccint problem description, which is followed by a description of the fix. > diff --git a/sql/sql_join_cache.cc b/sql/sql_join_cache.cc > index 620c52a3f40..65bedc3d337 100644 > --- a/sql/sql_join_cache.cc > +++ b/sql/sql_join_cache.cc > @@ -2622,7 +2622,7 @@ enum_nested_loop_state JOIN_CACHE::join_null_complements(bool skip_last) > get_record(); > /* The outer row is complemented by nulls for each inner table */ > restore_record(join_tab->table, s->default_values); > - mark_as_null_row(join_tab->table); > + mark_as_null_row(join_tab->table); In the future please make sure patches do not contain unrelated meaningless changes like this one. > rc= generate_full_extensions(get_curr_rec()); > if (rc != NESTED_LOOP_OK && rc != NESTED_LOOP_NO_MORE_ROWS) > goto finish; > diff --git a/mysql-test/main/merge.test b/mysql-test/main/merge.test > index 0485f3ed1c3..05b05287e1b 100644 > --- a/mysql-test/main/merge.test > +++ b/mysql-test/main/merge.test Why is the patch in the merge.test ? merge.test at the top says clearly "Test of MERGE TABLES". That is, this is a test for tables with ENGINE=MERGE, a feature not related to this bug. The right test files for this bug would be main/join_outer.test and main/derived.test, I would prefer the first one. > @@ -2886,3 +2886,50 @@ drop table tm, t; > --echo # > --echo # End of 10.8 tests > --echo # > + > +--echo # > +--echo # MDEV-28602 Wrong result with outer join, merged derived table and view > +--echo # > + > +drop table if exists t1, t2; ^^^ This is not needed as there is a DROP TABLE right above > +create table t1 ( > + Election int(10) unsigned NOT NULL > +); > + > +insert into t1 (Election) values (1); > + +create table t2 ( + VoteID int(10), + ElectionID int(10), + UserID int(10) +); + +insert into t2 (ElectionID, UserID) values (2, 30), (3, 30); +# INSERT INTO t2 (ElectionID, UserID) VALUES (1, 30); ^^ No need to have commented-out lines in the commit. +drop view if exists v1; ^^^ same as above: this is not needed. +create view v1 as select * from t1 + left join ( select 'Y' AS Voted, ElectionID from t2 ) AS T + on T.ElectionID = t1.Election +limit 9; +# limit X causes merge algorithm select as opposed to temp table +select * from v1; ... > diff --git a/sql/item.h b/sql/item.h > index 2d598546b91..8684477b558 100644 > --- a/sql/item.h > +++ b/sql/item.h > @@ -5531,18 +5531,42 @@ class Item_sp > > class Item_ref :public Item_ident > { > +public: > + > +#define NO_NULL_TABLE (reinterpret_cast<TABLE *>(0x1)) > + > + void set_null_ref_table() > + { > + null_ref_table= NO_NULL_TABLE; > + } > + > + bool check_null_ref() This works, but looks like a change that is 1. Too big 2. Looks like a partial implementation I'm looking at methods of Item_direct_view_ref: - save_val() - save_org_in_field() - save_in_result_field() - val_real() and other val_XXX() and they all follow the same pattern: if (check_null_ref()) produce SQL NULL; else Call Item_direct_ref::same_method(); This suggests we could add Item_direct_view_ref::save_in_field() and use the same approach. Conversely, if we move check_null_ref() to be in Item_ref, we will end up with: - check_null_ref() is in Item_ref - methods that make use of it are in Item_direct_view_ref() - methods of Item_ref() do not make check_null_ref() check, except for save_in_field() which does it. I think the former looks more logical than the latter. I've made the patch for it and I'll ask Sanja (the Item_ref/views expert) to check it. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

1 0

Re: [Maria-developers] 6a8268b7893: MDEV-28915: mysql_upgrade fails due to old_mode="", with "Cannot load from mysql.proc. The table is probably corrupted"
by Sergei Golubchik 28 Dec '22

28 Dec '22

Hi, Rucha, On Dec 28, Rucha Deodhar wrote: > revision-id: 6a8268b7893 (mariadb-10.6.11-62-g6a8268b7893) > parent(s): 4ca5a0ec98d > author: Rucha Deodhar > committer: Rucha Deodhar > timestamp: 2022-12-15 17:45:25 +0530 > message: > > MDEV-28915: mysql_upgrade fails due to old_mode="", with "Cannot load from > mysql.proc. The table is probably corrupted" > > Analysis: When mysql_upgrade runs statements for upgrade, characterset is > converted to utf8mb4 because server starts with old_mode that interprets > utf8 to utf8mb4, but mysql.proc table has "utf8mb3" as hardcoded, so > it crashes with corrupted table. > > Fix: Changed Table_check_intact::check() definition to allow both > utf8mb3 and utf8mb4 by checking prefix and changing the upgrade scripts > to explicitly use utf8mb3 > > diff --git a/mysql-test/main/mysql_upgrade-28915.opt b/mysql-test/main/mysql_upgrade-28915.opt > new file mode 100644 > index 00000000000..0be5567a64d > --- /dev/null > +++ b/mysql-test/main/mysql_upgrade-28915.opt > @@ -0,0 +1 @@ > +--old-mode=NO_DUP_KEY_WARNINGS_WITH_IGNORE it's a bit confusing, try to use --old-mode= that is, empty old_mode, otherwise one starts thinking what this bug has to do with NO_DUP_KEY_WARNINGS_WITH_IGNORE. > diff --git a/mysql-test/main/mysql_upgrade-28915.test b/mysql-test/main/mysql_upgrade-28915.test > new file mode 100644 > index 00000000000..61b1fc263cb > --- /dev/null > +++ b/mysql-test/main/mysql_upgrade-28915.test > @@ -0,0 +1,78 @@ > +--source include/mysql_upgrade_preparation.inc > + > +let $MYSQLD_DATADIR= `select @@datadir`; > + > +--echo # > +--echo # Beginning of 10.6 test > +--echo # > +--echo # MDEV-28915: mysql_upgrade fails due to old_mode="", with "Cannot load > +--echo # from mysql.proc. The table is probably corrupted" > + > +--echo # Show that tables are created with utf8mb3 even without UTF8_IS_UTF8MB3 (see the .opt file) I don't think you've shown it. mtr does not recreate system tables for every test. You need to force it to re-bootstrap with one of the options that do it, for example, --innodb-page-size=16k (search mtr for "rebootstrap" for details). But even that won't help, because it'll only add "rebootstrap" options to the server's command line when bootstrapping. May be a combination of .cnf and .opt can do what you want, I didn't try. > + > +SHOW CREATE TABLE mysql.proc; > +SHOW CREATE TABLE mysql.event; > + > +--exec $MYSQL_UPGRADE --force 2>&1 > + > +--remove_file $MYSQLD_DATADIR/mysql_upgrade_info > + > +SHOW CREATE TABLE mysql.proc; > +SHOW CREATE TABLE mysql.event; > + > +--echo # Emulate that tables were created with utf8mb4 by an older version > + > +ALTER TABLE mysql.proc MODIFY db CHAR(64) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL, > + MODIFY name CHAR(64) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci NOT NULL, > + MODIFY specific_name CHAR(64) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL, > + MODIFY definer VARCHAR(64) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL, > + MODIFY comment TEXT CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL, > + MODIFY character_set_client CHAR(32) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL, > + MODIFY collation_connection CHAR(32) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL, > + MODIFY db_collation CHAR(32) CHARACTER SET utf8mb4 COLLATE utf8mb4_general_ci DEFAULT NULL; > + > +ALTER TABLE mysql.event MODIFY db CHAR(64) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL DEFAULT '', > + MODIFY name CHAR(64) CHARACTER SET utf8mb4 NOT NULL DEFAULT '', > + MODIFY definer VARCHAR(384) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL DEFAULT '', > + MODIFY comment CHAR(64) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin NOT NULL DEFAULT '', > + MODIFY character_set_client CHAR(32) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL, > + MODIFY collation_connection CHAR(32) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL, > + MODIFY db_collation CHAR(32) CHARACTER SET utf8mb4 COLLATE utf8mb4_bin DEFAULT NULL; > + > +--echo # mysql_upgrade changes columns from utf8mb4 to utf8mb3 > + > +--exec $MYSQL_UPGRADE --force 2>&1 > +--remove_file $MYSQLD_DATADIR/mysql_upgrade_info > + > +--vertical_results > + > +SHOW CREATE TABLE mysql.proc; > +SHOW CREATE TABLE mysql.event; > + > +CREATE TABLE t1 (id1 INT, val1 VARCHAR(5)); > + > +DELIMITER |; > + > +CREATE PROCEDURE sp1 () > + BEGIN > + SELECT val1 FROM t1; > + END| You've made changes in sp.cc and event_db_repository.cc to ensure that these tables are usable even with utf8mb4 charset. But you don't test it if it works. You create a procedure and an event here, *after* tables were converted back to utf8mb3. you likely want to do it earlier, before mysql_upgrade, just after ALTER TABLE. > + > +DELIMITER ;| > + > +SELECT name, body_utf8, body FROM mysql.proc WHERE name like 'sp1'; > +CALL sp1(); > +SELECT name, body_utf8, body FROM mysql.proc WHERE name like 'sp1'; > + > +SET GLOBAL event_scheduler=ON; > + > +SELECT name, body_utf8, body FROM mysql.event; > +CREATE EVENT ev1 ON SCHEDULE EVERY 1 SECOND DO INSERT INTO t1 VALUES (1, 'abc'); > +SELECT name, body_utf8, body FROM mysql.event; > + > +SET GLOBAL event_scheduler=OFF; > +DROP EVENT ev1; > +DROP PROCEDURE sp1; > +DROP TABLE t1; > + > +--echo # end of 10.6 test Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 4ddf606debf: MDEV-30151 parse error 1=2 not between/in
by Sergei Golubchik 28 Dec '22

28 Dec '22

Hi, Alexander, I'm not sure I understand everything you did there, so questions below On Dec 28, Alexander Barkov wrote: > revision-id: 4ddf606debf (mariadb-10.3.37-59-g4ddf606debf) > parent(s): 8f30973234d > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2022-12-13 10:35:16 +0400 > message: > > MDEV-30151 parse error 1=2 not between/in > > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index 1f6485dac6a..ca614ff63e9 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -1687,7 +1687,8 @@ bool my_yyoverflow(short **a, YYSTYPE **b, size_t *yystacksize); > > %left PREC_BELOW_NOT > > -%nonassoc NOT_SYM > +/* The precendence of boolean NOT is in fact here. See the comment below. */ > + > %left '=' EQUAL_SYM GE '>' LE '<' NE > %nonassoc IS > %right BETWEEN_SYM > @@ -1699,6 +1700,24 @@ bool my_yyoverflow(short **a, YYSTYPE **b, size_t *yystacksize); > %left '*' '/' '%' DIV_SYM MOD_SYM > %left '^' > %left MYSQL_CONCAT_SYM > +/* > + Boolean negation has a special branch in "expr" starting with NOT_SYM. > + The precedence of logical negation is determined by the grammar itself > + (without using Bison terminal symbol precedence) in this order > + - Boolean factor (i.e. logical AND) > + - Boolean NOT > + - Boolean test (such as '=', IS NULL, IS TRUE) > + > + But we also need a precedence for NOT_SYM in other contexts, > + to shift (without reduce) in these cases: > + predicate <here> NOT IN ... > + predicate <here> NOT BETWEEN ... > + predicate <here> NOT LIKE ... > + predicate <here> NOT REGEXP ... > + If the precedence of NOT_SYM was low, it would reduce immediately > + after scanning "predicate" and then produce a syntax error on "NOT". > +*/ > +%nonassoc NOT_SYM > %nonassoc NEG '~' NOT2_SYM BINARY > %nonassoc COLLATE_SYM > > @@ -1938,6 +1957,7 @@ bool my_yyoverflow(short **a, YYSTYPE **b, size_t *yystacksize); > literal insert_ident order_ident temporal_literal > simple_ident expr sum_expr in_sum_expr > variable variable_aux > + boolean_test boolean_factor > predicate bit_expr parenthesized_expr > table_wild simple_expr column_default_non_parenthesized_expr udf_expr > primary_expr string_factor_expr mysql_concatenation_expr > @@ -9840,79 +9860,87 @@ expr: > MYSQL_YYABORT; > } > } > - | NOT_SYM expr %prec NOT_SYM > + | boolean_factor what does that achieve? > + ; > + > +boolean_factor: > + NOT_SYM boolean_factor > { > $$= negate_expression(thd, $2); > if (unlikely($$ == NULL)) > MYSQL_YYABORT; > } > - | expr IS TRUE_SYM %prec IS > + | boolean_test %prec PREC_BELOW_NOT > + ; > + > +boolean_test: > + boolean_test IS TRUE_SYM %prec IS > { > $$= new (thd->mem_root) Item_func_istrue(thd, $1); > if (unlikely($$ == NULL)) > MYSQL_YYABORT; > } > - | expr IS not TRUE_SYM %prec IS > + | boolean_test IS not TRUE_SYM %prec IS > { > $$= new (thd->mem_root) Item_func_isnottrue(thd, $1); > if (unlikely($$ == NULL)) > MYSQL_YYABORT; > } > - | expr IS FALSE_SYM %prec IS > + | boolean_test IS FALSE_SYM %prec IS > { > $$= new (thd->mem_root) Item_func_isfalse(thd, $1); > if (unlikely($$ == NULL)) > MYSQL_YYABORT; > } > - | expr IS not FALSE_SYM %prec IS > + | boolean_test IS not FALSE_SYM %prec IS > { > $$= new (thd->mem_root) Item_func_isnotfalse(thd, $1); > if (unlikely($$ == NULL)) > MYSQL_YYABORT; > } > - | expr IS UNKNOWN_SYM %prec IS > + | boolean_test IS UNKNOWN_SYM %prec IS > { > $$= new (thd->mem_root) Item_func_isnull(thd, $1); > if (unlikely($$ == NULL)) > MYSQL_YYABORT; > } > - | expr IS not UNKNOWN_SYM %prec IS > + | boolean_test IS not UNKNOWN_SYM %prec IS > { > $$= new (thd->mem_root) Item_func_isnotnull(thd, $1); > if (unlikely($$ == NULL)) > MYSQL_YYABORT; > } > - | expr IS NULL_SYM %prec PREC_BELOW_NOT > + | boolean_test IS NULL_SYM %prec PREC_BELOW_NOT can you try %prec IS here? I suspect that if this affects anything, it'll cause bugs > { > $$= new (thd->mem_root) Item_func_isnull(thd, $1); > if (unlikely($$ == NULL)) > MYSQL_YYABORT; > } > - | expr IS not NULL_SYM %prec IS > + | boolean_test IS not NULL_SYM %prec IS > { > $$= new (thd->mem_root) Item_func_isnotnull(thd, $1); > if (unlikely($$ == NULL)) > MYSQL_YYABORT; > } > - | expr EQUAL_SYM predicate %prec EQUAL_SYM > + | boolean_test EQUAL_SYM predicate %prec EQUAL_SYM > { > $$= new (thd->mem_root) Item_func_equal(thd, $1, $3); > if (unlikely($$ == NULL)) > MYSQL_YYABORT; > } > - | expr comp_op predicate %prec '=' > + | boolean_test comp_op predicate %prec '=' > { > $$= (*$2)(0)->create(thd, $1, $3); > if (unlikely($$ == NULL)) > MYSQL_YYABORT; > } > - | expr comp_op all_or_any '(' subselect ')' %prec '=' > + | boolean_test comp_op all_or_any '(' subselect ')' %prec '=' > { > $$= all_any_subquery_creator(thd, $1, $2, $3, $5); > if (unlikely($$ == NULL)) > MYSQL_YYABORT; > } > - | predicate > + | predicate %prec BETWEEN_SYM > ; > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 4d9f8a3c31e: MDEV-28669 addendum: additional tests and mtr changes
by Sergei Golubchik 27 Dec '22

27 Dec '22

Hi, Julius, On Dec 27, Julius Goryavsky wrote: > commit 4d9f8a3c31e > Author: Julius Goryavsky <julius.goryavsky(a)mariadb.com> > Date: Mon Dec 19 10:29:55 2022 +0100 > > diff --git a/mysql-test/mysql-test-run.pl b/mysql-test/mysql-test-run.pl > index 12afab4d28c..8df8198fc99 100755 > --- a/mysql-test/mysql-test-run.pl > +++ b/mysql-test/mysql-test-run.pl > @@ -313,7 +313,7 @@ my %mysqld_logs; > my $opt_debug_sync_timeout= 300; # Default timeout for WAIT_FOR actions. > my $warn_seconds = 60; > > -my $rebootstrap_re= '--innodb[-_](?:page[-_]size|checksum[-_]algorithm|undo[-_]tablespaces|log[-_]group[-_]home[-_]dir|data[-_]home[-_]dir)|data[-_]file[-_]path|force_rebootstrap'; > +my $rebootstrap_re= '--(?:loose[-_])?(?:innodb[-_](?:page[-_]size|file(?:[-_]format|per[-_]table)|compression[-_](?:default|algorithm)|checksum(?:s|[-_]algorithm)|undo[-_](?:directory|tablespaces)|(?:data|log[-_]group)[-_]home[-_]dir|buffer[-_]pool[-_]filename|data[-_]file[-_]path|default[-_](?:encryption[-_]key[-_]id|page[-_]encryption)|sys[-_]|(?:index|table)[-_]stats)|aria[-_]log[-_](?:dir[-_]path|file[-_]size))|force_rebootstrap'; 1. drop force_rebootstrap please 2. this code was already not very readable, but got fairly ridiculous now. Let's change the approach. E.g. put them all in a hash, like my %rebootstrap_check = map { $_ => 1 } qw( innodb-page-size innodb-file-format ... ); you'll need to normalize an option before looking it up in the hash > > sub testcase_timeout ($) { return $opt_testcase_timeout * 60; } > sub check_timeout ($) { return testcase_timeout($_[0]); } > @@ -2762,7 +2762,29 @@ sub mysql_server_start($) { > return; > } > > - my $datadir= $mysqld->value('datadir'); > + my $extra_opts= get_extra_opts($mysqld, $tinfo); > + > + # The test options can contain the --datadir parameter, but > + # the bootstrap function can only read datadir location from > + # a .cnf file. So we need to parse the options to get the > + # actual location of the data directory, considering both > + # the options and the .cnf file, and then store the resulting > + # value in a "$mysqld" hash - to use later, when we need to > + # know the actual location of the data directory: > + my $datadir=""; > + foreach my $opt ( @$extra_opts ) > + { > + if ($opt =~ /^--datadir=/) { > + $datadir = substr($opt, 10); > + last; strictly speaking, `last` is incorrect, there can be more than one --datadir > + } > + } > + # If datadir is not in the options, then read it from .cnf: > + if (! $datadir) { > + $datadir = $mysqld->value('datadir'); > + } > + $mysqld->{'datadir'} = $datadir; > + > if (not $opt_start_dirty) > { > > @@ -2785,17 +2807,59 @@ sub mysql_server_start($) { > } > } > > + # Run <tname>-master.sh > + if ($mysqld->option('#!run-master-sh') and > + defined $tinfo->{master_sh} and > + run_system('/bin/sh ' . $tinfo->{master_sh}) ) > + { > + $tinfo->{'comment'}= "Failed to execute '$tinfo->{master_sh}'"; > + return 1; > + } > + > + # Run <tname>-slave.sh > + if ($mysqld->option('#!run-slave-sh') and > + defined $tinfo->{slave_sh} and > + run_system('/bin/sh ' . $tinfo->{slave_sh})) > + { > + $tinfo->{'comment'}= "Failed to execute '$tinfo->{slave_sh}'"; > + return 1; > + } > + > my $mysqld_basedir= $mysqld->value('basedir'); > - my $extra_opts= get_extra_opts($mysqld, $tinfo); > > if ( $basedir eq $mysqld_basedir ) > { > if (! $opt_start_dirty) # If dirty, keep possibly grown system db > { > + # Find the name of the current section in the configuration > + # file and its suffix: > + my $section = $mysqld->{name}; > + my $server_name; > + my $suffix = ""; > + if (index($section, '.') != -1) { > + ($server_name, $suffix) = $section =~ /^\s*([^\s.]+)\s*\.\s*([^\s]+)/; > + } > + else { > + $server_name = $section =~ /^\s*([^\s]+)/; > + } Replace all 11 lines above with: my ($suffix) = ($mysqld->{name} =~ /\.(.*)/); > # Some InnoDB options are incompatible with the default bootstrap. > # If they are used, re-bootstrap > my @rebootstrap_opts; > @rebootstrap_opts = grep {/$rebootstrap_re/o} @$extra_opts if $extra_opts; > + # Let's store the additional bootstrap options in > + # the environment variable - they may be used later > + # in the mtr tests - for re-bootstrap or run the > + # recovery, etc: > + my $extra_text = "--datadir=$datadir"; that's rather random. Why do you add only this one option, that doesn't make sense. Better handle any additional requirements in the test. > + if (@rebootstrap_opts) { why do you need an if() ? > + $extra_text .= ' '.join(' ', @rebootstrap_opts); > + } > + if ($suffix) { > + $ENV{'MTR_BOOTSTRAP_OPTS_'.$suffix} = $extra_text; > + } > + else { > + $ENV{'MTR_BOOTSTRAP_OPTS'} = $extra_text; Is that even possible? > + } > if (@rebootstrap_opts) > { > mtr_verbose("Re-bootstrap with @rebootstrap_opts"); > diff --git a/mysql-test/suite/galera/disabled.def b/mysql-test/suite/galera/disabled.def > index 84babda2fa0..18e7c074059 100644 > --- a/mysql-test/suite/galera/disabled.def > +++ b/mysql-test/suite/galera/disabled.def > @@ -28,3 +28,4 @@ query_cache: MDEV-15805 Test failure on galera.query_cache > versioning_trx_id: MDEV-18590: galera.versioning_trx_id: Test failure: mysqltest: Result content mismatch > galera_wsrep_provider_unset_set: wsrep_provider is read-only for security reasons > pxc-421: wsrep_provider is read-only for security reasons > +galera_sst_rsync_innodb_nest : MDEV-29591 nesting innodb subdirectories in datadir causes SST to fail Why do you disable it if the whole commit is about fixing the test? How can I see that the test works? > diff --git a/mysql-test/suite/galera/r/galera_sst_rsync_innodb__innodb_dir,debug.rdiff b/mysql-test/suite/galera/r/galera_sst_rsync_innodb__innodb_dir,debug.rdiff > new file mode 100644 > index 00000000000..027f4860a39 > --- /dev/null > +++ b/mysql-test/suite/galera/r/galera_sst_rsync_innodb__innodb_dir,debug.rdiff > @@ -0,0 +1,114 @@ looks like a wrong file name? > +--- galera_sst_rsync_innodb_nest.result > ++++ galera_sst_rsync_innodb_nest.reject > +@@ -286,3 +286,111 @@ > + DROP TABLE t1; > + COMMIT; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] f984f49fba7: MDEV-30203 - deb fix piuparts
by Sergei Golubchik 21 Dec '22

21 Dec '22

Hi, Daniel, This is a review of `git diff 12786f0e779f 5f76981eb649`, that is, for commits: 5f76981eb64 rpm: packages that dont match component name 25eaa0aebfa MDEV-30205: [Cmtr: suite look in mariadb-test path 0981201f027 mtr - wsrep use mariadb executables eba0b35035a mtr: maradb names for *embedded and mariadb-client-test 974ca88f0fe deb: fix autotest - exclude for MDEV-30205 b19d75408bb Deb: Breaks/Replaces/COnflicts c60f0341950 MDEV-30203 deb breaks/replaces on compat packages f984f49fba7 MDEV-30203 - deb fix piuparts 310d025ff0d MDEV-30205: fix deb autopkgtest 89f6cf47ee2 rpm: fix packaging of compat/non-compat scripts links 494ed76d15a deb: mysqlhotcopy to mariadb-client-compat d03075f3a1f deb: server-core mysql* -> server-compat 9453812193d wsrep_sst_mariabackup to use mariadb-backup (and message accordingly) a4f843a7a5a Fix uncollected symlink eb9a469bf9a Add package recommended dependencies c135be62274 Remove duplicate entry 84c7cd29c61 Fix Debian packaging for mariabackup 5dbc7ae8bb4 MDEV-30203: compat packages - RPM remove mariabackup link eb4fa5d750f MDEV-30203: debian - make compat packages d4be0b9be73 MDEV-30203: Create mysql symlink packages (RPM) - scripts 441879fd493 MDEV-30203: Create mysql symlink packages (RPM) On Dec 19, Daniel Black wrote: > diff --git a/cmake/cpack_rpm.cmake b/cmake/cpack_rpm.cmake > index 339363b2169..7b48fd64f4c 100644 > --- a/cmake/cpack_rpm.cmake > +++ b/cmake/cpack_rpm.cmake > @@ -85,6 +85,35 @@ SET(CPACK_RPM_server_PACKAGE_DESCRIPTION "${CPACK_RPM_PACKAGE_DESCRIPTION}") > SET(CPACK_RPM_test_PACKAGE_SUMMARY "MariaDB database regression test suite") > SET(CPACK_RPM_test_PACKAGE_DESCRIPTION "${CPACK_RPM_PACKAGE_DESCRIPTION}") > > +# "set/append array" - append a set of strings, separated by a space > +MACRO(SETA var) > + FOREACH(v ${ARGN}) > + SET(${var} "${${var}} ${v}") > + ENDFOREACH() > +ENDMACRO(SETA) > + > +FOREACH(SYM_COMPONENT Server Client Test) > + SET(SYM ${SYM_COMPONENT}_symlinks) > + string(TOLOWER ${SYM} SYM) * why all cmake commands are uppercase, but `string` is lowercase? * SET is redundant, you can do STRING(TOLOWER ${SYM_COMPONENT}_symlinks SYM) * may be, better to use ${SYM_COMPONENT}-symlinks ? Like in MariaDB-server-symlinks ? > + SET(SYMCOMP ${SYM_COMPONENT}Symlinks) > + string(TOUPPER ${SYMCOMP} SYMCOMP_UPPER) > + SET(CPACK_COMPONENT_${SYMCOMP_UPPER}_GROUP "${SYM}") > + SET(CPACK_COMPONENTS_ALL "${CPACK_COMPONENTS_ALL}" "${SYMCOMP}") > + SET(CPACK_RPM_${SYM}_PACKAGE_SUMMARY "MySQL compatible symlinks for MariaDB database ${SYM_COMPONENT} binaries/scripts") > + SET(CPACK_RPM_${SYM}_PACKAGE_DESCRIPTION "${CPACK_RPM_PACKAGE_DESCRIPTION}") > + SET(CPACK_RPM_${SYM}_PACKAGE_ARCHITECTURE "noarch") > + SETA(CPACK_RPM_${SYM_COMPONENT}_PACKAGE_RECOMMENDS "MariaDB-${SYM}") You forgot to say that symlink package DEPENDS on the non-symlink package > +ENDFOREACH() > + > +# TODO change to 11.0 on rebase > +SETA(CPACK_RPM_client_symlinks_PACKAGE_CONFLICTS > + "MariaDB-client < 10.11.2" > + "MariaDB-server < 10.11.2") > +SETA(CPACK_RPM_server_symlinks_PACKAGE_CONFLICTS > + "MariaDB-server < 10.11.2") > +SETA(CPACK_RPM_test_symlinks_PACKAGE_CONFLICTS > + "MariaDB-test < 10.11.2") > + > # libmariadb3 > SET(CPACK_RPM_shared_PACKAGE_SUMMARY "LGPL MariaDB database client library") > SET(CPACK_RPM_shared_PACKAGE_DESCRIPTION "This is LGPL MariaDB client library that can be used to connect to MySQL > diff --git a/cmake/symlinks.cmake b/cmake/symlinks.cmake > index 3f3b4e4a9b5..a4eb6c2b42c 100644 > --- a/cmake/symlinks.cmake > +++ b/cmake/symlinks.cmake > @@ -12,7 +12,6 @@ endmacro() > REGISTER_SYMLINK("mariadb" "mysql") > REGISTER_SYMLINK("mariadb-access" "mysqlaccess") > REGISTER_SYMLINK("mariadb-admin" "mysqladmin") > -REGISTER_SYMLINK("mariadb-backup" "mariabackup") better not. there're scripts that might be using it. This is why I suggested `strncmp(my_progname, "mariadb", 7)` and not `strncmp(my_progname, "maria", 5)` in your PR > REGISTER_SYMLINK("mariadb-binlog" "mysqlbinlog") > REGISTER_SYMLINK("mariadb-check" "mysqlcheck") > REGISTER_SYMLINK("mariadb-client-test-embedded" "mysql_client_test_embedded") > diff --git a/debian/control b/debian/control > index 81418c13656..6b225ca4b3d 100644 > --- a/debian/control > +++ b/debian/control > @@ -548,7 +548,8 @@ Provides: default-mysql-client, > virtual-mysql-client > Recommends: libdbd-mariadb-perl | libdbd-mysql-perl, > libdbi-perl, > - libterm-readkey-perl > + libterm-readkey-perl, > + mariadb-client-compat Please, no. deb and rpm packages are different enough already. don't use different suffixes for them. Choose either -symlink or -compat (or something else) and use it consistently both in deb and in rpm. > Description: MariaDB database client binaries > MariaDB is a fast, stable and true multi-user, multi-threaded SQL database > server. SQL (Structured Query Language) is the most popular database query > @@ -558,6 +559,110 @@ Description: MariaDB database client binaries > This package includes the client binaries and the additional tools > innotop and mariadb-report (mysqlreport). > > +Package: mariadb-client-compat > +Architecture: all > +Depends: mariadb-client > +Multi-Arch: foreign > +Description: MySQL compatibility links to mariadb-client binaries/scripts. > +Conflicts: mariadb-client (<< ${source:Version}), > + mariadb-client-10.0, I wonder, would it work, if you won't set any Conflicts/Breaks, but only set Depends? It'd be a lot simpler that way (if it'll work). > + mariadb-client-10.1, > + mariadb-client-10.2, > + mariadb-client-10.3, > + mariadb-client-10.4, > + mariadb-client-10.5, > + mariadb-client-10.6, > + mariadb-client-10.7, > + mariadb-client-10.8, > + mariadb-client-5.1, > + mariadb-client-5.2, > + mariadb-client-5.3, > + mariadb-client-5.5, > + mariadb-client-core (<< ${source:Version}), > + mariadb-client-core-10.0, > + mariadb-client-core-10.1, > + mariadb-client-core-10.2, > + mariadb-client-core-10.3, > + mariadb-client-core-10.4, > + mariadb-client-core-10.5, > + mariadb-client-core-10.6, > + mariadb-client-core-10.7, > + mariadb-client-core-10.8, > + mariadb-server (<< ${source:Version}), > + mariadb-server-10.0, > + mariadb-server-10.1, > + mariadb-server-10.2, > + mariadb-server-10.3, > + mariadb-server-10.4, > + mariadb-server-10.5, > + mariadb-server-10.6, > + mariadb-server-10.7, > + mariadb-server-10.8, > + mysql-client (<< 5.0.51), > + mysql-client-5.0, > + mysql-client-5.1, > + mysql-client-5.5, > + mysql-client-5.6, > + mysql-client-5.7, > + mysql-client-8.0, > + mysql-client-core-5.0, > + mysql-client-core-5.1, > + mysql-client-core-5.5, > + mysql-client-core-5.6, > + mysql-client-core-5.7, > + mysql-client-core-8.0, > + mysql-server-core-5.7, > + mysql-server-core-8.0, > + percona-server-server-5.6, > + percona-server-server, > + percona-xtradb-cluster-server-5.6, > + percona-xtradb-cluster-server-5.7, > + percona-xtradb-cluster-server > +Breaks: mariadb-client (<< ${source:Version}), > + mariadb-client-10.0, > + mariadb-client-10.1, > + mariadb-client-10.2, > + mariadb-client-10.3, > + mariadb-client-10.4, > + mariadb-client-10.5, > + mariadb-client-10.6, > + mariadb-client-10.7, > + mariadb-client-10.8, > + mariadb-client-5.1, > + mariadb-client-core (<< ${source:Version}), > + mariadb-client-core-10.0, > + mariadb-client-core-10.1, > + mariadb-client-core-10.2, > + mariadb-client-core-10.3, > + mariadb-client-core-10.4, > + mariadb-client-core-10.5, > + mariadb-client-core-10.6, > + mariadb-client-core-10.7, > + mariadb-client-core-10.8, > + mariadb-server (<< ${source:Version}), > + mariadb-server-10.0, > + mariadb-server-10.1, > + mariadb-server-10.2, > + mariadb-server-10.3, > + mariadb-server-10.4, > + mariadb-server-10.5, > + mariadb-server-10.6, > + mariadb-server-10.7, > + mariadb-server-10.8, > + mysql-server-5.5, > + mysql-server-5.6, > + mysql-server-5.7, > + mysql-server-8.0, > + mysql-server-core-5.5, > + mysql-server-core-5.6, > + mysql-server-core-5.7, > + mysql-server-core-8.0, > + percona-server-server-5.6, > + percona-server-server, > + percona-xtradb-cluster-server-5.6, > + percona-xtradb-cluster-server-5.7, > + percona-xtradb-cluster-server > + > Package: mariadb-server-core > Architecture: any > Depends: mariadb-common (>= ${source:Version}), > diff --git a/debian/mariadb-backup.install b/debian/mariadb-backup.install > index e450f8f46a0..a288377ddd0 100644 > --- a/debian/mariadb-backup.install > +++ b/debian/mariadb-backup.install > @@ -1,6 +1,4 @@ > -usr/bin/mariabackup same here. needs mariadb-backup-compat > usr/bin/mariadb-backup > usr/bin/mbstream > -usr/share/man/man1/mariabackup.1 > usr/share/man/man1/mariadb-backup.1 > usr/share/man/man1/mbstream.1 > diff --git a/debian/mariadb-client.links b/debian/mariadb-client.links > index 62e3651daf5..8ccce12c609 100644 > --- a/debian/mariadb-client.links > +++ b/debian/mariadb-client.links > @@ -2,11 +2,6 @@ usr/bin/mariadb-check usr/bin/mariadb-analyze > usr/bin/mariadb-check usr/bin/mariadb-optimize > usr/bin/mariadb-check usr/bin/mariadb-repair > usr/bin/mariadb-check usr/bin/mariadbcheck > -usr/bin/mariadb-check usr/bin/mysqlanalyze > -usr/bin/mariadb-check usr/bin/mysqlcheck > -usr/bin/mariadb-check usr/bin/mysqloptimize > -usr/bin/mariadb-check usr/bin/mysqlrepair > -usr/bin/mariadb-report usr/bin/mysqlreport > usr/share/man/man1/mariadb-check.1.gz usr/share/man/man1/mariadb-analyze.1.gz > usr/share/man/man1/mariadb-check.1.gz usr/share/man/man1/mariadb-optimize.1.gz > usr/share/man/man1/mariadb-check.1.gz usr/share/man/man1/mariadb-repair.1.gz you forgot to move symlinks to man pages > diff --git a/debian/mariadb-server-compat.install b/debian/mariadb-server-compat.install > new file mode 100644 > index 00000000000..36701d8e0c6 > --- /dev/null > +++ b/debian/mariadb-server-compat.install > @@ -0,0 +1,12 @@ > +usr/bin/mysqld_multi > +usr/bin/mysqld_safe > +usr/bin/mysqld_safe_helper > +usr/bin/mysql_install_db > +usr/bin/mysql_upgrade > +usr/sbin/mysqld > +usr/share/man/man1/mysqld_multi.1 > +usr/share/man/man1/mysqld_safe.1 > +usr/share/man/man1/mysqld_safe_helper.1 great. do you do the same for rpms? I didn't see where manpages are moved to -symlinks packages in rpms. > +usr/share/man/man1/mysql_install_db.1 > +usr/share/man/man1/mysql_upgrade.1 > +usr/share/man/man8/mysqld.8 > diff --git a/debian/mariadb-test-compat.install b/debian/mariadb-test-compat.install > new file mode 100644 > index 00000000000..7b498d7d1c8 > --- /dev/null > +++ b/debian/mariadb-test-compat.install > @@ -0,0 +1 @@ > +usr/share/mysql/mysql-test/mysql-test-run.pl You forgot mysql-test-run, mysqltest, mysql_client_test, and their *embedded variants. But, you know, it's probably not worth a new package at all. It's the *test* package (not interesting for users, never used in production, so no immutability requirements). I suggest to keep those symlinks in the mariadb-test package. Your other PR will add a warning. And eventually we drop them. Optional/Recommended/Suggested - sounds like too much toubles for a *test* package. > diff --git a/mysql-test/lib/mtr_cases.pm b/mysql-test/lib/mtr_cases.pm > index fe202279ac7..ad1ac6fdf88 100644 > --- a/mysql-test/lib/mtr_cases.pm > +++ b/mysql-test/lib/mtr_cases.pm > @@ -387,14 +387,14 @@ sub collect_suite_name($$) > else > { > my @dirs = my_find_dir(dirname($::glob_mysql_test_dir), > - ["mysql-test/suite", @plugin_suitedirs ], > + ["mariadb-test/suite", "mysql-test/suite", @plugin_suitedirs ], Eh, no. That is, yes, but not in this set of commits, not in this MDEV > $suitename); > # > # if $suitename contained wildcards, we'll have many suites and > # their overlays here. Let's group them appropriately. > # > for (@dirs) { > - m@^.*/(?:mysql-test/suite|$plugin_suitedir_regex)/(.*)$@o or confess $_; > + m@^.*/(?:mariadb-test/suite|mysql-test/suite|$plugin_suitedir_regex)/(.*)$@o or confess $_; > push @{$suites{$1}}, $_; > } > } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 3

Re: [Maria-developers] MDEV-14443 test cases only
by Sergei Golubchik 19 Dec '22

19 Dec '22

Hi, Vicențiu, This is a review of _test cases only_. Mainly, to discuss the behavior. Random thoughts here and comments inline in the diff. There are two models for DENY that I can think of: "a hole in grants" and "a mask". The difference lies in where DENY is applied. In the "a hole in grants" model one has a certain set of grants (may be with "holes"), a role has another set of grants, and when one does SET ROLE, one gets a union of those grants. Like, one has SELECT on db1.* and DENY to db1.t1, and some role has SELECT on db1.t1 - by setting this role a user will have full SELECT on db1.* While in the "mask" model, denies are separate entities, when setting a role a union of denies is subtracted from the union of grants. In the example above a user would still be denied access to db1.t1. I personally see the "hole" model as more logical and easier to understand. It also allows to "patch" holes with roles, what the "mask" model cannot do. The "mask" model needs getting used to, the fact that after SET ROLE one can end up having less access than before. It's not clear who can DENY what and to whom. And DENY is very difficult to work around, esp. as one can DENY IGNORE DENIES. But it allows to DENY something to PUBLIC, and the "hole" model cannot really do that. And I think it'd be a popular use case. Ideally, I'd prefer to have some kind of a compromise model that's as simple to understand as a "hole", but allows to deny to public. Unfortunately, I don't see how to do that. You've implemented the "mask" behavior. Okay. It solves an important use case. More comments below: > diff --git a/mysql-test/suite/deny/columns.result b/mysql-test/suite/deny/columns.result > --- /dev/null > +++ b/mysql-test/suite/deny/columns.result > @@ -0,0 +1,126 @@ > +create user foo; > +create database deny_db; > +create table deny_db.t1 (a int, b int, secret int); > +create table deny_db.t2 (a2 int, b2 int, secret2 int); > +insert into deny_db.t2 values (100, 200, 300); > +grant all on *.* to foo; > +revoke ignore denies on *.* from foo; > +grant all on deny_db.* to foo; > +grant all on deny_db.t1 to foo; > +grant all on deny_db.t2 to foo; > +grant select (a, b, secret) on deny_db.t1 to foo; > +grant insert (a, b, secret) on deny_db.t1 to foo; > +grant update (a, b, secret) on deny_db.t1 to foo; > +grant references (a, b, secret) on deny_db.t1 to foo; > +grant select (a2, b2, secret2) on deny_db.t2 to foo; > +grant insert (a2, b2, secret2) on deny_db.t2 to foo; > +grant update (a2, b2, secret2) on deny_db.t2 to foo; > +grant references (a2, b2, secret2) on deny_db.t2 to foo; this is quite confusing, tons of redundant grants on different levels, it makes it very unclear of what causes what. > +# > +# Test select and insert ... select commands. > +# > +deny select (secret) on deny_db.t1 to foo; > +connect con1,localhost,foo,,; > +select * from deny_db.t1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for column 'secret' in table 't1' case in point. Normally, this should've been SELECT command denied to user 'foo'@'localhost' for table 't1' like in create database db; create table db.t1 (a int, b int, secret int); insert db.t1 values (1,2,100); create user foo@localhost; grant select (a,b) on db.t1 to foo@localhost; connect foo,localhost,foo; select a,b from db.t1; --error ER_TABLEACCESS_DENIED_ERROR select * from db.t1; --error ER_COLUMNACCESS_DENIED_ERROR select a,b,secret from db.t1; that is `select *` is ER_TABLEACCESS_DENIED_ERROR, and if I mention the column by name, then it's ER_COLUMNACCESS_DENIED_ERROR But you have so many redundant overlapping grants, that I really cannot see what error is correct in your test here. > +select secret from deny_db.t1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for column 'secret' in table 't1' > +# > +# These are different code paths. Check both. > +# > +insert into deny_db.t1 (a, b, secret) values (1, 2, 3); > +insert into deny_db.t1 values (1, 2, 3); > +# > +# Check INSERT SELECT as we need to account for 2 different kinds of > +# privileges. > +# > +insert into deny_db.t1 select * from deny_db.t1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for column 'secret' in table 't1' > +insert into deny_db.t1 select * from deny_db.t2; > +show full columns from deny_db.t1; > +Field Type Collation Null Key Default Extra Privileges Comment > +a int(11) NULL YES NULL select,insert,update,references > +b int(11) NULL YES NULL select,insert,update,references > +secret int(11) NULL YES NULL insert,update,references do you have a test where a denied column isn't shown by SHOW COLUMNS? that is, where you don't have other privileges on it? > +show full columns from deny_db.t2; > +Field Type Collation Null Key Default Extra Privileges Comment > +a2 int(11) NULL YES NULL select,insert,update,references > +b2 int(11) NULL YES NULL select,insert,update,references > +secret2 int(11) NULL YES NULL select,insert,update,references > +disconnect con1; > +connection default; > +revoke deny select (secret) on deny_db.t1 from foo; > +deny select(secret2) on deny_db.t2 to foo; > +connect con1,localhost,foo,,; > +insert into deny_db.t1 (a, b, secret) values (1, 2, 3); > +insert into deny_db.t1 values (1, 2, 3); > +insert into deny_db.t1 select * from deny_db.t1; > +insert into deny_db.t1 select * from deny_db.t2; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for column 'secret2' in table 't2' > +show full columns from deny_db.t1; > +Field Type Collation Null Key Default Extra Privileges Comment > +a int(11) NULL YES NULL select,insert,update,references > +b int(11) NULL YES NULL select,insert,update,references > +secret int(11) NULL YES NULL select,insert,update,references > +show full columns from deny_db.t2; > +Field Type Collation Null Key Default Extra Privileges Comment > +a2 int(11) NULL YES NULL select,insert,update,references > +b2 int(11) NULL YES NULL select,insert,update,references > +secret2 int(11) NULL YES NULL insert,update,references > +disconnect con1; > +# > +# Test insert commands > +# > +connection default; > +deny insert (secret) on deny_db.t1 to foo; > +connect con1,localhost,foo,,; > +insert into deny_db.t1 (a, b, secret) values (1, 2, 3); > +ERROR 42000: INSERT command denied to user 'foo'@'localhost' for column 'secret' in table 't1' > +insert into deny_db.t1 values (1, 2, 3); > +ERROR 42000: INSERT command denied to user 'foo'@'localhost' for column 'secret' in table 't1' didn't test this one, but likely should behave as select? that is, ER_TABLEACCESS_DENIED_ERROR please, test with GRANT (without DENY) before making any changes > +insert into deny_db.t1 select * from deny_db.t1; > +ERROR 42000: INSERT command denied to user 'foo'@'localhost' for column 'secret' in table 't1' > +insert into deny_db.t1 select * from deny_db.t2; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for column 'secret2' in table 't2' > +show full columns from deny_db.t1; > +Field Type Collation Null Key Default Extra Privileges Comment > +a int(11) NULL YES NULL select,insert,update,references > +b int(11) NULL YES NULL select,insert,update,references > +secret int(11) NULL YES NULL select,update,references > +disconnect con1; > +# > +# Test update commands. > +# > +connection default; > +deny select (secret) on deny_db.t1 to foo; > +deny select (secret2) on deny_db.t2 to foo; > +connect con1, localhost,foo,,; > +update deny_db.t1 set a=111, b=222 where secret=10; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for column 'secret' in table 't1' > +update deny_db.t1 set a=123 where a = (select secret2 from deny_db.t2); > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for column 'secret2' in table 't2' > +show full columns from deny_db.t1; > +Field Type Collation Null Key Default Extra Privileges Comment > +a int(11) NULL YES NULL select,insert,update,references > +b int(11) NULL YES NULL select,insert,update,references > +secret int(11) NULL YES NULL update,references > +show full columns from deny_db.t2; > +Field Type Collation Null Key Default Extra Privileges Comment > +a2 int(11) NULL YES NULL select,insert,update,references > +b2 int(11) NULL YES NULL select,insert,update,references > +secret2 int(11) NULL YES NULL insert,update,references > +disconnect con1; > +connection default; > +deny update (secret) on deny_db.t1 to foo; > +connect con1, localhost,foo,,; > +update deny_db.t1 set a=111, b=222; > +update deny_db.t1 set secret=333, a=111; > +ERROR 42000: UPDATE command denied to user 'foo'@'localhost' for column 'secret' in table 't1' > +update deny_db.t1 set secret=333; > +ERROR 42000: UPDATE command denied to user 'foo'@'localhost' for column 'secret' in table 't1' > +update deny_db.t1 set a=123, secret=333; > +ERROR 42000: UPDATE command denied to user 'foo'@'localhost' for column 'secret' in table 't1' > +disconnect con1; > +connection default; > +drop user foo; > +drop database deny_db; > diff --git a/mysql-test/suite/deny/columns_role.result b/mysql-test/suite/deny/columns_role.result > --- /dev/null > +++ b/mysql-test/suite/deny/columns_role.result > @@ -0,0 +1,57 @@ > +create database some_db; > +create view some_db.v1 as (select 1 as col); > +create user foo; > +create role bar; > +grant select on *.* to foo; > +grant show view on *.* to foo; > +grant bar to foo; > +grant select(col) on some_db.v1 to foo; > +revoke ignore denies on *.* from foo; why foo has ignore denies here? I don't see where it was granted. > +deny select (col) on some_db.v1 to bar; > +connect con1,localhost,foo,,; > +describe some_db.v1; > +Field Type Null Key Default Extra > +col int(1) NO 0 > +set role bar; > +describe some_db.v1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 'v1' this is very questionable. Should SET ROLE ever restrict what one can do? > +show create view some_db.v1; > +View Create View character_set_client collation_connection > +v1 CREATE ALGORITHM=UNDEFINED DEFINER=`root`@`localhost` SQL SECURITY DEFINER VIEW `some_db`.`v1` AS (select 1 AS `col`) latin1 latin1_swedish_ci > +disconnect con1; > +connection default; > +drop user foo; > +drop role bar; > +drop database some_db; > +# > +# Test if only role level denies are active. > +# > +create database some_db; > +create view some_db.v1 as (select 1 as col); > +create user foo; > +create role bar; > +grant bar to foo; > +grant select(col) on some_db.v1 to foo; > +deny select (col) on some_db.v1 to bar; > +connect con1,localhost,foo,,; > +# > +# Due to select(col) privilege on v1 we can see some_db here. > +# > +show databases; > +Database > +information_schema > +some_db > +test > +set role bar; > +# > +# Now the role's denies take effect and we can no longer see some_db. > +# > +show databases; > +Database > +information_schema > +test > +disconnect con1; > +connection default; > +drop user foo; > +drop role bar; > +drop database some_db; > diff --git a/mysql-test/suite/deny/deny_datastructures.result b/mysql-test/suite/deny/deny_datastructures.result > --- /dev/null > +++ b/mysql-test/suite/deny/deny_datastructures.result > @@ -0,0 +1,109 @@ > +create user foo; > +create role bar; > +# > +# Test deny format for global privileges. > +# > +deny select on *.* to foo; > +deny insert on *.* to bar; > +select user, host, JSON_EXTRACT(priv, '$.deny') from mysql.global_priv where user = 'foo'; > +user host JSON_EXTRACT(priv, '$.deny') > +foo % {"global": 1, "version_id": VERSION_ID} > +select user, host, JSON_EXTRACT(priv, '$.deny') from mysql.global_priv where user = 'bar'; > +user host JSON_EXTRACT(priv, '$.deny') > +bar {"global": 2, "version_id": VERSION_ID} > +flush privileges; > +deny select on some_db.* to foo; > +deny show view on some_db.* to foo; > +deny insert on some_other_db.* to foo; > +deny insert on some_other_db.* to bar; > +deny select on some_other_db.some_table to foo; > +deny insert on some_other_db.some_table_second to foo; > +deny update on some_other_db.some_table_third to foo; > +deny select(a, b) on some_other_db.some_table_third to foo; > +deny select, insert(a, b, c) on some_other_db.some_table_fourth to foo; > +select user, host, JSON_DETAILED(JSON_EXTRACT(priv, '$.deny')) from mysql.global_priv where user = 'foo'; > +user host JSON_DETAILED(JSON_EXTRACT(priv, '$.deny')) > +foo % { > + "global": 1, > + "db": > + [ > + > + { > + "name": "`some_db`", > + "access": 4194305 > + }, > + > + { > + "name": "`some_other_db`", > + "access": 2 > + } > + ], > + "table": > + [ > + > + { > + "name": "`some_other_db`.`some_table_second`", you sure you want that? you'd need to parse the name back when reading privileges, why would you need this extra complexity? I'd suggest something like "name" : [ "some_other_db", "some_table_second" ] (one component for db grants, three components for column grants) This way you don't need to parse the identifier to split on dots and remove backticks. You can even use [] for global grants and not split them into global/db/table/column lists, but have simply [ { "name ": [], "access": 1 }, { "name": ["some_db"], "access": 4194305 }, { "name": ["some_other_db"], "access": 2 } { "name": ["some_other_db", "some_table_second"], "access": 2 }, { "name": ["some_other_db", "some_table"], "access": 1 }, { "name": ["some_other_db", "some_table_third"], "access": 4 }, { "name": ["some_other_db", "some_table_fourth"], "access": 1 } { "name": ["some_other_db", "some_table_third", "a"], "access": 1 }, { "name": ["some_other_db", "some_table_fourth", "b"], "access": 2 }, { "name": ["some_other_db", "some_table_fourth", "c"], "access": 2 }, { "name": ["some_other_db", "some_table_third", "b"], "access": 1 }, ] May be it's easier if they're split into global/db/table/column, but there's defenitely no reason to put all identifier chain in one json property, quoted according to SQL rules. > + "access": 2 > + }, > + > + { > + "name": "`some_other_db`.`some_table`", > + "access": 1 > + }, > + > + { > + "name": "`some_other_db`.`some_table_third`", > + "access": 4 > + }, > + > + { > + "name": "`some_other_db`.`some_table_fourth`", > + "access": 1 > + } > + ], > + "column": > + [ > + > + { > + "name": "`some_other_db`.`some_table_third`.`a`", > + "access": 1 > + }, > + > + { > + "name": "`some_other_db`.`some_table_fourth`.`b`", > + "access": 2 > + }, > + > + { > + "name": "`some_other_db`.`some_table_fourth`.`c`", > + "access": 2 > + }, > + > + { > + "name": "`some_other_db`.`some_table_third`.`b`", > + "access": 1 > + }, > + > + { > + "name": "`some_other_db`.`some_table_fourth`.`a`", > + "access": 2 > + } > + ], > + "version_id": VERSION_ID > +} > +select user, host, JSON_DETAILED(JSON_EXTRACT(priv, '$.deny')) from mysql.global_priv where user = 'bar'; > +user host JSON_DETAILED(JSON_EXTRACT(priv, '$.deny')) > +bar { > + "global": 2, > + "db": > + [ > + > + { > + "name": "`some_other_db`", > + "access": 2 > + } > + ], > + "version_id": VERSION_ID > +} > +drop user foo; > +drop role bar; > diff --git a/mysql-test/suite/deny/dml.test b/mysql-test/suite/deny/dml.test > --- /dev/null > +++ b/mysql-test/suite/deny/dml.test > @@ -0,0 +1,241 @@ would be good to have a comment at the top of every test file you add to mention the 1) MDEV and 2) the purpose for that file to exist. E.g. in this case I wouldn't need to waste time guessing how dml.test is different from columns.test, if both have column-level grants. And even the content looks suspiciously the same. > +--source include/not_embedded.inc > + > +create user foo; > + > +create database deny_db; > +create table deny_db.t1 (a int, b int, secret int); > +insert into deny_db.t1 values (1, 2, 3); > + > +create table deny_db.t2 (a int, b int, secret int); > +insert into deny_db.t2 values (10, 20, 30); > + > +grant select (a) on deny_db.t1 to foo; > +grant insert (a) on deny_db.t1 to foo; > +grant update (a) on deny_db.t1 to foo; > +grant delete on deny_db.t1 to foo; > + > +--connect (con1,localhost,foo,,) > +insert into deny_db.t1 (a) values (1), (2), (3); > +disconnect con1; > + > +connection default; > +deny insert on deny_db.* to foo; > + > +--connect (con1,localhost,foo,,) > +--error ER_TABLEACCESS_DENIED_ERROR > +insert into deny_db.t1 (a) values (1), (2), (3); > +update deny_db.t1 set a=a*10; > +delete from deny_db.t1 where a = 20; > +disconnect con1; > + > +connection default; > +select * from deny_db.t1 order by a; > +truncate deny_db.t1; > + > +revoke deny insert on deny_db.* from foo; > + > + > +--connect (con1,localhost,foo,,) > +insert into deny_db.t1 (a) values (1), (2), (3); > +--echo # > +--echo # Test that revoking insert deny allows inserting only into allowed > +--echo # to insert columns. > +--echo # > +insert into deny_db.t1 (a) values (1), (2), (3); > + > +--error ER_COLUMNACCESS_DENIED_ERROR > +insert into deny_db.t1 (a,b) values (1,10), (2, 20), (3, 30); > +disconnect con1; > + > +connection default; > +--echo # > +--echo # Test delete/update command denies. > +--echo # > +deny select on deny_db.* to foo; > + > +--connect (con1,localhost,foo,,) > +--echo # > +--echo # Test not allowed to delete / update because of lack of select > +--echo # rights. > +--echo # > + > +show grants; > +--error ER_COLUMNACCESS_DENIED_ERROR > +update deny_db.t1 set a=a*10 where a = 1; Did you verify that in all cases the error is the same as if the privilege simply wasn't granted in the first place? In this case the error is correct, I've checked. > + > +--error ER_COLUMNACCESS_DENIED_ERROR > +delete from deny_db.t1 where a = 20; > +disconnect con1; > + > + > +connection default; > +--echo # > +--echo # Now deny update and delete directly, but keep select. > +--echo # > +revoke deny select on deny_db.* from foo; > +deny delete, update on deny_db.* to foo; > + > +--connect (con1,localhost,foo,,) > +--echo # > +--echo # Test not allowed to delete / update because of lack of delete/update > +--echo # rights. > +--echo # > + > +--error ER_TABLEACCESS_DENIED_ERROR > +update deny_db.t1 set a=a*10; > + > +--error ER_TABLEACCESS_DENIED_ERROR > +delete from deny_db.t1 where a = 20; > +disconnect con1; > + > +--echo # > +--echo # Test masking table level grants. > +--echo # > + > +connection default; > +grant select on deny_db.t1 to foo; > +grant insert on deny_db.t1 to foo; > +grant update on deny_db.t1 to foo; > +grant delete on deny_db.t1 to foo; > + > +--connect (con1,localhost,foo,,) > +insert into deny_db.t1 (a,b) values (1,10), (2, 20), (3, 30); > +disconnect con1; > + > +connection default; > +deny insert on deny_db.* to foo; > + > +--connect (con1,localhost,foo,,) > +--error ER_TABLEACCESS_DENIED_ERROR > +insert into deny_db.t1 (a,b) values (1,10), (2, 20), (3, 30); > + > +--error ER_TABLEACCESS_DENIED_ERROR > +update deny_db.t1 set a=a*10; > + > +--error ER_TABLEACCESS_DENIED_ERROR > +delete from deny_db.t1 where a = 20; > + > +disconnect con1; > + > +connection default; > +revoke deny update, delete on deny_db.* from foo; > + > +--connect (con1,localhost,foo,,) > +--error ER_TABLEACCESS_DENIED_ERROR > +insert into deny_db.t1 (a,b) values (1,10), (2, 20), (3, 30); > + > +update deny_db.t1 set a=a*10; > +delete from deny_db.t1 where a = 20; > +disconnect con1; > + > +--echo # > +--echo # Test masking database & global level grants. > +--echo # > +connection default; > +grant select, insert, update, delete on *.* to foo; > +grant select, insert, update, delete on deny_db.* to foo; > +grant select, insert, update, delete on deny_db.t1 to foo; > +deny select, insert on deny_db.* to foo; > + > +--connect (con1,localhost,foo,,) > +--error ER_TABLEACCESS_DENIED_ERROR > +insert into deny_db.t1 (a,b) values (1,10), (2, 20), (3, 30); > + > +--error ER_COLUMNACCESS_DENIED_ERROR > +update deny_db.t1 set a=a*10; > + > +--error ER_COLUMNACCESS_DENIED_ERROR > +delete from deny_db.t1 where a = 20; > +disconnect con1; > + > +connection default; > +revoke deny select on deny_db.* from foo; > +deny update, delete on deny_db.* to foo; > + > +--connect (con1,localhost,foo,,) > +--error ER_TABLEACCESS_DENIED_ERROR > +insert into deny_db.t1 (a,b) values (1,10), (2, 20), (3, 30); > + > +--error ER_TABLEACCESS_DENIED_ERROR > +update deny_db.t1 set a=a*10; > + > +--error ER_TABLEACCESS_DENIED_ERROR > +delete from deny_db.t1 where a = 20; > +disconnect con1; > + > +connection default; > +deny select on deny_db.* to foo; > + > +--connect (con1,localhost,foo,,) > +--error ER_TABLEACCESS_DENIED_ERROR > +insert into deny_db.t1 (a,b) values (1,10), (2, 20), (3, 30); > + > +--error ER_TABLEACCESS_DENIED_ERROR > +update deny_db.t1 set a=a*10; > + > +--error ER_TABLEACCESS_DENIED_ERROR > +delete from deny_db.t1 where a = 20; > +disconnect con1; > + > +--echo # > +--echo # Test revoking all denies. > +--echo # > +connection default; > +revoke deny select, insert, update, delete on deny_db.* from foo; by the way, what does REVOKE ALL PRIVILEGES do, does it remove denies? please, add a test. > +truncate deny_db.t1; > + > +--connect (con1,localhost,foo,,) > +insert into deny_db.t1 (a,b) values (1,10), (2, 20), (3, 30); > +update deny_db.t1 set a=a*10; > +select * from deny_db.t1; > +delete from deny_db.t1 where a = 20; > +disconnect con1; > + > +connection default; > + > +--echo # > +--echo # Test masking via table level denies > +--echo # > + > +deny select on deny_db.t1 to foo; > + > +--connect (con1,localhost,foo,,) > +insert into deny_db.t1 (a,b) values (1,10), (2, 20), (3, 30); > +--error ER_COLUMNACCESS_DENIED_ERROR > +update deny_db.t1 set a=a*10; > +--error ER_COLUMNACCESS_DENIED_ERROR > +update deny_db.t1 set a=a*10 where a = 10; > +--echo # > +--echo # Delete works without select. > +--echo # > +delete from deny_db.t1; > +--error ER_COLUMNACCESS_DENIED_ERROR > +delete from deny_db.t1 where a = 20; > + > +--error ER_TABLEACCESS_DENIED_ERROR > +select * from deny_db.t1; > +disconnect con1; > + > +connection default; > +revoke deny select on deny_db.t1 from foo; > +deny insert on deny_db.t1 to foo; > +deny update on deny_db.t1 to foo; > +deny delete on deny_db.t1 to foo; > + > +--connect (con1,localhost,foo,,) > +--error ER_TABLEACCESS_DENIED_ERROR > +insert into deny_db.t1 (a,b) values (1,10), (2, 20), (3, 30); > +--error ER_TABLEACCESS_DENIED_ERROR > +update deny_db.t1 set a=a*10; > +--error ER_TABLEACCESS_DENIED_ERROR > +update deny_db.t1 set a=a*10 where a = 10; > +--error ER_TABLEACCESS_DENIED_ERROR > +delete from deny_db.t1; > +--error ER_TABLEACCESS_DENIED_ERROR > +delete from deny_db.t1 where a = 20; > + > +select * from deny_db.t1; > + > +connection default; > +drop database deny_db; > +drop user foo; > diff --git a/mysql-test/suite/deny/global.result b/mysql-test/suite/deny/global.result > --- /dev/null > +++ b/mysql-test/suite/deny/global.result > @@ -0,0 +1,327 @@ > +create user foo; > +create database deny_db; > +create table deny_db.t1 (a int, b int, secret int); > +insert into deny_db.t1 values (1, 2, 3); > +create table deny_db.t2 (a int, b int, secret int); > +insert into deny_db.t2 values (10, 20, 30); > +# > +# Test global denies. > +# > +show databases; > +Database > +deny_db > +information_schema > +mtr > +mysql > +performance_schema > +sys > +test > +grant select (secret) on deny_db.t1 to foo; > +connect con1,localhost,foo,,; > +select * from deny_db2.t1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't1' > +show databases; > +Database > +deny_db > +information_schema > +test > +use deny_db; > +# > +# It is still possible to leak column names without denies. > +# > +select * from deny_db.t1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for column 'a' in table 't1' this is a bug. See my test above, it's ER_TABLEACCESS_DENIED_ERROR. Apparently, it returns ER_TABLEACCESS_DENIED_ERROR if there's at least one granted column before the non-granted column. That is, if you'd have create table deny_db.t1 (secret int, a int, b int); you'd get ER_TABLEACCESS_DENIED_ERROR. please fix it (where applicable, 10.3, perhaps?) > +select a from deny_db.t1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for column 'a' in table 't1' > +select b from deny_db.t1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for column 'b' in table 't1' > +select table_name, column_name from information_schema.columns > +where table_schema = 'deny_db'; > +table_name column_name > +t1 secret > +# > +# Invalid field error is returned if table exists. > +# > +select c from deny_db.t1; > +ERROR 42S22: Unknown column 'c' in 'field list' I feel like the whole column-leaking behavior is a bug too. This, likely, should be ER_COLUMNACCESS_DENIED_ERROR too > +# > +# Table access denied if table does exist but is not granted. > +# > +select c from deny_db.t2; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't2' > +# > +# Same error if table does not exist. > +# > +select c from deny_db.t_not_exists; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't_not_exists' > +select secret from deny_db.t1; > +secret > +3 > +# > +# Now apply deny. > +# > +connection default; > +deny select on *.* to foo; I see you reconnect to apply denies. What are the rules here? With grants, table/column grants work right away, while global and current-db grants are cached in the thd. but table/column denies aren't stored in separate tables, that's why the question. What are the rules here? > +disconnect con1; > +connect con1,localhost,foo,,; > +show databases; > +Database > +information_schema > +test > +use information_schema; > +use deny_db; > +ERROR 42000: Access denied for user 'foo'@'%' to database 'deny_db' > +select * from deny_db.t1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't1' > +select c from deny_db.t1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't1' > +select c from deny_db.t2; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't2' > +select c from deny_db.t_not_exists; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't_not_exists' > +select secret from deny_db.t1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't1' > +# > +# Grant table access then test deny. > +# > +connection default; > +grant select on deny_db.t1 to foo; > +connection con1; > +show databases; > +Database > +information_schema > +test > +use information_schema; > +use deny_db; > +ERROR 42000: Access denied for user 'foo'@'%' to database 'deny_db' > +select * from deny_db.t1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't1' > +select c from deny_db.t1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't1' > +select c from deny_db.t2; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't2' > +select c from deny_db.t_not_exists; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't_not_exists' > +select secret from deny_db.t1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't1' > +# > +# Grant database access then test deny. > +# > +connection default; > +grant select on deny_db.* to foo; > +connection con1; > +show databases; > +Database > +information_schema > +test > +use information_schema; > +use deny_db; > +ERROR 42000: Access denied for user 'foo'@'%' to database 'deny_db' > +select * from deny_db.t1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't1' > +select c from deny_db.t1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't1' > +select c from deny_db.t2; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't2' > +select c from deny_db.t_not_exists; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't_not_exists' > +select secret from deny_db.t1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't1' > +# > +# Grant global access then test deny. > +# > +connection default; > +grant select on *.* to foo; > +connection con1; > +show databases; > +Database > +information_schema > +test > +use information_schema; > +use deny_db; > +ERROR 42000: Access denied for user 'foo'@'%' to database 'deny_db' > +select * from deny_db.t1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't1' > +select c from deny_db.t1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't1' > +select c from deny_db.t2; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't2' > +select c from deny_db.t_not_exists; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't_not_exists' > +select secret from deny_db.t1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't1' > +connection default; > +# > +# Test underlying data loading. > +# > +select user, host, JSON_EXTRACT(priv, '$.deny') from mysql.global_priv where user = 'foo'; > +user host JSON_EXTRACT(priv, '$.deny') > +foo % {"global": 1, "version_id": 101100} > +show grants for foo; > +Grants for foo@% > +GRANT SELECT ON *.* TO `foo`@`%` > +DENY SELECT ON *.* TO `foo`@`%` > +GRANT SELECT ON `deny_db`.* TO `foo`@`%` > +GRANT SELECT, SELECT (secret) ON `deny_db`.`t1` TO `foo`@`%` meaning, the user will see what is denied to him? I expect it'll be a questionable item, some users will want denies to be not shown. Although I don't see how they cannot be. > +flush privileges; > +show grants for foo; > +Grants for foo@% > +GRANT SELECT ON *.* TO `foo`@`%` > +DENY SELECT ON *.* TO `foo`@`%` > +GRANT SELECT ON `deny_db`.* TO `foo`@`%` > +GRANT SELECT, SELECT (secret) ON `deny_db`.`t1` TO `foo`@`%` > +connection con1; > +show databases; > +Database > +information_schema > +test > +use information_schema; > +use deny_db; > +ERROR 42000: Access denied for user 'foo'@'%' to database 'deny_db' > +select * from deny_db.t1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't1' > +select c from deny_db.t1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't1' > +select c from deny_db.t2; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't2' > +select c from deny_db.t_not_exists; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't_not_exists' > +select secret from deny_db.t1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't1' > +disconnect con1; > +connection default; > +# > +# Now test insert not being denied. > +# > +grant insert(secret) on deny_db.t1 to foo; > +connect con1,localhost,foo,,; > +show databases; > +Database > +deny_db > +information_schema > +test > +insert into deny_db.t1(secret) values (10000); > +disconnect con1; > +connection default; > +drop database deny_db; > +drop user foo; > +# > +# Test denying all rights globally. please, add tests with DENY TO PUBLIC > +# User should still be able to connect and see information_schema > +# *at least*. > +# > +create user foo; > +create database some_db; > +grant select on *.* to foo; > +show grants for foo; > +Grants for foo@% > +GRANT SELECT ON *.* TO `foo`@`%` > +deny select on *.* to foo; > +show grants for foo; > +Grants for foo@% > +GRANT SELECT ON *.* TO `foo`@`%` > +DENY SELECT ON *.* TO `foo`@`%` > +deny all on *.* to foo; > +show grants for foo; > +Grants for foo@% > +GRANT SELECT ON *.* TO `foo`@`%` > +DENY ALL PRIVILEGES ON *.* TO `foo`@`%` > +connect con1,localhost,foo,,information_schema; > +show databases; > +Database > +information_schema > +disconnect con1; > +connection default; > +drop user foo; > +drop database some_db; > +############################################## > +# Test SELECT command interacting with deny. # > +############################################## > +create user foo; > +create user bar; > +create database some_db; > +create table some_db.t1 (a int, secret int); > +insert into some_db.t1 values (1, 100); > +use some_db; > +create view v1 as (select a from t1); > +create view v2 as (select secret from t1); > +connect con1,localhost,foo,,; > +select table_name, table_type from information_schema.tables where table_schema like 'some_db'; > +table_name table_type > +disconnect con1; > +connection default; > +grant select on *.* to foo; > +grant select on some_db.* to foo; > +grant select on some_db.t1 to foo; > +grant select(a) on some_db.t1 to foo; > +# > +# See what foo sees before denies. > +# > +connect con1,localhost,foo,,; > +select table_name, table_type from information_schema.tables where table_schema like 'some_db'; > +table_name table_type > +t1 BASE TABLE > +v1 VIEW > +v2 VIEW > +disconnect con1; > +connection default; > +show grants for foo; > +Grants for foo@% > +GRANT SELECT ON *.* TO `foo`@`%` > +GRANT SELECT ON `some_db`.* TO `foo`@`%` > +GRANT SELECT, SELECT (a) ON `some_db`.`t1` TO `foo`@`%` > +deny select on *.* to foo; > +show grants for foo; > +Grants for foo@% > +GRANT SELECT ON *.* TO `foo`@`%` > +DENY SELECT ON *.* TO `foo`@`%` > +GRANT SELECT ON `some_db`.* TO `foo`@`%` > +GRANT SELECT, SELECT (a) ON `some_db`.`t1` TO `foo`@`%` > +connect con1,localhost,foo,,; > +show grants; > +Grants for foo@% > +GRANT SELECT ON *.* TO `foo`@`%` > +DENY SELECT ON *.* TO `foo`@`%` > +GRANT SELECT ON `some_db`.* TO `foo`@`%` > +GRANT SELECT, SELECT (a) ON `some_db`.`t1` TO `foo`@`%` > +use some_db; > +ERROR 42000: Access denied for user 'foo'@'%' to database 'some_db' > +select * from some_db.t1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't1' > +select * from some_db.v1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 'v1' > +select * from some_db.v2; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 'v2' > +select table_name, table_type from information_schema.tables where table_schema like "some_db"; > +table_name table_type > +select table_name, column_name from information_schema.columns where table_schema like "some_db"; > +table_name column_name > +disconnect con1; > +connection default; > +grant insert(a) on some_db.t1 to bar; > +grant insert(a) on some_db.t1 to foo; > +connect con1,localhost,foo,,; > +select table_name, column_name from information_schema.columns where table_schema like "some_db"; > +table_name column_name > +t1 a > +disconnect con1; > +connect con2,localhost,bar,,; > +select table_name, column_name from information_schema.columns where table_schema like "some_db"; > +table_name column_name > +t1 a > +disconnect con2; > +connection default; > +deny insert on *.* to foo; > +deny insert on *.* to bar; > +connect con1,localhost,foo,,; > +select table_name, column_name from information_schema.columns where table_schema like "some_db"; > +table_name column_name > +disconnect con1; > +connect con2,localhost,bar,,; > +select table_name, column_name from information_schema.columns where table_schema like "some_db"; > +table_name column_name > +disconnect con2; > +connection default; > +drop user foo; > +drop user bar; > +drop database some_db; > diff --git a/mysql-test/suite/deny/ignore_denies.result b/mysql-test/suite/deny/ignore_denies.result > --- /dev/null > +++ b/mysql-test/suite/deny/ignore_denies.result > @@ -0,0 +1,76 @@ > +create user foo; > +create role bar; > +grant select, ignore denies on *.* to foo; > +deny select on *.* to foo; > +deny select on mysql.* to foo; > +deny select on mysql.global_priv to foo; > +deny select(user) on mysql.global_priv to foo; > +grant bar to foo; > +grant ignore denies on *.* to bar; > +connect con1,localhost,foo,,; > +# > +# The user should still be able to access mysql.global_priv due > +# to the IGNORE DENIES privilege being active. > +# > +select JSON_EXTRACT(priv, '$.deny') from mysql.global_priv where user = 'foo'; > +JSON_EXTRACT(priv, '$.deny') > +{"global": 1, "db": [{"name": "`mysql`", "access": 1}], "table": [{"name": "`mysql`.`global_priv`", "access": 1}], "column": [{"name": "`mysql`.`global_priv`.`user`", "access": 1}], "version_id": VERSION_ID} > +disconnect con1; > +connection default; > +# > +# Now test inheriting IGNORE DENIES from a role. > +# > +revoke ignore denies on *.* from foo; > +connect con1,localhost,foo,,; > +select JSON_EXTRACT(priv, '$.deny') from mysql.global_priv where user = 'foo'; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 'global_priv' > +# > +# Bar provides foo with IGNORE DENIES privilege. > +# > +set role bar; > +select JSON_EXTRACT(priv, '$.deny') from mysql.global_priv where user = 'foo'; > +JSON_EXTRACT(priv, '$.deny') > +{"global": 1, "db": [{"name": "`mysql`", "access": 1}], "table": [{"name": "`mysql`.`global_priv`", "access": 1}], "column": [{"name": "`mysql`.`global_priv`.`user`", "access": 1}], "version_id": VERSION_ID} > +disconnect con1; > +connection default; > +# > +# Denying IGNORE DENIES, overrides a user's ability to ignore denies. This is very questionable. One can DENY IGNORE DENIES TO PUBLIC and basically lock the server down for everyone, there will be no way to override that. (this is an observation, not ready to suggest a solution atm) > +# > +deny ignore denies on *.* to foo; > +grant ignore denies on *.* to foo; > +connect con1,localhost,foo,,; > +# > +# Foo can't ignore the SELECT priv deny. > +# > +select JSON_EXTRACT(priv, '$.deny') from mysql.global_priv where user = 'foo'; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 'global_priv' > +# > +# And neither if bar role provides the ignore deny priv. > +# > +set role bar; > +select JSON_EXTRACT(priv, '$.deny') from mysql.global_priv where user = 'foo'; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 'global_priv' > +disconnect con1; > +connection default; > +# > +# Now test what happens when IGNORE DENIES is denied from a role. > +# > +revoke deny ignore denies on *.* from foo; > +deny ignore denies on *.* to bar; > +connect con1,localhost,foo,,; > +# > +# Foo can ignore the SELECT priv deny when Bar isn't activated > +# > +select JSON_EXTRACT(priv, '$.deny') from mysql.global_priv where user = 'foo'; > +JSON_EXTRACT(priv, '$.deny') > +{"global": 1, "db": [{"name": "`mysql`", "access": 1}], "table": [{"name": "`mysql`.`global_priv`", "access": 1}], "column": [{"name": "`mysql`.`global_priv`.`user`", "access": 1}], "version_id": VERSION_ID} > +# > +# But when Bar is active, denies can not be ignored any more. > +# > +set role bar; > +select JSON_EXTRACT(priv, '$.deny') from mysql.global_priv where user = 'foo'; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 'global_priv' > +disconnect con1; > +connection default; > +drop user foo; > +drop role bar; > diff --git a/mysql-test/suite/deny/show_generic.result b/mysql-test/suite/deny/show_generic.result > --- /dev/null > +++ b/mysql-test/suite/deny/show_generic.result > @@ -0,0 +1,197 @@ > +# > +# This test checks that denies take part in SHOW commands properly. > +# > +create user foo; > +create database some_db; > +create table some_db.t1 (id int); > +connect con1,localhost,foo,,; > +show tables from some_db; > +ERROR 42000: Access denied for user 'foo'@'%' to database 'some_db' > +disconnect con1; > +connection default; > +grant insert on some_db.t1 to foo; > +connect con1,localhost,foo,,; > +show tables from some_db; > +Tables_in_some_db > +t1 > +disconnect con1; > +connection default; > +deny insert on some_db.* to foo; > +connect con1,localhost,foo,,; > +show databases; > +Database > +information_schema > +test > +show tables from some_db; > +ERROR 42000: Access denied for user 'foo'@'%' to database 'some_db' > +disconnect con1; > +connection default; > +# > +# Test that dropping a user does not give access to connections that > +# previously had elements denied. > +# > +# We will use two users, foo which will receive denies, bar which will > +# not. This will showcase the difference in behaviour. > +# > +create user bar; > +# > +# This global grant is cached in sctx->master_access on connection. > +# > +grant select on *.* to foo; > +grant select on *.* to bar; > +connect con1,localhost,foo,,; > +show tables from some_db; > +Tables_in_some_db > +t1 > +disconnect con1; > +connection default; > +# > +# Here we add a mask for some_db, it should no longer be visible to foo. > +# > +deny select on some_db.* to foo; > +connect con1,localhost,foo,,; > +use some_db; > +ERROR 42000: Access denied for user 'foo'@'%' to database 'some_db' > +show tables from some_db; > +ERROR 42000: Access denied for user 'foo'@'%' to database 'some_db' > +# > +# bar user has access to the database (no denies present) > +# > +connect con2,localhost,bar,,; > +use some_db; > +show tables from some_db; > +Tables_in_some_db > +t1 > +# > +# Do not disconnect foo & bar, but drop the users. > +# > +connection default; > +drop user foo; > +drop user bar; > +# > +# Our current running connection, because it featured denies previously > +# now doesn't have access. > +# > +connection con1; > +use some_db; > +ERROR 42000: Access denied for user 'foo'@'%' to database 'some_db' > +show tables from some_db; > +ERROR 42000: Access denied for user 'foo'@'%' to database 'some_db' > +disconnect con1; > +# > +# However, bar does not have any denies active. In this case we can keep > +# the previous MariaDB behaviour of using the "cache" within > +# sctx->master_access to grant access. This comment is confusing. As far as I can see both foo and bar preserve the behavior after `drop user`. Which is fine and what I was expected anyway. But comments imply there's some difference in behavior between them? > +# > +connection con2; > +use some_db; > +show tables from some_db; > +Tables_in_some_db > +t1 > +disconnect con2; > +connection default; > +drop database some_db; > +# > +# Test table level denies interacting with show tables > +# > +create database some_db; > +create user foo; > +create table some_db.t1 (a int, b int); > +create table some_db.t2 (a int, b int); > +grant select on *.* to foo; > +deny select on some_db.t1 to foo; > +deny insert on some_db.t2 to foo; > +select user, host, JSON_EXTRACT(priv, '$.deny') from mysql.global_priv > +where user = 'foo'; > +user host JSON_EXTRACT(priv, '$.deny') > +foo % {"table": [{"name": "`some_db`.`t2`", "access": 2}, {"name": "`some_db`.`t1`", "access": 1}], "version_id": VERSION_ID} > +connect con1,localhost,foo,,; > +show tables from some_db; > +Tables_in_some_db > +t2 > +show columns from some_db.t1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't1' > +show columns from some_db.t2; > +Field Type Null Key Default Extra > +a int(11) YES NULL > +b int(11) YES NULL > +select table_name, column_name, privileges from information_schema.columns > +where table_schema like 'some_db' order by table_name, column_name; > +table_name column_name privileges > +t2 a select > +t2 b select > +disconnect con1; > +connection default; > +grant insert on some_db.* to foo; > +connect con1,localhost,foo,,; > +show tables from some_db; > +Tables_in_some_db > +t1 > +t2 > +# > +# See MDEV-28783, this should not error out when global/db grants exist > +# (except for SELECT priv). > +# > +show columns from some_db.t1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't1' > +show columns from some_db.t2; > +Field Type Null Key Default Extra > +a int(11) YES NULL > +b int(11) YES NULL > +select table_name, column_name, privileges from information_schema.columns > +where table_schema like 'some_db' order by table_name, column_name; > +table_name column_name privileges > +t1 a insert > +t1 b insert > +t2 a select > +t2 b select > +disconnect con1; > +connection default; > +deny insert on some_db.t1 to foo; > +deny select on some_db.t2 to foo; > +connect con1,localhost,foo,,; > +# > +# some_db should still be visible, but it should show up as empty. > +# > +show databases; > +Database > +information_schema > +mtr > +mysql > +performance_schema > +some_db > +sys > +test > +show tables from some_db; > +Tables_in_some_db > +show columns from some_db.t1; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't1' > +show columns from some_db.t2; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't2' > +select table_name, column_name, privileges from information_schema.columns > +where table_schema like 'some_db' order by table_name, column_name; > +table_name column_name privileges > +disconnect con1; > +connection default; > +grant select(a) on some_db.t1 to foo; > +grant update(a) on some_db.t1 to foo; > +connect con1,localhost,foo,,; > +# > +# Update privilege on the column is not masked, only see a column. > +# > +show tables from some_db; > +Tables_in_some_db > +t1 > +show columns from some_db.t1; > +Field Type Null Key Default Extra > +a int(11) YES NULL > +show columns from some_db.t2; > +ERROR 42000: SELECT command denied to user 'foo'@'localhost' for table 't2' > +select table_name, column_name, privileges from information_schema.columns > +where table_schema like 'some_db' order by table_name, column_name; > +table_name column_name privileges > +t1 a update > +disconnect con1; > +connection default; > +drop user foo; > +drop database some_db; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] Review input for MDEV-21092, 21095, 29997
by Sergei Petrunia 08 Dec '22

08 Dec '22

Hi Rex, Please find below review input for the collection of patches for these MDEVs. First, please try to have each piece of functionality in its own commit. You can use "git rebase -i" to make your last commits be one commit, as well as re-order them. Then, you can do a carefully considered "git push --force" to overwrite the contents of your branch with the new set of commits. in prune_partitions(): + String parts; + String_list parts_list; + + make_used_partitions_str(thd->mem_root, prune_param.part_info, &parts, parts_list ); + trace_partition_pruning.add("partition_prune", parts.ptr()); This creates a list of used partitions regardless whether we need it or not (and typically trace=off, so we don't'). Please put this code into if (unlikely(thd->trace_started())) { ... } Consider this testcase: create table t2 (a int, b int) partition by hash(a) partitions 10; create table t3 (a int, b int) partition by hash(a) partitions 10; set optimizer_trace=1; explain format=json select * from t2,t3 where t2.a in (2,3,4) and t3.a in (4,5); select * from information_schema.optimizer_trace\G This gives: ... { "partition_prune": "p2,p3,p4" }, { "partition_prune": "p4,p5" }, ... which makes it impossible to see which table is pruning is done for. Let's make the trace be: { "prune_partitions" : { "table": "t2", "used_partitions": "p2,p3,p4" } } { "prune_partitions" : { "table": "t3", "used_partitions": "p4,p5" } } . in item_subselect.cc: + { + OPT_TRACE_TRANSFORM(thd, trace_wrapper, trace_transform, + in_subs->get_select_lex()->select_number, + "EXISTS (SELECT)", "IN (SELECT)"); + trace_transform.add( "upper_not", ( upper_not?true:false ) ); + } coding style: everything inside { } should be indented one level (=2 spaces) right. in sql_select.cc: "attached_conditions_summary": [ { "table": "t1", "attached": "t1.c < 30", "index": "t1.a < 10 and t1.b < 20" } Please change "index" to "index_condition". and "attached" to "attached_condition". Let's not print index_condition if it is null. (Yes, this is somewhat inconsistent with printing "attached":null) in item.cc: + #include "my_json_writer.h" + const char *dbug_print_optrace( ) dbug_print_optrace() is fine. I think it needs a comment like this: /* Debug printout: print the trace trace we're currently writing if any, and return it. */ (Feel free to fix poor English if necessary) Please fix the coding style in the function body as discussed on Slack. Please move #include to the other #includes at the top of the file. Can add a comment, "// For dbug_print_optrace" Would dbug_print_opt_trace() be a better name? + const char *dbug_print_items(Item *item) + { + *big_buf= 0; + + dbug_add_print_items( item ); + return big_buf; + } This function doesn't seem to be needed? For the rest of the functions: as discussed on Slack, we can add them in a separate commit if we decide that they're useful. Could you provide a very short text describing: The entry point (which function is to be called from gdb?). The use case. Is it "to print List<Item>" ? I don't recall seeing this data structure in the server... Is there a lot of need to print Item_cond and then print its children? This looks self-repetitive... I'd like to see a log of an example gdb session...

2 2

[Maria-developers] How to get started with contribution.
by Pinak faldu 29 Nov '22

29 Nov '22

Respected sir/madam, I am Pinak Faldu, a computer science undergraduate student. Currently, I am in my first year of a master's degree at Sinhgad College. I am new to open source contributions, but I am very familiar with ReactJS, MongoDB, MySql, Javascript, and C++. I would like to contribute to your organization, but I don't know where to begin. Could you please tell me how to get started? Hoping to hear from you soon. Regards Pinak

2 1

Re: [Maria-developers] 52f489ebccb: MDEV-29069 follow-up: support partially suitable keys
by Sergei Golubchik 02 Nov '22

02 Nov '22

Hi, Nikita, This review applies to the combined diff e2f8dff^..52f489e On Oct 04, Nikita Malyavin wrote: > diff --git a/sql/log_event.h b/sql/log_event.h > index 91a9eeee10f..8452e665c98 100644 > --- a/sql/log_event.h > +++ b/sql/log_event.h > @@ -5275,8 +5278,11 @@ class Rows_log_event : public Log_event > uchar *m_key; /* Buffer to keep key value during searches */ > KEY *m_key_info; /* Pointer to KEY info for m_key_nr */ > uint m_key_nr; /* Key number */ > + uint m_key_parts_suit; /* A number of key_parts suited to lookup */ that's very confusing name. Took me a while to understand what you meant. Better call it m_usable_key_parts > bool master_had_triggers; /* set after tables opening */ > > + uint key_parts_suit_event(const KEY *key) const; and this could be "get_usable_key_parts()" > + > int find_key(); // Find a best key to use in find_row() > int find_row(rpl_group_info *); > int write_row(rpl_group_info *, const bool); > diff --git a/sql/log_event_server.cc b/sql/log_event_server.cc > index 422d496d5c3..25705d13641 100644 > --- a/sql/log_event_server.cc > +++ b/sql/log_event_server.cc > @@ -6062,7 +6065,7 @@ int Rows_log_event::do_apply_event(rpl_group_info *rgi) > */ > sql_mode_t saved_sql_mode= thd->variables.sql_mode; > if (!is_auto_inc_in_extra_columns()) > - thd->variables.sql_mode= (rpl_data.is_online_alter() ? saved_sql_mode :0) > + thd->variables.sql_mode= (m_online_alter ? saved_sql_mode :0) wouldn't it be more correct to use saved_sql_mode also in normal replication? > | MODE_NO_AUTO_VALUE_ON_ZERO; > > // row processing loop > @@ -7949,14 +7952,35 @@ uint8 Write_rows_log_event::get_trg_event_map() > **************************************************************************/ > > #if defined(HAVE_REPLICATION) > -/* > - Compares table->record[0] and table->record[1] > +/** > + @brief Compares table->record[0] and table->record[1] > + > + @param masrter_columns a number of columns on the source replica, > + 0 if ONLINE ALTER TABLE > > - Returns TRUE if different. > + @returns true if different. > */ > -static bool record_compare(TABLE *table) > +static bool record_compare(TABLE *table, uint master_columns) > { > - bool result= FALSE; > + bool result= false; > + > + /* > + Determine if the optimized check is possible (and we can > + goto record_compare_exit). > + We should ensure that all extra columns (i.e. fieldnr > master_columns) > + have explicit values. If not, we will exclude them from comparison, > + as they can contain non-deterministic values. > + > + master_columns == 0 case is optimization for ONLINE ALTER to check > + all columns fast. > + */ > + bool all_values_set= master_columns == 0 > + && bitmap_is_set_all(&table->has_value_set); > + for (uint i = master_columns; all_values_set && i < table->s->fields; i++) > + { > + all_values_set= bitmap_is_set(&table->has_value_set, i); > + } wait, this doesn't make any sense. You only do the loop if all_values_set is true. But it is true only if bitmap_is_set_all(&table->has_value_set). There is no point in doing bitmap_is_set() for individual bits, when you already know that all bits are set :) shouldn't it simply be bool all_values_set= bitmap_is_set_all(&table->has_value_set); ? but see below, perhaps you don't need all_values_set at all > + > /** > Compare full record only if: > - there are no blob fields (otherwise we would also need > @@ -7969,40 +7993,43 @@ static bool record_compare(TABLE *table) > */ > if ((table->s->blob_fields + > table->s->varchar_fields + > - table->s->null_fields) == 0) > + table->s->null_fields) == 0 > + && all_values_set) > { > result= cmp_record(table,record[1]); > goto record_compare_exit; > } > > /* Compare null bits */ > - if (memcmp(table->null_flags, > - table->null_flags+table->s->rec_buff_length, > - table->s->null_bytes)) > + if (all_values_set && memcmp(table->null_flags, > + table->null_flags+table->s->rec_buff_length, > + table->s->null_bytes)) > { > - result= TRUE; // Diff in NULL value > + result= true; // Diff in NULL value > goto record_compare_exit; > } > > /* Compare fields */ > for (Field **ptr=table->field ; *ptr ; ptr++) > { > + Field *f= *ptr; > if (table->versioned() && (*ptr)->vers_sys_field()) > { > continue; > } > - /** > - We only compare field contents that are not null. > - NULL fields (i.e., their null bits) were compared > - earlier. > - */ > - if (!(*(ptr))->is_null()) > + /* > + We only compare fields that exist on the source (or in ONLINE > + ALTER case, that were in the original table). For reference, > + replica tables can also contain extra fields. > + */ > + if (f->field_index > master_columns && !f->has_explicit_value()) > + continue; same question as in a previous review. Do you mean that when f->field_index < master_columns, it's ok to compare values below even when !f->has_explicit_value() ? > + > + if (f->is_null() != f->is_null(table->s->rec_buff_length) > + || f->cmp_binary_offset(table->s->rec_buff_length)) may be it's better to create a fast copy of this function, like if (bitmap_is_set_all(&table->has_value_set)) { ... old record_compare() code starting from if ((table->s->blob_fields + table->s->varchar_fields + table->s->null_fields) == 0) } else { ... your code, only the loop over fields. } in this case you won't even need all_values_set or master_columns > { > - if ((*ptr)->cmp_binary_offset(table->s->rec_buff_length)) > - { > - result= TRUE; > - goto record_compare_exit; > - } > + result= true; > + goto record_compare_exit; > } > } > > @@ -8010,6 +8037,32 @@ static bool record_compare(TABLE *table) > return result; > } > > +/** > + Newly added fields with non-deterministic defaults (i.e. DEFAULT(RANDOM()), > + CURRENT_TIMESTAMP, AUTO_INCREMENT) should be excluded from key search. > + Basically we exclude all the default-filled fields based on > + has_explicit_value bitmap. > + */ > +uint Rows_log_event::key_parts_suit_event(const KEY *key) const > +{ > + uint master_columns= m_online_alter ? 0 : m_cols.n_bits; get_master_cols() > + uint p= 0; > + for (;p < key->ext_key_parts; p++) > + { > + Field *f= key->key_part[p].field; > + uint non_deterministic_default= f->default_value && > + f->default_value->flags | VCOL_NOT_STRICTLY_DETERMINISTIC; > + > + int next_number_field= f->table->next_number_field ? > + f->table->next_number_field->field_index : -1; eh? 1. why it's *in the loop* ? 2. why not simply || f == f->table->next_number_field or, to be extra safe || f->table->field[f->field_index] == f->table->next_number_field > + > + if (f->field_index >= master_columns I don't think this condition ^^^ is needed. Same as elsewhere, if f->field_index < master_columns but !bitmap_is_set, does it mean the keyseg is ok? I don't think so. > + && !bitmap_is_set(&m_table->has_value_set, f->field_index) > + && (non_deterministic_default || next_number_field == f->field_index)) > + break; > + } > + return p; > +} > > /** > Find the best key to use when locating the row in @c find_row(). > @@ -8024,13 +8077,26 @@ static bool record_compare(TABLE *table) > */ > int Rows_log_event::find_key() > { > - uint i, best_key_nr, last_part; > + uint i, best_key_nr; > KEY *key, *UNINIT_VAR(best_key); > ulong UNINIT_VAR(best_rec_per_key), tmp; > DBUG_ENTER("Rows_log_event::find_key"); > DBUG_ASSERT(m_table); > > + #ifdef DBUG_TRACE #ifndef DBUG_OFF > + auto _ = make_scope_exit([this]() { > + DBUG_EXECUTE_IF("rpl_report_chosen_key", > + push_warning_printf(m_table->in_use, > + Sql_condition::WARN_LEVEL_NOTE, > + ER_UNKNOWN_ERROR, "Key chosen: %d", > + m_key_nr == MAX_KEY ? > + -1 : m_key_nr);); > + }); > + #endif > + > + m_key_nr= MAX_KEY; > best_key_nr= MAX_KEY; > + uint parts_suit= 0; > > /* > Keys are sorted so that any primary key is first, followed by unique keys, > @@ -8041,33 +8107,38 @@ int Rows_log_event::find_key() > { > if (!m_table->s->keys_in_use.is_set(i)) > continue; > + parts_suit= key_parts_suit_event(key); > + if (!parts_suit) > + continue; > /* > We cannot use a unique key with NULL-able columns to uniquely identify > a row (but we can still select it for range scan below if nothing better > is available). > */ > - if ((key->flags & (HA_NOSAME | HA_NULL_PART_KEY)) == HA_NOSAME) > + if ((key->flags & (HA_NOSAME | HA_NULL_PART_KEY)) == HA_NOSAME && > + parts_suit == key->user_defined_key_parts) this doesn't make use of ext key parts. Should be something like if (parts_suit == key->ext_key_parts || (parts_suit >= key->user_defined_key_parts && (key->flags & ....))) > { > best_key_nr= i; > best_key= key; > + m_key_parts_suit= parts_suit; > break; > } > /* > We can only use a non-unique key if it allows range scans (ie. skip > FULLTEXT indexes and such). > */ > - last_part= key->user_defined_key_parts - 1; > DBUG_PRINT("info", ("Index %s rec_per_key[%u]= %lu", > - key->name.str, last_part, key->rec_per_key[last_part])); > - if (!(m_table->file->index_flags(i, last_part, 1) & HA_READ_NEXT)) > + key->name.str, parts_suit, key->rec_per_key[parts_suit])); > + if (!(m_table->file->index_flags(i, parts_suit, 1) & HA_READ_NEXT)) > continue; > > - tmp= key->rec_per_key[last_part]; > + tmp= key->rec_per_key[parts_suit - 1]; > if (best_key_nr == MAX_KEY || (tmp > 0 && tmp < best_rec_per_key)) > { > best_key_nr= i; > best_key= key; > best_rec_per_key= tmp; > + m_key_parts_suit= parts_suit; > } > } > > @@ -8494,12 +8568,20 @@ Delete_rows_log_event::do_before_row_operations(const Slave_reporting_capability > if (get_flags(STMT_END_F)) > status_var_increment(thd->status_var.com_stat[SQLCOM_DELETE]); > > + const uchar *curr_row_end= m_curr_row_end; > + unpack_row(rgi, m_table, m_width, m_curr_row, &m_cols, > + &curr_row_end, &m_master_reclength, m_rows_end); > + > + KEY *pk_info= m_table->key_info + m_table->s->primary_key; > if ((m_table->file->ha_table_flags() & HA_PRIMARY_KEY_REQUIRED_FOR_POSITION) && > - m_table->s->primary_key < MAX_KEY) > + m_table->s->primary_key < MAX_KEY && > + key_parts_suit_event(pk_info) == pk_info->ext_key_parts) > { > /* > We don't need to allocate any memory for m_key since it is not used. > */ > + m_key_nr= m_table->s->primary_key; > + m_key_parts_suit= pk_info->ext_key_parts; > return 0; > } > if (do_invoke_trigger()) already commented in the previous review that if "We don't need to allocate any memory" than you `return 0` early wthout doing do_invoke_trigger(). Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 11

Re: [Maria-developers] b2800c06b22: MDEV-28545 MyISAM reorganize partition corrupt older table format
by Sergei Golubchik 25 Oct '22

25 Oct '22

Hi, Alexander, Ok to push. One minor comment below: On Oct 25, Alexander Barkov wrote: > revision-id: b2800c06b22 (mariadb-10.4.26-45-gb2800c06b22) > parent(s): 34163154077 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2022-10-25 11:53:39 +0400 > message: > > MDEV-28545 MyISAM reorganize partition corrupt older table format > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index 6aae927800e..a621825c395 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -10068,6 +10068,18 @@ do_continue:; > > set_table_default_charset(thd, create_info, alter_ctx.db); > > + > +#ifdef WITH_PARTITION_STORAGE_ENGINE > + if (!fast_alter_partition) > +#endif To avoid ifdef's you can use if (IF_PARTITIONING(!fast_alter_partition, 1)) > + { > + /* > + We cannot alter partitions and change column data types at the same > + time. > + */ > + Create_field::upgrade_data_types(alter_info->create_list); > + } > + > if (create_info->check_fields(thd, alter_info, > table_list->table_name, table_list->db) || > create_info->fix_period_fields(thd, alter_info)) > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 152295d1cb1: rpl: extra DEFAULTs should not be UPDATEd
by Sergei Golubchik 24 Oct '22

24 Oct '22

Hi, Nikita, As we've discussed on slack, this is wrong. The (serialized) behavior should be as if all DML happened before ALTER. Because if some DML would've happened after ALTER, it would be able to see the new table definition and refer to new columns. On Oct 24, Nikita Malyavin wrote: > revision-id: 152295d1cb1 (mariadb-10.6.1-525-g152295d1cb1) > parent(s): 90274a7428e > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2022-10-14 23:17:59 +0300 > message: > > rpl: extra DEFAULTs should not be UPDATEd > > diff --git a/sql/log_event_server.cc b/sql/log_event_server.cc > index 25705d13641..a72005d70ab 100644 > --- a/sql/log_event_server.cc > +++ b/sql/log_event_server.cc > @@ -8832,11 +8832,17 @@ Update_rows_log_event::do_exec_row(rpl_group_info *rgi) > message= thd->wsrep_info; > #endif /* WSREP_PROC_INFO */ > > - /* this also updates m_curr_row_end */ > thd_proc_info(thd, message); > - if (unlikely((error= unpack_current_row(rgi, &m_cols_ai)))) > + Field **default_field= m_table->default_field; > + m_table->default_field= NULL; > + /* this also updates m_curr_row_end */ > + error= unpack_current_row(rgi, &m_cols_ai); > + m_table->default_field= default_field; > + > + if (unlikely(error)) > goto err; > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] MDEV-29624 MDEV-29655 Fix ASAN errors on pushdown of derived table
by Sergey Petrunia 23 Oct '22

23 Oct '22

Hi Oleg, > commit 4cad1e16a85114815075e5706d6a8faf5e8fc2e5 > Author: Oleg Smirnov <olernov(a)gmail.com> > Date: Wed Oct 19 13:26:19 2022 +0400 > > MDEV-29624 MDEV-29655 Fix ASAN errors on pushdown of derived table > > Deallocation of TABLE_LIST::dt_handler and TABLE_LIST::pushdown_derived > was performed in multiple places if code. This not only made the code > more difficult to maintain but also led to memory leaks and > ASAN heap-use-after-free errors. > This commit puts deallocation of TABLE_LIST::dt_handler and > TABLE_LIST::pushdown_derived to the single point - JOIN::cleanup() I think the patch is ok to push except for this part: > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index a91b4571b21..4b8956132e8 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -14029,11 +14030,11 @@ void JOIN::cleanup(bool full) > delete pushdown_query; > pushdown_query= 0; > > - if (!join_tab) > + List_iterator<TABLE_LIST> li(*join_list); > + TABLE_LIST *table_ref; > + while ((table_ref= li++)) > { > - List_iterator<TABLE_LIST> li(*join_list); > - TABLE_LIST *table_ref; > - while ((table_ref= li++)) > + if (!join_tab) > { > if (table_ref->table && > table_ref->jtbm_subselect && > @@ -14043,6 +14044,14 @@ void JOIN::cleanup(bool full) > table_ref->table= NULL; > } > } > + if (table_ref->pushdown_derived) > + { > + delete table_ref->pushdown_derived; > + table_ref->pushdown_derived= NULL; > + } > + delete table_ref->dt_handler; > + table_ref->dt_handler= NULL; > + > } > } > /* Restore ref array to original state */ What do you think of moving the cleanup to JOIN_TAB::cleanup()? I did the following and it seems to work: diff --git a/sql/sql_select.cc b/sql/sql_select.cc index a91b4571b21..11cf3e13365 100644 --- a/sql/sql_select.cc +++ b/sql/sql_select.cc @@ -13529,6 +13530,16 @@ void JOIN_TAB::cleanup() select= 0; delete quick; quick= 0; + if (table && table->pos_in_table_list) + { + TABLE_LIST *tl= table->pos_in_table_list; + + delete tl->pushdown_derived; + tl->pushdown_derived= NULL; + + delete tl->dt_handler; + tl->dt_handler= NULL; + } if (rowid_filter) { delete rowid_filter; BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

1 0

Re: [Maria-developers] 39e20ca7e28: MDEV-29748 ASAN errors or server crash in File_parser::parse upon concurrent view operations
by Sergei Golubchik 21 Oct '22

21 Oct '22

Hi, Oleksandr, Looks good. Just one question below: On Oct 19, Oleksandr Byelkin wrote: > revision-id: 39e20ca7e28 (mariadb-10.3.36-107-g39e20ca7e28) > parent(s): d6707ab11f6 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2022-10-18 16:38:09 +0200 > message: > > MDEV-29748 ASAN errors or server crash in File_parser::parse upon concurrent view operations > > Read the version of the view share when we read definition to prevent > simultaniouse access to a view table SHARE (and so its MEM_ROOT) > from different threads. > > diff --git a/sql/sql_view.cc b/sql/sql_view.cc > index 68a8410f559..1e4a740f871 100644 > --- a/sql/sql_view.cc > +++ b/sql/sql_view.cc > @@ -1175,19 +1175,24 @@ static int mysql_register_view(THD *thd, TABLE_LIST *view, > bool mariadb_view_version_get(TABLE_SHARE *share) > { > DBUG_ASSERT(share->is_view); > + DBUG_ASSERT(share->tabledef_version.length == 0); > > if (!(share->tabledef_version.str= > (uchar*) alloc_root(&share->mem_root, > MICROSECOND_TIMESTAMP_BUFFER_SIZE))) > return TRUE; > - share->tabledef_version.length= 0; // safety if the drfinition file is brocken > > DBUG_ASSERT(share->view_def != NULL); > if (share->view_def->parse((uchar *) &share->tabledef_version, NULL, > view_timestamp_parameters, 1, > &file_parser_dummy_hook)) > + { > + // safety if the definition file is brocken > + share->tabledef_version.length= 0; when can view_def->parse() fail? _Only_ if the definition is invalid? Can it succeed for one TABLE and fail for another, both of the same TABLE_SHARE? > return TRUE; > + } > DBUG_ASSERT(share->tabledef_version.length == MICROSECOND_TIMESTAMP_BUFFER_SIZE-1); > + > return FALSE; > } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 4

Re: [Maria-developers] 99dcf061c8c: MDEV-29481 mariadb-upgrade prints confusing statement
by Sergei Golubchik 21 Oct '22

21 Oct '22

Hi, Alexander, ok to push! On Oct 21, Alexander Barkov wrote: > revision-id: 99dcf061c8c (mariadb-10.4.26-55-g99dcf061c8c) > parent(s): 1be451ca797 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2022-10-21 14:31:02 +0400 > message: > > MDEV-29481 mariadb-upgrade prints confusing statement > > This is a new version of the patch instead of the reverted: > > MDEV-28727 ALTER TABLE ALGORITHM=NOCOPY does not work after upgrade > > Ignore the difference in key packing flags HA_BINARY_PACK_KEY and HA_PACK_KEY > during ALTER to allow ALGORITHM=INSTANT and ALGORITHM=NOCOPY in more cases. > > If for some reasons (e.g. due to a bug fix such as MDEV-20704) these > cumulative (over all segments) flags in KEY::flags are different for > the old and new table inside compare_keys_but_name(), the difference > in HA_BINARY_PACK_KEY and HA_PACK_KEY in KEY::flags is not really important: > > MyISAM and Aria can handle such cases well: per-segment flags are stored in > MYI and MAI files anyway and they are read during ha_myisam::open() > ha_maria::open() time. So indexes get opened with correct per-segment > flags that were calculated during the table CREATE time, no matter > what the old (CREATE time) and new (ALTER TIME) per-index compression > flags are, and no matter if they are equal or not. > > All other engine ignore key compression flags, so this change > is safe for other engines as well. > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] c83f87ece4a: MDEV-15990 REPLACE on a precise-versioned table returns duplicate key error (ER_DUP_ENTRY)
by Sergei Golubchik 15 Oct '22

15 Oct '22

Hi, Nikita, Sorry, this comes very late. But the bug is still present in 10.10, and the filter sorts by the date precisely to highlight these old bugs. So, here you are. First comment, please, port it to the latest 10.3. I wasn't able to find the repository with the fix, but the patch itself is avaliable from the pull request as https://github.com/MariaDB/server/commit/c83f87ece4a.patch so it can be salvaged. More comments - see below. On Oct 15, Nikita Malyavin wrote: > From: Nikita Malyavin <nikitamalyavin(a)gmail.com> > Date: Fri, 20 Jul 2018 00:24:40 +1000 > Subject: [PATCH] MDEV-15990: REPLACE on a precise-versioned table returns > * handle zero value in `row_start` `trx_id` > * refuse to do more than one versioned insert on the same transaction and same row > * generalize versioned cases with and without triggers > > diff --git a/sql/sql_insert.cc b/sql/sql_insert.cc > index dda469a1426d5..cd2690128de67 100644 > --- a/sql/sql_insert.cc > +++ b/sql/sql_insert.cc > @@ -1928,32 +1928,15 @@ int write_record(THD *thd, TABLE *table,COPY_INFO *info) > For system versioning wa also use path through delete since we would > save nothing through this cheating. > */ > - if (last_uniq_key(table,key_nr) && > + if (last_uniq_key(table,key_nr) && !table->versioned() && ok! > !table->file->referenced_by_foreign_key() && > (!table->triggers || !table->triggers->has_delete_triggers())) > { > - if (table->versioned(VERS_TRX_ID) && table->vers_write) > - { > - bitmap_set_bit(table->write_set, table->vers_start_field()->field_index); > - table->vers_start_field()->store(0, false); > - } > - if (unlikely(error= table->file->ha_update_row(table->record[1], > - table->record[0])) && > - error != HA_ERR_RECORD_IS_THE_SAME) > + error= table->file->ha_update_row(table->record[1], table->record[0]); > + if (unlikely(error && error != HA_ERR_RECORD_IS_THE_SAME)) > goto err; > if (likely(!error)) > - { > info->deleted++; > - if (table->versioned(VERS_TIMESTAMP) && table->vers_write) > - { > - store_record(table, record[2]); > - error= vers_insert_history_row(table); > - restore_record(table, record[2]); > - if (unlikely(error)) > - goto err; > - } > - } > - else > error= 0; // error was HA_ERR_RECORD_IS_THE_SAME > thd->record_first_successful_insert_id_in_cur_stmt(table->file->insert_id_for_cur_row); > /* > @@ -1969,23 +1952,22 @@ int write_record(THD *thd, TABLE *table,COPY_INFO *info) > TRG_ACTION_BEFORE, TRUE)) > goto before_trg_err; > > - if (!table->versioned(VERS_TIMESTAMP)) > + if (table->versioned(VERS_TRX_ID) && table->vers_write) > + { > + bitmap_set_bit(table->write_set, table->vers_start_field()->field_index); this is conceptually wrong, you cannot set table->write_set in the middle of statement execution. that is you can and it might even work for some engines, but it breaks the API. > + table->vers_start_field()->store(0, false); > + } > + > + if (!table->versioned()) > error= table->file->ha_delete_row(table->record[1]); > else > - { > - store_record(table, record[2]); > - restore_record(table, record[1]); > - table->vers_update_end(); why? is the row_end updated somewhere else? > error= table->file->ha_update_row(table->record[1], > table->record[0]); > - restore_record(table, record[2]); > - } > - if (unlikely(error)) > + if (unlikely(error && error != HA_ERR_RECORD_IS_THE_SAME)) > goto err; > - if (!table->versioned(VERS_TIMESTAMP)) > + if (likely(!error)) > info->deleted++; > - else > - info->updated++; > + error= 0; > if (!table->file->has_transactions()) > thd->transaction.stmt.modified_non_trans_table= TRUE; > if (table->triggers && > @@ -1995,6 +1977,17 @@ int write_record(THD *thd, TABLE *table,COPY_INFO *info) > trg_error= 1; > goto ok_or_after_trg_err; > } > + if (table->versioned(VERS_TIMESTAMP) && table->vers_write) > + { > + store_record(table, record[2]); > + error = vers_insert_history_row(table); if you do that, you shouldn't have been doing ha_update_row() above, as far as I can see. What is that update above for? > + restore_record(table, record[2]); > + if (unlikely(error)) > + goto err; > + } > + if (table->versioned()) > + goto after_trg_n_copied_inc; > + > /* Let us attempt do write_row() once more */ > } > } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 19772f9c277: MDEV-29748 ASAN errors or server crash in File_parser::parse upon concurrent view operations
by Sergei Golubchik 13 Oct '22

13 Oct '22

Hi, Oleksandr, On Oct 13, Oleksandr Byelkin wrote: > revision-id: 19772f9c277 (mariadb-10.3.36-90-g19772f9c277) > parent(s): e05ab0cfc5f > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2022-10-12 15:59:46 +0200 > message: > > MDEV-29748 ASAN errors or server crash in File_parser::parse upon concurrent view operations > > Prevent simultaniouse access to a view table SHARE (and so its MEM_ROOT) from different threads. If you're writing into share, why not to do it when a share is populated, then you wouldn't need a mutex. And it'd be a logical place to initialize the share anyway. Doing it at some arbitrary point in time later looks strange. I mean, why not to call this mariadb_view_version_get after the line share->view_def= sql_parse_prepare(&pathstr, &share->mem_root, true); where it logically belongs? > + mysql_mutex_lock(&share->LOCK_share); > > if (!(share->tabledef_version.str= > (uchar*) alloc_root(&share->mem_root, > MICROSECOND_TIMESTAMP_BUFFER_SIZE))) Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] b1b38b64598: MDEV-29181 Potential corruption on FK update on a table with vcol index
by Sergei Golubchik 10 Oct '22

10 Oct '22

Hi, Nikita, On Aug 30, Nikita Malyavin wrote: > revision-id: b1b38b64598 (mariadb-10.5.17-4-gb1b38b64598) > parent(s): 3b656ac8c17 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2022-08-23 00:27:39 +0300 > message: > > MDEV-29181 Potential corruption on FK update on a table with vcol > index > > vc_templ->mysql_table concept is completely broken. This table pointer > persists with a global access and can be overwritten by arbitrary > thread. Like it wasn't enough, it also could become invalid after > eviction from tc cache. this is a bit misleading. I agree it's completely broken, though :) But not "this table pointer", because, of course it persists with a global access and can be overwritten and so on. This is by design, it's protected by mysql_table_query_id. The problem is that table (dict_table_t) argument of innodb_find_table_for_vc() points to a shared data structure, not thread local, as I believed. So vc_templ is shared too. I suggest to rewrite the comment like vc_templ->mysql_table concept is completely broken. table argument of innodb_find_table_for_vc() points to a shared data structure dict_table_t, and thus vc_templ is shared too and cannot be used for a thread-local cache. > This new solution simply does the following: > * Sets up a referenced table list in TABLE instance (sql_base.cc) > * Iterates through it along with dict_table_t::referenced_set > (row_upd_check_references_constraints) > * Passes corresponding prebuilt through a call chain to > row_ins_foreign_check_on_constraint > * Sets up newly created upd_node_t::prebuilt field and uses it > accordingly is cascade update. Is this upd_node_t::prebuilt used anywhere? As a prebuilt, I mean. I couldn't find it (it's a complex patch, so I couldn've missed it). As far as I can see, it's only used to store a pointer to TABLE. So it seems to me than a simpler fix for this bug could be: * remove vc_templ caching (mysql_table and mysql_table_query_id) * store TABLE* in upd_node_t. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

3 6

Re: [Maria-developers] dcfc60ea2ee: MDEV-29481 mariadb-upgrade prints confusing statement
by Sergei Golubchik 05 Oct '22

05 Oct '22

Hi, Alexander, See a couple of questions below: On Oct 04, Alexander Barkov wrote: > revision-id: dcfc60ea2ee (mariadb-10.4.26-28-gdcfc60ea2ee) > parent(s): 6c2c825e503 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2022-10-03 10:32:42 +0400 > message: > > MDEV-29481 mariadb-upgrade prints confusing statement > > This is a new version of the patch instead of the reverted: > > MDEV-28727 ALTER TABLE ALGORITHM=NOCOPY does not work after upgrade > diff --git a/mysql-test/main/alter_table_debug_aria.test b/mysql-test/main/alter_table_debug_aria.test > new file mode 100644 > index 00000000000..22df57686e1 > --- /dev/null > +++ b/mysql-test/main/alter_table_debug_aria.test > @@ -0,0 +1,36 @@ > +--source include/have_debug.inc > +--source include/have_maria.inc > + > +--echo # > +--echo # Start of 10.4 tests > +--echo # > + > +--echo # > +--echo # MDEV-28727 ALTER TABLE ALGORITHM=NOCOPY does not work after upgrade > +--echo # > + > +CREATE TABLE t1 (a DOUBLE NOT NULL, KEY(a)) ENGINE=ARIA; > +ALTER TABLE t1 MODIFY a DOUBLE NOT NULL DEFAULT 10, ALGORITHM=NOCOPY; > +DROP TABLE t1; > + > +CREATE TABLE t1 (a DOUBLE NOT NULL, KEY(a)) ENGINE=ARIA; > +ALTER TABLE t1 MODIFY a DOUBLE NOT NULL DEFAULT 10, ALGORITHM=INSTANT; > +DROP TABLE t1; > + > +SET SESSION debug_dbug='+d,emulate_pre_mdev_20704'; > +CREATE TABLE t1 (a DOUBLE NOT NULL, KEY(a)) ENGINE=ARIA; Why did you add these debug-only tests with the emulate_pre_mdev_20704 hack, when you have perfect non-debug tests with real pre-MDEV-20704 created tables ? > +SET SESSION debug_dbug=''; > +--error ER_ALTER_OPERATION_NOT_SUPPORTED > +ALTER TABLE t1 MODIFY a DOUBLE NOT NULL DEFAULT 10, ALGORITHM=INSTANT; > +DROP TABLE t1; > + > +SET SESSION debug_dbug='+d,emulate_pre_mdev_20704'; > +CREATE TABLE t1 (a DOUBLE NOT NULL, KEY(a)) ENGINE=ARIA; > +SET SESSION debug_dbug=''; > +--error ER_ALTER_OPERATION_NOT_SUPPORTED > +ALTER TABLE t1 MODIFY a DOUBLE NOT NULL DEFAULT 10, ALGORITHM=NOCOPY; > +DROP TABLE t1; > + > +--echo # > +--echo # End of 10.4 tests > +--echo # > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index 2436031200a..c192a4553a9 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -6748,11 +6753,13 @@ Compare_keys compare_keys_but_name(const KEY *table_key, const KEY *new_key, > const KEY *const new_pk, > const KEY *const old_pk) > { > - if (table_key->algorithm != new_key->algorithm) > + ulong key_flag_mask= HA_KEYFLAG_MASK | > + table->file->key_pack_flags_supported(); I don't understand. You said that these KEY->flags do not affect what the engine does. Why to compare them at all? > + > + if ((table_key->flags & key_flag_mask) != (new_key->flags & key_flag_mask)) > return Compare_keys::NotEqual; > > - if ((table_key->flags & HA_KEYFLAG_MASK) != > - (new_key->flags & HA_KEYFLAG_MASK)) > + if (table_key->algorithm != new_key->algorithm) > return Compare_keys::NotEqual; > > if (table_key->user_defined_key_parts != new_key->user_defined_key_parts) Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] e2f8dffc056: MDEV-29069 ER_KEY_NOT_FOUND on online autoinc addition + concurrent DELETE
by Sergei Golubchik 03 Oct '22

03 Oct '22

Hi, Nikita, Few comments below On Aug 27, Nikita Malyavin wrote: > commit e2f8dffc056 > Author: Nikita Malyavin <nikitamalyavin(a)gmail.com> > Date: Mon Jul 25 23:52:24 2022 +0300 > > diff --git a/sql/log_event_server.cc b/sql/log_event_server.cc > index 422d496d5c3..ca17c5c07a8 100644 > --- a/sql/log_event_server.cc > +++ b/sql/log_event_server.cc > @@ -7949,14 +7952,35 @@ uint8 Write_rows_log_event::get_trg_event_map() > **************************************************************************/ > > #if defined(HAVE_REPLICATION) > -/* > - Compares table->record[0] and table->record[1] > +/** > + @brief Compares table->record[0] and table->record[1] > + > + @param masrter_columns a number of columns on the source replica, > + 0 if ONLINE ALTER TABLE > > - Returns TRUE if different. > + @returns true if different. > */ > -static bool record_compare(TABLE *table) > +static bool record_compare(TABLE *table, uint master_columns) > { > - bool result= FALSE; > + bool result= false; > + > + /* > + Determine if the optimized check is possible (and we can > + goto record_compare_exit). > + We should ensure that all extra columns (i.e. fieldnr > master_columns) > + have explicit values. If not, we will exclude them from comparison, > + as they can contain non-deterministic values. > + > + master_columns == 0 case is optimization for ONLINE ALTER to check > + all columns fast. > + */ > + bool all_values_set= master_columns == 0 > + && bitmap_is_set_all(&table->has_value_set); > + for (uint i = master_columns; all_values_set && i < table->s->fields; i++) > + { > + all_values_set= bitmap_is_set(&table->has_value_set, i); > + } it'd be safer and faster to do all_values_set= bitmap_is_set_all(&table->has_value_set); and, please, add replication (not online alter) tests: 1. slave has more columns 2. binlog row format is "minimal", this might trigger it too > + > /** > Compare full record only if: > - there are no blob fields (otherwise we would also need > @@ -7969,47 +7993,68 @@ static bool record_compare(TABLE *table) > */ > if ((table->s->blob_fields + > table->s->varchar_fields + > - table->s->null_fields) == 0) > + table->s->null_fields) == 0 > + && all_values_set) > { > result= cmp_record(table,record[1]); > goto record_compare_exit; > } > > /* Compare null bits */ > - if (memcmp(table->null_flags, > + if (all_values_set && memcmp(table->null_flags, > table->null_flags+table->s->rec_buff_length, > table->s->null_bytes)) > { > - result= TRUE; // Diff in NULL value > + result= true; // Diff in NULL value > goto record_compare_exit; > } > > /* Compare fields */ > for (Field **ptr=table->field ; *ptr ; ptr++) > { > + Field *f= *ptr; > if (table->versioned() && (*ptr)->vers_sys_field()) > { > continue; > } > - /** > - We only compare field contents that are not null. > - NULL fields (i.e., their null bits) were compared > - earlier. > + /* > + We only compare fields that exist on the source (or in ONLINE > + ALTER case, that were in the original table). For reference, > + replica tables can also contain extra fields. > */ > - if (!(*(ptr))->is_null()) > - { > - if ((*ptr)->cmp_binary_offset(table->s->rec_buff_length)) > + if (f->field_index > master_columns && !f->has_explicit_value()) > + continue; why do you check for f->field_index > master_columns? isn't !f->has_explicit_value() the one that decided? in what case will be ok to compare values when f->field_index < master_columns but no explicit value was set? > + > + if (f->is_null() != f->is_null(table->s->rec_buff_length) > + || f->cmp_binary_offset(table->s->rec_buff_length)) > { > - result= TRUE; > + result= true; > goto record_compare_exit; > } > } > - } > > record_compare_exit: > return result; > } > > +/** > + Newly added fields with non-deterministic defaults (i.e. DEFAULT(RANDOM()), > + CURRENT_TIMESTAMP, AUTO_INCREMENT) should be excluded from key search. > + Basically we exclude all the default-filled fields based on > + has_explicit_value bitmap. > + */ > +bool Rows_log_event::key_suits_event(const KEY *key) const > +{ > + uint master_columns= m_online_alter ? 0 : m_cols.n_bits; > + for (uint p= 0; p < key->ext_key_parts; p++) > + { > + uint field_idx= key->key_part[p].fieldnr - 1; > + if (field_idx >= master_columns > + && !bitmap_is_set(&m_table->has_value_set, field_idx)) > + return false; > + } > + return true; > +} > > /** > Find the best key to use when locating the row in @c find_row(). > @@ -8030,6 +8075,18 @@ int Rows_log_event::find_key() > DBUG_ENTER("Rows_log_event::find_key"); > DBUG_ASSERT(m_table); > > + #ifdef DBUG_TRACE > + auto _ = make_scope_exit([this]() { > + DBUG_EXECUTE_IF("rpl_report_chosen_key", > + push_warning_printf(m_table->in_use, > + Sql_condition::WARN_LEVEL_NOTE, > + ER_UNKNOWN_ERROR, "Key chosen: %d", > + m_key_nr == MAX_KEY ? > + -1 : m_key_nr);); > + }); > + #endif > + > + m_key_nr= MAX_KEY; it would've been simpler and shorter to do it the old style, with a goto > best_key_nr= MAX_KEY; > > /* > @@ -8041,6 +8098,8 @@ int Rows_log_event::find_key() > { > if (!m_table->s->keys_in_use.is_set(i)) > continue; > + if (!key_suits_event(key)) > + continue; 1. again, add a replication test, please. when, e.g., slave has more columns and a unique index over new columns. 2. going over all keyparts for every event is rather redundant. you can calculate the set of usable keys once and put it, for example, in m_table->keys_in_use_for_query. > /* > We cannot use a unique key with NULL-able columns to uniquely identify > a row (but we can still select it for range scan below if nothing better > @@ -8221,7 +8280,8 @@ int Rows_log_event::find_row(rpl_group_info *rgi) > DBUG_DUMP("record[0]", table->record[0], table->s->reclength); > > if ((table->file->ha_table_flags() & HA_PRIMARY_KEY_REQUIRED_FOR_POSITION) && > - table->s->primary_key < MAX_KEY) > + table->s->primary_key < MAX_KEY && > + m_key_nr == table->s->primary_key) add a replication test for that: slave has an extra column which is a PK, and HA_PRIMARY_KEY_REQUIRED_FOR_POSITION is set (InnoDB) > { > /* > Use a more efficient method to fetch the record given by > @@ -8494,12 +8554,18 @@ Delete_rows_log_event::do_before_row_operations(const Slave_reporting_capability > if (get_flags(STMT_END_F)) > status_var_increment(thd->status_var.com_stat[SQLCOM_DELETE]); > > + const uchar *curr_row_end= m_curr_row_end; > + unpack_row(rgi, m_table, m_width, m_curr_row, &m_cols, > + &curr_row_end, &m_master_reclength, m_rows_end); > + > if ((m_table->file->ha_table_flags() & HA_PRIMARY_KEY_REQUIRED_FOR_POSITION) && > - m_table->s->primary_key < MAX_KEY) > + m_table->s->primary_key < MAX_KEY > + && key_suits_event(&m_table->key_info[m_table->s->primary_key])) > { > /* > We don't need to allocate any memory for m_key since it is not used. > */ > + m_key_nr= m_table->s->primary_key; > return 0; > } This is very strange if(). 1. it's only for delete, not for update 2. it's before invoking triggers (bug) 3. it's wrong if the master table has a different primary key (bug) I'd suggest to remove it completely. An optimization of avoiding a malloc can be done inside find_key() after the best key is found. > if (do_invoke_trigger()) Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 5265f7001c6: MDEV-17124: mariadb 10.1.34, views and prepared statements: ERROR 1615 (HY000): Prepared statement needs to be re-prepared
by Sergei Golubchik 28 Sep '22

28 Sep '22

Hi, Oleksandr, Few minor comments, see below On Sep 27, Oleksandr Byelkin wrote: > revision-id: 5265f7001c6 (mariadb-10.3.36-60-g5265f7001c6) > parent(s): 86953d3df0a > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2022-09-27 10:23:31 +0200 > message: > > MDEV-17124: mariadb 10.1.34, views and prepared statements: ERROR 1615 (HY000): Prepared statement needs to be re-prepared > > The problem is that if table definition cache (TDC) is full of real tables > which are in tables cache, view definition can not stay there so will be > removed by its own underlying tables. > In situation above old mechanism of detection matching definition in PS > and current version always require reprepare and so prevent executing > the PS. > > One work around is to increase TDC, other - improve version check for > views/triggers (which is done here). Now in suspicious cases we check: > - timestamp (microseconds) of the view to be sure that version really > have changed; > - time (microseconds) of creation of a trigger related to time > (microseconds) of statement preparation. > diff --git a/sql/table.cc b/sql/table.cc > index 506195127b2..f289c2052cb 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -8861,6 +8861,63 @@ bool TABLE_LIST::is_with_table() > return derived && derived->with_element; > } > > + > +/** > + Check if the definition are the same. > + > + If versions do not match it check definitions (with checking and setting > + trigger definition versions (times) > + > + @sa check_and_update_table_version() > +*/ > + > +bool TABLE_LIST::is_the_same_definition(THD* thd, TABLE_SHARE *s) > +{ > + enum enum_table_ref_type tp= s->get_table_ref_type(); > + if (m_table_ref_type == tp) > + { > + bool res= m_table_ref_version == s->get_table_ref_version(); > + > + /* > + If definition is different check content version > + */ if res == true, why do you need to check tabledef_version? > + if (tabledef_version.length && > + tabledef_version.length == s->tabledef_version.length && > + memcmp(tabledef_version.str, s->tabledef_version.str, > + tabledef_version.length) == 0) > + { > + if (table && table->triggers) > + { > + my_hrtime_t hr_stmt_prepare= thd->hr_prepare_time; > + if (hr_stmt_prepare.val) > + for(uint i= 0; i < TRG_EVENT_MAX; i++) > + for (uint j= 0; j < TRG_ACTION_MAX; j++) > + { > + Trigger *tr= > + table->triggers->get_trigger((trg_event_type)i, > + (trg_action_time_type)j); > + if (tr) > + if (hr_stmt_prepare.val <= tr->hr_create_time.val) > + { > + set_tabledef_version(s); > + return FALSE; > + } > + } > + } > + set_table_id(s); > + return TRUE; > + } > + else > + tabledef_version.length= 0; > + return res; > + } > + else > + set_tabledef_version(s); why not set_table_id() ? > + return FALSE; > +} > + > + > uint TABLE_SHARE::actual_n_key_parts(THD *thd) > { > return use_ext_keys && > diff --git a/sql/sql_trigger.h b/sql/sql_trigger.h > index ae3d1738b16..f7a2ccf2abc 100644 > --- a/sql/sql_trigger.h > +++ b/sql/sql_trigger.h > @@ -198,7 +198,7 @@ class Table_triggers_list: public Sql_alloc > */ > List<ulonglong> definition_modes_list; > /** Create times for triggers */ > - List<ulonglong> create_times; > + List<my_hrtime_t> hr_create_times; I suspect it might be UB. You use FILE_OPTIONS_ULLLIST for hr_create_times, perhaps it's safer to use List<ulonglong> here. > > List<LEX_CSTRING> definers_list; > > @@ -328,4 +328,14 @@ bool mysql_create_or_drop_trigger(THD *thd, TABLE_LIST *tables, bool create); > extern const char * const TRG_EXT; > extern const char * const TRN_EXT; > > + > +/** > + Make time compatible with MySQL 5.7 trigger time. > +*/ > + > +inline my_hrtime_t make_hr_time(my_time_t time, ulong time_sec_part) > +{ > + return my_hrtime_t({((ulonglong) time)*1000000 + time_sec_part}); > +} better put it next to hrtime_to_time/etc and, please, always 'static inline' > + > #endif /* SQL_TRIGGER_INCLUDED */ > diff --git a/sql/sql_view.cc b/sql/sql_view.cc > index 17dea643144..177d01a312e 100644 > --- a/sql/sql_view.cc > +++ b/sql/sql_view.cc > @@ -1154,7 +1163,32 @@ static int mysql_register_view(THD *thd, TABLE_LIST *view, > DBUG_RETURN(error); > } > > +/** > + Check is TABLE_LEST and SHARE match > + @param[in] view TABLE_LIST of the view > + @param[in] share Share object of view > + > + @return false on error or misspatch Eh. I don't understad this comment at all. First, I thought you made two typos, s/is/if/ and s/misspatch/mismatch/. But this function doesn't check if something matches, if gets the view version, as the name says, the name's good. But the comment seems to be from is_the_same_definition() > +*/ > + > +bool mariadb_view_version_get(TABLE_SHARE *share) > +{ > + DBUG_ASSERT(share->is_view); > + > + if (!(share->tabledef_version.str= > + (uchar*) alloc_root(&share->mem_root, > + MICROSECOND_TIMESTAMP_BUFFER_SIZE))) > + return TRUE; > + share->tabledef_version.length= 0; // safety if the drfinition file is brocken > > + DBUG_ASSERT(share->view_def != NULL); > + if (share->view_def->parse((uchar *) &share->tabledef_version, NULL, > + view_timestamp_parameters, 1, > + &file_parser_dummy_hook)) > + return TRUE; > + DBUG_ASSERT(share->tabledef_version.length == MICROSECOND_TIMESTAMP_BUFFER_SIZE-1); > + return FALSE; > +} > > /** > read VIEW .frm and create structures Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 3

Re: [Maria-developers] 349283c5e7a: MDEV-17124: mariadb 10.1.34, views and prepared statements: ERROR 1615 (HY000): Prepared statement needs to be re-prepared
by Sergei Golubchik 27 Sep '22

27 Sep '22

Hi, Oleksandr, On Jan 27, Oleksandr Byelkin wrote: > commit 349283c5e7a > Author: Oleksandr Byelkin <sanja(a)mariadb.com> > Date: Wed Apr 17 15:50:59 2019 +0200 > > MDEV-17124: mariadb 10.1.34, views and prepared statements: ERROR 1615 (HY000): Prepared statement needs to be re-prepared > > The problem is that if table definition cache (TDC) is full of real tables > which are in tables cache, view definition can not stay there so will be > removed by its own underlying tables. > In situation above old mechanism of detection matching definition in PS > and current version always require reprepare and so prevent executing > the PS. > > One work around is to increase TDC, other - improve version check for > views/triggers (which is done here). Now in suspicious cases we check: > - timestamp (ms) of the view to be sure that version really have changed; > - time (ms) of creation of a trigger related to time (ms) of statement > preparation. not "ms" but "us" or "µs". microseconds, not millisecons. > diff --git a/mysql-test/r/ps_ddl.result b/mysql-test/r/ps_ddl.result > index e69c6e06193..005c78a0319 100644 > --- a/mysql-test/r/ps_ddl.result > +++ b/mysql-test/r/ps_ddl.result > @@ -831,8 +831,8 @@ a b c > 20 40 100 > 30 60 150 > call p_verify_reprepare_count(1); > -SUCCESS > - > +ERROR > +Expected: 1, actual: 0 oops, forgot to update? > # Check that we properly handle ALTER VIEW statements. > execute stmt; > a b c > @@ -882,9 +882,9 @@ a b c > 10 50 60 > 20 100 120 > 30 150 180 > -call p_verify_reprepare_count(1); > -SUCCESS > - > +call p_verify_reprepare_count(0); > +ERROR > +Expected: 0, actual: 1 oops, updated too much? > execute stmt; > a b c > 10 50 60 > diff --git a/sql/sql_class.h b/sql/sql_class.h > index 3f0fba8fc10..b9bb9c978fb 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -1079,6 +1079,7 @@ class Statement: public ilink, public Query_arena > > LEX_STRING name; /* name for named prepared statements */ > LEX *lex; // parse tree descriptor > + ulonglong ms_prepare_time; // time of preparation in microseconds "ms" here too :( Better just say my_hrtime_t prepare_time; > /* > Points to the query associated with this statement. It's const, but > we need to declare it char * because all table handlers are written > diff --git a/sql/sql_prepare.cc b/sql/sql_prepare.cc > index 64e4cd30561..a4a24cc1a80 100644 > --- a/sql/sql_prepare.cc > +++ b/sql/sql_prepare.cc > @@ -4177,6 +4177,9 @@ bool Prepared_statement::prepare(const char *packet, uint packet_len) > Query_arena *old_stmt_arena; > DBUG_ENTER("Prepared_statement::prepare"); > DBUG_ASSERT(m_sql_mode == thd->variables.sql_mode); > + > + // The same format as for triggers to compare > + ms_prepare_time= my_hrtime().val; do it at the end instead? > /* > If this is an SQLCOM_PREPARE, we also increase Com_prepare_sql. > However, it seems handy if com_stmt_prepare is increased always, > @@ -4519,6 +4523,22 @@ Prepared_statement::execute_loop(String *expanded_query, > DBUG_ASSERT(thd->get_stmt_da()->sql_errno() == ER_NEED_REPREPARE); > thd->clear_error(); > > + { > + /* > + Check if we too fast with reprepare: > + we can be so fast that: > + 1) make change of a trigger, > + 2) prepare, > + 3) try to exacute and reprepare > + in 1 microsecond, so we will wait till > + next microsecond before last reprepare > + */ > + while (first_prepared == my_hrtime().val) > + { > + pthread_yield(); > + } > + } I wouldn't bother. A more likely case is when a system time goes back, e.g. if there's an ntpd running and your loop doesn't help there. I suggest not to do anything special in this case. And if an ::execute() will open a trigger or a view with a timestamp >= than ms_prepare_time, it will cause reprepare every time. Which is exactly the behavior before your bugfix - opening of a trigger or a view causes reprepare - and it is not a problem unless one has a very small table definition cache. > + > error= reprepare(); > > if (! error) /* Success */ > diff --git a/sql/parse_file.cc b/sql/parse_file.cc > index 999f53bd681..92e7d82af1c 100644 > --- a/sql/parse_file.cc > +++ b/sql/parse_file.cc > @@ -174,11 +174,11 @@ write_parameter(IO_CACHE *file, uchar* base, File_option *parameter) > { > /* string have to be allocated already */ > LEX_STRING *val_s= (LEX_STRING *)(base + parameter->offset); > - time_t tm= my_time(0); > - > - get_date(val_s->str, GETDATE_DATE_TIME|GETDATE_GMT|GETDATE_FIXEDLENGTH, > - tm); > - val_s->length= PARSE_FILE_TIMESTAMPLENGTH; > + ulonglong tm= my_hrtime().val; add a comment here, "number of microseconds since Epoch, timezone-independent" or something like that > + // Paded to 19 characters for compatibility > + val_s->length= snprintf(val_s->str, MICROSECOND_TIMESTAMP_BUFFER_SIZE, > + "%019lld", tm); > + DBUG_ASSERT(val_s->length == MICROSECOND_TIMESTAMP_BUFFER_SIZE-1); > if (my_b_write(file, (const uchar *)val_s->str, > PARSE_FILE_TIMESTAMPLENGTH)) > DBUG_RETURN(TRUE); > @@ -835,15 +835,15 @@ File_parser::parse(uchar* base, MEM_ROOT *mem_root, > /* string have to be allocated already */ > LEX_STRING *val= (LEX_STRING *)(base + parameter->offset); > /* yyyy-mm-dd HH:MM:SS = 19(PARSE_FILE_TIMESTAMPLENGTH) characters */ a comment is now wrong > - if (ptr[PARSE_FILE_TIMESTAMPLENGTH] != '\n') > + if (ptr[MICROSECOND_TIMESTAMP_BUFFER_SIZE-1] != '\n') > { > my_error(ER_FPARSER_ERROR_IN_PARAMETER, MYF(0), > parameter->name.str, line); > DBUG_RETURN(TRUE); > } > - memcpy(val->str, ptr, PARSE_FILE_TIMESTAMPLENGTH); > - val->str[val->length= PARSE_FILE_TIMESTAMPLENGTH]= '\0'; > - ptr+= (PARSE_FILE_TIMESTAMPLENGTH+1); > + memcpy(val->str, ptr, MICROSECOND_TIMESTAMP_BUFFER_SIZE-1); > + val->str[val->length= MICROSECOND_TIMESTAMP_BUFFER_SIZE-1]= '\0'; > + ptr+= MICROSECOND_TIMESTAMP_BUFFER_SIZE; > break; > } > case FILE_OPTIONS_STRLIST: > diff --git a/sql/table.h b/sql/table.h > index 14d6b787b4e..82508388f84 100644 > --- a/sql/table.h > +++ b/sql/table.h > @@ -2159,7 +2175,7 @@ struct TABLE_LIST > LEX_STRING source; /* source of CREATE VIEW */ > LEX_STRING view_db; /* saved view database */ > LEX_STRING view_name; /* saved view name */ > - LEX_STRING timestamp; /* GMT time stamp of last operation */ > + LEX_STRING ms_timestamp; /* GMT time stamp of last operation */ better keep "timestamp" name. Or "hrtimestamp" if you'd like > st_lex_user definer; /* definer of view */ > ulonglong file_version; /* version of file's field set */ > ulonglong mariadb_version; /* version of server on creation */ > diff --git a/sql/table.cc b/sql/table.cc > index ca6ce02e4f2..24412966b36 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -8480,6 +8480,53 @@ bool TABLE_LIST::is_with_table() > } > > > +bool TABLE_LIST::is_table_ref_id_equal(THD* thd, TABLE_SHARE *s) > +{ > + enum enum_table_ref_type tp= s->get_table_ref_type(); > + if (m_table_ref_type == tp) > + { > + bool res= m_table_ref_version == s->get_table_ref_version(); > + > + /* > + If definition is different check content version > + */ > + if (tabledef_version.length && > + tabledef_version.length == s->tabledef_version.length && > + memcmp(tabledef_version.str, s->tabledef_version.str, > + tabledef_version.length) == 0) > + { > + if (table && table->triggers) > + { > + > + ulonglong ms_stmt_prepare= thd->ms_prepare_time; > + if (ms_stmt_prepare) > + for(uint i= 0; i < TRG_EVENT_MAX; i++) > + for (uint j= 0; j < TRG_ACTION_MAX; j++) > + { > + Trigger *tr= > + table->triggers->get_trigger((trg_event_type)i, > + (trg_action_time_type)j); > + if (tr) > + if (ms_stmt_prepare <= tr->ms_create_time) > + { > + set_tabledef_version(s); > + return FALSE; > + } > + } > + } > + set_table_id(s); > + return TRUE; > + } > + else > + tabledef_version.length= 0; > + return res; > + } > + else > + set_tabledef_version(s); > + return FALSE; > +} this is now a very confusing function. it doesn't only check is_table_ref_id_equal, it also sets some of them (tabledef or both tabledef and ref_id), and under unclear conditions. Could you please comment it? And you don't check that view's timestamp is in the past. If it's not, it can be changed still within the same microsecond. > + > + > uint TABLE_SHARE::actual_n_key_parts(THD *thd) > { > return use_ext_keys && > diff --git a/sql/sql_trigger.h b/sql/sql_trigger.h > index e2e6e1b7acc..70f246d4e0d 100644 > --- a/sql/sql_trigger.h > +++ b/sql/sql_trigger.h > @@ -112,7 +112,7 @@ class Trigger :public Sql_alloc > GRANT_INFO subject_table_grants; > sql_mode_t sql_mode; > /* Store create time. Can't be mysql_time_t as this holds also sub seconds */ > - ulonglong create_time; > + ulonglong ms_create_time; // Create time timestamp in microseconds my_hrtime_t create_time; > trg_event_type event; > trg_action_time_type action_time; > uint action_order; > @@ -195,7 +195,7 @@ class Table_triggers_list: public Sql_alloc > */ > List<ulonglong> definition_modes_list; > /** Create times for triggers */ > - List<ulonglong> create_times; > + List<ulonglong> ms_create_times; ditto > > List<LEX_STRING> definers_list; > > @@ -331,4 +331,14 @@ bool mysql_create_or_drop_trigger(THD *thd, TABLE_LIST *tables, bool create); > extern const char * const TRG_EXT; > extern const char * const TRN_EXT; > > + > +/** > + Make time compatible with MySQL 5.7 trigger time. > +*/ > + > +inline ulonglong make_ms_time(my_time_t time, ulong time_sec_part) make_hrtime. and better put it together with hrtime_to_time, hrtime_from_time, etc > +{ > + return ((ulonglong) time)*1000000 + time_sec_part; > +} > + > #endif /* SQL_TRIGGER_INCLUDED */ > diff --git a/sql/sql_view.cc b/sql/sql_view.cc > index 020830483c5..3b51f6d2451 100644 > --- a/sql/sql_view.cc > +++ b/sql/sql_view.cc > @@ -1131,7 +1141,31 @@ static int mysql_register_view(THD *thd, TABLE_LIST *view, > DBUG_RETURN(error); > } > > +/** > + Check is TABLE_LEST and SHARE match > + @param[in] view TABLE_LIST of the view > + @param[in] share Share object of view > + > + @return false on error or misspatch > +*/ > > +bool mariadb_view_version_get(TABLE_SHARE *share) > +{ > + DBUG_ASSERT(share->is_view); > + > + if (!(share->tabledef_version.str= > + (uchar*) alloc_root(&share->mem_root, > + MICROSECOND_TIMESTAMP_BUFFER_SIZE))) > + return TRUE; > + > + DBUG_ASSERT(share->view_def != NULL); > + if (share->view_def->parse((uchar *) &share->tabledef_version, NULL, > + view_timestamp_parameters, 1, > + &file_parser_dummy_hook)) > + return TRUE; > + DBUG_ASSERT(share->tabledef_version.length == MICROSECOND_TIMESTAMP_BUFFER_SIZE-1); If you open an old view with YYYY-MM-DD... timestamp, where do you convert it? If you open a view without a timestamp at all, where do you handle it? > + return FALSE; > +} > > /** > read VIEW .frm and create structures > diff --git a/sql/sql_show.cc b/sql/sql_show.cc > index 9483db9eff9..138fa4bc631 100644 > --- a/sql/sql_show.cc > +++ b/sql/sql_show.cc > @@ -6724,13 +6724,15 @@ static bool store_trigger(THD *thd, Trigger *trigger, > table->field[14]->store(STRING_WITH_LEN("OLD"), cs); > table->field[15]->store(STRING_WITH_LEN("NEW"), cs); > > - if (trigger->create_time) > + if (trigger->ms_create_time) > { > + /* timestamp is in microseconds */ > table->field[16]->set_notnull(); > thd->variables.time_zone->gmt_sec_to_TIME(&timestamp, > - (my_time_t)(trigger->create_time/100)); > - /* timestamp is with 6 digits */ > - timestamp.second_part= (trigger->create_time % 100) * 10000; > + (my_time_t) > + (trigger->ms_create_time/ > + 1000000)); > + timestamp.second_part= trigger->ms_create_time % 1000000; hrtime_to_time() and hrtime_sec_part() > ((Field_temporal_with_date*) table->field[16])->store_time_dec(&timestamp, > 2); > } > @@ -9810,12 +9812,14 @@ static bool show_create_trigger_impl(THD *thd, Trigger *trigger) > trigger->db_cl_name.length, > system_charset_info); > > - if (trigger->create_time) > + if (trigger->ms_create_time) > { > MYSQL_TIME timestamp; > thd->variables.time_zone->gmt_sec_to_TIME(&timestamp, > - (my_time_t)(trigger->create_time/100)); > - timestamp.second_part= (trigger->create_time % 100) * 10000; > + (my_time_t) > + (trigger->ms_create_time/ > + 1000000)); > + timestamp.second_part= trigger->ms_create_time % 1000000; ditto > p->store(&timestamp, 2); > } > else Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 5

Re: [Maria-developers] commit 192fea6b104b249bb4764635bb3a52fb9c8a2148 MDEV-27128
by Alexey Botchkov 19 Sep '22

19 Sep '22

Hi, Rucha. I decided i dislike the overall design of the struct json_schema. Seems better to do it with C++ classes using virtual methods. So instead of structures we'd have classes as json_schema_const json_schema_enum json_scnema_string json_schema_number etc. they all inherited from the class json_schema_item { virtual parse() virtual check() } Parsing the schema we can behave like this: json_parse_schema() checks all the keys on the current level looking for 'const' 'enum' and 'typ[e' keywords only. If it gets 'const' or 'enum' it creates the json_schema_const/enum respectively For the "type" it creates the appropriate json_schema_xxx and callst ->parse() with the same JSON object. So the parse would go over it once again looking for the type-specific keywords. That way we don't need the hash_list as the destructor of the json_schema_xxx class would free it's resources. Minor comments. You compare keys like strncmp((const char*)curr_key, "const", key_len) all over the code. But it is not correct. That way 'cons' or 'con' or 'co' or even 'c' - they all will be considered equal to the 'const'. Just add the 'key_cmp()' function that would compare key lengths then do the strncmp. > double val= je->s.cs->strntod((char *) je->value, je->value_len, &end; You do this even if the value_type is not number. Not very safe. Better to avoid doing that here at all. That queriy just crashes the server. Didn't yet found why SET @schema_array= '{ "typ": "arra", "prefixItem": [ { "typ": "objec"}, { "typ": "objec", "propertie": { "number1": {"typ":"numbe"}, "string1": {"typ":"strin"}, "obj1" : {"typ":"objec", "propertie": { "key1" : {"type":"string"} } } }, "required":["obj1"] } ], "items":false }'; SELECT JSON_SCHEMA_VALID(@schema_array, '[{"key1":"value1"}, {"number1":1, "string1":"str1", "obj1":{"key1":"val1"}}]'); Feel free to discuss the design details! Best regards. HF

1 0

Re: [Maria-developers] e0e75de899f: MDEV-26715 Windows/installer - allow passwordless login for root
by Sergei Golubchik 13 Sep '22

13 Sep '22

Hi, Vladislav, Note, despite what the subject says, below is not for one commit, but for the diff over them all. Just a couple of questions and a comment: > diff --git a/libmariadb b/libmariadb > index a6665e65c05..22f7ed12b46 160000 > --- a/libmariadb > +++ b/libmariadb > @@ -1 +1 @@ > -Subproject commit a6665e65c05c00f9df983a0f788a643ca7bc084c > +Subproject commit 22f7ed12b4645a9fc5d66974551359373090c86b are auth_gssapi plugins using only standard Windows libraries? > diff --git a/mysql-test/main/mysql_install_db_win_utf8.test b/mysql-test/main/mysql_install_db_win_utf8.test > index fc67b66d3ca..0dc75af3914 100644 > --- a/mysql-test/main/mysql_install_db_win_utf8.test > +++ b/mysql-test/main/mysql_install_db_win_utf8.test > @@ -1,6 +1,6 @@ > --source include/windows.inc > --source include/check_utf8_cli.inc > - > +use mysql; Why? > # Create database in tmp directory using mysql_install_db.exe, > # and start server from this directory. > let $ddir= $MYSQLTEST_VARDIR/tmp/датадир; > diff --git a/plugin/auth_gssapi/mysql-test/auth_gssapi/suite.pm b/plugin/auth_gssapi/mysql-test/auth_gssapi/suite.pm > index e77ba05cb5c..71a6bf33b7b 100644 > --- a/plugin/auth_gssapi/mysql-test/auth_gssapi/suite.pm > +++ b/plugin/auth_gssapi/mysql-test/auth_gssapi/suite.pm > @@ -1,8 +1,9 @@ > + > package My::Suite::AuthGSSAPI; > > @ISA = qw(My::Suite); > > -return "No AUTH_GSSAPI plugin" unless $ENV{AUTH_GSSAPI_SO}; > +return "No AUTH_GSSAPI plugin" unless ($ENV{AUTH_GSSAPI_SO} or ($^O eq 'MSWin32')); The second condition normally is something like or $::mysqld_variables{gssapi} eq "ON"; > > return "Not run for embedded server" if $::opt_embedded_server; > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] f5a8327d016: MDEV-29458: Role grant commands do not propagate all grants
by Sergei Golubchik 12 Sep '22

12 Sep '22

Hi, Vicentiu, On Sep 12, Vicentiu Ciorbaru wrote: > revision-id: f5a8327d016 (mariadb-10.3.36-26-gf5a8327d016) > parent(s): fa75057b238 > author: Vicențiu Ciorbaru > committer: Vicențiu Ciorbaru > timestamp: 2022-09-05 16:40:10 +0300 > message: > > MDEV-29458: Role grant commands do not propagate all grants > > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index ba457083b75..3806fb31aa6 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -2741,7 +2741,9 @@ static void acl_update_role(const char *rolename, ulong privileges) > { > ACL_ROLE *role= find_acl_role(rolename); > if (role) > + { > role->initial_role_access= role->access= privileges; > + } > } I wish you put this and the tab-to-spaces change in a separate commit. (even better - tab-to-spaces in a separate commit and this hunk - nowhere :) > @@ -9720,6 +9725,14 @@ static int handle_grant_struct(enum enum_acl_lists struct_no, bool drop, > remove_ptr_from_dynarray(&grantee->role_grants, acl_role); > } > > + /* Remove all of the role_grants from this user. */ > + while (acl_role->role_grants.elements) > + delete_dynamic_element(&acl_role->role_grants, > + acl_role->role_grants.elements - 1); this is a weird way of deleting all elements in the array. What's wrong with delete_dynamic(&acl_role->role_grants); ? > + > + /* all grants must be revoked from this role by now. propagate this */ > + propagate_role_grants(acl_role, PRIVS_TO_MERGE::ALL); > + > my_hash_delete(&acl_roles, (uchar*) acl_role); > DBUG_RETURN(1); > } > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 3f9e6c9c111: MDEV-22133 handle_fatal_signal (sig=11) on optimized builds in handle_grant_table instead of ERROR | Buffer overflow (on optimized builds)
by Sergei Golubchik 09 Sep '22

09 Sep '22

Hi, Oleksandr, On Sep 09, Oleksandr Byelkin wrote: > revision-id: 3f9e6c9c111 (mariadb-10.3.35-36-g3f9e6c9c111) > parent(s): 5feb60ce186 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2022-06-24 15:37:25 +0200 > message: > > MDEV-22133 handle_fatal_signal (sig=11) on optimized builds in handle_grant_table instead of ERROR | Buffer overflow (on optimized builds) > > Return an error if we can not read a table required for ACL. > > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index 4c68b4505a4..38d2e9c35a1 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -7420,6 +7420,11 @@ static bool grant_load(THD *thd, > while (!p_table->file->ha_index_next(p_table->record[0])); > } > } > + else > + { > + sql_print_error("Missing system table mysql.procs_priv; " > + "please run mysql_upgrade to create it"); > + } I don't think so. The comment earlier says p_table= procs_priv.table(); // this can be NULL and if "this can be NULL" then it's not an error if it is. > > end_unlock_p: > if (p_table) > @@ -10041,6 +10046,8 @@ static int handle_grant_data(THD *thd, Grant_tables& tables, bool drop, > } > > /* Handle stored routines table. */ > + if (tables.procs_priv_table().table_exists()) > + { This is ok, but also, please, check that all accesses to other optional tables are properly protected. See FIRST_OPTIONAL_TABLE define, all tables below it may not exist. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] ed477a6d30c: MDEV-29056 Replica thread reports error on ALTER ONLINE after LOCK WRITE
by Sergei Golubchik 09 Sep '22

09 Sep '22

Hi, Nikita, No, in this case (and commonly for collateral changes) I'd rather prefer to refine the original (878a92e) commit to not cause the MDEV-29056 in the first place. Ignoring LOCK= clause is logically correct and that's what the old code was doing anyway. But perhaps ALTER TABLE still needs to check that ALGORITHM= and LOCK= clauses are compatible? On Aug 27, Nikita Malyavin wrote: > revision-id: ed477a6d30c (mariadb-10.6.1-521-ged477a6d30c) > parent(s): 56206e60315 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2022-07-26 00:41:00 +0300 > message: > > MDEV-29056 Replica thread reports error on ALTER ONLINE after LOCK WRITE > > Commit 878a92e (MDEV-28943) changes the behavior of ALTER TABLE under > prelocking. It now ignores passed LOCK= value in that case. > > However, LOCK TABLES command is never replicated, so the replica node > remains unaware of it. > > The solution would be to guess on the replica side on the mode used by > master. To make a deduction reliable, master's locked_tables_mode state > is passed for query log events. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 68ebe0e2f5e: MDEV-29446 Change SHOW CREATE TABLE to display default collation
by Sergei Golubchik 09 Sep '22

09 Sep '22

Hi, Alexander, On Sep 09, Alexander Barkov wrote: > revision-id: 68ebe0e2f5e (mariadb-10.3.36-25-g68ebe0e2f5e) > parent(s): f501f815bc7 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2022-09-07 08:10:58 +0400 > message: > > MDEV-29446 Change SHOW CREATE TABLE to display default collation > > diff --git a/sql/sql_show.cc b/sql/sql_show.cc > index a019c3c9dbd..cdf0bc01b6a 100644 > --- a/sql/sql_show.cc > +++ b/sql/sql_show.cc > @@ -2253,18 +2247,6 @@ int show_create_table(THD *thd, TABLE_LIST *table_list, String *packet, > { > packet->append(STRING_WITH_LEN(" CHARACTER SET ")); > packet->append(field->charset()->csname); > - } > - /* > - For string types dump collation name only if > - collation is not primary for the given charset > - > - For generated fields don't print the COLLATE clause if > - the collation matches the expression's collation. > - */ > - if (!(field->charset()->state & MY_CS_PRIMARY) && > - (!field->vcol_info || > - field->charset() != field->vcol_info->expr->collation.collation)) why, I think, this second condition could still apply, no need to remove it. > - { > packet->append(STRING_WITH_LEN(" COLLATE ")); > packet->append(field->charset()->name); > } Note that in 10.10 you need to fix json_table code inside a big if() condition: ... (!(m_explicit_cs->state & MY_CS_PRIMARY) && (str->append(STRING_WITH_LEN(" COLLATE ")) || str->append(&m_explicit_cs->coll_name))))) || ... Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] MDEV-25080 Allow pushdown of queries involving UNIONs in outer select to foreign engines
by Oleg Smirnov 09 Sep '22

09 Sep '22

Hello! The implementation for MDEV-25080 is ready and pushed to the *preview-10.11-mdev-25080-union-pushdown *branch. The FederatedX storage engine is adjusted accordingly. The example of how it works can be found at *mysql-test/federated/federatedx_create_handlers.test:* --echo # Pushdown of the whole UNION SELECT * from federated.t1 UNION SELECT * from federated.t2; EXPLAIN SELECT * from federated.t1 UNION SELECT * from federated.t2; --echo # Pushdown of a part of the UNION SELECT * from federated.t1 UNION SELECT * from t3; EXPLAIN SELECT * from federated.t1 UNION SELECT * from t3; -- Best regards, ------------------------- *Oleg Smirnov* Sr. Software Engineer MariaDB Server Team

1 0

[Maria-developers] Cosmetic input re: MDEV-25080 Allow pushdown of UNIONs to foreign engines
by Sergey Petrunia 06 Sep '22

06 Sep '22

Hi Oleg, Please find below coding style/etc input re the patch for MDEV-25080. (See the MDEV for other comments, more are coming). > diff --git a/sql/select_handler.cc b/sql/select_handler.cc > index 795ed8eb641..99bd2c0cf7f 100644 > --- a/sql/select_handler.cc > +++ b/sql/select_handler.cc > @@ -114,10 +125,7 @@ bool select_handler::send_data() > bool select_handler::send_eof() > { > DBUG_ENTER("select_handler::send_eof"); > - > - if (select->join->result->send_eof()) > - DBUG_RETURN(true); > - DBUG_RETURN(false); > + DBUG_RETURN(result->send_eof()); > } What is this change for? If it's just the code cleanup and we HAVE to do it (I don't see an urgent reason), please do it in a separate commit. Reasons against putting this change into this commit: When one sees this change as part of a meaningful patch, they start to examine the change to see what difference it makes. After one minute, one realizes the old and new code are the same. The time and attention have already been wasted, though. > diff --git a/sql/sql_explain.cc b/sql/sql_explain.cc > index 7c3918bfd20..6726e1b28f1 100644 > --- a/sql/sql_explain.cc > +++ b/sql/sql_explain.cc > @@ -555,6 +575,12 @@ int Explain_union::print_explain(Explain_query *query, > } > > if (!using_tmp) > + /* > + The union operation may not employ a temporary table, for example, > + for UNION ALL, in that case the results of the query are sent directly > + to the output. So there is no actual UNION operation and we don't need > + to print the line in the EXPLAIN output. > + */ > return 0; Please add { } braces as the if-part takes more than one line now. > diff --git a/sql/sql_lex.cc b/sql/sql_lex.cc > index 9382625a747..437acbb749c 100644 > --- a/sql/sql_lex.cc > +++ b/sql/sql_lex.cc > @@ -5549,9 +5552,13 @@ void st_select_lex::set_explain_type(bool on_the_fly) > type= "MATERIALIZED UNION"; > else > { > - type= is_uncacheable ? "UNCACHEABLE UNION": "UNION"; > + type= is_uncacheable ? "UNCACHEABLE UNION" > + : "UNION"; > Again: please don't just re-format the code > @@ -5561,15 +5568,14 @@ void st_select_lex::set_explain_type(bool on_the_fly) > { > bool uses_cte= false; > for (JOIN_TAB *tab= first_linear_tab(join, WITHOUT_BUSH_ROOTS, > - WITH_CONST_TABLES); > - tab; > - tab= next_linear_tab(join, tab, WITHOUT_BUSH_ROOTS)) > + WITH_CONST_TABLES); > + tab; tab= next_linear_tab(join, tab, WITHOUT_BUSH_ROOTS)) > { and here. > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index 45fed46e375..2310583074b 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -1843,6 +1843,8 @@ int JOIN::optimize() > join_optimization_state init_state= optimization_state; > if (select_lex->pushdown_select) > { > + if (optimization_state == JOIN::OPTIMIZATION_DONE) > + return 0; Please add and assert that after this we have optimization_state==JOIN::NOT_OPTIMIZED. > // Do same as JOIN::optimize_inner does: > fields= &select_lex->item_list; > > @@ -4885,7 +4887,6 @@ void JOIN::cleanup_item_list(List<Item> &items) const > DBUG_VOID_RETURN; > } > > - Please put this back, the coding style says to use two empty lines between the functions. > /** > @brief > Look for provision of the select_handler interface by a foreign engine .. > -select_handler *find_select_handler(THD *thd, > - SELECT_LEX* select_lex) > +select_handler *find_select_handler(THD *thd, SELECT_LEX *select_lex) ^ Don't reformat. > { > - if (select_lex->next_select()) > - return 0; > if (select_lex->master_unit()->outer_select()) > return 0; > > @@ -4921,15 +4922,14 @@ select_handler *find_select_handler(THD *thd, > { > tbl= select_lex->join->tables_list; > } > - else if (thd->lex->query_tables && > - thd->lex->query_tables->next_global) > + else if (thd->lex->query_tables && thd->lex->query_tables->next_global) > { > tbl= thd->lex->query_tables->next_global; > } > else > return 0; > > - for (;tbl; tbl= tbl->next_global) > + for (; tbl; tbl= tbl->next_global) > { > if (!tbl->table) > continue; > @@ -4937,12 +4937,14 @@ select_handler *find_select_handler(THD *thd, > if (!ht->create_select) > continue; > select_handler *sh= ht->create_select(thd, select_lex); > - return sh; > + if (sh) > + return sh; > } > return 0; > } Don't reformat. meaningful changes are ok. > > > + Remove the 3rd empty line. > /** > An entry point to single-unit select (a select without UNION). > ... diff --git a/sql/sql_union.cc b/sql/sql_union.cc index e8bdbcba2f2..4cb9f3e025a 100644 --- a/sql/sql_union.cc +++ b/sql/sql_union.cc @@ -1294,6 +1294,52 @@ bool init_item_int(THD* thd, Item_int* &item) return true; } > +select_handler *find_unit_handler(THD *thd, > + SELECT_LEX_UNIT *unit) ^^^ coding style. ... > @@ -2053,6 +2106,7 @@ void st_select_lex_unit::optimize_bag_operation(bool is_outer_distinct) > bag_set_op_optimized= true; > } > > +select_handler *find_select_handler(THD *thd, SELECT_LEX *select_lex); Please don't have function declaration in the middle of a file... > @@ -2272,14 +2350,14 @@ bool st_select_lex_unit::exec() > if (union_result->flush()) > { > thd->lex->current_select= lex_select_save; > - DBUG_RETURN(1); > + return true; Please fix identation. > diff --git a/storage/federatedx/federatedx_pushdown.cc b/storage/federatedx/federatedx_pushdown.cc > index 664f0570238..492055773cb 100644 > --- a/storage/federatedx/federatedx_pushdown.cc > +++ b/storage/federatedx/federatedx_pushdown.cc > @@ -163,13 +163,13 @@ void ha_federatedx_derived_handler::print_error(int, unsigned long) > } > > > -static select_handler* > -create_federatedx_select_handler(THD* thd, SELECT_LEX *sel) > +template<typename T> > +static select_handler *create_federatedx_handler(THD *thd, T *sel_lex) > { > if (!use_pushdown) > return 0; > Do we need to generate that much code? What if there was a function like bool check_pushdown_allowed(select_lex_node *node) which could process both SELECT_LEX and SELECT_LEX_UNIT objects. Then, we would need two one-liner functions: create_select_handler(type S) { if (check_pushdown_allowed(S)) return new ha_federatedx_select_handler(S) } BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

1 0

Re: [Maria-developers] 489a7fba324: MDEV-29322 ASAN heap-use-after-free in Query_log_event::do_apply_event
by Sergei Golubchik 04 Sep '22

04 Sep '22

Hi, Andrei, I still don't understand this. 1. Why rgi->options_to_bin_log is only set for GTID_EVENT? Is there a guarantee that there always be a GTID_EVENT before a QUERY_EVENT? What if gtids aren't enabled? 2. how do you guarantee that all query events for a previous value of options_written_to_bin_log have been applied before updating it in the rgi? 3. and if you do guarantee it, why is it atomic? No query events = nobody reads options_written_to_bin_log concurrently. And why not to do a simple patch, like the one I've attached? (note, it's an intentionally minimal patch to show the fix, practically I'd accompany it with a small cleanup patch) Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org On Sep 02, Andrei Elkin wrote: > > > Query_log_event cannot just randomly use some > > options_written_to_bin_log, it must use options_written_to_bin_log > > from the Format_description_log_event that came from the binlog file > > where the Query_log_event was read from. > > [ Not to engage you into a separate discussion, > still why 'randomly'? In the quoted description p.1 says > the Query_log_event uses `options_written_to_bin_log` from the > immediately preceding-in-binlog FD. ] > > > If rli->relay_log.description_event_for_exec is not that > > Format_description_log_event, and assuming that the correct > > Format_description_log_event is not stored anywhere, the Query_log_event > > has to remember options_written_to_bin_log itself, > > It's an option, but the chosen one avoids such remembering per each > Query event. Instead ... > > > there's not much we can do about it. > > > > Or it can apply the mask in the constructor and store not flags2 but > > the new value of thd->variables.option_bits. > > > > Or it can extend flags2 to cover all OPTIONS_WRITTEN_TO_BINLOG. > > ... of taking care of individual events the patch basically goes along > the base to use a placeholder that > A. lasts (unlike FD instance) the whole slave session time, and > B. the stored `options_written_to_binlog` is guaranteed to be a copy > of the most recent FD before the Query > > To help you to understand the method, here is a sequence diagram, > where on the top line > the slave SQL thread, its Relay-log-info object, a similar to that Worker's instance, > and a Worker thread. > > > SQL thread RLI/RGI Worker RGI Worker > | . . . > --->+ read FD . . i > | . . . > +--------------> > stores options > . > --->+ read Query > | . > +-------------------------->+ > finds Worker RGI > to store in theree a copy FD.options > | . > +-------------------------------------------->+ > schedules Query Worker | > . | > .<----------------+ > reads options | > | applies > ... Query > > It must be clear the A holds. > To prove B let after Query a second FD^2 is read by the SQL thread and > that FD^2 is from a different version server. > Then the SQL thread always have a wait-for-workers-to-become-idle > condition, so execution would go like this: > > --->+ read Query . ... > | . | > +-------------------------->+ | > finds Worker RGI | > to store in theree a copy FD.options | > | | > --->+ read FD^2 | applying of > | | Query > ... | > ... wait for Worker to complete Query | > ... | > +<============================================| > | > +-------------->. > | stores FD^2 > | options > ... > > So FD^2 can't affect queries from earlier binlogs. > > I am all ears to hear about your feedback. > > Cheers, > > Andrei

2 2

Re: [Maria-developers] 489a7fba324: MDEV-29322 ASAN heap-use-after-free in Query_log_event::do_apply_event
by Sergei Golubchik 01 Sep '22

01 Sep '22

Hi, Andrei, On Sep 01, Andrei wrote: > revision-id: 489a7fba324 (mariadb-10.5.17-10-g489a7fba324) > parent(s): 0324bde8469 > author: Andrei > committer: Andrei > timestamp: 2022-09-01 14:15:11 +0300 > message: > > MDEV-29322 ASAN heap-use-after-free in Query_log_event::do_apply_event > > The ASAN report was reported in the parallel slave execution of a Query > event and implicitly involved (so also parallelly run) Format-Description > event. > The Query actually had dependency on a preceding "old" FD whose > instance got destructed, so to cause the ASAN error. > > The case is fixed with copying of the value of > `options_written_to_bin_log` the sensitive member > of FD event into worker's stable placeholders, so FD change > won't affect Queries dependent on "old" instances. > > The method observes the following properties: > 0. each worker gets its private copy; > 1. the copied value gets a part of the execution context for any > Queries that follow the FD in binlog; > 2. any new FD forces a new copy which does not any affect on > the past (in binlog order) Queries which might be still in > execution. Sorry, but this doesn't make any sense. The value of options_written_to_bin_log depends on the version of the server that has written that binlog file. Query_log_event cannot just randomly use some options_written_to_bin_log, it must use options_written_to_bin_log from the Format_description_log_event that came from the binlog file where the Query_log_event was read from. If rli->relay_log.description_event_for_exec is not that Format_description_log_event, and assuming that the correct Format_description_log_event is not stored anywhere, the Query_log_event has to remember options_written_to_bin_log itself, there's not much we can do about it. Or it can apply the mask in the constructor and store not flags2 but the new value of thd->variables.option_bits. Or it can extend flags2 to cover all OPTIONS_WRITTEN_TO_BINLOG. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 5154e224a97: MDEV-5215 Granted to PUBLIC
by Sergei Golubchik 25 Aug '22

25 Aug '22

Hi, Oleksandr, > commit 5154e224a97 > Author: Oleksandr Byelkin <sanja(a)mariadb.com> > Date: Mon Dec 13 16:15:21 2021 +0100 > > diff --git a/scripts/mysql_system_tables.sql b/scripts/mysql_system_tables.sql > index fd2f1c95dda..a1721913b2c 100644 > --- a/scripts/mysql_system_tables.sql > +++ b/scripts/mysql_system_tables.sql > @@ -47,6 +47,10 @@ INSERT IGNORE INTO global_priv SELECT * FROM tmp_user_sys WHERE 0 <> @need_sys_u > DROP TABLE tmp_user_sys; > > > +-- This special "role" needed for GRAND ... TO PUBLIC > +INSERT IGNORE INTO global_priv (Host,User,Priv) VALUES ('', 'PUBLIC', concat('{"access":0,"version_id":',regexp_replace(regexp_replace(@@version, '\\b\\d\\b', '0\\0'), '\\D', ''),',"is_role":true}')); 1. why here and not in mysql_system_tables_data.sql ? 2. why this version magic? mysql_system_tables_data.sql doesn't do it for other accounts 3. why do you do it at all? I thought the server can create this role as needed > + > + > CREATE DEFINER='mariadb.sys'@'localhost' SQL SECURITY DEFINER VIEW IF NOT EXISTS user AS SELECT > Host, > User, > @@ -95,7 +99,9 @@ CREATE DEFINER='mariadb.sys'@'localhost' SQL SECURITY DEFINER VIEW IF NOT EXISTS > ELT(IFNULL(JSON_VALUE(Priv, '$.is_role'), 0) + 1, 'N', 'Y') AS is_role, > IFNULL(JSON_VALUE(Priv, '$.default_role'), '') AS default_role, > CAST(IFNULL(JSON_VALUE(Priv, '$.max_statement_time'), 0.0) AS DECIMAL(12,6)) AS max_statement_time > - FROM global_priv; > + FROM global_priv > +-- Do not show special role for GRANT TO PUBLIC > + WHERE not (Host = "" and User = "PUBLIC"); Why? I don't think it's wrong to show a new PUBLIC role. > > -- Remember for later if user table already existed > set @had_user_table= @@warning_count != 0; > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index 601cae2e945..aa9bd91ec52 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -316,6 +318,13 @@ static bool show_table_and_column_privileges(THD *, const char *, const char *, > static int show_routine_grants(THD *, const char *, const char *, > const Sp_handler *sph, char *, int); > > +static ACL_ROLE *acl_public= NULL; > + > +inline privilege_t public_access() > +{ > + return (acl_public ? acl_public->access : NO_ACL); > +} Why not to have acl_public always not null? e.g. define it as static ACL_ROLE acl_public; so it's not even a pointer. And don't store it in the dynamic array. Or make it like static ACL_ROLE acl_public_dummy, *acl_public= &acl_public_dummy; then you can keep he real role in the dynamic array, but still acl_public will never be null. but in the standard PUBLIC is not a role at all, so may be it'd be simpler not to store it in the dynamic array > + > class Grant_tables; > class User_table; > class Proxies_priv_table; > @@ -3148,7 +3183,7 @@ bool acl_getroot(Security_context *sctx, const char *user, const char *host, > } > else // Role, not User > { > - ACL_ROLE *acl_role= find_acl_role(user); > + ACL_ROLE *acl_role= find_acl_role(user, false); are there tests for that? > if (acl_role) > { > res= 0; > @@ -3161,6 +3196,26 @@ bool acl_getroot(Security_context *sctx, const char *user, const char *host, > } > } > > + /* > + PUBLIC magic: > + > + Note: for usual user privileges of 3 component merged together: > + 1) user privileges > + 2) set role privileges > + 3) public privileges > + But for this routine (used in Security_context::change_security_context) > + only 2 component merged: > + 1) user OR role privileges we are switching to because if we're switching to a user, there can be no set role? I feel this comment is more confusing, then helping, really, I'd suggest to remove it completely > + 2) public privileges > + */ > + if (acl_public) > + { > + if (ACL_DB *acl_db= acl_db_find(db, public_name.str, "", "", FALSE)) > + sctx->db_access|= acl_db->access; > + > + sctx->master_access|= acl_public->access; > + } > + > mysql_mutex_unlock(&acl_cache->lock); > DBUG_RETURN(res); > } > @@ -3320,20 +3375,22 @@ int acl_setrole(THD *thd, const char *rolename, privilege_t access) > /* merge the privileges */ > Security_context *sctx= thd->security_ctx; > sctx->master_access= access; > - if (thd->db.str) > - sctx->db_access= acl_get(sctx->host, sctx->ip, sctx->user, thd->db.str, FALSE); > - > if (!strcasecmp(rolename, "NONE")) > { > thd->security_ctx->priv_role[0]= 0; > } > else > { > - if (thd->db.str) > - sctx->db_access|= acl_get("", "", rolename, thd->db.str, FALSE); > /* mark the current role */ > strmake_buf(thd->security_ctx->priv_role, rolename); > } > + if (thd->db.str) > + sctx->db_access= acl_get_all3(sctx, thd->db.str, FALSE); > + > + // PUBLIC magic > + if (acl_public) > + sctx->master_access|= acl_public->access; why not to do it in check_user_can_set_role()? Like - *access = acl_user->access | role->access; + *access = acl_user->access | role->access | acl_public->access; > + > return 0; > } > > @@ -3347,9 +3404,13 @@ static uchar* check_get_key(ACL_USER *buff, size_t *length, > > static void acl_update_role(const char *rolename, const privilege_t privileges) > { > - ACL_ROLE *role= find_acl_role(rolename); > + ACL_ROLE *role= find_acl_role(rolename, true); > if (role) > + { > role->initial_role_access= role->access= privileges; > + if (strcasecmp(rolename, public_name.str) == 0) > + acl_public= role; why? I mean, in what case can acl_public != role here? > + } > } > > > @@ -3638,6 +3703,23 @@ privilege_t acl_get(const char *host, const char *ip, > DBUG_RETURN(db_access & host_access); > } > > +/* > + Check if there is access for the host/user, role, public on the database > +*/ > + > +privilege_t acl_get_all3(Security_context *sctx, const char *db, > + bool db_is_patern) > +{ > + privilege_t access= acl_get(sctx->host, sctx->ip, > + sctx->priv_user, db, db_is_patern); > + if (sctx->priv_role[0]) > + access|= acl_get("", "", sctx->priv_role, db, db_is_patern); > + if (acl_public) > + access|= acl_get("", "", public_name.str, db, db_is_patern); how can this be optimized, considering that in most cases and for most db names there will be no grants to role and public? A couple of ideas: maintain counters total_number_of_db_grants_to_roles ad total_number_of_db_grants_to_public. This is rather crude. keep grants to users, to roles, and to public in separate caches. or in different dynamic arrays (won't help much in a default setup) keep propery per ACL_DB or shared between all ACL_DB's of one DB? that'd be rather tricky to maintain > + return access; > +} > + > + > /* > Check if there are any possible matching entries for this host > > @@ -3759,7 +3841,7 @@ static bool add_role_user_mapping(const char *uname, const char *hname, > const char *rname) > { > ACL_USER_BASE *grantee= find_acl_user_base(uname, hname); > - ACL_ROLE *role= find_acl_role(rname); > + ACL_ROLE *role= find_acl_role(rname, true); why? > > if (grantee == NULL || role == NULL) > return 1; > @@ -4247,7 +4329,7 @@ bool is_acl_user(const char *host, const char *user) > if (*host) // User > res= find_user_exact(host, user) != NULL; > else // Role > - res= find_acl_role(user) != NULL; > + res= find_acl_role(user, false) != NULL; this disallows DEFINER PUBLIC, right? I couldn't find a test for that. > > mysql_mutex_unlock(&acl_cache->lock); > return res; > @@ -7537,11 +7620,11 @@ bool mysql_grant_role(THD *thd, List <LEX_USER> &list, bool revoke) > if (user->host.str) > hostname= user->host; > else > - if ((role_as_user= find_acl_role(user->user.str))) > + if ((role_as_user= find_acl_role(user->user.str, false))) Why is it false here? It looks like it disallows GRANT role TO PUBLIC > hostname= empty_clex_str; > else > { > - if (is_invalid_role_name(username.str)) > + if (check_role_name(username.str, false) == ROLE_NAME_INVALID) > { > append_user(thd, &wrong_users, &username, &empty_clex_str); > result= 1; > @@ -8280,19 +8355,14 @@ bool check_grant(THD *thd, privilege_t want_access, TABLE_LIST *tables, > if (any_combination_will_do) > continue; > > - t_ref->grant.grant_table_user= grant_table; // Remember for column test > - t_ref->grant.grant_table_role= grant_table_role; > - t_ref->grant.version= grant_version; > - t_ref->grant.privilege|= grant_table ? grant_table->privs : NO_ACL; > - t_ref->grant.privilege|= grant_table_role ? grant_table_role->privs : NO_ACL; > + t_ref->grant.privilege|= t_ref->grant.aggregate_privs(); > t_ref->grant.want_privilege= ((want_access & COL_ACLS) & ~t_ref->grant.privilege); > > if (!(~t_ref->grant.privilege & want_access)) > continue; > > - if ((want_access&= ~((grant_table ? grant_table->cols : NO_ACL) | > - (grant_table_role ? grant_table_role->cols : NO_ACL) | > - t_ref->grant.privilege))) > + if ((want_access&= ~(t_ref->grant.aggregate_cols() | > + t_ref->grant.privilege))) > { > goto err; // impossible > } 1. why is this if() here, if it's impossible? 2. I'm a bit worried that you set t_ref->grant.version early when it's still possible to goto err. Old code was setting the version very late after all checks have succeeded. > @@ -8335,6 +8405,49 @@ static void check_grant_column_int(GRANT_TABLE *grant_table, const char *name, > } > } > > +inline privilege_t GRANT_INFO::aggregate_privs() > +{ > + return (grant_table_user ? grant_table_user->privs : NO_ACL) | > + (grant_table_role ? grant_table_role->privs : NO_ACL) | > + (grant_public ? grant_public->privs : NO_ACL); > +} > + > +inline privilege_t GRANT_INFO::aggregate_cols() > +{ > + return (grant_table_user ? grant_table_user->cols : NO_ACL) | > + (grant_table_role ? grant_table_role->cols : NO_ACL) | > + (grant_public ? grant_public->cols : NO_ACL); > +} > + > +void GRANT_INFO::refresh(const Security_context *sctx, > + const char *db, const char *table) > +{ > + if (version != grant_version) > + read(sctx, db, table); > +} > + > +void GRANT_INFO::read(const Security_context *sctx, > + const char *db, const char *table) > +{ > +#ifdef EMBEDDED_LIBRARY > + grant_table_user= grant_table_role= grant_public= NULL; > +#else > + grant_table_user= > + table_hash_search(sctx->host, sctx->ip, db, > + sctx->priv_user, > + table, FALSE); /* purecov: inspected */ you don't need to preserve purecov comments when moving the code around > + grant_table_role= > + sctx->priv_role[0] ? table_hash_search("", NULL, db, > + sctx->priv_role, > + table, TRUE) : NULL; > + grant_public= > + acl_public ? table_hash_search("", NULL, db, > + public_name.str, > + table, TRUE) : NULL; > +#endif > + version= grant_version; /* purecov: inspected */ > +} > + > /* > Check column rights in given security context > > @@ -9533,6 +9626,13 @@ static bool show_global_privileges(THD *thd, ACL_USER_BASE *acl_entry, > want_access= ((ACL_ROLE *)acl_entry)->initial_role_access; > else > want_access= acl_entry->access; > + > + // suppress "GRANT USAGE ON *.* TO `PUBLIC`" why? I agree that it's pointless, but disabling just it is kind of difficult to explain. > + if (!(want_access & ~GRANT_ACL) && > + acl_entry->user.length == public_name.length && > + strcasecmp(acl_entry->user.str, public_name.str) == 0) > + return FALSE; > + > if (test_all_bits(want_access, (GLOBAL_ACLS & ~ GRANT_ACL))) > global.append(STRING_WITH_LEN("ALL PRIVILEGES")); > else if (!(want_access & ~GRANT_ACL)) > @@ -11015,10 +11116,15 @@ bool mysql_drop_user(THD *thd, List <LEX_USER> &list, bool handle_as_role) > { > int rc; > user_name= get_current_user(thd, tmp_user_name, false); > - if (!user_name) > + if (!user_name || (handle_as_role && > + (strcasecmp(user_name->user.str, > + public_name.str) == 0))) better add this condition to the following if(), not to this one. Like: - if (handle_as_role != user_name->is_role()) + if (handle_as_role != user_name->is_role() || + (handle_as_role && check_role_name(...) == ROLE_NAME_INVALID)) > { > thd->clear_error(); > - append_str(&wrong_users, STRING_WITH_LEN("CURRENT_ROLE")); > + if (!user_name) > + append_str(&wrong_users, STRING_WITH_LEN("CURRENT_ROLE")); > + else > + append_str(&wrong_users, public_name.str, public_name.length); > result= TRUE; > continue; > } > diff --git a/sql/sql_show.cc b/sql/sql_show.cc > index 6290a47ee30..143e647f74c 100644 > --- a/sql/sql_show.cc > +++ b/sql/sql_show.cc > @@ -5304,7 +5300,7 @@ int get_all_tables(THD *thd, TABLE_LIST *tables, COND *cond) > &thd->col_access, NULL, 0, 1) || > (!thd->col_access && check_grant_db(thd, db_name->str))) || > sctx->master_access & (DB_ACLS | SHOW_DB_ACL) || > - acl_get(sctx->host, sctx->ip, sctx->priv_user, db_name->str, 0)) > + acl_get_all3(sctx, db_name->str, 0)) Was it a bug? That one couldn't see db's if privileges were granted via roles? > #endif > { > Dynamic_array<LEX_CSTRING*> table_names(PSI_INSTRUMENT_MEM); > diff --git a/mysql-test/main/public_basic.test b/mysql-test/main/public_basic.test > new file mode 100644 > index 00000000000..3f5993dfe97 > --- /dev/null > +++ b/mysql-test/main/public_basic.test > @@ -0,0 +1,93 @@ > +SHOW GRANTS FOR PUBLIC; > + > +--echo # it is not PUBLIC but an user > +--echo # (this should work as it allowed for roles for example) > +create user PUBLIC; > +create user PUBLIC@localhost; > +GRANT SELECT on test.* to PUBLIC@localhost; > +drop user PUBLIC@localhost; > +drop user PUBLIC; > + > +--echo # preinstalled PUBLIC > +GRANT SELECT on test.* to PUBLIC; > +GRANT SELECT on mysql.db to PUBLIC; > +--replace_regex /"version_id"\:[0-9]+/"version_id":VERSION/ > +select * from mysql.global_priv where user="PUBLIC" ; > + > +SHOW GRANTS FOR PUBLIC; > + > +GRANT UPDATE on test.* to PUBLIC; > +GRANT UPDATE on mysql.db to PUBLIC; > + > +SHOW GRANTS FOR PUBLIC; > + > +REVOKE SELECT on test.* from PUBLIC; > +REVOKE SELECT on mysql.db from PUBLIC; > + > +SHOW GRANTS FOR PUBLIC; > + > +REVOKE UPDATE on test.* from PUBLIC; > +REVOKE UPDATE on mysql.db from PUBLIC; > + > +--error ER_NONEXISTING_GRANT > +REVOKE UPDATE on test.* from PUBLIC; > +--error ER_NONEXISTING_TABLE_GRANT > +REVOKE UPDATE on mysql.db from PUBLIC; > + > +SHOW GRANTS FOR PUBLIC; > + > +--echo # automaticly added PUBLIC > +delete from mysql.global_priv where user="PUBLIC"; > +flush privileges; > +select * from mysql.global_priv where user="PUBLIC" ; > +GRANT SELECT on test.* to PUBLIC; > +GRANT SELECT on mysql.db to PUBLIC; > +--replace_regex /"version_id"\:[0-9]+/"version_id":VERSION/ > +select * from mysql.global_priv where user="PUBLIC" ; > + > +SHOW GRANTS FOR PUBLIC; > + > +GRANT UPDATE on test.* to PUBLIC; > +GRANT UPDATE on mysql.db to PUBLIC; > + > +SHOW GRANTS FOR PUBLIC; > + > +REVOKE SELECT on test.* from PUBLIC; > +REVOKE SELECT on mysql.db from PUBLIC; > + > +SHOW GRANTS FOR PUBLIC; > + > +REVOKE UPDATE on test.* from PUBLIC; > +REVOKE UPDATE on mysql.db from PUBLIC; > + > +SHOW GRANTS FOR PUBLIC; > + > +--error ER_INVALID_ROLE > +GRANT XXXXXX TO CURRENT_USER; > +--echo # following should fail with the same error as above > +--error ER_INVALID_ROLE > +GRANT PUBLIC TO CURRENT_USER; > + > +--error ER_INVALID_ROLE > +REVOKE XXXXXX FROM CURRENT_USER; > +--echo # following should fail with the same error as above > +--error ER_INVALID_ROLE > +REVOKE PUBLIC FROM CURRENT_USER; > +--error ER_CANNOT_USER > + > +drop role XXXXXX; > +--echo # following should fail with the same error as above > +--error ER_CANNOT_USER > +drop role PUBLIC; > + > +--error ER_INVALID_ROLE > +SET ROLE XXXXXX; > +--echo # following should fail with the same error as above > +--error ER_INVALID_ROLE > +SET ROLE PUBLIC; > + > +--error ER_INVALID_ROLE > +SET DEFAULT ROLE XXXXXX; > +--echo # following should fail with the same error as above > +--error ER_INVALID_ROLE > +SET DEFAULT ROLE PUBLIC; This is good, but one can update global_priv table directly and put the default_role=PUBLIC there. Please, add a test for that. and also a couple of tests for `public` (in backticks) > diff --git a/mysql-test/main/public_privileges.test b/mysql-test/main/public_privileges.test > new file mode 100644 > index 00000000000..a542376f05c > --- /dev/null > +++ b/mysql-test/main/public_privileges.test > @@ -0,0 +1,293 @@ > +--echo # > +--echo # Test DB/TABLE/COLUMN privileges in queries > +--echo # > + > +SHOW GRANTS FOR PUBLIC; > + > +create user testuser; > +create database testdb1; > +use testdb1; > +create table t1 (a int, b int); > +insert into t1 values (1,2); > +create database testdb2; > +use testdb2; > +create table t2 (a int, b int); > +insert into t2 values (1,2); > +create table t3 (a int, b int); > +insert into t3 values (1,2); > + > +connect (testuser,localhost,testuser,,); > +connection testuser; "connect testuser" also switches to this connection, the following "connection testuser" is redundant > +--error ER_TABLEACCESS_DENIED_ERROR > +select * from testdb1.t1; > +--error ER_TABLEACCESS_DENIED_ERROR > +select * from testdb2.t2; > +--error ER_TABLEACCESS_DENIED_ERROR > +select b from testdb2.t3; > +--error ER_TABLEACCESS_DENIED_ERROR > +select a from testdb2.t3; > + > +connection default; > + > +GRANT SELECT ON testdb1.* to PUBLIC; > +GRANT SELECT ON testdb2.t2 to PUBLIC; > +GRANT SELECT (b) ON testdb2.t3 to PUBLIC; > + > +disconnect testuser; > +connect (testuser,localhost,testuser,,); > +connection testuser; > +select * from testdb1.t1; > +select * from testdb2.t2; > +select b from testdb2.t3; > +--error ER_COLUMNACCESS_DENIED_ERROR > +select a from testdb2.t3; > + > +connection default; > + > +--replace_column 1 # 3 # 6 # 7 # > +SHOW PROCESSLIST; why? > + > +connection default; > + > +use test; > +disconnect testuser; > +REVOKE SELECT ON testdb1.* from PUBLIC; > +REVOKE SELECT ON testdb2.t2 from PUBLIC; > +REVOKE SELECT (b) ON testdb2.t3 from PUBLIC; may be `revoke all privileges, grant option` ? looks like a good place to have it tested > +drop user testuser; > +drop database testdb1; > +drop database testdb2; > + > +--echo # > +--echo # test global process list privilege and EXECUTE db level > +--echo # > + > +create user testuser; > +create database testdb; > +use testdb; > +create procedure p1 () select 1; > + > +connect (testuser,localhost,testuser,,); > +connection testuser; > + > +--replace_column 1 # 3 # 6 # 7 # > +SHOW PROCESSLIST; > +--error ER_PROCACCESS_DENIED_ERROR > +call testdb.p1(); > + > +connection default; > + > +GRANT PROCESS ON *.* to PUBLIC; > +GRANT EXECUTE ON testdb.* to PUBLIC; > + > +disconnect testuser; > +connect (testuser,localhost,testuser,,); > +connection testuser; > + > +--replace_column 1 # 3 # 6 # 7 # > +SHOW PROCESSLIST; > +call testdb.p1(); > + > +connection default; > + > +--replace_column 1 # 3 # 6 # 7 # > +SHOW PROCESSLIST; > + > +connection default; > + > +use test; > +disconnect testuser; > +REVOKE PROCESS ON *.* from PUBLIC; > +REVOKE EXECUTE ON testdb.* from PUBLIC; > +drop user testuser; > +drop database testdb; > + > +--echo # > +--echo # test DB privilege to allow USE statement > +--echo # > + > +create user testuser; > +create database testdb; > + > +connect (testuser,localhost,testuser,,); > +connection testuser; > + > +--error ER_DBACCESS_DENIED_ERROR > +use testdb; > + > +connection default; > + > +GRANT LOCK TABLES ON testdb.* to PUBLIC; > + > +disconnect testuser; > +connect (testuser,localhost,testuser,,); > +connection testuser; > + > +use testdb; > + > +connection default; > + > +use test; > +disconnect testuser; > +REVOKE LOCK TABLES ON testdb.* from PUBLIC; > +drop user testuser; > +drop database testdb; > + > + > +--echo # > +--echo # test DB privilege to allow USE statement (as above) > +--echo # test current db privileges > +--echo # > + > +create user testuser; > +create database testdb; > +use testdb; > +create table t1 (a int); > +insert into t1 values (1); > +GRANT LOCK TABLES ON testdb.* to PUBLIC; > + > +connect (testuser,localhost,testuser,,); > +connection testuser; > + > +use testdb; > +--error ER_TABLEACCESS_DENIED_ERROR > +update t1 set a=a+1; > + > +connection default; > + > +GRANT UPDATE,SELECT ON testdb.* to PUBLIC; > + > +disconnect testuser; > +connect (testuser,localhost,testuser,,); > +connection testuser; > + > +use testdb; > +update t1 set a=a+1; > + > +connection default; > +select * from testdb.t1; > + > +use test; > +disconnect testuser; > +REVOKE LOCK TABLES ON testdb.* from PUBLIC; > +REVOKE UPDATE,SELECT ON testdb.* from PUBLIC; > +drop user testuser; > +drop database testdb; > + > + > +--echo # > +--echo # test DB privilege to allow USE statement (as above) > +--echo # test table/column privileges in current DB > +--echo # > + > +create user testuser; > +create database testdb; > +use testdb; > +create table t1 (a int); > +insert into t1 values (1); > +create table t2 (a int, b int); > +insert into t2 values (1,2); > +GRANT LOCK TABLES ON testdb.* to PUBLIC; > + > +connect (testuser,localhost,testuser,,); > +connection testuser; > + > +use testdb; > +--error ER_TABLEACCESS_DENIED_ERROR > +delete from t1; > +--error ER_TABLEACCESS_DENIED_ERROR > +select b from t2; > +--error ER_TABLEACCESS_DENIED_ERROR > +select a from t2; > + > +connection default; > + > +GRANT DELETE ON testdb.t1 to PUBLIC; > +GRANT SELECT (a) ON testdb.t2 to PUBLIC; > + > +disconnect testuser; > +connect (testuser,localhost,testuser,,); > +connection testuser; > + > +use testdb; > +delete from t1; > +select a from t2; > +--error ER_COLUMNACCESS_DENIED_ERROR > +select b from t2; > + > +connection default; > +select * from testdb.t1; > + > +use test; > +disconnect testuser; > +REVOKE LOCK TABLES ON testdb.* from PUBLIC; > +REVOKE DELETE ON testdb.t1 from PUBLIC; > +REVOKE SELECT (a) ON testdb.t2 from PUBLIC; > +drop user testuser; > +drop database testdb; > + > +--echo # > +--echo # test function privilege > +--echo # > + > +create user testuser; > +create database testdb; > +use testdb; > +create function f1() returns int return 2; > + > +connect (testuser,localhost,testuser,,); > +connection testuser; > + > +--error ER_PROCACCESS_DENIED_ERROR > +alter function testdb.f1 comment "A stupid function"; > +--error ER_PROCACCESS_DENIED_ERROR > +select testdb.f1(); > + > +connection default; > + > +GRANT ALTER ROUTINE ON testdb.* to PUBLIC; > + > +disconnect testuser; > +connect (testuser,localhost,testuser,,); > +connection testuser; > + > +alter function testdb.f1 comment "A stupid function"; > +--error ER_PROCACCESS_DENIED_ERROR > +select testdb.f1(); > + > +connection default; > + > +use test; > +disconnect testuser; > +REVOKE ALTER ROUTINE ON testdb.* from PUBLIC; > +drop function testdb.f1; > +drop user testuser; > +drop database testdb; > + > +--echo # > +--echo # bug with automatically added PUBLIC role > +--echo # > + > +--echo # automaticly added PUBLIC > +delete from mysql.global_priv where user="PUBLIC"; > +flush privileges; > +GRANT SELECT on test.* to PUBLIC; > + > +REVOKE SELECT on test.* from PUBLIC; > + > +create user testuser; > +create database testdb1; > +use testdb1; > +create table t1 (a int, b int); > +insert into t1 values (1,2); > + > +connect (testuser,localhost,testuser,,); > +connection testuser; > +--error ER_TABLEACCESS_DENIED_ERROR > +select * from testdb1.t1; > + > +connection default; > + > +disconnect testuser; > +drop user testuser; > +drop database testdb1; > diff --git a/mysql-test/suite/roles/none_public.test b/mysql-test/suite/roles/none_public.test > index a0ec2315cfc..d0ba62d226b 100644 > --- a/mysql-test/suite/roles/none_public.test > +++ b/mysql-test/suite/roles/none_public.test > @@ -19,8 +19,9 @@ grant select on *.* to none; > grant public to role1; > --error ER_INVALID_ROLE > grant role1 to public; ^^^ this shouldn't fail > ---error ER_INVALID_ROLE > -grant select on *.* to public; > +# PUBLIC is legal role > +#--error ER_INVALID_ROLE > +#grant select on *.* to public; you can just remove it, commenting out isn't better > --error ER_INVALID_ROLE > grant role1 to current_role; > @@ -40,16 +41,18 @@ revoke select on *.* from public; > > --error ER_INVALID_ROLE > show grants for none; > ---error ER_INVALID_ROLE > -show grants for public; > +# PUBLIC is legal role > +#--error ER_INVALID_ROLE > +#show grants for public; > > --error ER_INVALID_ROLE > create definer=none view test.v1 as select 1; > ---error ER_INVALID_ROLE > -create definer=public view test.v1 as select 1; > +# PUBLIC is legal role > +#--error ER_INVALID_ROLE > +#create definer=public view test.v1 as select 1; eh, no, PUBLIC isn't a valid definer, as far as I understand. > > drop role role1; > > -insert mysql.global_priv values ('', 'none', '{"is_role":true}'), ('', 'public', '{"is_role":true}'); > +insert mysql.global_priv values ('', 'none', '{"is_role":true}'); > flush privileges; > -delete from mysql.global_priv where host=''; > +delete from mysql.global_priv where host='' and user='none'; > diff --git a/mysql-test/main/sp_notembedded.result b/mysql-test/main/sp_notembedded.result > index e03361598a6..239f8acb7f3 100644 > --- a/mysql-test/main/sp_notembedded.result > +++ b/mysql-test/main/sp_notembedded.result > @@ -345,7 +345,6 @@ show grants; > Grants for foo1@localhost > GRANT USAGE ON *.* TO `foo1`@`localhost` IDENTIFIED BY PASSWORD '-F3A2A51A9B0F2BE2468926B4132313728C250DBF' > GRANT CREATE ROUTINE ON `test`.* TO `foo1`@`localhost` > -GRANT EXECUTE, ALTER ROUTINE ON PROCEDURE `test`.`spfoo` TO `foo1`@`localhost` Why did this line disappear? > connection default; > disconnect foo; > drop procedure spfoo; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] bcfc424d859: MDEV-5215 prerequisite: remove test and test_* database hacks in the test suite
by Sergei Golubchik 25 Aug '22

25 Aug '22

Hi, Oleksandr, On Aug 25, Oleksandr Byelkin wrote: > revision-id: bcfc424d859 (mariadb-10.6.1-483-gbcfc424d859) > parent(s): 1c3182a99e5 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2022-08-16 11:49:41 +0200 > message: > > MDEV-5215 prerequisite: remove test and test_* database hacks in the test suite let's say, it's ok. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 1c3182a99e5: MDEV-5215 prerequisite of prerequisite: if DB is not mentioned in connect ignore errors of switching to it
by Sergei Golubchik 25 Aug '22

25 Aug '22

Hi, Oleksandr, On Aug 25, Oleksandr Byelkin wrote: > revision-id: 1c3182a99e5 (mariadb-10.6.1-482-g1c3182a99e5) > parent(s): 7253cdf8928 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2022-08-16 11:49:41 +0200 > message: > > MDEV-5215 prerequisite of prerequisite: if DB is not mentioned in connect ignore errors of switching to it > > diff --git a/mysql-test/main/init_connect.test b/mysql-test/main/init_connect.test > index e96d02fe0d1..60549c3458e 100644 > --- a/mysql-test/main/init_connect.test > +++ b/mysql-test/main/init_connect.test > @@ -10,31 +10,31 @@ > > --source include/add_anonymous_users.inc > > -connect (con0,localhost,root,,); > +connect (con0,localhost,root,,test); why did you specify a db explicitly (here and in many other places)? I thought the whole point was to modify mysqltest in a way that would avoid the need of changing tests. > connection con0; > select hex(@a); Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] b3844107287: MDEV-16546 System versioning setting to allow history modification
by Sergei Golubchik 24 Aug '22

24 Aug '22

Hi, Aleksey! On Jun 08, Aleksey Midenkov wrote: > revision-id: b3844107287 (mariadb-10.5.2-424-gb3844107287) > parent(s): 27d66d644cf > author: Aleksey Midenkov <midenok(a)gmail.com> > committer: Aleksey Midenkov <midenok(a)gmail.com> > timestamp: 2021-03-01 17:43:34 +0300 > message: > > MDEV-16546 System versioning setting to allow history modification > > 1. force_fields_visible session variable makes system-invisible and > user-invisible fields visible on next table open. FLUSH TABLES > required for already opened tables. * system_versioning_insert_history * allows pseudocolumns ROW_START and ROW_END be specified in INSERT * doesn't affect invisible columns * FLUSH TABLES requirememnt is rather confusing, but let's have it, at least until I could suggest something better > 2. If secure_timestamp allows to modify timestamp variable then > following DML commands: INSERT, INSERT..SELECT and LOAD DATA can > specify row_start and row_end system field values. when system_versioning_insert_history=1 > 3. Cleaned up select_insert::send_data() from setting vers_write as > this parameter is now set on TABLE initialization. I don't see where it's set. This commit removes two table->vers_write= table->versioned(); and adds one from_field->table->vers_write= false; in Item_field::fix_fields. But where is vers_write set to true? Was it done in some other commit? > diff --git a/mysql-test/suite/versioning/r/insert.result b/mysql-test/suite/versioning/r/insert.result > index 2645d0184e8..00d1bb705c4 100644 > --- a/mysql-test/suite/versioning/r/insert.result > +++ b/mysql-test/suite/versioning/r/insert.result > @@ -112,3 +112,95 @@ x y > 9 9001 > drop table t1; > drop table t2; > +# > +# MDEV-16546 System versioning setting to allow history modification > +# > +# restart: --secure-timestamp=NO > +set @@session.time_zone='+00:00'; > +create table t1(x int) with system versioning; > +insert into t1(x) values (1); > +insert into t1(x, row_start, row_end) values (2, '1980-01-01 00:00:00', '1980-01-01 00:00:01'); > +ERROR 42S22: Unknown column 'row_start' in 'field list' > +set session force_fields_visible= 1; > +flush tables; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `x` int(11) DEFAULT NULL, > + `row_start` timestamp(6) GENERATED ALWAYS AS ROW START, > + `row_end` timestamp(6) GENERATED ALWAYS AS ROW END, shouldn't be visible here, only in insert and load. > + PERIOD FOR SYSTEM_TIME (`row_start`, `row_end`) > +) ENGINE=DEFAULT_ENGINE DEFAULT CHARSET=latin1 WITH SYSTEM VERSIONING > +insert into t1(x, row_start, row_end) values (3, '1980-01-01 00:00:00', '1980-01-01 00:00:01'); please, add tests for insert into t1 values (4) this should work, row_start/row_end must be mentioned explicitly. Also insert into t1 set x=5, row_start=..., row_end=... this should work too. And when only one row_start or row_end is mentioned as well. But update t1 set row_start=... insert into t1 (x, row_start, row_end) values (...) on duplicate key update row_start=... this should fail. While replace load data ... replace should work, but as DELETE+INSERT (I mean that DELETE in versioned tables doesn't actually delete history). If that's too difficult, then an error too. > +update t1 set x= x + 1; > +select x, check_row_ts(row_start, row_end) from t1 for system_time all order by x; ... > diff --git a/mysql-test/suite/versioning/t/insert.opt b/mysql-test/suite/versioning/t/insert.opt > new file mode 100644 > index 00000000000..a74d68957ef > --- /dev/null > +++ b/mysql-test/suite/versioning/t/insert.opt > @@ -0,0 +1 @@ > +--secure-timestamp=yes why is this? > -- source suite/versioning/common_finish.inc > diff --git a/mysql-test/suite/versioning/t/insert2.opt b/mysql-test/suite/versioning/t/insert2.opt > new file mode 100644 > index 00000000000..a74d68957ef > --- /dev/null > +++ b/mysql-test/suite/versioning/t/insert2.opt > @@ -0,0 +1 @@ > +--secure-timestamp=yes and this? > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index d23b190b2c5..93f8a8434e4 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -7604,6 +7604,20 @@ void THD::set_last_commit_gtid(rpl_gtid &gtid) > #endif > } > > +bool THD::vers_modify_sys_field() const don't call it vers_modify_sys_field, your previous name vers_modify_history was better. There may be other sys fields that will never be directly writable (like ROWID, for example). > +{ > + if (lex->sql_command != SQLCOM_INSERT && > + lex->sql_command != SQLCOM_INSERT_SELECT && > + lex->sql_command != SQLCOM_LOAD) > + return false; I don't think this is enough to reject insert ... on duplicate key update row_start=xxx > + if (opt_secure_timestamp >= SECTIME_REPL || this is fixed in a later commit, so ok > + (opt_secure_timestamp == SECTIME_SUPER && > + !(security_ctx->master_access & SUPER_ACL))) > + return false; > + return true; > +} > + > + > void > wait_for_commit::reinit() > { > diff --git a/sql/sql_class.h b/sql/sql_class.h > index 2913a4a8350..520ea1d63bf 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -820,6 +820,7 @@ typedef struct system_variables > > vers_asof_timestamp_t vers_asof_timestamp; > ulong vers_alter_history; > + my_bool force_fields_visible; system_versioning_insert_history > } SV; > > /** > @@ -5400,6 +5401,7 @@ class THD: public THD_count, /* this must be first */ > Item *sp_prepare_func_item(Item **it_addr, uint cols= 1); > bool sp_eval_expr(Field *result_field, Item **expr_item_ptr); > > + bool vers_modify_sys_field() const; vers_modify_history > }; > > > diff --git a/sql/sql_insert.cc b/sql/sql_insert.cc > index f0b84ab126f..84c2ae4ab36 100644 > --- a/sql/sql_insert.cc > +++ b/sql/sql_insert.cc > @@ -2014,7 +2014,7 @@ int write_record(THD *thd, TABLE *table, COPY_INFO *info, select_result *sink) > !table->file->referenced_by_foreign_key() && > (!table->triggers || !table->triggers->has_delete_triggers())) > { > - if (table->versioned(VERS_TRX_ID)) > + if (table->versioned(VERS_TRX_ID) && table->vers_write) does it even make sense? I mean direct insertion of transaction ids. > { > bitmap_set_bit(table->write_set, table->vers_start_field()->field_index); > table->file->column_bitmaps_signal(); > @@ -2029,7 +2029,7 @@ int write_record(THD *thd, TABLE *table, COPY_INFO *info, select_result *sink) > info->deleted++; > if (!table->file->has_transactions()) > thd->transaction->stmt.modified_non_trans_table= TRUE; > - if (table->versioned(VERS_TIMESTAMP)) > + if (table->versioned(VERS_TIMESTAMP) && table->vers_write) This (and the previous hunk) are in the DUP_REPLACE branch. There should be tests for these changes. > { > store_record(table, record[2]); > error= vers_insert_history_row(table); > diff --git a/sql/sys_vars.cc b/sql/sys_vars.cc > index f817f9086a9..4429308a097 100644 > --- a/sql/sys_vars.cc > +++ b/sql/sys_vars.cc > @@ -6737,3 +6737,7 @@ static Sys_var_ulonglong Sys_max_rowid_filter_size( > SESSION_VAR(max_rowid_filter_size), CMD_LINE(REQUIRED_ARG), > VALID_RANGE(1024, (ulonglong)~(intptr)0), DEFAULT(128*1024), > BLOCK_SIZE(1)); > + > +static Sys_var_mybool Sys_force_fields_visible( > + "force_fields_visible", "Make invisible fields visible on next table open", "system_versioning_insert_history" "Allows direct inserts into ROW_START and ROW_END columns" > + SESSION_VAR(force_fields_visible), CMD_LINE(OPT_ARG), DEFAULT(FALSE)); > diff --git a/sql/table.cc b/sql/table.cc > index f4bdbdeac5a..520f3738c9e 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -4024,6 +4024,9 @@ enum open_frm_error open_table_from_share(THD *thd, TABLE_SHARE *share, > { > if (!((*field_ptr)= share->field[i]->clone(&outparam->mem_root, outparam))) > goto err; > + if (thd->variables.force_fields_visible && > + (*field_ptr)->invisible <= INVISIBLE_SYSTEM) > + (*field_ptr)->invisible= VISIBLE; Not really, system_versioning_insert_history should not affect visibility of anything besides ROW_START and ROW_END pseudocolumns in the INSERT and LOAD DATA statements. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 5

Re: [Maria-developers] Review request for MDEV-12325 Unexpected data type and truncation when using CTE
by Alexander Barkov 06 Aug '22

06 Aug '22

Hello Monty. The patch looks OK for me. I have just one question below: On 8/5/22 7:15 PM, Michael Widenius wrote: > Hi! > > Just to make things easy, here is the patch: > commit a556c4d972eaa09bb8ad586b9bda6cfb2a7b574a > Author: Monty <monty(a)mariadb.org> > Date: Fri Aug 5 17:57:27 2022 +0300 > > MDEV-12325 Unexpected data type and truncation when using CTE > > When creating a recursive CTE, the column types are taken from the > non recursive part of the CTE (this is according to the SQL standard). > > This patch adds code to abort the CTE if the calculated values in the > recursive part does not fit in the fields in the created temporary table. > > The new code only affects recursive CTE, so it should not cause any notable > problems for old applications. > <cut> > diff --git a/sql/sql_union.cc b/sql/sql_union.cc > index 48d8c16db68..d271a919a4e 100644 > --- a/sql/sql_union.cc > +++ b/sql/sql_union.cc > @@ -111,6 +111,8 @@ int select_unit::send_data(List<Item> &values) > { > int rc; > int not_reported_error= 0; > + enum_check_fields save_count_cuted_fields= thd->count_cuted_fields; > + > if (unit->offset_limit_cnt) > { // using limit offset,count > unit->offset_limit_cnt--; > @@ -118,8 +120,10 @@ int select_unit::send_data(List<Item> &values) > } > if (thd->killed == ABORT_QUERY) > return 0; > + > if (table->no_rows_with_nulls) > table->null_catch_flags= CHECK_ROW_FOR_NULLS_TO_REJECT; > + > if (intersect_mark) > { > fill_record(thd, table, table->field + 1, values, TRUE, FALSE); > @@ -127,6 +131,8 @@ int select_unit::send_data(List<Item> &values) > } > else > fill_record(thd, table, table->field, values, TRUE, FALSE); > + thd->count_cuted_fields= save_count_cuted_fields; > + > if (unlikely(thd->is_error())) > { > rc= 1; > @@ -297,7 +303,13 @@ bool select_unit::send_eof() > Why do you need above changes in select_unit::send_data()? Shouldn't it be enough just to have below changes in select_union_recursive::send_data()? > int select_union_recursive::send_data(List<Item> &values) > { > + Abort_on_warning_instant_set handle_abort(thd, 1); > + enum_check_fields save_count_cuted_fields= thd->count_cuted_fields; > + > + /* For recursive CTE's give warnings for wrong field info */ > + thd->count_cuted_fields= CHECK_FIELD_WARN; > int rc= select_unit::send_data(values); > + thd->count_cuted_fields= save_count_cuted_fields; > > if (rc == 0 && > write_err != HA_ERR_FOUND_DUPP_KEY && <cut>

1 1

Re: [Maria-developers] a507f126b03: MDEV-29159 Patch for MDEV-28918 introduces more inconsistency than it solves, breaks usability
by Sergei Golubchik 05 Aug '22

05 Aug '22

Hi, Alexander, a thought about the error message text and a couple of questions about changes in results, please, see below: On Aug 05, Alexander Barkov wrote: > revision-id: a507f126b03 (mariadb-10.7.4-39-ga507f126b03) > parent(s): 97d16c7544c > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2022-08-05 09:42:46 +0400 > message: > > MDEV-29159 Patch for MDEV-28918 introduces more inconsistency than it solves, breaks usability > > 1. Store assignment failures on incompatible data types now cause errors if: > - STRICT_ALL_TABLES or STRICT_TRANS_TABLES sql_mode is used, and > - IGNORE is not used > > Otherwise, only a warning is raised and the statement continues. > > 2. Changing the error/warning test as follows: > > -ERROR HY000: Illegal parameter data types inet6 and int for operation 'SET' > +ERROR HY000: Unknown CAST(int AS inet6) in assignment of 'target' Better not to write 'CAST(int AS inet6)' if there's no explicit CAST() function, I think it might be confusing. I'd say Cannot cast int as inet6 in assignment of 'target' > so in case of a big table it's easier to see which column has the problem. > The new error text is aslo applied to SP variables. s/aslo/also/ > diff --git a/mysql-test/main/sp-vars.result b/mysql-test/main/sp-vars.result > index 127a0993e2b..feef67ccb1d 100644 > --- a/mysql-test/main/sp-vars.result > +++ b/mysql-test/main/sp-vars.result > @@ -1026,11 +1026,11 @@ BEGIN > SELECT arg; > END| > CALL p1((1, 2)); > -ERROR HY000: Illegal parameter data types tinyint and row for operation 'SET' > +ERROR 21000: Operand should contain 1 column(s) Why not "Unknown CAST" like elsewhere? > diff --git a/mysql-test/main/type_geometry_mix_int.result b/mysql-test/main/type_geometry_mix_int.result > index e5d00c63727..0d0a21a8d29 100644 > --- a/mysql-test/main/type_geometry_mix_int.result > +++ b/mysql-test/main/type_geometry_mix_int.result > @@ -6,6 +6,7 @@ > # > CREATE TABLE t1 (target GEOMETRY DEFAULT POINT(1,1), source INT DEFAULT 0); > # Start of type_store_assignment_incompatible.inc > +SET @sql_mode_save= (SELECT @@sql_mode); strange, why not `SET @sql_mode_save= @@sql_mode` ? > SET @source_type= (SELECT COLUMN_TYPE FROM INFORMATION_SCHEMA.COLUMNS > WHERE COLUMN_NAME='source' > AND TABLE_NAME='t1' > @@ -47,6 +52,7 @@ INSERT INTO t3 VALUES > (2, > (SELECT source FROM t2 ORDER BY id LIMIT 1), > (SELECT source FROM t2 ORDER BY id LIMIT 1)); > +ERROR 22003: Cannot get geometry object from data you send to the GEOMETRY field What's that? It wasn't an error, and now it is? > SELECT * FROM v3; > id target source > 1 POINT(1 1) 0 Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 2

Re: [Maria-developers] f3fc58149dd: MDEV-29159 Patch for MDEV-28918 introduces more inconsistency than it solves, breaks usability
by Sergei Golubchik 05 Aug '22

05 Aug '22

Hi, Alexander, A couple of comments: On Aug 03, Alexander Barkov wrote: > revision-id: f3fc58149dd (mariadb-10.7.4-39-gf3fc58149dd) > parent(s): 97d16c7544c > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2022-08-03 11:28:31 +0400 > message: > > MDEV-29159 Patch for MDEV-28918 introduces more inconsistency than it solves, breaks usability > > Store assignment failures on incompatible data types now cause errors if: > - STRICT_ALL_TABLES or STRICT_TRANS_TABLES sql_mode is used, and > - IGNORE is not used > > Otherwise, only a warning is raised and the statement continues. > > TODO: add MTR tests for "loose" modes. > > diff --git a/sql/field.cc b/sql/field.cc > index 249269a6b1d..e997e2cbd69 100644 > --- a/sql/field.cc > +++ b/sql/field.cc > @@ -964,7 +964,7 @@ Type_handler::aggregate_for_result_traditional(const Type_handler *a, > } > > > -bool Field::check_assignability_from(const Type_handler *from) const > +bool Field::check_assignability_from(const Type_handler *from, bool error) const "error" is very confusing here, I had to read the whole patch to understand what it means. Better names: * only_warning * ignore * strict * issue_error > { > /* > Using type_handler_for_item_field() here to get the data type handler > @@ -982,6 +982,15 @@ bool Field::check_assignability_from(const Type_handler *from) const > type_handler_for_item_field()); > if (th.aggregate_for_result(from->type_handler_for_item_field())) > { > + if (!error) > + { > + THD *thd= get_thd(); > + push_warning_printf(thd, Sql_condition::WARN_LEVEL_WARN, > + ER_ILLEGAL_PARAMETER_DATA_TYPES2_FOR_OPERATION, > + ER_THD(thd, ER_ILLEGAL_PARAMETER_DATA_TYPES2_FOR_OPERATION), > + type_handler()->name().ptr(), from->name().ptr(), "SET"); > + return false; > + } > my_error(ER_ILLEGAL_PARAMETER_DATA_TYPES2_FOR_OPERATION, MYF(0), > type_handler()->name().ptr(), from->name().ptr(), "SET"); > return true; if you wouldn't want to change the error message in a followup commit, I'd suggested something like my_error(ER_ILLEGAL_PARAMETER_DATA_TYPES2_FOR_OPERATION, MYF(error ? 0 : ME_WARNING), type_handler()->name().ptr(), from->name().ptr(), "SET"); return error; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

[Maria-developers] Review: MDEV-29075 Changing explicit_defaults_for_timestamp within stored procedure works inconsistently
by Alexander Barkov 01 Aug '22

01 Aug '22

Hello Sergei, I have a couple of suggestions: > commit 7b8304045272111a6f4d44196d6b37cbfef06f37 > Author: Sergei Golubchik <serg(a)mariadb.org> > Date: Wed Jul 20 17:31:48 2022 +0200 > > MDEV-29075 Changing explicit_defaults_for_timestamp within stored procedure works inconsistently > > move the OPTION_EXPLICIT_DEF_TIMESTAMP check from the parsing step > to the execution > <cut> > > diff --git a/sql/field.cc b/sql/field.cc > index 2e0c70d3d13..23ebb07b7f7 100644 > --- a/sql/field.cc > +++ b/sql/field.cc > @@ -10824,6 +10824,7 @@ Column_definition::Column_definition(THD *thd, Field *old_field, > comment= old_field->comment; > vcol_info= old_field->vcol_info; > option_list= old_field->option_list; > + explicitly_nullable= !(old_field->flags & NOT_NULL_FLAG); This introduces asymmetry in how NULL and NOT NULL attributes are handled. I suggest considering these ways: 1. Change "bool explicitly_nullable" to "bool explicit_nullability", which will be: - false if nothing was specified - true if NULL or NOT NULL was specified. 2. Or don't store NOT_NULL_FLAG in Column_definition::flags at all. Add a new Column_definition enum member with three values for: - not specified - explicit NULL - explicit NOT NULL As discussed on slack, the latter will need some changes. But I think not that much, as only Column_definition needs changes, while Field does not seem to need at this point. > compression_method_ptr= 0; > versioning= VERSIONING_NOT_SET; > invisible= old_field->invisible; > diff --git a/sql/field.h b/sql/field.h > index 5d5be5204cd..87a30bd9f95 100644 > --- a/sql/field.h > +++ b/sql/field.h > @@ -5281,7 +5281,7 @@ class Column_definition: public Sql_alloc, > uint flags, pack_length; > List<String> interval_list; > engine_option_value *option_list; > - > + bool explicitly_nullable; > > /* > This is additinal data provided for any computed(virtual) field. > @@ -5303,7 +5303,7 @@ class Column_definition: public Sql_alloc, > comment(null_clex_str), > on_update(NULL), invisible(VISIBLE), char_length(0), > flags(0), pack_length(0), > - option_list(NULL), > + option_list(NULL), explicitly_nullable(false), > vcol_info(0), default_value(0), check_constraint(0), > versioning(VERSIONING_NOT_SET), period(NULL) > { > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index 3eeabe20c71..8c31f988a05 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -2316,6 +2316,8 @@ void promote_first_timestamp_column(List<Create_field> *column_definitions) > if (column_definition.is_timestamp_type() || // TIMESTAMP > column_definition.unireg_check == Field::TIMESTAMP_OLD_FIELD) // Legacy > { > + if (!column_definition.explicitly_nullable) > + column_definition.flags|= NOT_NULL_FLAG; I think this does not work well with PS. On the very first execution the field becomes NOT NULL forever, if the current OPTION_EXPLICIT_DEF_TIMESTAMP says so. > DBUG_PRINT("info", ("field-ptr:%p", column_definition.field)); > if (first && > (column_definition.flags & NOT_NULL_FLAG) != 0 && // NOT NULL, > diff --git a/sql/sql_type.cc b/sql/sql_type.cc > index 44a51428280..dc0c0b1e0cf 100644 > --- a/sql/sql_type.cc > +++ b/sql/sql_type.cc > @@ -4258,18 +4258,6 @@ void Type_handler_temporal_with_date::Item_update_null_value(Item *item) const > (void) item->get_date(thd, &ltime, Datetime::Options(thd)); > } > > -bool > -Type_handler_timestamp_common:: > -Column_definition_set_attributes(THD *thd, > - Column_definition *def, > - const Lex_field_type_st &attr, > - column_definition_type_t type) const > -{ > - Type_handler::Column_definition_set_attributes(thd, def, attr, type); > - if (!(thd->variables.option_bits & OPTION_EXPLICIT_DEF_TIMESTAMP)) > - def->flags|= NOT_NULL_FLAG; > - return false; > - > > void Type_handler_string_result::Item_update_null_value(Item *item) const > { > diff --git a/sql/sql_type.h b/sql/sql_type.h > index 7ff4bc64679..e5f54535ece 100644 > --- a/sql/sql_type.h > +++ b/sql/sql_type.h > @@ -6655,11 +6655,6 @@ class Type_handler_timestamp_common: public Type_handler_temporal_with_date > bool Item_func_min_max_get_date(THD *thd, Item_func_min_max*, > MYSQL_TIME *, date_mode_t fuzzydate) > const override; > - bool Column_definition_set_attributes(THD *thd, > - Column_definition *def, > - const Lex_field_type_st &attr, > - column_definition_type_t type) > - const override; > }; > > > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index 5416cec49b5..f326d8c0cc5 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -5831,7 +5831,6 @@ field_def: > | opt_generated_always AS virtual_column_func > { > Lex->last_field->vcol_info= $3; > - Lex->last_field->flags&= ~NOT_NULL_FLAG; // undo automatic NOT NULL for timestamps > } > vcol_opt_specifier vcol_opt_attribute > { > @@ -6313,7 +6312,12 @@ attribute_list: > ; > > attribute: > - NULL_SYM { Lex->last_field->flags&= ~ NOT_NULL_FLAG; $$.init(); } > + NULL_SYM > + { > + Lex->last_field->flags&= ~NOT_NULL_FLAG; > + Lex->last_field->explicitly_nullable= true; > + $$.init(); > + } > | DEFAULT column_default_expr { Lex->last_field->default_value= $2; $$.init(); } > | ON UPDATE_SYM NOW_SYM opt_default_time_precision > {

3 3

[Maria-developers] review for (MDEV-28632) bugfix: DEFAULT NULL was allowed for NOT NULL columns
by Alexander Barkov 29 Jul '22

29 Jul '22

Hello Sergei, > commit 5a362d486b30fdaf3c7a360737331767154b4ee8 > Author: Sergei Golubchik <serg(a)mariadb.org> > Date: Mon Jul 18 22:53:27 2022 +0200 > > bugfix: DEFAULT NULL was allowed for NOT NULL columns > > this was detected during parsing (so NOT NULL DEFAULT NULL was an error), > but if a column was made NOT NULL later (e.g. as a part of primary key, > in mysql_prepare_create_table()), DEFAULT NULL was accepted and ignored. > > to fix this the NOT NULL DEFAULT NULL was moved into > mysql_prepare_create_table(). I'm not sure that in case of auto_increment we should really disallow DEFAULT NULL under terms of this TIMESTAMP task. Look at this script: DROP TABLE IF EXISTS t1; CREATE TABLE t1 (a INT NOT NULL AUTO_INCREMENT PRIMARY KEY); INSERT INTO t1 VALUES (NULL); Query OK, 1 row affected (0.012 sec) NULL can be inserted without errors. Yes, it's inserted with a side effect - it's replaced to the next auto increment value. But why this side effect makes NULL an invalid default? Note, I'm not against this change. Stricter behavior is good :) But I'm worried that: - It can break compatibility on one hand. - It goes without it's own MDEV on the other hand. This change deserves it's own MDEV (with motivation and consequences explained) and should be properly documented and mentioned in release notes. <cut> > index 712b8629c09..d8471894db9 100644 > --- a/mysql-test/main/view.result > +++ b/mysql-test/main/view.result > @@ -4082,12 +4082,12 @@ DROP TABLE t1,t2; > # MDEV-6251: SIGSEGV in query optimizer (in set_check_materialized > # with MERGE view) > # > -CREATE TABLE t1 (a1 INT(11) NOT NULL DEFAULT NULL AUTO_INCREMENT PRIMARY KEY); > -CREATE TABLE t2 (b1 INT(11) NOT NULL DEFAULT NULL AUTO_INCREMENT PRIMARY KEY); > -CREATE TABLE t3 (c1 INT(11) NOT NULL DEFAULT NULL AUTO_INCREMENT PRIMARY KEY); > -CREATE TABLE t4 (d1 INT(11) NOT NULL DEFAULT NULL AUTO_INCREMENT PRIMARY KEY); > -CREATE TABLE t5 (e1 INT(11) NOT NULL DEFAULT NULL AUTO_INCREMENT PRIMARY KEY); > -CREATE TABLE t6 (f1 INT(11) NOT NULL DEFAULT NULL AUTO_INCREMENT PRIMARY KEY); > +CREATE TABLE t1 (a1 INT(11) NOT NULL AUTO_INCREMENT PRIMARY KEY); > +CREATE TABLE t2 (b1 INT(11) NOT NULL AUTO_INCREMENT PRIMARY KEY); > +CREATE TABLE t3 (c1 INT(11) NOT NULL AUTO_INCREMENT PRIMARY KEY); > +CREATE TABLE t4 (d1 INT(11) NOT NULL AUTO_INCREMENT PRIMARY KEY); > +CREATE TABLE t5 (e1 INT(11) NOT NULL AUTO_INCREMENT PRIMARY KEY); > +CREATE TABLE t6 (f1 INT(11) NOT NULL AUTO_INCREMENT PRIMARY KEY); It seems you removed all cases with the old syntax. So now it's not recorded in MTR what happens on this statement: CREATE TABLE t1 (a INT NOT NULL DEFAULT NULL AUTO_INCREMENT PRIMARY KEY); Please cover this scenario in some relevant *.test file. <cut> > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index cf0bbfb1130..3eeabe20c71 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc <cut> > @@ -3647,6 +3643,15 @@ mysql_prepare_create_table(THD *thd, HA_CREATE_INFO *create_info, > { > Field::utype type= (Field::utype) MTYP_TYPENR(sql_field->unireg_check); > > + if (sql_field->default_value && > + sql_field->default_value->expr->type() == Item::NULL_ITEM && > + sql_field->flags & NOT_NULL_FLAG && > + !(sql_field->flags & AUTO_INCREMENT_FLAG)) > + { > + my_error(ER_INVALID_DEFAULT, MYF(0), sql_field->field_name.str); > + DBUG_RETURN(TRUE); > + } ^^^ now obvious errors are caught only during EXECUTE, although it's already clear at the PREPARE stage that the DEFAULT is wrong: MariaDB [test]> PREPARE stmt FROM 'CREATE TABLE T1 (a INT NOT NULL DEFAULT NULL)'; Query OK, 0 rows affected (0.000 sec) Statement prepared MariaDB [test]> EXECUTE stmt; ERROR 1067 (42000): Invalid default value for 'a' I suggest we still catch non-unambiguous cases (when nothing depends on system variables) during PREPARE. Inside some existing or a new Type_handler method. Also, please add PS tests: - for unambiguous scenarios like here - for ambiguous scenarios, e.g. with TIMESTAMP columns changing OPTION_EXPLICIT_DEF_TIMESTAMP between PREPARE and EXECUTE. It seems this code needs to be changed: > /* > Set NO_DEFAULT_VALUE_FLAG if this field doesn't have a default value and > it is NOT NULL, not an AUTO_INCREMENT field, not a TIMESTAMP and not > updated trough a NOW() function. > */ > if (!sql_field->default_value && > !sql_field->has_default_function() && > (sql_field->flags & NOT_NULL_FLAG) && > (!sql_field->is_timestamp_type() || > (thd->variables.option_bits & OPTION_EXPLICIT_DEF_TIMESTAMP))&& > !sql_field->vers_sys_field()) > { > sql_field->flags|= NO_DEFAULT_VALUE_FLAG; > sql_field->pack_flag|= FIELDFLAG_NO_DEFAULT; > } It can only add NO DEFAULT related flags. Shouldn't it also remove NO DEFAULT flags if the current PS EXECUTE time option OPTION_EXPLICIT_DEF_TIMESTAMP is unset? <cut> > diff --git a/storage/spider/mysql-test/spider/bg/t/spider_fixes.test b/storage/spider/mysql-test/spider/bg/t/spider_fixes.test > index b222f494ba1..41f87e61416 100644 > --- a/storage/spider/mysql-test/spider/bg/t/spider_fixes.test > +++ b/storage/spider/mysql-test/spider/bg/t/spider_fixes.test > @@ -294,14 +294,14 @@ if ($USE_CHILD_GROUP2) > --disable_query_log > echo CREATE TABLE ta_l ( > a int(11) NOT NULL DEFAULT '0', > - b char(1) DEFAULT NULL, > - c datetime DEFAULT NULL, > + b char(1), > + c datetime, > PRIMARY KEY (a, b, c) > ) MASTER_1_ENGINE MASTER_1_CHARSET MASTER_1_COMMENT5_2_1; > eval CREATE TABLE ta_l ( > a int(11) NOT NULL DEFAULT '0', > - b char(1) DEFAULT NULL, > - c datetime DEFAULT NULL, > + b char(1), > + c datetime, > PRIMARY KEY (a, b, c) > ) $MASTER_1_ENGINE $MASTER_1_CHARSET $MASTER_1_COMMENT5_2_1; > --enable_query_log Why this change ^^^ ? > diff --git a/storage/spider/mysql-test/spider/t/spider_fixes.test b/storage/spider/mysql-test/spider/t/spider_fixes.test > index 47bc225d614..75f108f1164 100644 > --- a/storage/spider/mysql-test/spider/t/spider_fixes.test > +++ b/storage/spider/mysql-test/spider/t/spider_fixes.test > @@ -294,14 +294,14 @@ if ($USE_CHILD_GROUP2) > --disable_query_log > echo CREATE TABLE ta_l ( > a int(11) NOT NULL DEFAULT '0', > - b char(1) DEFAULT NULL, > - c datetime DEFAULT NULL, > + b char(1), > + c datetime, > PRIMARY KEY (a, b, c) > ) MASTER_1_ENGINE MASTER_1_CHARSET MASTER_1_COMMENT5_2_1; > eval CREATE TABLE ta_l ( > a int(11) NOT NULL DEFAULT '0', > - b char(1) DEFAULT NULL, > - c datetime DEFAULT NULL, > + b char(1), > + c datetime, > PRIMARY KEY (a, b, c) > ) $MASTER_1_ENGINE $MASTER_1_CHARSET $MASTER_1_COMMENT5_2_1; > --enable_query_log Same question.

2 2

Re: [Maria-developers] Move on with Gtid strict mode
by Kristian Nielsen 13 Jul '22

13 Jul '22

Andrei Elkin <andrei.elkin(a)mariadb.com> writes: > Kristian, salve! Hi Andrei, thanks for interesting comments and discussion! >> 3. To solve (2), we recover the old master with >> --tc-heuristic-recover=rollback. This makes it rollback (=discard) any >> transaction that was not fully committed to disk (does it also roll back the >> binary log for such transactions? I don't recall this detail). >> >> 4. Because of (3), there might be a transaction that originated on the old >> master (has it's server_id), but which is now missing from that server (now >> a slave) as it was deliberately discarded. >> >> 5. To solve (4), we will use circular replication > > Well, the circularity is not necessary for (4). The ex-master just needs > to be configured as slave. Right, I see now. > Also let me call in the following this stage of receiving/accepting own > events as 'slave recovery'. But thinking more, there's another point that I don't understand, around the use of --tc-heuristic-recover=rollback. From memory, and also from reading docs on https://mariadb.com/kb/en/heuristic-recovery-with-the-transaction-coordinat…, this will forcibly roll back any prepared-but-not-committed InnoDB transaction, but _not_ roll back any corresponding binlog entry. Doesn't this leave the binlog inconsistent with the storage engine? Of the last few transactions in the binlog, some may be in InnoDB, some may not. And if so, doesn't this make the binlog useless for deciding which of our own transactions to ignore? When the old-master-now-slave gets one of its own events that's also in the binlog, we don't know if it is in the engine. If we apply it, we may end up with it double. If we do not apply it, we may be missing it. Is there some mechanims used in this feature/MDEV that handles this correctly that I'm missing? >> So isn't there a better way to achieve the underlying use case? For example, >> why not just use the new MASTER_DEMOTE_TO_SLAVE option to set the correct >> slave position from which to replicate from in the old master (now >> slave)? > > That's how it should normally do, yet it's an optimization (to save on > traffic) part of the solution. Interesting. Can you elaborate on this optimization - how does it save on traffic? > In the worst case `gtid_slave_pos` can be even empty for the > ex-master's domain, so the ex-master slave might keep receiving a bulk > of own events, only to ignore the most of them. > And that's the patch's base code logics as well. I guess there's something here that I'm not understanding. If MASTER_DEMOTE_TO_SLAVE is used to copy the gtid_binlog_pos into the gtid_slave_pos, how can the gtid_slave_pos be empty for the ex-master's domain? > ... , `gtid_ignore_duplicate = OFF` would have to go in my perception. > And if it goes, we'd remain with the only "choice" of > `gtid_ignore_duplicate = ON` :-) so the option would be gone. Hm. So are you suggesting that the current behaviour when --gtid-ignore-duplicates=ON becomes the default behaviour - in fact the only behaviour? As I remember it, the --gtid-ignore-duplicates=ON option is a very dangerous option unless the servers are managed very strictly and correctly. In fact I was expecting to find a dire warning written by myself about this, but didn't find anything in a quick web search ;-) I think the main problem is if an incorrect transaction done on a slave somewhere introduces an out-of-order GTID sequence, Then the replication will silently ignore potentially a large number of transactions that have their sequence number smaller than the incorrect out-of-order transaction. Out-of-order will normally be an error. But mistakes happen, and it is bad if this causes replication to become silently corrupt, as opposed to flagging the error (assuming --gtid-strict-mode=ON). What do you think? Again, it's cool that things are being worked on in MariaDB GTID, and I hope my comments don't come out as purely negative. While MariaDB/MySQL replication is in some ways overly complex, this is also what gives it the enormous flexibility for advanced users to do crazy^H^H^H^H^Hfantastic stuff! So there's some leeway for allowing complexity. But there are also some very delicate pitfalls that the original design tried hard to avoid, with decisions that may not always be obvious. So anything I can do there to help with history and design principles, I'm happy to assist with... - Kristian.

1 0

Re: [Maria-developers] ce5cc8fb905: MDEV-29021 ALTER TABLE fails when a stored virtual column is dropped+added
by Sergei Golubchik 13 Jul '22

13 Jul '22

Hi, Nikita, This is good, but I think fill_extra_persistent_columns() shouldn't be used at all. It doesn't handle default values. I've changed rpl_alter_extra_persistent test as -alter table t1 add column z1 int as(a+1) virtual, add column z2 int as (a+2) persistent; +alter table t1 add column z1 int as(a+1) virtual, add column z2 int default (a+2); and, of course, new column z2 wasn't updated properly. An easy fix would be to use your code in all cases and remove fill_extra_persistent_columns(). Could you do that too, please? And add the test case too. May be just, append , add column z3 int default (a+2) On Jul 07, Nikita Malyavin wrote: > revision-id: ce5cc8fb905 (mariadb-10.6.1-506-gce5cc8fb905) > parent(s): 49ad875902e > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2022-07-04 15:55:03 +0300 > message: > > MDEV-29021 ALTER TABLE fails when a stored virtual column is dropped+added > > We shouldn't rely on `fill_extra_persistent_columns`, as it only updates > fields which have an index > cols->n_bits (replication bitmap width). > > Normal update_virtual_fields+update_default_fields shoudl be done. > > diff --git a/sql/rpl_record.cc b/sql/rpl_record.cc > index 17731e64600..42f476fc112 100644 > --- a/sql/rpl_record.cc > +++ b/sql/rpl_record.cc > @@ -412,6 +412,26 @@ int unpack_row(rpl_group_info *rgi, TABLE *table, uint const colcnt, > { > copy->do_copy(copy); > } > + if (table->default_field) > + { > + error= table->update_default_fields(table->in_use->lex->ignore); > + if (unlikely(error)) > + DBUG_RETURN(error); > + } > + if (table->vfield) > + { > + error= table->update_virtual_fields(table->file, VCOL_UPDATE_FOR_WRITE); > + if (unlikely(error)) > + DBUG_RETURN(error); > + } > + } > + else > + { > + /* > + Add Extra slave persistent columns > + */ > + if (unlikely(error= fill_extra_persistent_columns(table, cols->n_bits))) > + DBUG_RETURN(error); > } > > /* > @@ -439,12 +459,6 @@ int unpack_row(rpl_group_info *rgi, TABLE *table, uint const colcnt, > } > } > > - /* > - Add Extra slave persistent columns > - */ > - if (unlikely(error= fill_extra_persistent_columns(table, cols->n_bits))) > - DBUG_RETURN(error); > - > /* > We should now have read all the null bytes, otherwise something is > really wrong. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 4

Re: [Maria-developers] 3e2d297830b: MDEV-29013 ER_KEY_NOT_FOUND/lock timeout upon online alter with long unique
by Sergei Golubchik 13 Jul '22

13 Jul '22

Hi, Nikita, On Jul 08, Nikita Malyavin wrote: > revision-id: 3e2d297830b (mariadb-10.6.1-511-g3e2d297830b) > parent(s): c1feb8c3c32 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2022-07-05 15:20:04 +0300 > message: > > MDEV-29013 ER_KEY_NOT_FOUND/lock timeout upon online alter with long unique > > 1. ER_KEY_NOT_FOUND > Some virtual columns were not updated because were not included in read_set > to the moment of calculation. > > In Row_logs_event::do_apply_event some fields are excluded based on m_cols > value, the number of replicated rows. the number of replicated *columns* > We can't rely on this. Basically, we'd be satisfied, if all columns were > just set, at least for now. > > 2. ER_LOCK_WAIT_TIMEOUT > This is a long unique specific problem. > > Sometimes, lookup_handler is created for to->file. To properly free it, > ha_reset should be called. It is usually done by calling > close_thread_table, but ALTER TABLE makes it differently. Hence, a single > ha_reset call is added to mysql_alter_table. > > Also, the lifetime of lookup_handler is corrected: it's been allocated on > thd->mem_root, which in case of online alter > is event's mem_root, and can vary other ways. > It is changed to table->mem_root, which is more appropriate: > ha_reset is called only when table is closed, or in a few additional cases, > which correspond to a statement end. So memory leaks are unlikely here. > > diff --git a/sql/handler.cc b/sql/handler.cc > index d71c0fc83a3..fbebfd27203 100644 > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -3212,7 +3212,7 @@ int handler::create_lookup_handler() > handler *tmp; > if (lookup_handler != this) > return 0; > - if (!(tmp= clone(table->s->normalized_path.str, table->in_use->mem_root))) > + if (!(tmp= clone(table->s->normalized_path.str, &table->mem_root))) that's wrong, the lifetime of a lookup handler is one statement, it's destroyed at the end. You cannot keep allocating new handlers on the table->mem_root for every statement. > return 1; > lookup_handler= tmp; > return lookup_handler->ha_external_lock(table->in_use, F_RDLCK); > diff --git a/sql/log_event_server.cc b/sql/log_event_server.cc > index 89ca2a41937..2c19bf67586 100644 > --- a/sql/log_event_server.cc > +++ b/sql/log_event_server.cc > @@ -6007,6 +6007,11 @@ int Rows_log_event::do_apply_event(rpl_group_info *rgi) > if (m_width == table->s->fields && bitmap_is_set_all(&m_cols)) > set_flags(COMPLETE_ROWS_F); > > + Rpl_table_data rpl_data{}; > + if (rgi) rgi->get_table_data(table, &rpl_data); > + > + if (!rpl_data.copy_fields) > + { > /* > Set tables write and read sets. > > @@ -6027,17 +6027,19 @@ int Rows_log_event::do_apply_event(rpl_group_info *rgi) > bitmap_set_all(table->read_set); > if (get_general_type_code() == DELETE_ROWS_EVENT || > get_general_type_code() == UPDATE_ROWS_EVENT) > bitmap_intersect(table->read_set,&m_cols); > > bitmap_set_all(table->write_set); > > /* WRITE ROWS EVENTS store the bitmap in m_cols instead of m_cols_ai */ > MY_BITMAP *after_image= ((get_general_type_code() == UPDATE_ROWS_EVENT) ? > &m_cols_ai : &m_cols); > bitmap_intersect(table->write_set, after_image); > > this->slave_exec_mode= slave_exec_mode_options; // fix the mode > + } I'm sorry, I don't quite understand. "some fields are excluded based on m_cols value" - what columns were excluded? What was the value of m_cols here? > table->rpl_write_set= table->write_set; > > // Do event specific preparations > error= do_before_row_operations(rli); > @@ -6045,8 +6052,6 @@ int Rows_log_event::do_apply_event(rpl_group_info *rgi) > extra columns on the slave. In that case, do not force > MODE_NO_AUTO_VALUE_ON_ZERO. > */ > - Rpl_table_data rpl_data{}; > - if (rgi) rgi->get_table_data(table, &rpl_data); > sql_mode_t saved_sql_mode= thd->variables.sql_mode; > if (!is_auto_inc_in_extra_columns()) > thd->variables.sql_mode= (rpl_data.copy_fields ? saved_sql_mode : 0) > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index 29c4d09cd39..a3efcdd9785 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -11011,6 +11011,9 @@ do_continue:; > !(table->file->ha_table_flags() & HA_REUSES_FILE_NAMES) && > !(new_table->file->ha_table_flags() & > HA_REUSES_FILE_NAMES)); > + > + // Close lookup_handler. > + new_table->file->ha_reset(); please, put it not here, but inside the following thd->drop_temporary_table() call. It'll also fix Sanja's bug MDEV-26433. And please coordinate with him to put it on the same place so that git would detect a conflict and we could merge it to have only one reset call in that method, not two. > /* > Close the intermediate table that will be the new table, but do > not delete it! Even though MERGE tables do not have their children > @@ -11854,8 +11857,11 @@ copy_data_between_tables(THD *thd, TABLE *from, TABLE *to, > > // We'll be filling from->record[0] from row events > bitmap_set_all(from->write_set); > - // We restore bitmaps, because update event is going to mess up with them. > + // Use all columns. UPDATE/DELETE events reset read_set and write_set to > + // def_*_set after each row operation, so using all_set won't work. > to->default_column_bitmaps(); > + bitmap_set_all(&to->def_read_set); > + bitmap_set_all(&to->def_write_set); > > end_read_record(&info); > init_read_record_done= false; > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 73c4fd41afc: Do not ignore sql_mode when replicating
by Sergei Golubchik 13 Jul '22

13 Jul '22

Hi, Nikita, On Jul 07, Nikita Malyavin wrote: > > diff --git a/sql/log_event_server.cc b/sql/log_event_server.cc > index aba846453e9..89ca2a41937 100644 > --- a/sql/log_event_server.cc > +++ b/sql/log_event_server.cc > @@ -6045,9 +6045,12 @@ int Rows_log_event::do_apply_event(rpl_group_info *rgi) > extra columns on the slave. In that case, do not force > MODE_NO_AUTO_VALUE_ON_ZERO. > */ > + Rpl_table_data rpl_data{}; > + if (rgi) rgi->get_table_data(table, &rpl_data); > sql_mode_t saved_sql_mode= thd->variables.sql_mode; > if (!is_auto_inc_in_extra_columns()) > - thd->variables.sql_mode= MODE_NO_AUTO_VALUE_ON_ZERO; > + thd->variables.sql_mode= (rpl_data.copy_fields ? saved_sql_mode : 0) you start using this condition in many places and it's quite cryptic for those, who don't know the code. I'd suggest to add method, like struct Rpl_table_data { ... bool is_online_alter() { return copy_fields != NULL; } ... }; btw, it fits very well in your Rpl_table_data simplification. > + | MODE_NO_AUTO_VALUE_ON_ZERO; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] Move on with Gtid strict mode
by Kristian Nielsen 12 Jul '22

12 Jul '22

Hi Andrei, Good to hear from you, and to see that things are still going on with MariaDB GTID replication! I tried looking at the MDEVs that you referred to. Here is how I understand the motivation for this, correct me if I'm wrong: 1. We're considering a slave with "lossless semi-sync slave", where phantom reads are avoided by delaying storage engine commit until at least one slave has acknowledged a transaction. 2. If a master crashes, and comes back up after recovery, we want to prevent it from having any extra transactions that are not replicated to the rest of the replication servers (one of which will have been promoted as a new master). Such a transaction would prevent demoting the old master as a slave, as it would contain a rouge transaction not known to the rest of the replication topology. 3. To solve (2), we recover the old master with --tc-heuristic-recover=rollback. This makes it rollback (=discard) any transaction that was not fully committed to disk (does it also roll back the binary log for such transactions? I don't recall this detail). 4. Because of (3), there might be a transaction that originated on the old master (has it's server_id), but which is now missing from that server (now a slave) as it was deliberately discarded. 5. To solve (4), we will use circular replication so that the missing transaction gets replicated back to the old master (now slave). And we want to have a slave replicate its own transactions (replicate_same_sever_id), which is normally not done. And we want to use the server's binlog to know which transactions to replicate and which to ignore. Is that correctly understood? I must admit, this sounds horribly complex to me. So my question would be, do we really need to do this to solve the fundamental use case? In my belief, the main drawback with MariaDB replication and GTID is the complexity of especially the more advanced setups, which is barely feasible for even a skilled DBA to fully understand. Does this come from trying to port over some specific MySQL replication/GTID feature to MariaDB GTID? One important point is that MariaDB GTID was originally designed to keep the concept of master/binlog and slave/relaylog cleanly separated. A master transaction needs only be written to the binlog, there is no need to update any replication state (=overhead). Similarly, a slave transaction needs only be replicated to the storage engine, there is no need to write it to the binlog (=overhead). IIRC, MySQL (GTID) replication required the binlog (and --log-slave-updates) to be enabled on a slave) This is one of the improvements in the design of MariaDB GTID. So I don't particularly like the idea of now involving the binlog in how a slave replicates transactions. On the contrary, the MASTER_USE_GTID=current_pos feature is one thing I regret from the original GTID design, precisely because it violates the principle. And I was happy to see work to deprecate this, which I think is the prefered direction. So isn't there a better way to achieve the underlying use case? For example, why not just use the new MASTER_DEMOTE_TO_SLAVE option to set the correct slave position from which to replicate from in the old master (now slave)? And then use the existing --gtid-ignore-duplicate option? Since this is GTID-based circular replication, there must already be configured a separate gtid_domain_id for each master? Then you suggest to change the current --gtid-strict-mode and --gtid-ignore-duplicate options to a single option. I feel less strongly about this change; as you say, --gtid-strict-mode=OFF + --gtid-ignore-duplicates=ON may not make much sense. Still, I wonder what is the the need for such a change? This would be a not-backwards-compatible change, with little real benefit, as the semantics would be the same? Also, the --gtid-strict-mode option is an option that is highly recommended for any GTID use, and which should preferably be as simple as possible to understand to be able to be widely used by DBAs. On the contrary --gtid-ignore-duplicates is a very complex option only useful for very complex replication topologies, and with many subtle pitfalls that is very hard to fully understand. So there is real benefit to keeping them separate. As to the --gtid-strict-mode=OFF. The main original motivation for this option (and its default value of "OFF") was to be able to enable GTID by default, for any server installation upgraded (to was it 10.0 that introduced GTID? Long time back!). This is another big improvement (so I think) in MariaDB GTID over MySQL, that GTID is available by default, and compatible with any replication topology that works with old-style MySQL replication. Is it really so that no-one is using old-style, non-GTID replication any more, with all its sloppy setups possible, Sure, then non-strict GTID mode can be deprecated, I suppose - but it seems doubtful that this would be the case for *all* users? I hope I managed to explain where I didn't fully understand the motivation and need for some of these proposals? I'd be very interested to understand better any underlying necessities that I didn't catch from your mail, and maybe refine my comments. And once again, great to see the work still going on with MariaDB replication and GTID! Hope this helps, - Kristian. Andrei Elkin <andrei.elkin(a)mariadb.com> writes: > Two points of concern of this email deal with observations around GTID strict mode. > > > 1. About gtid_ignore_duplicates > > Introduced by > > hash: 2c2478b82260 > author: unknown > date: 2014-03-09 10:27:38 > > MDEV-5804: If same GTID is ... > Before, the arrival of same GTID twice in multi-source replication > would cause double-apply or in gtid strict mode an error. > > with a clear motivation to overcome the gtid strict mode error policy. > That is `gtid_ignore_duplicates = ON' softens `gtid_strict_mode = ON' > to ignore a duplicate gtid without a slave error/interruption. > As `gtid_strict_mode = OFF', `gtid_ignore_duplicates = ON' combination is > hardly ever relevant/practical, the `gtid_ignore_duplicates = ON' then can be fairly > perceived as `gtid_strict_mode=ON' 's "mild" submode to suggest > enum `gtid_stict_mode' { OFF, ON, "ON_BUT_IGNORE_DUPLICATES_RATHER_THAT_ERROR" } > which would obsolete `gtid_ignore_duplicates'. > > Why I'd prefer this change actually also deals with the circular replication > that - MDEV-28609 - forces us to do searching for duplicates > in the slave's binlog. > MDEV-28609 presents a case where a same server-id transaction received > by a semisync slave requires a more sophisticated decision making than > before (the same server-id based) for weather it to be ignored or > accepted. > > The accept policy of even own server-id gtids (transactions) is necessary for > MDEV-21117 semisync slave recovery, and the legacy > `replicate_same_sever_id' filtering can't help being too coarse as we need to consider > the gtid seq_no. > So under those conditions the same server-id transaction is made to be accepted > when it does not exist in the slave's binlog. > Otherwise, when it does exist in binlog, it'd be (like) a duplicate, arriving through > a circular topology from a "source" of the slave local binlog, only > to be ignored, like the same server-id gtids normally are. > The ignore policy in this case corresponds to the proposed > `gtid_stict_mode = "ON_BUT_IGNORE..."'. > > It should be naturally default as it inherits its pre-gtid equivalent of > `replicate_same_sever_id = NO' > and is safe (the slave's gtid state) for a real multi-source configuration as well. > > `gtid_stict_mode = ON' still would make sense as a method to > catch misconfigurations of the likes of Change-Master's do,ignore- domain_id ones. > > Your take? > > 2. But it's not clear what `gtid_strict_mode = OFF' is for. > > The ON mode is pretty flexible allowing gtid seq_no holes that is setting the gtid state > to any greater than the current (logical timestamp) value. > To set the gtid "clock" backward is still possible though requires slave restart. > > So how would you comment on an idea to deprecate OFF? > > Cheers, > > Andrei

1 0

Re: [Maria-developers] [Commits] 78370ea: MDEV-28965 Assertion failure when preparing UPDATE with derived table in WHERE
by Oleksandr Byelkin 11 Jul '22

11 Jul '22

Hi, Igor! Generally all looks good, but see my notes and couple of questions below. On Thu, Jul 7, 2022 at 6:57 AM IgorBabaev <igor(a)mariadb.com> wrote: > revision-id: 78370ea0fd21f66a4084d488ddcebac541249241 > (mariadb-10.6.1-477-g78370ea) > parent(s): 2db18fdb3d68d906fbd188ec570a64502ba55849 > author: Igor Babaev > committer: Igor Babaev > timestamp: 2022-07-06 21:57:23 -0700 > message: > > MDEV-28965 Assertion failure when preparing UPDATE with derived table in > WHERE > > This patch fixes not only the assertion failure in the function > Field_iterator_table_ref::set_field_iterator() but also: > - fixes the problem of forced materialization of derived tables used > in subqueries contained in WHERE clauses of single-table and multi-table > UPDATE and DELETE statements > - fixes the problem of MDEV-17954 that prevented execution of multi-table > DELETE statements if they use in their WHERE clauses references to > the tables that are updated. > > The patch must be considered a complement to the patch for MDEV-28883. > > --- > mysql-test/main/delete_use_source.result | 184 ++++++++++- > mysql-test/main/delete_use_source.test | 120 ++++++++ > mysql-test/main/derived.result | 336 > +++++++++++++++++++++ > mysql-test/main/derived.test | 210 +++++++++++++ > mysql-test/main/derived_cond_pushdown.result | 27 +- > mysql-test/main/derived_cond_pushdown.test | 2 +- > mysql-test/main/multi_update.result | 2 - > mysql-test/main/multi_update.test | 2 - > mysql-test/main/subselect.result | 1 - > mysql-test/main/subselect.test | 1 - > mysql-test/main/subselect_no_exists_to_in.result | 1 - > mysql-test/main/subselect_no_mat.result | 1 - > mysql-test/main/subselect_no_opts.result | 1 - > mysql-test/main/subselect_no_scache.result | 1 - > mysql-test/main/subselect_no_semijoin.result | 1 - > .../engines/iuds/r/update_delete_number.result | 22 +- > .../suite/engines/iuds/t/update_delete_number.test | 20 +- > sql/opt_subselect.cc | 56 +++- > sql/sql_base.cc | 41 ++- > sql/sql_delete.cc | 63 ++-- > sql/sql_delete.h | 14 +- > sql/sql_lex.cc | 28 +- > sql/sql_lex.h | 1 + > sql/sql_update.cc | 21 +- > sql/sql_update.h | 16 +- > sql/sql_yacc.yy | 13 + > sql/table.cc | 10 +- > 27 files changed, 1106 insertions(+), 89 deletions(-) > > [skip] > > --- a/mysql-test/suite/engines/iuds/t/update_delete_number.test > +++ b/mysql-test/suite/engines/iuds/t/update_delete_number.test > @@ -285,8 +285,18 @@ SELECT * FROM t1,t2 WHERE t2.c1=t1.c2; > DELETE FROM a1, a2 USING t1 AS a1 INNER JOIN t2 AS a2 WHERE a2.c1=a1.c2; > --sorted_result > SELECT * FROM t1,t2 WHERE t2.c1=t1.c2; > ---error ER_UPDATE_TABLE_USED > -DELETE FROM t1,t2 using t1,t2 where t1.c1=(select c1 from t1); > +TRUNCATE TABLE t1; > +TRUNCATE TABLE t2; > +INSERT INTO t1 VALUES(254,127,1),(0,-128,2),(1,127,3),(3,NULL,5); > +INSERT INTO t2 VALUES(127,255,1),(127,1,2),(-128,0,3),(-1,NULL,5); > +# After the patch for MDEV-28883 this should not report > +# --error ER_UPDATE_TABLE_USED anymore > Why is the above left as a comment? (the same in other cases) [skip] > diff --git a/sql/opt_subselect.cc b/sql/opt_subselect.cc > index fa338f0..ac47566 100644 > --- a/sql/opt_subselect.cc > +++ b/sql/opt_subselect.cc > @@ -30,6 +30,8 @@ > #include "sql_base.h" > #include "sql_const.h" > #include "sql_select.h" > +#include "sql_update.h" > +#include "sql_delete.h" > it would be nice to have comment why you include it as it only for one function as I can see, something like: #include "sql_update.h" // processing_as_multitable_update_prohibited (the same for delete and for including in #include "sql_update.h" & "sql_base.cc") #include "filesort.h" > #include "opt_subselect.h" > #include "sql_test.h" > @@ -532,6 +534,48 @@ bool is_materialization_applicable(THD *thd, > Item_in_subselect *in_subs, > return FALSE; > } > > [skip] > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > index 2b41b78..447573b 100644 > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -47,6 +47,8 @@ > #include "sql_prepare.h" > #include "sql_statistics.h" > #include "sql_cte.h" > +#include "sql_update.h" > +#include "sql_delete.h" > see above > #include <m_ctype.h> > #include <my_dir.h> > #include <hash.h> > @@ -1164,7 +1166,7 @@ TABLE_LIST* find_dup_table(THD *thd, TABLE_LIST > *table, TABLE_LIST *table_list, > /* > If we found entry of this table or table of SELECT which already > processed in derived table or top select of > multi-update/multi-delete > - (exclude_from_table_unique_test) or prelocking placeholder. > + (exclude_from_table_unique_test) or prelocking placeholder. > Please fix formatting above > */ > DBUG_PRINT("info", > ("found same copy of table or table which we should > skip")); > @@ -1175,16 +1177,43 @@ TABLE_LIST* find_dup_table(THD *thd, TABLE_LIST > *table, TABLE_LIST *table_list, > We come here for queries of type: > INSERT INTO t1 (SELECT tmp.a FROM (select * FROM t1) as tmp); > > - Try to fix by materializing the derived table > + Try to fix by materializing the derived table if one can't do > without it. > */ > TABLE_LIST *derived= res->belong_to_derived; > if (derived->is_merged_derived() && !derived->derived->is_excluded()) > { > - DBUG_PRINT("info", > + bool materialize= true; > + if (thd->lex->sql_command == SQLCOM_UPDATE) > + { > + Sql_cmd_update *cmd= (Sql_cmd_update *) (thd->lex->m_sql_cmd); > + if (cmd->is_multitable()) > + materialize= false; > + else if (!cmd->processing_as_multitable_update_prohibited(thd)) > + { > + cmd->set_as_multitable(); > + materialize= false; > + } > + } > + else if (thd->lex->sql_command == SQLCOM_DELETE) > + { > + Sql_cmd_delete *cmd= (Sql_cmd_delete *) (thd->lex->m_sql_cmd); > + if (cmd->is_multitable()) > + materialize= false; > + if (!cmd->processing_as_multitable_delete_prohibited(thd)) > + { > + cmd->set_as_multitable(); > + materialize= false; > + } > + } > + if (materialize) > + { > + DBUG_PRINT("info", > ("convert merged to materialization to resolve the > conflict")); > - derived->change_refs_to_fields(); > - derived->set_materialized_derived(); > - goto retry; > + derived->change_refs_to_fields(); > + derived->set_materialized_derived(); > + // derived->field_translation= 0; > above comment has no big sens, looks not cleaned up some chenges? > + goto retry; > + } > } > } > DBUG_RETURN(res); > > [skip] > diff --git a/sql/sql_delete.h b/sql/sql_delete.h > index e1d5044..ffb8173 100644 > --- a/sql/sql_delete.h > +++ b/sql/sql_delete.h > @@ -44,11 +44,12 @@ class Sql_cmd_delete final : public Sql_cmd_dml > { > public: > Sql_cmd_delete(bool multitable_arg) > - : multitable(multitable_arg), save_protocol(NULL) {} > + : orig_multitable(multitable_arg), save_protocol(NULL) > + { multitable= orig_multitable; } > why do not use the same initializers? : multitable(multitable_arg), rig_multitable(multitable_arg), save_protocol(NULL) {} > enum_sql_command sql_command_code() const override > { > - return multitable ? SQLCOM_DELETE_MULTI : SQLCOM_DELETE; > + return orig_multitable ? SQLCOM_DELETE_MULTI : SQLCOM_DELETE; > } > > DML_prelocking_strategy *get_dml_prelocking_strategy() > > [skip] > diff --git a/sql/sql_lex.cc b/sql/sql_lex.cc > index 78c067a..4059c0e 100644 > --- a/sql/sql_lex.cc > +++ b/sql/sql_lex.cc > @@ -37,6 +37,8 @@ > #include "sql_partition.h" > #include "sql_partition_admin.h" // > Sql_cmd_alter_table_*_part > #include "event_parse_data.h" > +#include "sql_update.h" > +#include "sql_delete.h" > see above > > void LEX::parse_error(uint err_number) > { > @@ -4037,9 +4039,8 @@ bool LEX::can_use_merged() > SYNOPSIS > LEX::can_not_use_merged() > > - @param no_update_or_delete Set to 1 if we can't use merge with > multiple-table > - updates, like when used from > - TALE_LIST::init_derived() > + @param forced_no_merge_for_update_delete Set to 1 if we can't use merge > with > + multiple-table updates/deletes > > DESCRIPTION > Temporary table algorithm will be used on all SELECT levels for > queries > @@ -4050,7 +4051,7 @@ bool LEX::can_use_merged() > TRUE - VIEWs with MERGE algorithms can be used > */ > > -bool LEX::can_not_use_merged(bool no_update_or_delete) > +bool LEX::can_not_use_merged(bool forced_no_merge_for_update_delete) > { > switch (sql_command) { > case SQLCOM_CREATE_VIEW: > @@ -4064,18 +4065,29 @@ bool LEX::can_not_use_merged(bool > no_update_or_delete) > return TRUE; > > case SQLCOM_UPDATE_MULTI: > - case SQLCOM_DELETE_MULTI: > - if (no_update_or_delete) > + if (forced_no_merge_for_update_delete) > return TRUE; > /* Fall through */ > > case SQLCOM_UPDATE: > - if (no_update_or_delete && m_sql_cmd && > - (m_sql_cmd->sql_command_code() == SQLCOM_UPDATE_MULTI || > + if (forced_no_merge_for_update_delete && > + (((Sql_cmd_update *) m_sql_cmd)->is_multitable() || > query_tables->is_multitable())) > return TRUE; > + return FALSE; > + > + case SQLCOM_DELETE_MULTI: > + if (forced_no_merge_for_update_delete) > + return TRUE; > /* Fall through */ > > + case SQLCOM_DELETE: > + if (forced_no_merge_for_update_delete && > + (((Sql_cmd_delete *) m_sql_cmd)->is_multitable() || > + query_tables->is_multitable())) > + return TRUE; > + return FALSE; > + > default: > return FALSE; > } > diff --git a/sql/sql_lex.h b/sql/sql_lex.h > index e733b4f..402e3bd 100644 > > [skip] > diff --git a/sql/sql_update.cc b/sql/sql_update.cc > index 6b14c4f..7769777 100644 > --- a/sql/sql_update.cc > +++ b/sql/sql_update.cc > @@ -2807,6 +2807,23 @@ bool multi_update::send_eof() > > > /** > + @brief Check whether conversion to multi-table update is prohibited > + > + @param thd global context the processed statement > + @returns true if conversion is prohibited, false otherwise > + > + @todo > + Introduce handler level flag for storage engines that would prohibit > + such conversion for any single-table update. > +*/ > + > +bool Sql_cmd_update::processing_as_multitable_update_prohibited(THD *thd) > +{ > + return false; > +} > + > + > +/** > @brief Perform precheck of table privileges for update statements > > @param thd global context the processed statement > @@ -2889,7 +2906,9 @@ bool Sql_cmd_update::prepare_inner(THD *thd) > "updating and querying the same temporal periods table"); > DBUG_RETURN(TRUE); > } > - multitable= true; > + if (!table_list->is_multitable() && > + !processing_as_multitable_update_prohibited(thd)) > + multitable= true; > } > } > > diff --git a/sql/sql_update.h b/sql/sql_update.h > index d0fc7cb..4aff77a 100644 > --- a/sql/sql_update.h > +++ b/sql/sql_update.h > @@ -46,12 +46,12 @@ class Sql_cmd_update final : public Sql_cmd_dml > { > public: > Sql_cmd_update(bool multitable_arg) > - : multitable(multitable_arg) > - { } > + : orig_multitable(multitable_arg) > + { multitable= orig_multitable; } > why do not use construction for both? Sql_cmd_update(bool multitable_arg) : multitable(multitable_arg) : orig_multitable(multitable_arg) {} > > enum_sql_command sql_command_code() const override > { > - return multitable ? SQLCOM_UPDATE_MULTI : SQLCOM_UPDATE; > + return orig_multitable ? SQLCOM_UPDATE_MULTI : SQLCOM_UPDATE; > } > > DML_prelocking_strategy *get_dml_prelocking_strategy() > @@ -59,6 +59,12 @@ class Sql_cmd_update final : public Sql_cmd_dml > return &multiupdate_prelocking_strategy; > } > > + bool processing_as_multitable_update_prohibited(THD *thd); > + > + bool is_multitable() { return multitable; } > + > + void set_as_multitable() { multitable= true; } > + > protected: > /** > @brief Perform precheck of table privileges for update statements > @@ -89,13 +95,15 @@ class Sql_cmd_update final : public Sql_cmd_dml > */ > bool multitable; > > + /* Original value of the 'multitable' flag set by constructor */ > + const bool orig_multitable; > + > /* The prelocking strategy used when opening the used tables */ > Multiupdate_prelocking_strategy multiupdate_prelocking_strategy; > > public: > /* The list of the updating expressions used in the set clause */ > List<Item> *update_value_list; > - > }; > > #endif /* SQL_UPDATE_INCLUDED */ > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index a587a37..13dc602 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -13370,8 +13370,21 @@ delete_single_table: > YYPS->m_lock_type, > YYPS->m_mdl_type, > NULL, > + 0))) > + MYSQL_YYABORT; > + Select->table_list.save_and_clear(&Lex->auxiliary_table_list); > + Lex->table_count= 1; > Why it is not enough to have this counter (table_count) set by open_tables? May be it would be better to use mysql_init_select() to set everything needed by SELECT for sure? > + Lex->query_tables= 0; > + Lex->query_tables_last= &Lex->query_tables; > + if (unlikely(!Select-> > + add_table_to_list(thd, $2, NULL, > TL_OPTION_UPDATING, > + YYPS->m_lock_type, > + YYPS->m_mdl_type, > + NULL, > $3))) > MYSQL_YYABORT; > + Lex->auxiliary_table_list.first->correspondent_table= > + Lex->query_tables; > YYPS->m_lock_type= TL_READ_DEFAULT; > YYPS->m_mdl_type= MDL_SHARED_READ; > } > [skip]

1 0

[Maria-developers] Account locked and USERS view?
by Martijn Tonies (Upscene Productions) 06 Jul '22

06 Jul '22

Hi, Any reason why the ‘account locked’ attribute isn’t included in the USERS view? https://mariadb.com/kb/en/account-locking/ It could be added like this: if(json_value(mysql.global_priv.Priv, '$.account_locked') = 1, 'Y', 'N') AS is_locked With regards, Martijn Tonies Upscene Productions https://www.upscene.com Database Workbench - developer tool for Oracle, SQL Server, PostgreSQL, MySQL, InterBase, NexusDB and Firebird.

1 0

Re: [Maria-developers] MDEV-28881 Server crashes in Dep_analysis_context::create_table_value
by Sergey Petrunia 05 Jul '22

05 Jul '22

Hi Oleg, > commit 812f8cbfb3a7a02705f65d8bb7e30ae2faf13b5d > Author: Oleg Smirnov <olernov(a)gmail.com> > Date: Tue Jun 21 16:20:17 2022 +0700 > > MDEV-28881 Server crashes in Dep_analysis_context::create_table_value > > SELECT_LEX::first_select()->join may be NULL for some scenarios. > This commit adds a check for this. Looking into Jira I can infer that the patch fixes two issues, but the commit comment doesn't say that. Please avoid doing this in the future. Please split the commit into two: - first one fixing the SELECT_LEX::first_select()->join == NULL part - the second fixing the memory management. For the second part, I think it's not a good approach when most of the Table Elimination data resides in a MEM_ROOT with exception of std::set which keeps its data on the heap. I was wondering if there's a way to get STL containers to reside on MEM_ROOT as well, and found that yes there is, and MySQL actually already has it: https://dev.mysql.com/worklog/task/?id=6074 https://github.com/mysql/mysql-server/blob/8.0/sql/mem_root_allocator.h Can we copy mem_root_allocator.h from them (in a separate commit) and then fix memory management here by making std::set use it? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

1 0

Re: [Maria-developers] 9ec0b36ee8b: MDEV-28838 password_reuse_check plugin mixes username and password
by Sergei Golubchik 04 Jul '22

04 Jul '22

Hi, Oleksandr, On Jul 04, Oleksandr Byelkin wrote: > revision-id: 9ec0b36ee8b (mariadb-10.7.4-27-g9ec0b36ee8b) > parent(s): ac0af4ec4ab > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2022-06-29 17:25:44 +0200 > message: > > MDEV-28838 password_reuse_check plugin mixes username and password > > To prevent the problem of mixing user name and password and > host name and user name we add length of the hostname and user name > to the hash. > > diff --git a/plugin/password_reuse_check/password_reuse_check.c b/plugin/password_reuse_check/password_reuse_check.c > index ff0364ce007..4d07cda878a 100644 > --- a/plugin/password_reuse_check/password_reuse_check.c > +++ b/plugin/password_reuse_check/password_reuse_check.c > @@ -165,9 +166,11 @@ static int validate(const MYSQL_CONST_LEX_STRING *username, > return 1; > } > > - memcpy(buff, hostname->str, hostname->length); > - memcpy(buff + hostname->length, username->str, username->length); > - memcpy(buff + hostname->length + username->length, password->str, > + int2store(buff, hostname->length); > + int2store(buff + 2, username->length); > + memcpy(buff + 4, hostname->str, hostname->length); > + memcpy(buff + 4 + hostname->length, username->str, username->length); > + memcpy(buff + 4 + hostname->length + username->length, password->str, > password->length); why? let's use the standard length,string format, why risk inventing something unusual here for no benefits at all? I'd do like: static char *store_str(char *to, MYSQL_CONST_LEX_STRING from) { int2store(to, from.length); memcpy(to+2, from.str, from.length); return to+2+from.length; } and buff= store_str(buff, hostname); buff= store_str(buff, username); buff= store_str(buff, password); > buff[key_len]= 0; > memset(hash, 0, sizeof(hash)); > @@ -235,7 +238,7 @@ maria_declare_plugin(password_reuse_check) > 0x0100, ^^^ this should be 0x0101 (or 0x0200 as I suggest below) > NULL, > sysvars, > - "1.0", > + "1.1", make it 2.0 please, this is an incompatible change > MariaDB_PLUGIN_MATURITY_GAMMA > } > maria_declare_plugin_end; > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] e9293e31ec6: MDEV-28930 ALTER TABLE Deadlocks with parallel TL_WRITE
by Sergei Golubchik 30 Jun '22

30 Jun '22

Hi, Nikita, This is very good. Just one comment, see below: On Jun 29, Nikita Malyavin wrote: > revision-id: e9293e31ec6 (mariadb-10.6.1-499-ge9293e31ec6) > parent(s): 6cfcb36d027 > author: Nikita Malyavin > committer: Nikita Malyavin > timestamp: 2022-06-29 20:16:19 +0300 > message: > > MDEV-28930 ALTER TABLE Deadlocks with parallel TL_WRITE > > ALTER ONLINE TABLE acquires table with TL_READ. Myisam normally acquires > TL_WRITE for DML, which makes it hang until table is freed. > > We deadlock once ALTER upgrades its MDL lock. > > Solution: > Unlock table earlier. We don't need to hold TL_READ once we finished > copying. Relay log replication requires no data locks on `from` table. > > diff --git a/sql/log_event_server.cc b/sql/log_event_server.cc > index 425de469a7f..90f6283d9df 100644 > --- a/sql/log_event_server.cc > +++ b/sql/log_event_server.cc > @@ -5734,7 +5734,7 @@ int Rows_log_event::do_apply_event(rpl_group_info *rgi) > used in the transaction and proceed with execution of the actual > event. > */ > - if (!thd->lock) > + if (!thd->lock && rgi->tables_to_lock_count) > { > /* > Lock_tables() reads the contents of thd->lex, so they must be Instead of adding a second condition to the execution path for every row event in the replication, consider this: - if (!thd->lock) + if (!thd->open_tables) this will work identically for replication, I hope, but will be always false in your case, as you still have the table open. I mean, that's the idea. I run the rpl suite with it, but haven't done any further testing, so not completely sure it'll work. When making this change, don't forget to update the comment right above this if(). Ok to push into bb-10.10-MDEV-16329 after that. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 3

[Maria-developers] How to know when a FUNCTION is AGGREGATE?
by Martijn Tonies (Upscene Productions) 17 Jun '22

17 Jun '22

Hi all, Since MariaDB 10.3, you can use an AGGREGATE stored function. https://mariadb.com/kb/en/stored-aggregate-functions/ When examining the meta data, how does one distinguish between aggregate and normal stored functions? Can’t find anything in the information_schema about this. With regards, Martijn Tonies Upscene Productions https://www.upscene.com Database Workbench - developer tool for Oracle, SQL Server, PostgreSQL, MySQL, InterBase, NexusDB and Firebird.

3 3

Re: [Maria-developers] 49ecf935415: MDEV-27009 Add UCA-14.0.0 collations
by Sergei Golubchik 16 Jun '22

16 Jun '22

Hi, Alexander, On Mar 14, Alexander Barkov wrote: > revision-id: 49ecf935415 (mariadb-10.6.1-335-g49ecf935415) > parent(s): c67789f63c8 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2022-02-28 14:04:58 +0400 > message: > > MDEV-27009 Add UCA-14.0.0 collations please, list all user visible changes there. Mainly that collations are now decoupled from charsets. New syntax in CREATE TABLE, changes in I_S tables, etc. > diff --git a/mysql-test/include/ctype_utf_uca1400_ids.inc b/mysql-test/include/ctype_utf_uca1400_ids.inc > new file mode 100644 > index 00000000000..09cf49fc0e7 > --- /dev/null > +++ b/mysql-test/include/ctype_utf_uca1400_ids.inc > @@ -0,0 +1,17 @@ file names are confusing. better rename ctype_ucs_uca1400_ids.inc to something like ctype_convert_uca1400_ids and ctype_utf_uca1400_ids to ctype_set_names_uca1400_ids or something like that, to show what they do. > + > +--disable_ps_protocol > +--enable_metadata > +DELIMITER $$; > +FOR rec IN (SELECT COLLATION_NAME > + FROM INFORMATION_SCHEMA.COLLATION_CHARACTER_SET_APPLICABILITY > + WHERE CHARACTER_SET_NAME=@charset > + AND COLLATION_NAME RLIKE 'uca1400' > + ORDER BY ID) > +DO > + EXECUTE IMMEDIATE CONCAT('SET NAMES ',@charset,' COLLATE ', rec.COLLATION_NAME); > + SELECT rec.COLLATION_NAME; > +END FOR; > +$$ > +DELIMITER ;$$ > +--disable_metadata > +--enable_ps_protocol > diff --git a/include/m_ctype.h b/include/m_ctype.h > index 4c6628b72b3..706764ead2a 100644 > --- a/include/m_ctype.h > +++ b/include/m_ctype.h > @@ -34,7 +34,9 @@ enum loglevel { > extern "C" { > #endif > > -#define MY_CS_NAME_SIZE 32 > +#define MY_CS_CHARACTER_SET_NAME_SIZE 32 > +#define MY_CS_COLLATION_NAME_SIZE 64 That's FULL_COLLATION_NAME_SIZE, right? > + > #define MY_CS_CTYPE_TABLE_SIZE 257 > #define MY_CS_TO_LOWER_TABLE_SIZE 256 > #define MY_CS_TO_UPPER_TABLE_SIZE 256 > @@ -240,6 +242,46 @@ typedef enum enum_repertoire_t > } my_repertoire_t; > > > +/* ID compatibility */ > +typedef enum enum_collation_id_type > +{ > + MY_COLLATION_ID_TYPE_PRECISE= 0, > + MY_COLLATION_ID_TYPE_COMPAT_100800= 1 > +} my_collation_id_type_t; > + > + > +/* Collation name display modes */ > +typedef enum enum_collation_name_mode > +{ > + MY_COLLATION_NAME_MODE_FULL= 0, > + MY_COLLATION_NAME_MODE_CONTEXT= 1 > +} my_collation_name_mode_t; > + > + > +/* Level flags */ > +#define MY_CS_LEVEL_BIT_PRIMARY 0x00 > +#define MY_CS_LEVEL_BIT_SECONDARY 0x01 > +#define MY_CS_LEVEL_BIT_TERTIARY 0x02 > +#define MY_CS_LEVEL_BIT_QUATERNARY 0x03 > + > +#define MY_CS_COLL_LEVELS_S1 (1<<MY_CS_LEVEL_BIT_PRIMARY) > + > +#define MY_CS_COLL_LEVELS_AI_CS (1<<MY_CS_LEVEL_BIT_PRIMARY)| \ > + (1<<MY_CS_LEVEL_BIT_TERTIARY) > + > +#define MY_CS_COLL_LEVELS_S2 (1<<MY_CS_LEVEL_BIT_PRIMARY)| \ > + (1<<MY_CS_LEVEL_BIT_SECONDARY) > + > +#define MY_CS_COLL_LEVELS_S3 (1<<MY_CS_LEVEL_BIT_PRIMARY)| \ > + (1<<MY_CS_LEVEL_BIT_SECONDARY) | \ > + (1<<MY_CS_LEVEL_BIT_TERTIARY) AI_CS and S3 don't seem to be used yet > + > +#define MY_CS_COLL_LEVELS_S4 (1<<MY_CS_LEVEL_BIT_PRIMARY)| \ > + (1<<MY_CS_LEVEL_BIT_SECONDARY) | \ > + (1<<MY_CS_LEVEL_BIT_TERTIARY) | \ > + (1<<MY_CS_LEVEL_BIT_QUATERNARY) > + > + > /* Flags for strxfrm */ > #define MY_STRXFRM_LEVEL1 0x00000001 /* for primary weights */ > #define MY_STRXFRM_LEVEL2 0x00000002 /* for secondary weights */ > diff --git a/sql/sql_alter.cc b/sql/sql_alter.cc > index 86c6e9a27f8..9ddd482ad57 100644 > --- a/sql/sql_alter.cc > +++ b/sql/sql_alter.cc > @@ -546,6 +546,7 @@ bool Sql_cmd_alter_table::execute(THD *thd) > > result= mysql_alter_table(thd, &select_lex->db, &lex->name, > &create_info, > + lex->create_info.default_charset_collation, I don't see why you need a new argument here. It's create_info.default_charset_collation, so, mysql_alter_table already gets it in create_info. All other mysql_alter_table invocations also take create_info argument and can get default_charset_collation from there > first_table, > &alter_info, > select_lex->order_list.elements, > diff --git a/sql/sql_partition_admin.cc b/sql/sql_partition_admin.cc > index fb1ae0d5fc7..4188dde252b 100644 > --- a/sql/sql_partition_admin.cc > +++ b/sql/sql_partition_admin.cc > @@ -211,6 +211,7 @@ bool compare_table_with_partition(THD *thd, TABLE *table, TABLE *part_table, > part_table->use_all_columns(); > table->use_all_columns(); > if (unlikely(mysql_prepare_alter_table(thd, part_table, &part_create_info, > + Lex_maybe_default_charset_collation(), Same. Can be in part_create_info > &part_alter_info, &part_alter_ctx))) > { > my_error(ER_TABLES_DIFFERENT_METADATA, MYF(0)); > diff --git a/sql/sql_i_s.h b/sql/sql_i_s.h > index bed2e886718..5ff06d32231 100644 > --- a/sql/sql_i_s.h > +++ b/sql/sql_i_s.h > @@ -162,6 +162,11 @@ class Yesno: public Varchar > { > public: > Yesno(): Varchar(3) { } > + static LEX_CSTRING value(bool val) > + { > + return val ? Lex_cstring(STRING_WITH_LEN("Yes")) : > + Lex_cstring(); > + } eh... please, rename the class from Yesno to something like Yesempty or Yes_or_empty, something that says that the second should not be Lex_cstring(STRING_WITH_LEN("No")) > }; > > > diff --git a/sql/table.cc b/sql/table.cc > index a683a78ff49..c28cb2bd928 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -3491,6 +3493,16 @@ int TABLE_SHARE::init_from_sql_statement_string(THD *thd, bool write, > else > thd->set_n_backup_active_arena(arena, &backup); > > + /* > + THD::reset_db() does not set THD::db_charset, > + so it keeps pointing to the character set and collation > + of the current database, rather than the database of the > + new initialized table. Hmm, is that correct? Could you check other invocation of thd->reset_db()? Perhaps they all need to switch charset? In that case it should be done inside THD::reset_db(). Or may be they have to use mysql_change_db_impl() instead? > + Let's call get_default_db_collation() before reset_db(). > + This forces the db.opt file to be loaded. > + */ > + db_cs= get_default_db_collation(thd, db.str); > + > thd->reset_db(&db); > lex_start(thd); > > @@ -3498,6 +3510,11 @@ int TABLE_SHARE::init_from_sql_statement_string(THD *thd, bool write, > sql_unusable_for_discovery(thd, hton, sql_copy)))) > goto ret; > > + if (!(thd->lex->create_info.default_table_charset= > + thd->lex->create_info.default_charset_collation. > + resolved_to_character_set(db_cs, db_cs))) > + DBUG_RETURN(true); How could this (and similar if()'s in other files) fail? > + > thd->lex->create_info.db_type= hton; > #ifdef WITH_PARTITION_STORAGE_ENGINE > thd->work_part_info= 0; // For partitioning > diff --git a/sql/mysys_charset.h b/sql/mysys_charset.h > new file mode 100644 > index 00000000000..86eaeedd432 > --- /dev/null > +++ b/sql/mysys_charset.h > @@ -0,0 +1,44 @@ > +#ifndef MYSYS_CHARSET > +#define MYSYS_CHARSET > + > +/* Copyright (c) 2021, MariaDB Corporation. > + > + This program is free software; you can redistribute it and/or modify > + it under the terms of the GNU General Public License as published by > + the Free Software Foundation; version 2 of the License. > + > + This program is distributed in the hope that it will be useful, > + but WITHOUT ANY WARRANTY; without even the implied warranty of > + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + GNU General Public License for more details. > + > + You should have received a copy of the GNU General Public License > + along with this program; if not, write to the Free Software > + Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1335 USA */ > + > + > +#include "my_sys.h" > + > + > +class Charset_loader_mysys: public MY_CHARSET_LOADER > +{ > +public: > + Charset_loader_mysys() > + { > + my_charset_loader_init_mysys(this); > + } > + void raise_unknown_collation_error(const char *name, > + CHARSET_INFO *name_cs) const; > + CHARSET_INFO *get_charset(const char *cs_name, uint cs_flags, myf my_flags); > + CHARSET_INFO *get_exact_collation(const char *name, myf utf8_flag); > + CHARSET_INFO *get_contextually_typed_collation(CHARSET_INFO *cs, > + const char *name); > + CHARSET_INFO *get_contextually_typed_collation(const char *name); > + CHARSET_INFO *get_contextually_typed_collation_or_error(CHARSET_INFO *cs, > + const char *name); > + CHARSET_INFO *find_default_collation(CHARSET_INFO *cs); > + CHARSET_INFO *find_bin_collation_or_error(CHARSET_INFO *cs); > +}; you can have C++ code in mysys too, you know, no need to put it in sql/mysys* > + > +#endif // MYSYS_CHARSET > + > diff --git a/strings/ctype-simple.c b/strings/ctype-simple.c > index b579f0af203..d09dfba86ed 100644 > --- a/strings/ctype-simple.c > +++ b/strings/ctype-simple.c > @@ -1940,13 +1941,26 @@ my_bool my_propagate_complex(CHARSET_INFO *cs __attribute__((unused)), > } > > > +void my_ci_set_strength(struct charset_info_st *cs, uint strength) > +{ > + DBUG_ASSERT(strength > 0 && strength <= MY_STRXFRM_NLEVELS); don't use && in asserts, please create two separate asserts instead: DBUG_ASSERT(strength > 0); DBUG_ASSERT(strength <= MY_STRXFRM_NLEVELS); > + cs->levels_for_order= ((1 << strength) - 1); why do you still use the old concept of "strength"? Why not to use bitmap consistently everywhere? > +} > + > + > +void my_ci_set_level_flags(struct charset_info_st *cs, uint flags) > +{ > + DBUG_ASSERT(flags < (1<<MY_STRXFRM_NLEVELS)); > + cs->levels_for_order= flags; > +} > + > /* > Normalize strxfrm flags > > SYNOPSIS: > my_strxfrm_flag_normalize() > + cs - the CHARSET_INFO pointer > flags - non-normalized flags > - nlevels - number of levels > > NOTES: > If levels are omitted, then 1-maximum is assumed. > diff --git a/sql/handler.h b/sql/handler.h > index 8ad521e189a..1e82f37b1e7 100644 > --- a/sql/handler.h > +++ b/sql/handler.h > @@ -2409,7 +2386,32 @@ struct Table_specification_st: public HA_CREATE_INFO, > { > HA_CREATE_INFO::options= 0; > DDL_options_st::init(); > + default_charset_collation.init(); > + } > + > + bool > + add_alter_list_item_convert_to_charset(const Lex_charset_collation_st &cl) > + { > + /* > + cs cannot be NULL, as sql_yacc.yy translates > + CONVERT TO CHARACTER SET DEFAULT > + to > + CONVERT TO CHARACTER SET <character-set-of-the-current-database> > + TODO: Shouldn't we postpone resolution of DEFAULT until the > + character set of the table owner database is loaded from its db.opt? > + */ > + DBUG_ASSERT(cl.charset_collation()); > + DBUG_ASSERT(!cl.is_contextually_typed_collation()); > + alter_table_convert_to_charset= cl.charset_collation(); > + default_charset_collation.Lex_charset_collation_st::operator=(cl); looks quite ugly. can you do, like, default_charset_collation.set(cl) ? > + used_fields|= (HA_CREATE_USED_CHARSET | HA_CREATE_USED_DEFAULT_CHARSET); > + return false; > } > + bool add_table_option_default_charset(CHARSET_INFO *cs); > + bool add_table_option_default_collation(const Lex_charset_collation_st &cl); > + bool resolve_db_charset_and_collation(THD *thd, > + const LEX_CSTRING &db, > + bool is_alter); > }; > > > diff --git a/strings/ctype-uca1400data.h b/strings/ctype-uca1400data.h > new file mode 100644 > index 00000000000..da95dcfde54 > --- /dev/null > +++ b/strings/ctype-uca1400data.h > @@ -0,0 +1,44151 @@ > +/* > + Generated from allkeys.txt version '14.0.0' > +*/ if it's generated, do you need to check it in? perhaps it should be generated during the build? you've checked in allkeys1400.txt anyway. > +static const uint16 uca1400_p000[]= { /* 0000 (4 weights per char) */ > +0x0000,0x0000,0x0000,0x0000, 0x0000,0x0000,0x0000,0x0000, /* 0000 */ > +0x0000,0x0000,0x0000,0x0000, 0x0000,0x0000,0x0000,0x0000, /* 0002 */ > diff --git a/sql/sql_lex.cc b/sql/sql_lex.cc > index 6ca10267187..d115401a855 100644 > --- a/sql/sql_lex.cc > +++ b/sql/sql_lex.cc > @@ -542,6 +542,30 @@ bool LEX::add_alter_list(LEX_CSTRING name, LEX_CSTRING new_name, bool exists) > } > > > +bool LEX::add_alter_list_item_convert_to_charset( > + THD *thd, > + CHARSET_INFO *cs, > + const Lex_charset_collation_st &cl) > +{ > + if (!cs) > + { > + Lex_charset_collation_st tmp; > + tmp.set_charset_collate_default(thd->variables.collation_database); Hmm, what if one is doing ALTER TABLE db.test CHARSET DEFAULT and current db is not `db` but `test` ? > + if (!(cs= tmp.charset_collation())) > + return true; // Should not actually happen assert? > + } > + > + Lex_explicit_charset_opt_collate tmp(cs, false); > + if (tmp.merge_opt_collate_or_error(cl) || > + create_info.add_alter_list_item_convert_to_charset( > + Lex_charset_collation(tmp))) > + return true; > + > + alter_info.flags|= ALTER_CONVERT_TO; > + return false; > +} > + > + > void LEX::init_last_field(Column_definition *field, > const LEX_CSTRING *field_name) > { > @@ -11871,29 +11869,41 @@ CHARSET_INFO *Lex_collation_st::find_default_collation(CHARSET_INFO *cs) > "def" is the upper level CHARACTER SET clause (e.g. of a table) > */ > CHARSET_INFO * > -Lex_collation_st::resolved_to_character_set(CHARSET_INFO *def) const > +Lex_charset_collation_st::resolved_to_character_set(CHARSET_INFO *def) const > { > DBUG_ASSERT(def); > - if (m_type != TYPE_CONTEXTUALLY_TYPED) > - { > - if (!m_collation) > - return def; // Empty - not typed at all > - return m_collation; // Explicitly typed > + > + switch (m_type) { > + case TYPE_EMPTY: > + return def; > + case TYPE_CHARACTER_SET: > + DBUG_ASSERT(m_ci); > + return m_ci; > + case TYPE_COLLATE_EXACT: > + DBUG_ASSERT(m_ci); > + return m_ci; > + case TYPE_COLLATE_CONTEXTUALLY_TYPED: > + break; > } > > // Contextually typed > - DBUG_ASSERT(m_collation); > + DBUG_ASSERT(m_ci); > > - if (m_collation == &my_charset_bin) // CHAR(10) BINARY > - return find_bin_collation(def); > + Charset_loader_mysys loader; > + if (is_contextually_typed_binary_style()) // CHAR(10) BINARY > + return loader.find_bin_collation_or_error(def); > > - if (m_collation == &my_charset_latin1) // CHAR(10) COLLATE DEFAULT > - return find_default_collation(def); > + if (is_contextually_typed_collate_default()) // CHAR(10) COLLATE DEFAULT > + return loader.find_default_collation(def); > + > + const LEX_CSTRING context_name= collation_name_context_suffix(); I'd rather put this in assert, not in if(). Like - if (!strncasecmp(context_name.str, STRING_WITH_LEN("uca1400_"))) + DBUG_ASSERT(!strncasecmp(context_cl_name.str, STRING_WITH_LEN("uca1400_"))); > + if (!strncasecmp(context_name.str, STRING_WITH_LEN("uca1400_"))) > + return loader.get_contextually_typed_collation_or_error(def, > + context_name.str); > > /* > - Non-binary and non-default contextually typed collation. > + Non-binary, non-default, non-uca1400 contextually typed collation. > We don't have such yet - the parser cannot produce this. > - But will have soon, e.g. "uca1400_as_ci". > */ > DBUG_ASSERT(0); > return NULL; > @@ -11944,58 +11972,106 @@ bool Lex_collation_st:: > CHAR(10) BINARY .. COLLATE latin1_bin > CHAR(10) COLLATE uca1400_as_ci .. COLLATE latin1_bin > */ > - if (collation() == &my_charset_latin1 && > - !(cl.collation()->state & MY_CS_PRIMARY)) > + if (is_contextually_typed_collate_default() && > + !cl.charset_collation()->default_flag()) > { > - my_error(ER_CONFLICTING_DECLARATIONS, MYF(0), > - "COLLATE ", "DEFAULT", "COLLATE ", > - cl.collation()->coll_name.str); > + error_conflicting_collations_or_styles(*this, cl); > return true; > } > - if (collation() == &my_charset_bin && > - !(cl.collation()->state & MY_CS_BINSORT)) > + > + if (is_contextually_typed_binary_style() && > + !cl.charset_collation()->binsort_flag()) > { > - my_error(ER_CONFLICTING_DECLARATIONS, MYF(0), > - "", "BINARY", "COLLATE ", cl.collation()->coll_name.str); > + error_conflicting_collations_or_styles(*this, cl); > return true; > } > *this= cl; > return false; > } > > - if (cl.is_contextually_typed_collation()) > - { > + DBUG_ASSERT(0); > + return false; > +} > + > + > +bool > +Lex_explicit_charset_opt_collate:: > + merge_collate_or_error(const Lex_charset_collation_st &cl) > +{ > + DBUG_ASSERT(cl.type() != Lex_charset_collation_st::TYPE_CHARACTER_SET); > + > + switch (cl.type()) { > + case Lex_charset_collation_st::TYPE_EMPTY: > + return false; > + case Lex_charset_collation_st::TYPE_CHARACTER_SET: > + DBUG_ASSERT(0); > + return false; > + case Lex_charset_collation_st::TYPE_COLLATE_EXACT: > /* > - EXPLICIT + CONTEXT > - CHAR(10) COLLATE latin1_bin .. COLLATE DEFAULT - not supported > - CHAR(10) COLLATE latin1_bin .. COLLATE uca1400_as_ci - not yet > + EXPLICIT + EXPLICIT > + CHAR(10) CHARACTER SET latin1 .. COLLATE latin1_bin > + CHAR(10) CHARACTER SET latin1 COLLATE latin1_bin .. COLLATE latin1_bin > + CHAR(10) COLLATE latin1_bin .. COLLATE latin1_bin > + CHAR(10) COLLATE latin1_bin .. COLLATE latin1_bin > + CHAR(10) CHARACTER SET latin1 BINARY .. COLLATE latin1_bin > */ > - DBUG_ASSERT(0); // Not possible yet > + if (m_with_collate && m_ci != cl.charset_collation()) > + { > + my_error(ER_CONFLICTING_DECLARATIONS, MYF(0), > + "COLLATE ", m_ci->coll_name.str, > + "COLLATE ", cl.charset_collation()->coll_name.str); > + return true; > + } > + if (!my_charset_same(m_ci, cl.charset_collation())) > + { > + my_error(ER_COLLATION_CHARSET_MISMATCH, MYF(0), > + cl.charset_collation()->coll_name.str, m_ci->cs_name.str); > + return true; > + } > + m_ci= cl.charset_collation(); > + m_with_collate= true; > return false; > - } > > - /* > - EXPLICIT + EXPLICIT > - CHAR(10) CHARACTER SET latin1 .. COLLATE latin1_bin > - CHAR(10) CHARACTER SET latin1 COLLATE latin1_bin .. COLLATE latin1_bin > - CHAR(10) COLLATE latin1_bin .. COLLATE latin1_bin > - CHAR(10) COLLATE latin1_bin .. COLLATE latin1_bin > - CHAR(10) CHARACTER SET latin1 BINARY .. COLLATE latin1_bin > - */ > - if (type() == TYPE_EXPLICIT && collation() != cl.collation()) > - { > - my_error(ER_CONFLICTING_DECLARATIONS, MYF(0), > - "COLLATE ", collation()->coll_name.str, > - "COLLATE ", cl.collation()->coll_name.str); > - return true; > - } > - if (!my_charset_same(collation(), cl.collation())) > - { > - my_error(ER_COLLATION_CHARSET_MISMATCH, MYF(0), > - cl.collation()->coll_name.str, collation()->cs_name.str); > - return true; > + case Lex_charset_collation_st::TYPE_COLLATE_CONTEXTUALLY_TYPED: > + if (cl.is_contextually_typed_collate_default()) > + { > + /* > + SET NAMES latin1 COLLATE DEFAULT; > + ALTER TABLE t1 CONVERT TO CHARACTER SET latin1 COLLATE DEFAULT; > + */ > + CHARSET_INFO *tmp= Charset_loader_mysys().find_default_collation(m_ci); > + if (!tmp) > + return true; > + m_ci= tmp; > + m_with_collate= true; > + return false; > + } > + else > + { > + /* > + EXPLICIT + CONTEXT > + CHAR(10) COLLATE latin1_bin .. COLLATE DEFAULT not possible yet > + CHAR(10) COLLATE latin1_bin .. COLLATE uca1400_as_ci > + */ > + > + const LEX_CSTRING context_cl_name= cl.collation_name_context_suffix(); > + if (!strncasecmp(context_cl_name.str, STRING_WITH_LEN("uca1400_"))) Like above, better DBUG_ASSERT(!strncasecmp(context_cl_name.str, STRING_WITH_LEN("uca1400_"))) > + { > + CHARSET_INFO *tmp; > + Charset_loader_mysys loader; > + if (!(tmp= loader.get_contextually_typed_collation_or_error(m_ci, > + context_cl_name.str))) > + return true; > + m_with_collate= true; > + m_ci= tmp; > + return false; > + } > + > + DBUG_ASSERT(0); // Not possible yet > + return false; > + } > } > - *this= cl; > + DBUG_ASSERT(0); > return false; > } > > diff --git a/strings/ctype-uca.c b/strings/ctype-uca.c > index b89916f3b20..3e6b4e4ce43 100644 > --- a/strings/ctype-uca.c > +++ b/strings/ctype-uca.c > @@ -30542,7 +30613,7 @@ static const char vietnamese[]= > Myanmar, according to CLDR Revision 8900. > http://unicode.org/cldr/trac/browser/trunk/common/collation/my.xml > */ > -static const char myanmar[]= "[shift-after-method expand][version 5.2.0]" > +static const char myanmar[]= "[shift-after-method expand]" What's going on with myanmar? You removed a version here and added &my_uca_v520 below in its charset_info_st. What does this change mean? > /* Tones */ > "&\\u108C" > "<\\u1037" > @@ -37627,7 +37825,7 @@ struct charset_info_st my_charset_utf32_myanmar_uca_ci= > NULL, /* to_lower */ > NULL, /* to_upper */ > NULL, /* sort_order */ > - NULL, /* uca */ > + &my_uca_v520, /* uca */ What does this change? > NULL, /* tab_to_uni */ > NULL, /* tab_from_uni */ > &my_unicase_unicode520,/* caseinfo */ Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 3

[Maria-developers] Review #3 for: MDEV-26278 Elimination: consider GROUP BY as unique key
by Sergey Petrunia 13 Jun '22

13 Jun '22

Hi Oleg, Please fix compile errors on this platform: https://buildbot.mariadb.org/#/builders/146/builds/13126/steps/5/logs/stdio also please apply the attached diff that fixes the coding style and wording of comments (input and/or adjustments are welcome). After this, the patch is ok to push. The next steps are to create a branch with name like preview-10.10-mdev26278-table-elimination and to officially submit it for preview tree build and testing. > commit 7275bf4939248959b5017314d6b0393296831c0c (HEAD, origin/bb-10.9-MDEV-26278-look) > Author: Oleg Smirnov <olernov(a)gmail.com> > Date: Thu Feb 17 22:53:37 2022 +0700 > > MDEV-26278 Add functionality to eliminate derived tables from the query > > Elimination of unnecessary tables from SQL queries is already present > in MariaDB. But it only works for regular tables and not for derived ones. > > Imagine we have a view: > CREATE VIEW v1 AS SELECT a, b, max(c) AS maxc FROM t1 GROUP BY a, b > > Due to "GROUP BY a, b" the values of combinations {a, b} are unique, > and this fact can be treated as like derived table "v1" has a unique key > on fields {a, b}. > > Suppose we have a SQL query: > SELECT t2.* FROM t2 LEFT JOIN v1 ON t2.a=v1.a and t2.b=v1.b > > 1. Since {v1.a, v1.b} is unique and both these fields are bound to t2, > "v1" is functionally dependent on t2. > This means every record of "t2" will be either joined with > a single record of "v1" or NULL-complemented. > 2. No fields of "v1" are present on the SELECT list > > These two facts allow the server to completely exclude (eliminate) > the derived table "v1" from the query. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

2 1

Re: [Maria-developers] 8c2c0e6079a: MDEV-27265 Improve contraction performance in UCA collations
by Sergei Golubchik 11 Jun '22

11 Jun '22

Hi, Alexander, Why haven't you used HASH? On Jun 11, Alexander Barkov wrote: > revision-id: 8c2c0e6079a (mariadb-10.9.1-13-g8c2c0e6079a) > parent(s): 27fa5c10eff > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2022-05-26 13:17:34 +0400 > message: > > MDEV-27265 Improve contraction performance in UCA collations > > Adding a hash table for contractions. > > The old code iterated through all items in MY_CONTRACTIONS, > and was much slower, especially for those contractions > in the end of the list. > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] MDEV-26278 Elimination: consider GROUP BY as unique key
by Sergey Petrunia 09 Jun '22

09 Jun '22

Hi Oleg, Please find the review below. > commit 20609d374b3a82c46291ee5e87cfadd26de47817 > Author: Oleg Smirnov <olernov(a)gmail.com> > Date: Thu Feb 17 22:53:37 2022 +0700 > > MDEV-26278 Elimination: consider GROUP BY as unique key Please provide a more verbose description. We should aim for it to provide sufficient info for one to understand what the optimization does and how that is achieved. Some general comments: The patch uses this formatting for multi-line comments: > + /* GROUP BY expression is considered as a synthetic "unique key" > + for the derived table. Add this key as a dependency */ While the coding style and actual MariaDB code would use this: /* GROUP BY expression is considered as a synthetic "unique key" for the derived table. Add this key as a dependency */ Please use this style for all comments. Also, the patch uses the term "synthetic unique key". I think the term is not very good: It can be confused with "synthetic primary key", also I read "synthetic" as something that actually exists, while in our case the key doesn't exist. Would using "Pseudo-key" be better? > diff --git a/sql/table.h b/sql/table.h > index 497502b2d06..fd598556cc3 100644 > --- a/sql/table.h > +++ b/sql/table.h > @@ -2872,6 +2872,9 @@ struct TABLE_LIST > } > } > > + void mark_derived_as_eliminated() { m_is_derived_eliminated= true; } > + bool is_derived_eliminated() const { return m_is_derived_eliminated; } > + So, TABLE_LIST objects are created non-eliminated and then they are eliminated, and that is forever. TABLE_LIST object has the same lifetime as the statement, that is, it will survive multiple executions of a prepared statement. On the other hand, Table Elimination optimization is done in eliminate_tables() call which is invoked for every statement execution. This looks like a dangerous mismatch. I'll continue below > private: > bool prep_check_option(THD *thd, uint8 check_opt_type); > bool prep_where(THD *thd, Item **conds, bool no_where_clause); > diff --git a/sql/sql_lex.cc b/sql/sql_lex.cc > index 3bf6d9abf57..02ef5cd7ad7 100644 > --- a/sql/sql_lex.cc > +++ b/sql/sql_lex.cc > @@ -11786,7 +11786,7 @@ bool SELECT_LEX_UNIT::explainable() const > EXPLAIN/ANALYZE unit, when: > (1) if it's a subquery - it's not part of eliminated WHERE/ON clause. > (2) if it's a CTE - it's not hanging (needed for execution) > - (3) if it's a derived - it's not merged > + (3) if it's a derived - it's not merged or eliminated > if it's not 1/2/3 - it's some weird internal thing, ignore it > */ > return item ? > @@ -11795,6 +11795,7 @@ bool SELECT_LEX_UNIT::explainable() const > derived && derived->derived_result && > !with_element->is_hanging_recursive(): // (2) > derived ? > - derived->is_materialized_derived() : // (3) > + derived->is_materialized_derived() && // (3) > + !derived->is_derived_eliminated() : > false; > } I think, there's no need for m_is_derived_eliminated. You can have this check in is_derived_eliminated(): this->derived->table->map & this->outer_select()->join->eliminated_tables Please try doing this. > diff --git a/mysql-test/main/table_elim.result b/mysql-test/main/table_elim.result > index deff0623370..842330f3713 100644 > --- a/mysql-test/main/table_elim.result > +++ b/mysql-test/main/table_elim.result > @@ -704,3 +704,252 @@ LIMIT 1; > PostID Voted > 1 NULL > DROP TABLE t1,t2; > +# > +# MDEV-26278: Table elimination does not work across derived tables > +# > +create table t1 (a int, b int); > +insert into t1 select seq, seq+10 from seq_1_to_10; > +create table t11 ( > +a int not null, > +b int, > +key(a) > +); > +insert into t11 select A.seq, A.seq+B.seq > +from > +seq_1_to_100 A, > +seq_1_to_1000 B; Do you really need 100K rows table? I think 10K rows will be just as good. > diff --git a/sql/opt_table_elimination.cc b/sql/opt_table_elimination.cc > index 8c4720bdec4..fe2155191ca 100644 > --- a/sql/opt_table_elimination.cc > +++ b/sql/opt_table_elimination.cc ... > @@ -278,6 +286,8 @@ class Dep_value_field : public Dep_value > Dep_module_key *key_dep; > /* Otherwise, this and advance */ > uint equality_no; > + > + Dep_module_synthetic_key *synth_key_dep; Please add a comment for the above that we should try that one, too. @@ -447,6 +463,55 @@ const size_t Dep_module::iterator_size= > MY_MAX(Dep_module_expr::iterator_size, Dep_module_key::iterator_size); > > > +/* A synthetic unique key module for a derived table. > + For example, a derived table > + SELECT t11.a, count(*) from t1 LEFT JOIN t2 ON t1.id = t2.fk GROUP BY t11.a > + has unique values in its first field (t11.a) due to GROUP BY expression > + so this can be considered as a unique key for this derived table > +*/ > + > +class Dep_module_synthetic_key : public Dep_module > +{ > +public: > + Dep_module_synthetic_key(Dep_value_table *table_arg, > + std::vector<field_index_t>&& field_indexes) > + : table(table_arg), derived_table_field_indexes(field_indexes) > + { > + unbound_args= static_cast<uint>(field_indexes.size()); > + } > + > + Dep_value_table * table; Coding style: "*table". ... > @@ -1603,7 +1674,73 @@ Dep_value_table *Dep_analysis_context::create_table_value(TABLE *table) > key_list= &(key_dep->next_table_key); > } > } > - return table_deps[table->tablenr]= tbl_dep; > + > + auto select_unit= table_list->get_unit(); > + SELECT_LEX *first_select= nullptr; > + if (select_unit) Please move the code that creates the pseudo-key into a separate function. ... > + > +int Dep_analysis_context::find_field_in_list(List<Item> &fields_list, > + Item *field) Please add a one-line comment describing what function does. > @@ -1786,6 +1941,18 @@ Dep_module* Dep_value_field::get_next_unbound_module(Dep_analysis_context *dac, > } > else > di->key_dep= NULL; > + > + Dep_module_synthetic_key *synth_key_dep= di->synth_key_dep; > + if (synth_key_dep && !synth_key_dep->is_applicable() && > + std::find(synth_key_dep->derived_table_field_indexes.begin(), > + synth_key_dep->derived_table_field_indexes.end(), > + field->field_index) != std::end(synth_key_dep->derived_table_field_indexes)) Please introduce a method like bool Dep_module_synthetic_key::covers_field(int field_index); Will this also allow to make Dep_module_synthetic_key::derived_table_field_indexes a private member? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

1 1

[Maria-developers] Introduction of Weijun
by Weijun Huang 07 Jun '22

07 Jun '22

Hi everyone! My name is Weijun Huang and I was selected for GSoC to work with MariaDB. I'm from China and I am studying at Technical University of Denmark, majoring in Human-centered Artificial Intelligence. My nickname on Zulip is Weijun Huang, and on Github it's Weijun-H. My project for GSoC in 2022 is called "Add Regression Functions for MariaDB." I've already started to work with MariaDB Server. Now I'm doing the implementation of REGR_SXX, REGR_SYY, and REGR_SXY I'm looking forward to working on this project. It would be an excellent experience for me! Best regard Weijun Huang

1 0

Re: [Maria-developers] 2fdcfaaba09: MDEV-27832 disable binary logging for SQL SERVICE.
by Sergei Golubchik 06 May '22

06 May '22

Hi, Alexey, Looks good. Minor comments below. Ok to push after addressing them. On May 06, Alexey Botchkov wrote: > revision-id: 2fdcfaaba09 (mariadb-10.7.2-8-g2fdcfaaba09) > parent(s): 33fd136c61b > author: Alexey Botchkov > committer: Alexey Botchkov > timestamp: 2022-05-05 15:03:38 +0400 > message: > > MDEV-27832 disable binary logging for SQL SERVICE. > > Binary logging is now disabled for the queries run by SQL SERVICE. > The binlogging can be turned on with the 'SET SQL_LOG_BIN=On' query. > > diff --git a/mysql-test/suite/plugins/t/test_sql_service.test b/mysql-test/suite/plugins/t/test_sql_service.test > index b80d78fe6e5..0fd86fccd1a 100644 > --- a/mysql-test/suite/plugins/t/test_sql_service.test > +++ b/mysql-test/suite/plugins/t/test_sql_service.test > @@ -58,3 +73,7 @@ drop table t1; > > uninstall plugin test_sql_service; > > +--replace_column 2 # 5 # > +--replace_regex /xid=[0-9]+/xid=XX/ /GTID [0-9]+-[0-9]+-[0-9]+/GTID #-#-#/ > +SHOW BINLOG EVENTS LIMIT 3,100; use --source include/show_binlog_events.inc It does all the replaces, provides a unified output everywhere, it'll be updated if show binlog event output changes. No need to duplicate that. > + > diff --git a/sql/sql_prepare.cc b/sql/sql_prepare.cc > index e025147c71e..d3b77fbece3 100644 > --- a/sql/sql_prepare.cc > +++ b/sql/sql_prepare.cc > @@ -5617,12 +5617,26 @@ class Protocol_local : public Protocol_text > THD *new_thd; > Security_context empty_ctx; > > + my_bool do_log_bin; > + > Protocol_local(THD *thd_arg, THD *new_thd_arg, ulong prealloc) : > Protocol_text(thd_arg, prealloc), > cur_data(0), first_data(0), data_tail(&first_data), alloc(0), > - new_thd(new_thd_arg) > + new_thd(new_thd_arg), do_log_bin(FALSE) > {} > > + void set_binlog_vars(my_bool *sav_log_bin) > + { > + *sav_log_bin= thd->variables.sql_log_bin; > + thd->variables.sql_log_bin= do_log_bin; > + thd->set_binlog_bit(); > + } > + void restore_binlog_vars(my_bool &sav_log_bin) this doesn't need to be a reference, just a normal pass-by-value will do. > + { > + do_log_bin= thd->variables.sql_log_bin; > + thd->variables.sql_log_bin= sav_log_bin; > + thd->set_binlog_bit(); > + } > protected: > bool net_store_data(const uchar *from, size_t length); > bool net_store_data_cs(const uchar *from, size_t length, Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 9d6be81c0da: MDEV-27832 disable binary logging for SQL SERVICE.
by Sergei Golubchik 05 May '22

05 May '22

Hi, Alexey, On Mar 02, Alexey Botchkov wrote: > revision-id: 9d6be81c0da (mariadb-10.7.2-8-g9d6be81c0da) > parent(s): 33fd136c61b > author: Alexey Botchkov > committer: Alexey Botchkov > timestamp: 2022-02-15 13:37:59 +0400 > message: > > MDEV-27832 disable binary logging for SQL SERVICE. > > Binary logging is now disabled for the queries run by SQL SERVICE. > The binlogging can be turned on with the 'SET SQL_LOG_BIN=On' query. > > diff --git a/mysql-test/suite/plugins/r/test_sql_service.result b/mysql-test/suite/plugins/r/test_sql_service.result > index 00f0411b665..943bb239e5f 100644 > --- a/mysql-test/suite/plugins/r/test_sql_service.result > +++ b/mysql-test/suite/plugins/r/test_sql_service.result > @@ -2,6 +2,19 @@ install plugin test_sql_service soname 'test_sql_service'; > show status like 'test_sql_service_passed'; > Variable_name Value > Test_sql_service_passed 1 > +set global test_sql_service_execute_sql_global= 'create table test.t1 select 1 as a, @@SQL_LOG_BIN'; > +set global test_sql_service_execute_sql_local= 'insert into test.t1 select 2 as a, @@SQL_LOG_BIN'; > +set global test_sql_service_execute_sql_global= 'SET SQL_LOG_BIN=1'; > +set global test_sql_service_execute_sql_global= 'insert into test.t1 select 3 as a, @@SQL_LOG_BIN'; > +set global test_sql_service_execute_sql_global= 'SET SQL_LOG_BIN=0'; > +set global test_sql_service_execute_sql_global= 'insert into test.t1 select 4 as a, @@SQL_LOG_BIN'; > +select * from t1 order by a; this is not a test. Please, enable binlog and see whether queries are actually logged. Also, add a test for, say, sql_auto_is_null: set global test_sql_service_execute_sql_global='set sql_auto_is_null=1'; set global test_sql_service_execute_sql_global='insert test.t1 select 5, @@sql_auto_is_null'; select * from t1 where a=5; > +a @@SQL_LOG_BIN > +1 0 > +2 0 > +3 1 > +4 0 > +drop table t1; > set global test_sql_service_run_test= 1; > show status like 'test_sql_service_passed'; > Variable_name Value > diff --git a/sql/sql_prepare.cc b/sql/sql_prepare.cc > index e025147c71e..48681ac0b6f 100644 > --- a/sql/sql_prepare.cc > +++ b/sql/sql_prepare.cc > @@ -5617,12 +5617,30 @@ class Protocol_local : public Protocol_text > THD *new_thd; > Security_context empty_ctx; > > + my_bool do_log_bin; > + > Protocol_local(THD *thd_arg, THD *new_thd_arg, ulong prealloc) : > Protocol_text(thd_arg, prealloc), > cur_data(0), first_data(0), data_tail(&first_data), alloc(0), > - new_thd(new_thd_arg) > + new_thd(new_thd_arg), do_log_bin(FALSE) > {} > > + void set_binlog_vars(my_bool *sav_log_bin, ulonglong *sav_bits) > + { > + *sav_log_bin= thd->variables.sql_log_bin; > + *sav_bits= thd->variables.option_bits; > + > + if ((thd->variables.sql_log_bin= do_log_bin)) > + thd->variables.option_bits|= OPTION_BIN_LOG; > + else > + thd->variables.option_bits&= ~OPTION_BIN_LOG; > + } > + void restore_binlog_vars(my_bool &sav_log_bin, ulonglong &sav_bits) > + { > + do_log_bin= thd->variables.sql_log_bin; > + thd->variables.sql_log_bin= sav_log_bin; > + thd->variables.option_bits= sav_bits; that's what sql_auto_is_null test is for.You destroy all other option_bits here. Better don't save/restore option_bits, but use the same if() as in set_binlog_vars(). Or, even better, reuse fix_sql_log_bin_after_update(). I'd suggest to make it a method in THD > + } > protected: > bool net_store_data(const uchar *from, size_t length); > bool net_store_data_cs(const uchar *from, size_t length, > @@ -6230,12 +6248,17 @@ loc_advanced_command(MYSQL *mysql, enum enum_server_command command, > Ed_connection con(p->thd); > Security_context *ctx_orig= p->thd->security_ctx; > MYSQL_LEX_STRING sql_text; > + my_bool log_bin_orig; > + ulonglong option_bits_orig; > + p->set_binlog_vars(&log_bin_orig, &option_bits_orig); > + > DBUG_ASSERT(current_thd == p->thd); > sql_text.str= (char *) arg; > sql_text.length= arg_length; > p->thd->security_ctx= &p->empty_ctx; > result= con.execute_direct(p, sql_text); > p->thd->security_ctx= ctx_orig; > + p->restore_binlog_vars(log_bin_orig, option_bits_orig); > } > if (skip_check) > result= 0; > @@ -6391,6 +6414,9 @@ extern "C" MYSQL *mysql_real_connect_local(MYSQL *mysql) > new_thd->security_ctx->skip_grants(); > new_thd->query_cache_is_applicable= 0; > new_thd->variables.wsrep_on= 0; > + new_thd->variables.sql_log_bin= 0; > + new_thd->variables.option_bits&= ~OPTION_BIN_LOG; > + > /* > TOSO: decide if we should turn the auditing off > for such threads. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] 4ba5b444b4b: MDEV-28437: Assertion `!eliminated' failed in Item_subselect::exec
by Sergei Golubchik 04 May '22

04 May '22

Hi, Sergei, On May 04, Sergei Petrunia wrote: > revision-id: 4ba5b444b4b (mariadb-10.2.43-97-g4ba5b444b4b) > parent(s): 70555454b4c > author: Sergei Petrunia > committer: Sergei Petrunia > timestamp: 2022-05-04 12:24:48 +0300 > message: > > MDEV-28437: Assertion `!eliminated' failed in Item_subselect::exec > > (This is the assert that was added in fix for MDEV-26047) > > Table elimination may remove an ON expression from an outer join. > However SELECT_LEX::update_used_tables() will still call > > item->walk(&Item::eval_not_null_tables) > > for eliminated expressions. If the subquery is constant and cheap > Item_cond_and will attempt to evaluate it, which will trigger an > assert. > The fix is not to call update_used_tables() or eval_not_null_tables() > for ON expressions that were eliminated. > > diff --git a/sql/sql_lex.cc b/sql/sql_lex.cc > index 22ee8801e3a..8e718f2a942 100644 > --- a/sql/sql_lex.cc > +++ b/sql/sql_lex.cc > @@ -4229,7 +4229,7 @@ void SELECT_LEX::update_used_tables() > } > } > while ((embedding= embedding->embedding)); > - if (tl->on_expr) > + if (tl->on_expr && !is_eliminated_table(join->eliminated_tables, tl)) > { > tl->on_expr->update_used_tables(); > tl->on_expr->walk(&Item::eval_not_null_tables, 0, NULL); > @@ -4253,8 +4253,11 @@ void SELECT_LEX::update_used_tables() > if (embedding->on_expr && > embedding->nested_join->join_list.head() == tl) > { can embedding be eliminated? if yes - does it make sense to do anything in tl's below that eliminated embedding? Perhaps it should be (in the do..while above), like do { bool maybe_null; if ((maybe_null= MY_TEST(embedding->outer_join))) { tl->table->maybe_null= maybe_null; break; } + if (is_eliminated_table(join->eliminated_tables, embedding)) + tl= embedding->embedding; } while ((embedding= embedding->embedding)); > + if (!is_eliminated_table(join->eliminated_tables, embedding)) > + { > embedding->on_expr->update_used_tables(); > embedding->on_expr->walk(&Item::eval_not_null_tables, 0, NULL); > + } > } > tl= embedding; > embedding= tl->embedding; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 85cc56875e9: MDEV-19398: Assertion `item1->type() == Item::FIELD_ITEM ...
by Sergei Golubchik 03 May '22

03 May '22

Hi, Sergei, The solutions looks fine Two comments below, about the implementation. ok to push after addressing them On May 03, Sergei Petrunia wrote: > revision-id: 85cc56875e9 (mariadb-10.2.43-75-g85cc56875e9) > parent(s): 3b6c04f44c4 > author: Sergei Petrunia > committer: Sergei Petrunia > timestamp: 2022-05-03 14:06:27 +0300 > message: > > MDEV-19398: Assertion `item1->type() == Item::FIELD_ITEM ... > > Window Functions code tries to minimize the number of times it > needs to sort the select's resultset by finding "compatible" > OVER (PARTITION BY ... ORDER BY ...) clauses. > > This employs compare_order_elements(). That function assumed that > the order expressions are Item_field-derived objects (that refer > to a temp.table). But this is not always the case: one can > construct queries order expressions are arbitrary item expressions. > > Add handling for such expressions: sort them according to the window > specification they appeared in. > This means we cannot detect that two compatible PARTITION BY clauses > that use expressions can share the sorting step. > But at least we won't crash. > can share the same sort > > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > index 989ca0c8803..457849a7569 100644 > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -8624,6 +8624,7 @@ bool st_select_lex::add_window_def(THD *thd, > fields_in_window_functions+= win_part_list_ptr->elements + > win_order_list_ptr->elements; > } > + win_def->win_spec_number= window_specs.elements; > return (win_def == NULL || window_specs.push_back(win_def)); > } > > @@ -8651,6 +8652,7 @@ bool st_select_lex::add_window_spec(THD *thd, > win_order_list_ptr->elements; > } > thd->lex->win_spec= win_spec; > + win_spec->win_spec_number= window_specs.elements; > return (win_spec == NULL || window_specs.push_back(win_spec)); > } > > diff --git a/sql/sql_window.cc b/sql/sql_window.cc > index 3ef751bc5b9..fbf7aca0197 100644 > --- a/sql/sql_window.cc > +++ b/sql/sql_window.cc > @@ -312,15 +312,44 @@ setup_windows(THD *thd, Ref_ptr_array ref_pointer_array, TABLE_LIST *tables, > #define CMP_GT 2 // Greater then > > static > -int compare_order_elements(ORDER *ord1, ORDER *ord2) > +int compare_order_elements(ORDER *ord1, int weight1, > + ORDER *ord2, int weight2) > { > if (*ord1->item == *ord2->item && ord1->direction == ord2->direction) > return CMP_EQ; > Item *item1= (*ord1->item)->real_item(); > Item *item2= (*ord2->item)->real_item(); > - DBUG_ASSERT(item1->type() == Item::FIELD_ITEM && > - item2->type() == Item::FIELD_ITEM); > - ptrdiff_t cmp= ((Item_field *) item1)->field - ((Item_field *) item2)->field; > + > + bool item1_field= (item1->type() == Item::FIELD_ITEM); > + bool item2_field= (item2->type() == Item::FIELD_ITEM); > + > + ptrdiff_t cmp; > + if (item1_field && item2_field) > + cmp= ((Item_field *) item1)->field - ((Item_field *) item2)->field; 1. why is this order stable? I'd rather sort by item->field->field_index 2. please, add an assert, that item1->field->table == item2->field->table > + else if (item1_field && !item2_field) > + return CMP_LT; > + else if (!item1_field && item2_field) > + return CMP_LT; > + else > + { > + /* > + Ok, item1_field==NULL and item2_field==NULL. > + We're not able to compare Item expressions. Order them according to > + their passed "weight" (which comes from Window_spec::win_spec_number): > + */ > + if (weight1 != weight2) > + cmp= weight1 - weight2; > + else > + { > + /* > + The weight is the same. That is, the elements come from the same > + window specification... This shouldn't happen. > + */ > + DBUG_ASSERT(0); > + cmp= item1 - item2; > + } > + } > + > if (cmp == 0) > { Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 1c0731e0544: MDEV-26695: Number of an invalid row is not calculated for table value constructor
by Sergei Golubchik 26 Apr '22

26 Apr '22

Hi, Rucha, Looks good. please also check that row_number values match what's in a warning, and then ok to push! On Apr 26, Rucha Deodhar wrote: > On Tue, Apr 26, 2022 at 12:41 AM Sergei Golubchik <serg(a)mariadb.org> wrote: > > what if you combine it with union? > > create table t (a char(1)) values ('a'),('b') union values ('foo'); > > > > please, add a test for that. > > > > > It gives 3 as row the number (in warning). Added a test for it. > Please find it here: 1c0731e0544db4d2f5c107473776bbd6d4d3e2f0 > > > On Apr 25, Rucha Deodhar wrote: > > > revision-id: 1c0731e0544 (mariadb-10.7.1-33-g1c0731e0544) > > > parent(s): 5e6fd4e8043 > > > author: Rucha Deodhar > > > committer: Rucha Deodhar > > > timestamp: 2022-01-23 13:48:18 +0530 > > > message: > > > > > > MDEV-26695: Number of an invalid row is not calculated for table value > > > constructor > > > > > > Analysis: counter does not increment while sending rows for table value > > > constructor and so row_number assumes the default value (0 in this case). > > > Fix: Increment the counter to avoid counter using default value. > > > > > > diff --git a/sql/sql_tvc.cc b/sql/sql_tvc.cc > > > index bdaf6829fbd..7db0ab217ca 100644 > > > --- a/sql/sql_tvc.cc > > > +++ b/sql/sql_tvc.cc > > > @@ -422,7 +422,9 @@ bool table_value_constr::exec(SELECT_LEX *sl) > > > DBUG_ENTER("table_value_constr::exec"); > > > List_iterator_fast<List_item> li(lists_of_values); > > > List_item *elem; > > > + THD *cur_thd= sl->parent_lex->thd; > > > ha_rows send_records= 0; > > > + int rc=0; > > > > > > if (select_options & SELECT_DESCRIBE) > > > DBUG_RETURN(false); > > > @@ -438,10 +440,10 @@ bool table_value_constr::exec(SELECT_LEX *sl) > > > > > > while ((elem= li++)) > > > { > > > + cur_thd->get_stmt_da()->inc_current_row_for_warning(); > > > if (send_records >= sl->master_unit()->lim.get_select_limit()) > > > break; > > > - int rc= > > > - result->send_data_with_check(*elem, sl->master_unit(), > > send_records); > > > + rc= result->send_data_with_check(*elem, sl->master_unit(), > > send_records); > > > > you don't need to do that anymore (it was needed in your first patch, > > but not here) > > > > Removed. > > > > if (!rc) > > > send_records++; > > > else if (rc > 0) > > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 1c0731e0544: MDEV-26695: Number of an invalid row is not calculated for table value constructor
by Sergei Golubchik 25 Apr '22

25 Apr '22

Hi, Rucha, what if you combine it with union? create table t (a char(1)) values ('a'),('b') union values ('foo'); please, add a test for that. On Apr 25, Rucha Deodhar wrote: > revision-id: 1c0731e0544 (mariadb-10.7.1-33-g1c0731e0544) > parent(s): 5e6fd4e8043 > author: Rucha Deodhar > committer: Rucha Deodhar > timestamp: 2022-01-23 13:48:18 +0530 > message: > > MDEV-26695: Number of an invalid row is not calculated for table value > constructor > > Analysis: counter does not increment while sending rows for table value > constructor and so row_number assumes the default value (0 in this case). > Fix: Increment the counter to avoid counter using default value. > > diff --git a/sql/sql_tvc.cc b/sql/sql_tvc.cc > index bdaf6829fbd..7db0ab217ca 100644 > --- a/sql/sql_tvc.cc > +++ b/sql/sql_tvc.cc > @@ -422,7 +422,9 @@ bool table_value_constr::exec(SELECT_LEX *sl) > DBUG_ENTER("table_value_constr::exec"); > List_iterator_fast<List_item> li(lists_of_values); > List_item *elem; > + THD *cur_thd= sl->parent_lex->thd; > ha_rows send_records= 0; > + int rc=0; > > if (select_options & SELECT_DESCRIBE) > DBUG_RETURN(false); > @@ -438,10 +440,10 @@ bool table_value_constr::exec(SELECT_LEX *sl) > > while ((elem= li++)) > { > + cur_thd->get_stmt_da()->inc_current_row_for_warning(); > if (send_records >= sl->master_unit()->lim.get_select_limit()) > break; > - int rc= > - result->send_data_with_check(*elem, sl->master_unit(), send_records); > + rc= result->send_data_with_check(*elem, sl->master_unit(), send_records); you don't need to do that anymore (it was needed in your first patch, but not here) > if (!rc) > send_records++; > else if (rc > 0) Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 585cd1f52e7: MDEV-26875: Wrong user in SET DEFAULT ROLE error
by Sergei Golubchik 25 Apr '22

25 Apr '22

Hi, Anel, On Apr 25, Anel Husakovic wrote: > revision-id: 585cd1f52e7 (mariadb-10.5.14-10-g585cd1f52e7) > parent(s): 52b32c60c26 > author: Anel Husakovic > committer: Anel Husakovic > timestamp: 2022-02-14 13:59:24 +0100 > message: > > MDEV-26875: Wrong user in SET DEFAULT ROLE error > > - Caused by 7c02e8717de5, where 957cb7b7ba35 introduced the bug. > > Reviewed by: > > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index c4e66cf5d73..e83bc5635dc 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -3277,10 +3277,14 @@ static int check_user_can_set_role(THD *thd, const char *user, > check_role_is_granted_callback, > NULL) == -1)) > { > - /* Role is not granted but current user can see the role */ > - my_printf_error(ER_INVALID_ROLE, "User %`s@%`s has not been granted role %`s", > - MYF(0), thd->security_ctx->priv_user, > - thd->security_ctx->priv_host, rolename); > + /* If the SET ROLE is applied on the anonymous user, host is null */ > + if (!host) > + my_printf_error(ER_INVALID_ROLE, "User %`s@%`s has not been granted role %`s", > + MYF(0), thd->security_ctx->priv_user, thd->security_ctx->priv_host, rolename); I don't understand it. 1) what does it mean? why for anonymous user you print priv_user@priv_host ? 2) do you have any tests for that? There are none in the commit > + else > + /* Role is not granted but current user can see the role */ > + my_printf_error(ER_INVALID_ROLE, "User %`s@%`s has not been granted role %`s", > + MYF(0), user, host, rolename); > } > else > { Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 5f8ffc7271a: MDEV-20207: Assertion `! is_set()' failed in Diagnostics_area::set_eof_status upon HANDLER READ
by Sergei Golubchik 25 Apr '22

25 Apr '22

Hi, Rucha, The fix is ok to push. But as a general comment, better try to define variables in the smallest scope where they're used. In particular here it could've been { MY_BITMAP *old_map= dbug_tmp_use_all_columns(table, &table->write_set); - (void) item->save_in_field(key_part->field, 1); + int res= item->save_in_field(key_part->field, 1); because you don't need `res` outside of that scope. Makes the code easier to read. Of course, there can be exceptions. For example, if the variable is an object and its initialization is very expensive. But whether you move `int res` declaration inside or not - it's ok to push, no need to ask for another review. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org On Apr 25, Rucha Deodhar wrote: > revision-id: 5f8ffc7271a (mariadb-10.2.40-251-g5f8ffc7271a) > parent(s): 2cbf92522b5 > author: Rucha Deodhar > committer: Rucha Deodhar > timestamp: 2022-01-27 01:18:19 +0530 > message: > > MDEV-20207: Assertion `! is_set()' failed in Diagnostics_area::set_eof_status > upon HANDLER READ > > Analysis: The error state is not stored while checking condition and key name. > Fix: Return true while checking condition and key name if error is reported > because geometry object can't be created from the data in the index value for > HANDLER READ. > > diff --git a/sql/sql_handler.cc b/sql/sql_handler.cc > index ec3756eceba..7c002a68d9e 100644 > --- a/sql/sql_handler.cc > +++ b/sql/sql_handler.cc > @@ -554,6 +554,7 @@ mysql_ha_fix_cond_and_key(SQL_HANDLER *handler, > { > THD *thd= handler->thd; > TABLE *table= handler->table; > + int res= 0; > if (cond) > { > /* This can only be true for temp tables */ > @@ -613,8 +614,10 @@ mysql_ha_fix_cond_and_key(SQL_HANDLER *handler, > if (!in_prepare) > { > MY_BITMAP *old_map= dbug_tmp_use_all_columns(table, &table->write_set); > - (void) item->save_in_field(key_part->field, 1); > + res= item->save_in_field(key_part->field, 1); > dbug_tmp_restore_column_map(&table->write_set, old_map); > + if (res) > + return 1; > } > key_len+= key_part->store_length; > keypart_map= (keypart_map << 1) | 1;

1 0

Re: [Maria-developers] 14cc679c95e: MDEV-27831 Let the SQL SERVICE user set the current user name.
by Sergei Golubchik 22 Apr '22

22 Apr '22

Hi, Alexey, On Mar 04, Alexey Botchkov wrote: > revision-id: 14cc679c95e (mariadb-10.7.2-8-g14cc679c95e) > parent(s): 33fd136c61b > author: Alexey Botchkov > committer: Alexey Botchkov > timestamp: 2022-02-15 11:35:18 +0400 > message: > > MDEV-27831 Let the SQL SERVICE user set the current user name. > > The 'user' argument added to the mysql_real_connect_local. I think this is wrong on many levels. A plugin name is already known inside the plugin, you shouldn't force the plugin to pass its own name as an argument in multiple places, the server should determine it automatically. You should not set current_user to an arbitrary string for audit plugin to see it. current_user is the name of the user account and it's used in many places as such. Try, for example, to create a view or a stored procedure. Who will be a definer? Setting only user() might be ok. Setting @@proxy_user or @@external_user is even better, if your audit plugin can show them. @@external_user would be the best, I think it's purely informational. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 3

Re: [Maria-developers] 96b40a5e2f5: MDEV-27744 InnoDB: Failing assertion: !cursor->index->is_committed() in row0ins.cc (from row_ins_sec_index_entry_by_modify) | Assertion `0' failed in row_upd_sec_index_entry (debug) | Corruption
by Sergei Golubchik 22 Apr '22

22 Apr '22

Hi, Alexander, Some comments/questions here. I'll need to look at the patch again after I understand why you did it this way better. On Apr 18, Alexander Barkov wrote: > revision-id: 96b40a5e2f5 (mariadb-10.3.33-61-g96b40a5e2f5) > parent(s): 7355f7b1f5c > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2022-04-07 16:22:17 +0400 > message: > > MDEV-27744 InnoDB: Failing assertion: !cursor->index->is_committed() in row0ins.cc (from row_ins_sec_index_entry_by_modify) | Assertion `0' failed in row_upd_sec_index_entry (debug) | Corruption A detailed comment here is a must. What was the problem, how it was fixed. > diff --git a/sql/item_func.h b/sql/item_func.h > index 754b1cd1eb2..d844e45280a 100644 > --- a/sql/item_func.h > +++ b/sql/item_func.h > @@ -35,6 +35,36 @@ extern "C" /* Bug in BSDI include file */ > #include <cmath> > > > +template<class FUNC> > +FUNC* item_func_create_2_3_args(THD *thd, const LEX_CSTRING &name, > + List<Item> *item_list) > +{ > + int arg_count= item_list ? item_list->elements : 0; > + > + switch (arg_count) { > + case 2: > + { > + Item *param_1= item_list->pop(); > + Item *param_2= item_list->pop(); > + return new (get_thd_memroot(thd)) FUNC(thd, param_1, param_2); > + } > + case 3: > + { > + Item *param_1= item_list->pop(); > + Item *param_2= item_list->pop(); > + Item *param_3= item_list->pop(); > + return new (get_thd_memroot(thd)) FUNC(thd, param_1, param_2, param_3); A constructor that takes a List<Item> wouldn't need a switch. And wouldn't need a template either, your create_check_args_ge() isn't a template. > + break; > + } > + default: > + my_error(ER_WRONG_PARAMCOUNT_TO_NATIVE_FCT, MYF(0), name.str); > + break; > + } > + > + return NULL; > +} > + > + > class Item_func :public Item_func_or_sum > { > void sync_with_sum_func_and_with_field(List<Item> &list); > @@ -56,8 +86,41 @@ class Item_func :public Item_func_or_sum > bool check_argument_types_can_return_text(uint start, uint end) const; > bool check_argument_types_can_return_date(uint start, uint end) const; > bool check_argument_types_can_return_time(uint start, uint end) const; > + > + void print_schema_name_if_needed(String *to) const > + { > + const LEX_CSTRING schema_name= schema_name_cstring(); > + if (schema_name.length) > + { > + to->append(schema_name); > + to->append('.'); > + } this should check that schema_name isn't the same as the schema in the path (implicit path, assumed by the current sql_mode). > + } > + void print_maybe_qualified_name(String *to) const > + { > + print_schema_name_if_needed(to); > + to->append(func_name()); > + } > + > public: > > + // Print an error message for a builtin-schema qualified function call > + static void wrong_param_count_error(const LEX_CSTRING &schema_name, > + const LEX_CSTRING &func_name); > + > + // Check that the number of arguments is greater or equal to "expected" > + static bool create_check_args_ge(const LEX_CSTRING &name, > + const List<Item> *item_list, > + uint expected) > + { > + if (!item_list || item_list->elements < expected) > + { > + my_error(ER_WRONG_PARAMCOUNT_TO_NATIVE_FCT, MYF(0), name.str); what's the point of your wrong_param_count_error() if you aren't using it? > + return true; > + } > + return false; > + } > + > table_map not_null_tables_cache; > > enum Functype { UNKNOWN_FUNC,EQ_FUNC,EQUAL_FUNC,NE_FUNC,LT_FUNC,LE_FUNC, > diff --git a/sql/item_strfunc.cc b/sql/item_strfunc.cc > index 75abd9906cd..e45fe6ef6d7 100644 > --- a/sql/item_strfunc.cc > +++ b/sql/item_strfunc.cc > @@ -3204,6 +3239,39 @@ static String *default_pad_str(String *pad_str, CHARSET_INFO *collation) > return pad_str; > } > > + > +Item_func_lpad* > +Item_func_lpad::create(THD *thd, const LEX_CSTRING &name, > + List<Item> *item_list) > +{ > + return item_func_create_2_3_args<Item_func_lpad>(thd, name, item_list); may be item_func_create_2_3_args should be in Item_func? Like create_check_args_ge() is? and may be create_check_args_ge() should be a template too? > +} > + > + > +Item_func_lpad_oracle* > +Item_func_lpad_oracle::create(THD *thd, const LEX_CSTRING &name, > + List<Item> *item_list) > +{ > + return item_func_create_2_3_args<Item_func_lpad_oracle>(thd, name, item_list); > +} > + > + > +Item_func_rpad* > +Item_func_rpad::create(THD *thd, const LEX_CSTRING &name, > + List<Item> *item_list) > +{ > + return item_func_create_2_3_args<Item_func_rpad>(thd, name, item_list); > +} > + > + > +Item_func_rpad_oracle* > +Item_func_rpad_oracle::create(THD *thd, const LEX_CSTRING &name, > + List<Item> *item_list) > +{ > + return item_func_create_2_3_args<Item_func_rpad_oracle>(thd, name, item_list); > +} > + > + > bool Item_func_pad::fix_length_and_dec() > { > if (arg_count == 3) > diff --git a/sql/sql_lex.cc b/sql/sql_lex.cc > index 7f1b362d91d..6b0a351f76e 100644 > --- a/sql/sql_lex.cc > +++ b/sql/sql_lex.cc > @@ -1542,7 +1541,18 @@ int Lex_input_stream::lex_one_token(YYSTYPE *yylval, THD *thd) > if (lex->parsing_options.lookup_keywords_after_qualifier) > next_state= MY_LEX_IDENT_OR_KEYWORD; > else > - next_state= MY_LEX_IDENT_START; // Next is ident (not keyword) > + { > + /* > + Next is: > + - A qualified func with a special syntax: > + mariadb_schema.REPLACE('a','b','c') > + mariadb_schema.SUSTRING('a',1,2) > + mariadb_schema.TRIM('a') > + - Or an identifier otherwise. No keyword lookup is done, > + all keywords are treated as identifiers. > + */ > + next_state= MY_LEX_IDENT_OR_QUALIFIED_SPECIAL_FUNC; I don't understand why you did it in the lexer and not in the parser, like - | REPLACE '(' expr ',' expr ',' expr ')' + | opt_schema REPLACE '(' expr ',' expr ',' expr ')' or (if the above won't work in LARL(1) parser) - | ident_cli '.' ident_cli '(' opt_expr_list ')' + | ident_cli '.' func_2nd_part with + func_2nd_part: ident_cli '(' opt_expr_list ')' + | REPLACE '(' expr ',' expr ',' expr ')' etc (not reviewing lexer changes anymore, until I understand the above) > + } > if (!ident_map[(uchar) yyPeek()]) // Probably ` or " > next_state= MY_LEX_START; > return((int) c); Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 4

Re: [Maria-developers] 17548c8a8b6: MDEV-25477 Auto-create breaks replication when triggering event was not replicated
by Sergei Golubchik 20 Apr '22

20 Apr '22

Hi, Aleksey, Basically, it's fine, but I still didn't like that there's a new flag in THD to mark a statement that has to be logged. So I tried to change to log_current_statement. I know you tried that and you told me what tests fails, rpl.create_or_replace_*. I debugged that a bit, and I think the problem is that select_create inherits from select_insert, so CREATE ... SELECT runs your code that is only supposed to be run for INSERT ODKU or REPLACE. This change makes all tests pass (after removing vers_created_partitions): @@ -4225,7 +4225,8 @@ bool select_insert::prepare_eof() thd->clear_error(); else errcode= query_error_code(thd, killed_status == NOT_KILLED); - StatementBinlog stmt_binlog(thd, thd->binlog_need_stmt_format(trans_table)); + StatementBinlog stmt_binlog(thd, !can_rollback_data() && + thd->binlog_need_stmt_format(trans_table)); res= thd->binlog_query(THD::ROW_QUERY_TYPE, thd->query(), thd->query_length(), trans_table, FALSE, FALSE, errcode); On Apr 10, Aleksey Midenkov wrote: > revision-id: 17548c8a8b6 (mariadb-10.6.1-250-g17548c8a8b6) > parent(s): 6282e025a0e > author: Aleksey Midenkov > committer: Aleksey Midenkov > timestamp: 2022-03-07 23:49:35 +0300 > message: > > MDEV-25477 Auto-create breaks replication when triggering event was not replicated > > diff --git a/sql/sql_class.h b/sql/sql_class.h > index 686e6e70766..4fed1bf7355 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -2914,6 +2914,13 @@ class THD: public THD_count, /* this must be first */ > int binlog_flush_pending_rows_event(bool stmt_end, bool is_transactional); > int binlog_remove_pending_rows_event(bool clear_maps, bool is_transactional); > > + bool binlog_need_stmt_format(bool is_transactional) const > + { > + return vers_created_partitions && !binlog_get_pending_rows_event(is_transactional); > + } > + > + bool binlog_for_noop_dml(bool transactional_table); > + > /** > Determine the binlog format of the current statement. > > @@ -3557,6 +3564,7 @@ class THD: public THD_count, /* this must be first */ > bool tablespace_op; /* This is TRUE in DISCARD/IMPORT TABLESPACE */ > /* True if we have to log the current statement */ > bool log_current_statement; > + uint vers_created_partitions; > /** > True if a slave error. Causes the slave to stop. Not the same > as the statement execution error (is_error()), since > diff --git a/sql/sql_insert.cc b/sql/sql_insert.cc > index 707b8a0d3bf..25551dfa197 100644 > --- a/sql/sql_insert.cc > +++ b/sql/sql_insert.cc > @@ -4223,6 +4225,7 @@ bool select_insert::prepare_eof() > thd->clear_error(); > else > errcode= query_error_code(thd, killed_status == NOT_KILLED); > + StatementBinlog stmt_binlog(thd, thd->binlog_need_stmt_format(trans_table)); > res= thd->binlog_query(THD::ROW_QUERY_TYPE, > thd->query(), thd->query_length(), > trans_table, FALSE, FALSE, errcode); Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 8

Re: [Maria-developers] 93493a0e9b5: MDEV-24176 Server crashes after insert in the table with virtual column generated using date_format() and if()
by Sergei Golubchik 17 Apr '22

17 Apr '22

Hi, Aleksey! Could you please explain in the commit comment what the bug was and what you are fixing? I wasn't able to understad that, so I had to debug those crashes myself. So, the reason is, objects allocated in the wrong arena, right? And it was done inconsistently, on different code paths the "re-fix" context was different. I agree that encapsulating all the context in one Vcol_expr_context is a good idea, it'll allow to have the same, correct, context everywhere. But always using expr_arena doesn't necessarily seem to be a correct choice. E.g. Arg_comparator creates Item_cache_temporal on every fix_field. If vcol expr is re-fixed all the time, expr_arena will grow continuously. Same for charset conversion, CASE, IN, etc. I suspect that if the vcol expr is always re-fixed, it has to use stmt arena. Why do you re-fix fields? I can hardly imagine that an Item_field can find itself in a different table. And if it's always in the same table, why to force it to find this table over and over? We've already discussed VCOL_TABLE_ALIAS, no need to go there again. On Jan 06, Aleksey Midenkov wrote: > revision-id: 93493a0e9b5 (mariadb-10.2.40-194-g93493a0e9b5) > parent(s): a565e63c54c > author: Aleksey Midenkov > committer: Aleksey Midenkov > timestamp: 2021-12-29 00:46:31 +0300 > message: > > MDEV-24176 Server crashes after insert in the table with virtual > column generated using date_format() and if() > > When table is reopened from cache vcol_info contains stale > expression. We refresh expression via TABLE::vcol_fix_exprs() but > first we must prepare a proper context (Vcol_expr_context) and do some > preparations which meet some requirements: > > 1. fields must not be fixed, we unfix them with cleanup() via > cleanup_processor. > > 2. Item_ident must have cached_table cleared. Again, this is stale > data which was already freed. cached_table interferes with > find_field_in_tables(). > > We cannot clear cached_table directly in Item_ident::cleanup() > method. Lots of spaghetti logic depends on cached_table existence even > after cleanup() done. > > 3. If Item_ident has table_name non-NULL we trigger expr update. That > is done via Item_ident::check_vcol_func_processor() and > VCOL_TABLE_ALIAS flag. > > (4-7) The below conditions are prepared by Vcol_expr_context and used in > both fix_session_expr_for_read() and TABLE::vcol_fix_exprs(): > > 4. Any fix_expr() must be done on expr_arena as there may be new items > created. It was a bug in fix_session_expr_for_read(). It was just not > reproduced because there were no second refix. Now refix is done for > more cases so it does reproduce. Tests affected: vcol.binlog > > 5. Also name resolution context must be narrowed to the single table > with all crutches in place. Tests affected: vcol.update > > 6. sql_mode must be clean and not fail expr update. > > sql_mode such as MODE_NO_BACKSLASH_ESCAPES, MODE_NO_ZERO_IN_DATE, etc > must not affect vcol expression update. If the table was created > successfully any further evaluation must not fail. Tests affected: > main.func_like > > 7. Warnings are disabled during expr update. It is assumed that these > warnings were printed before during initialization phase or later > during execution phase. Tests affected: vcol.wrong_arena > > Dictionary: refresh, update and refix are the synonyms for > vcol_fix_exprs() or fix_session_expr_for_read() calls. > > 8. cleanup_excluding_fields_processor() removed. It was just a quick > hack to exclude wrongly working Item_field from processing. Now it > works due to correct execution environment (Vcol_expr_context). > Related to MDEV-10355. > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 12

Re: [Maria-developers] 62d3496969d: MDEV-24176 Server crashes after insert in the table with virtual column generated using date_format() and if()
by Sergei Golubchik 12 Apr '22

12 Apr '22

Hi, Aleksey, This is a review of `git diff 5fbbdbc85ba3 62d3496969db` Basically, no new comments, see below: On Apr 07, Aleksey Midenkov wrote: > diff --git a/libmariadb b/libmariadb > index f6c3d9fd2af..735a7299dba 160000 > --- a/libmariadb > +++ b/libmariadb > @@ -1 +1 @@ > -Subproject commit f6c3d9fd2af5d17db64cc996574aa312efd70fcf > +Subproject commit 735a7299dbae19cc2b82b9697becaf90e9b43047 you've mistakenly checked in a submodule > diff --git a/sql/field.cc b/sql/field.cc > index f18fb25ebe3..04c492e6c69 100644 > --- a/sql/field.cc > +++ b/sql/field.cc > @@ -2416,6 +2416,13 @@ int Field::set_default() > if (default_value) > { > Query_arena backup_arena; > + /* > + TODO: this imposes memory leak until table flush when save_in_field() > + does expr_arena allocation. F.ex. case from main.default: > + > + CREATE TABLE t1 (a INT DEFAULT CONCAT('1 ')); > + INSERT INTO t1 VALUES (DEFAULT); I wasn't able to repeat that. In your branch (didn't try in vanilla 10.3) and for the above test case I see no additional allocations in table->mem_root after the table is opened. > + */ > table->in_use->set_n_backup_active_arena(table->expr_arena, &backup_arena); > int rc= default_value->expr->save_in_field(this, 0); > table->in_use->restore_active_arena(table->expr_arena, &backup_arena); > diff --git a/sql/table.cc b/sql/table.cc > index 08d91678b25..d22da17c31a 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -2970,36 +2988,133 @@ static bool fix_vcol_expr(THD *thd, Virtual_column_info *vcol) ... > +bool Vcol_expr_context::init() > +{ > + /* > + As this is vcol expression we must narrow down name resolution to > + single table. > + */ > + if (init_lex_with_single_table(thd, table, &lex)) I've already asked about it in a separate email, so here I'm just marking a line that we haven't fully agreed on yet > + { > + my_error(ER_OUT_OF_RESOURCES, MYF(0)); > + table->map= old_map; > + return true; > + } > + > + lex.sql_command= old_lex->sql_command; > + thd->variables.sql_mode= 0; > + > + TABLE_LIST const *tl= table->pos_in_table_list; > + DBUG_ASSERT(table->pos_in_table_list); > + > + if (table->pos_in_table_list->security_ctx) > thd->security_ctx= tl->security_ctx; > - bool res= fix_session_vcol_expr(thd, vcol); > + > + inited= true; > + return false; > +} > + Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] e3e59c9e38d: MDEV-28266 Crash in Field_string::type_handler when calling procedures
by Sergei Golubchik 10 Apr '22

10 Apr '22

Hi, Oleksandr, On Apr 10, Oleksandr Byelkin wrote: > revision-id: e3e59c9e38d (mariadb-10.6.6-83-ge3e59c9e38d) > parent(s): 4e1ca388381 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2022-04-08 10:37:17 +0200 > message: > > MDEV-28266 Crash in Field_string::type_handler when calling procedures > > In case of only metadata request the cursor lack the call to redirect > Field::orig_table to the temporary table. > > diff --git a/sql/sql_cursor.cc b/sql/sql_cursor.cc > index 0add5845558..155346341f1 100644 > --- a/sql/sql_cursor.cc > +++ b/sql/sql_cursor.cc > @@ -183,6 +183,16 @@ int mysql_open_cursor(THD *thd, select_result *result, > { > Materialized_cursor *materialized_cursor= > result_materialize->materialized_cursor; > + if (result_materialize->view_structure_only()) > + { > + /* > + If it was view_structure_only then only send metadata call was made > + and there was no other call. So we have to "close" the cursor > + explicitly (which redirect orig_table in the fields, and untie > + it with closed tables) > + */ > + materialized_cursor->on_table_fill_finished(); > + } > > if ((rc= materialized_cursor->open(0))) > { > delete materialized_cursor; > goto end; > } normally on_table_fill_finished() is called from Select_materialize::send_eof() and Select_materialize::abort_result_set(). Note that directly after your added code there's materialized_cursor->open(), and there we have int Materialized_cursor::open(JOIN *join __attribute__((unused))) { ... if (!rc) { thd->server_status|= SERVER_STATUS_CURSOR_EXISTS; result->send_eof(); } else { result->abort_result_set(); } That is, one could expect that even without your added code, on_table_fill_finished() would be called. The problem is that result here is sp_cursor::Select_fetch_into_spvars, and sp_cursor::Select_fetch_into_spvars::send_eof() is simply { return FALSE; } So, as a fix I'd suggest to remove on_table_fill_finished() from send_eof/abort_result_set, and put it unconditionally after the if(), to make it work no matter what select_result was chosen. Like this: ====================================== --- a/sql/sql_cursor.cc +++ b/sql/sql_cursor.cc @@ -80,19 +80,7 @@ class Select_materialize: public select_unit Select_materialize(THD *thd_arg, select_result *result_arg): select_unit(thd_arg), result(result_arg), materialized_cursor(0) {} virtual bool send_result_set_metadata(List<Item> &list, uint flags); - bool send_eof() - { - if (materialized_cursor) - materialized_cursor->on_table_fill_finished(); - return false; - } - - void abort_result_set() - { - if (materialized_cursor) - materialized_cursor->on_table_fill_finished(); - } - + bool send_eof() { return false; } bool view_structure_only() const { return result->view_structure_only(); @@ -333,6 +321,8 @@ int Materialized_cursor::open(JOIN *join result->abort_result_set(); } + on_table_fill_finished(); + return rc; } ====================================== with the commit comment saying that "on_table_fill_finished() should always be done at the end of open(), even if result is not Select_materialize, but, for example, Select_fetch_into_spvars" Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] a10facb656b: MDEV-27065 Partitioning tables with custom data directories moves data back to default directory
by Sergei Golubchik 29 Mar '22

29 Mar '22

Hi, Nayuta, On Mar 29, Nayuta Yanagisawa wrote: > revision-id: a10facb656b (mariadb-10.2.43-48-ga10facb656b) > parent(s): 33ff18627ea > author: Nayuta Yanagisawa > committer: Nayuta Yanagisawa > timestamp: 2022-03-29 23:43:16 +0900 > message: > > MDEV-27065 Partitioning tables with custom data directories moves data back to default directory > > --- > .../suite/parts/r/partition_exch_qa_14.result | 22 ++++++++++++++++++++++ > sql/ha_partition.cc | 16 ++++++++++++++++ > 2 files changed, 38 insertions(+) > > diff --git a/mysql-test/suite/parts/r/partition_exch_qa_14.result b/mysql-test/suite/parts/r/partition_exch_qa_14.result > index 1420982436a..52839644f55 100644 > --- a/mysql-test/suite/parts/r/partition_exch_qa_14.result > +++ b/mysql-test/suite/parts/r/partition_exch_qa_14.result > @@ -9,6 +9,9 @@ PARTITION BY RANGE (a) > (PARTITION p0 VALUES LESS THAN (10) DATA DIRECTORY = 'MYSQLTEST_VARDIR/mysql-test-data-dir' INDEX DIRECTORY = 'MYSQLTEST_VARDIR/mysql-test-idx-dir', > PARTITION p1 VALUES LESS THAN (100) DATA DIRECTORY = 'MYSQLTEST_VARDIR/mysql-test-data-dir' INDEX DIRECTORY = 'MYSQLTEST_VARDIR/mysql-test-idx-dir', > PARTITION p2 VALUES LESS THAN (1000) DATA DIRECTORY = 'MYSQLTEST_VARDIR/mysql-test-data-dir' INDEX DIRECTORY = 'MYSQLTEST_VARDIR/mysql-test-idx-dir'); > +Warnings: > +Warning 1618 <DATA DIRECTORY> option ignored > +Warning 1618 <INDEX DIRECTORY> option ignored This is very confusing. There are four DATA DIRECTORY and four INDEX DIRECTORY in the statement. Which one is ignored? (yes, *I* know, the table level one) > @@ -148,6 +154,9 @@ PARTITION BY RANGE (a) > (PARTITION p0 VALUES LESS THAN (10) , > PARTITION p1 VALUES LESS THAN (100) , > PARTITION p2 VALUES LESS THAN (1000) ); > +Warnings: > +Warning 1618 <DATA DIRECTORY> option ignored > +Warning 1618 <INDEX DIRECTORY> option ignored and here DATA/INDEX DIRECTORY behaves not like other options. For a normal option table-level value would apply to all partitions. That is CREATE TABLE t1 (...) OPT="VAL (PARTITION p0 VALUES LESS THAN (10) , PARTITION p1 VALUES LESS THAN (100) , PARTITION p2 VALUES LESS THAN (1000) ); would be the same as CREATE TABLE t1 (...) (PARTITION p0 VALUES LESS THAN (10) OPT="VAL, PARTITION p1 VALUES LESS THAN (100) OPT="VAL, PARTITION p2 VALUES LESS THAN (1000) OPT="VAL); Is that correct? > CREATE TABLE tsp (a INT, b VARCHAR(55), PRIMARY KEY (a)) DATA DIRECTORY = 'MYSQLTEST_VARDIR/mysql-test-data-dir' INDEX DIRECTORY = 'MYSQLTEST_VARDIR/mysql-test-idx-dir' ENGINE = MYISAM > PARTITION BY RANGE (a) > SUBPARTITION BY HASH(a) Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 88f8aa20bba: MDEV-23486 RBR can bypass secure_timestamp=YES
by Sergei Golubchik 26 Mar '22

26 Mar '22

Hi, Aleksey! On Jan 09, Aleksey Midenkov wrote: > revision-id: 88f8aa20bba (mariadb-10.3.31-79-g88f8aa20bba) > parent(s): d5451867af1 > author: Aleksey Midenkov > committer: Aleksey Midenkov > timestamp: 2021-12-23 04:15:39 +0300 > message: > > MDEV-23486 RBR can bypass secure_timestamp=YES > > Pass to a slave whether master table is system-versioned via > TM_SYSTEM_VERSIONED flag. That deprecates m_vers_from_plain > detection. If secure_timestamp == YES or master did not supply row_end > data treat the row as current data and generate timestamps for it. I don't understand why does it matter whether the master is system versioned or not. secure_timestamp=YES means the table has to store actual timestamps when rows were changed. It doesn't trust the master to set timestamps. So it should not matter whether master table is versioned or not, if you don't trust the master, you cannot trust its TM_SYSTEM_VERSIONED flag either. Even if the table was system versioned on the master, the master could've had secure_timestamp=NO and a completely fake history. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] MDEV-24560 SIGSEGV in st_join_table::cleanup
by Sergey Petrunia 25 Mar '22

25 Mar '22

Hi Oleg, > commit 4f41909c3ee1d7e42c56947c9a9156fb78c22d8a > Author: Oleg Smirnov <olernov(a)gmail.com> > Date: Fri Mar 11 21:18:34 2022 +0700 > > MDEV-24560 SIGSEGV in st_join_table::cleanup > > If JOIN::create_postjoin_aggr_table encounters errors during execution > then free_tmp_table() is then called twice for JOIN_TAB::aggr. > The solution is to initialize JOIN_TAB::aggr and JOIN_TAB::table only > on successful completion of JOIN::create_postjoin_aggr_table > > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index c044656e916..8c07014d981 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -4037,10 +4037,6 @@ JOIN::create_postjoin_aggr_table(JOIN_TAB *tab, List<Item> *table_fields, > !tables_list); > if (tab > join_tab) > (tab - 1)->next_select= sub_select_postjoin_aggr; > - if (!(tab->aggr= new (thd->mem_root) AGGR_OP(tab))) > - goto err; > - tab->table= table; > - table->reginfo.join_tab= tab; > > /* if group or order on first table, sort first */ > if ((group_list && simple_group) || > @@ -4089,7 +4085,10 @@ JOIN::create_postjoin_aggr_table(JOIN_TAB *tab, List<Item> *table_fields, > order= NULL; > } > } > - > + if (!(tab->aggr= new (thd->mem_root) AGGR_OP(tab))) > + goto err; > + tab->table= table; > + table->reginfo.join_tab= tab; > DBUG_RETURN(false); I see a potential issue in the patch: The moved code does this assignment: > tab->table= table; Let's look inside this if-branch: >> if ((group_list && simple_group) || >> (implicit_grouping && select_lex->have_window_funcs())) The code inside if() { ... } part seems to be fine: it does not touch the contents of 'tab', in particular tab->table. Let's look at the coed inside the else {... } part: It has this condition: >> if (!group_list && !table->distinct && order && simple_order && >> tab == join_tab + const_tables) >> { Note that tab==join_tab+const_tables. Then, a few lines below: >> add_sorting_to_table(join_tab + const_tables, order)) This is essentially add_sorting_to_table(tab, order). Inside JOIN::add_sorting_to_table one can find this: TABLE *table= tab->table; if ((tab == join_tab + const_tables) && table->pos_in_table_list && table->pos_in_table_list->is_sjm_scan_table()) that is, we access tab->table and tab->table->pos_in_table_list. Which are not assigned yet (because your patch moves the assignment to happen later). I've tried to construct an example to demonstrate this issue, but was not successful. I've made this change: diff --git a/sql/sql_select.cc b/sql/sql_select.cc index c044656e916..5a9a7d7e73f 100644 --- a/sql/sql_select.cc +++ b/sql/sql_select.cc @@ -4080,6 +4080,8 @@ JOIN::create_postjoin_aggr_table(JOIN_TAB *tab, List<Item> *table_fields, tab == join_tab + const_tables) { DBUG_PRINT("info",("Sorting for order")); + fprintf(stderr,"AAABBBCCC:%s\n", thd->query()); + DBUG_ASSERT(0); THD_STAGE_INFO(thd, stage_sorting_for_order); if (ordered_index_usage != ordered_index_order_by && and ran the entire testsuite with it. There were no failures. I'm not sure if this is deadcode or we just don't have enough test coverage. I suggest to assume the latter and change the patch to avoid this issue. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://petrunia.net

1 0

Re: [Maria-developers] 3be1e491fb5: MDEV-27743 Remove Lex::charset
by Sergei Golubchik 22 Mar '22

22 Mar '22

Hi, Alexander, All comments from the previous review were addressed and there were no other changes. So, ok to push. On Mar 22, Alexander Barkov wrote: > revision-id: 3be1e491fb5 (mariadb-10.6.1-383-g3be1e491fb5) > parent(s): c69defd3b61 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2022-03-21 21:33:30 +0400 > message: > > MDEV-27743 Remove Lex::charset > > This patch also fixes: > > MDEV-27690 Crash on `CHARACTER SET csname COLLATE DEFAULT` in column definition > MDEV-27853 Wrong data type on column `COLLATE DEFAULT` and table `COLLATE some_non_default_collation` > MDEV-28067 Multiple conflicting column COLLATE clauses are not rejected > MDEV-28118 Wrong collation of `CAST(.. AS CHAR COLLATE DEFAULT)` > MDEV-28119 Wrong column collation on MODIFY + CONVERT > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 026e52bad84: MDEV-27743 Remove Lex::charset
by Sergei Golubchik 21 Mar '22

21 Mar '22

Hi, Alexander, Please, see below On Mar 18, Alexander Barkov wrote: > revision-id: 026e52bad84 (mariadb-10.6.1-369-g026e52bad84) > parent(s): 369498d6b8c > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2022-03-18 12:26:03 +0400 > message: > > MDEV-27743 Remove Lex::charset > > This patch also fixes: > > MDEV-27690 Crash on `CHARACTER SET csname COLLATE DEFAULT` in column definition > MDEV-27853 Wrong data type on column `COLLATE DEFAULT` and table `COLLATE some_non_default_collation` > MDEV-28067 Multiple conflicting column COLLATE clauses are not rejected > MDEV-28118 Wrong collation of `CAST(.. AS CHAR COLLATE DEFAULT)` > MDEV-28119 Wrong column collation on MODIFY + CONVERT > > diff --git a/mysql-test/main/ctype_collate_column.test b/mysql-test/main/ctype_collate_column.test > new file mode 100644 > index 00000000000..6a7cf2cc84c > --- /dev/null > +++ b/mysql-test/main/ctype_collate_column.test > @@ -0,0 +1,637 @@ ... > + > +DELIMITER $$; > +CREATE PROCEDURE p1(dt TEXT, cs TEXT, cl0 TEXT, cl1 TEXT, cl2 TEXT, tcs TEXT) > +BEGIN > + DECLARE errstate TEXT DEFAULT NULL; > + DECLARE errno INT DEFAULT NULL; > + DECLARE errmsg TEXT DEFAULT NULL; > + DECLARE query TEXT; > + DECLARE result TEXT; > + DECLARE CONTINUE HANDLER FOR SQLEXCEPTION > + BEGIN > + GET DIAGNOSTICS CONDITION 1 errstate = RETURNED_SQLSTATE, > + errno = MYSQL_ERRNO, errmsg = MESSAGE_TEXT; > + END; > + SET query= CONCAT('CREATE TABLE t1 (a ', dt, ' ', cs, ' ', cl0, > + ' NOT NULL ',cl1, > + ' DEFAULT '''' ', cl2, > + ') ', tcs, ' ENGINE=Memory'); > + EXECUTE IMMEDIATE query; > + IF errmsg IS NOT NULL > + THEN > + SET result=CONCAT('ERROR: ', > + COALESCE(errstate,'<NULL>'), ' ', > + COALESCE(errno,'<NULL>'), ' ', > + COALESCE(errmsg,'<NULL>')); > + INSERT INTO results (dt,cs,cl0,cl1,cl2,tcs,query,result) > + VALUES (dt,cs,cl0,cl1,cl2,tcs,query,result); > + ELSE > + FOR row IN (SELECT CONCAT(COLUMN_TYPE, > + ' CHARACTER SET ', CHARACTER_SET_NAME, > + ' COLLATE ', COLLATION_NAME) AS result > + FROM INFORMATION_SCHEMA.COLUMNS > + WHERE TABLE_SCHEMA='test' AND TABLE_NAME='t1') > + DO > + INSERT INTO results (dt,cs,cl0,cl1,cl2,tcs,query,result) > + VALUES (dt,cs,cl0,cl1,cl2,tcs,query,row.result); > + END FOR; > + DROP TABLE t1; > + END IF; > +END; > +$$ > +DELIMITER ;$$ > + > + > +DELIMITER $$; > +CREATE PROCEDURE p3(dt TEXT) > +BEGIN > + FOR row IN ( > + SELECT cs, cl0, cl1.cl1 AS cl1, cl2.cl1 AS cl2, tcs > + FROM cs, cl0, cl1, cl1 AS cl2, tcs > + ORDER BY cs, cl0, cl1, cl2, tcs > + ) > + DO > + CALL p1('char(10)', row.cs, row.cl0, row.cl1, row.cl2, row.tcs); minor detail: here ^^^ you want to use dt, not 'char(10)' > + END FOR; > +END; > +$$ > +DELIMITER ;$$ > + > + > +--disable_column_names > +CALL p3('char(10)'); > +--enable_column_names > + > +SELECT * FROM results_statistics; isn't it redundant? you select all rows from result anyway. If something will change, the result of that select will already cause the the to fail. Why do you need second mismatch in the result/reject. > + > +--vertical_results > +SELECT query, result, '' AS `` FROM results > +ORDER BY dt, cs, cl0, cl1, cl2, tcs; > +--horizontal_results > + > +DROP PROCEDURE p1; > +DROP PROCEDURE p3; > + > +DROP TABLE cs, cl0, cl1, tcs; ... > +# CREATE TABLE t1 > +# ( > +# a CHAR(10) BINARY NOT NULL DEFAULT '' > +# ) CHARACTER SET cs COLLATE cs_bin; > + > +DELETE FROM results > +WHERE cs='' > + AND cl0='BINARY' > + AND cl1='' > + AND cl2='' > + AND tcs LIKE 'CHARACTER SET%' > + AND result NOT LIKE 'ERROR%' > + AND result RLIKE CONCAT('COLLATE ', tcs_character_set_name, '_bin'); > +SELECT ROW_COUNT(); ... > +DROP FUNCTION is_collate_clause_with_explicit_default_collation; > +DROP FUNCTION is_collate_clause_with_explicit_collation; > +DROP FUNCTION is_conflicting_charset_explicit_collate_explicit; > +DROP FUNCTION is_conflicting_collate_explicit2; > +DROP FUNCTION is_conflicting_collate_default_collate_explicit; > +DROP FUNCTION collate_cs_default_collation; I think the main result is `select * from results` (with order by, etc). It's great, that you added that. Everything else is redundant, I suggest to remove other selects and deletes. > + > + > +--echo # > +--echo # End of 10.9 tests > +--echo # > diff --git a/sql/lex_charset.h b/sql/lex_charset.h > new file mode 100644 > index 00000000000..d95f0bc1920 > --- /dev/null > +++ b/sql/lex_charset.h > @@ -0,0 +1,202 @@ ... > +#define LEX_CHARSET_COLLATION_TYPE_BITS 2 > + static_assert(((1<<LEX_CHARSET_COLLATION_TYPE_BITS)-1) >= > + TYPE_COLLATE_CONTEXTUALLY_TYPED, > + "Lex_charset_collation_st::Type bits check"); > + > +protected: > + CHARSET_INFO *m_ci; > + Type m_type; > +public: > + static CHARSET_INFO *find_bin_collation(CHARSET_INFO *cs); > + static CHARSET_INFO *find_default_collation(CHARSET_INFO *cs); > +public: ... > + void set_contextually_typed_binary_style() > + { > + m_ci= &my_collation_contextually_typed_binary; > + m_type= TYPE_COLLATE_CONTEXTUALLY_TYPED; > + } > + bool is_contextually_typed_collate_default() const > + { > + return m_ci == &my_collation_contextually_typed_default && > + m_type == TYPE_COLLATE_CONTEXTUALLY_TYPED; what does it mean when only one of these two expressions is true? Say, the first true, the second false? Or the first false, the second true? > + } > + bool is_contextually_typed_binary_style() const > + { > + return m_ci == &my_collation_contextually_typed_binary && > + m_type == TYPE_COLLATE_CONTEXTUALLY_TYPED; > + } ... > diff --git a/strings/ctype-bin.c b/strings/ctype-bin.c > index bb746ad90b0..a5c9129ebea 100644 > --- a/strings/ctype-bin.c > +++ b/strings/ctype-bin.c > @@ -630,3 +630,69 @@ struct charset_info_st my_charset_bin = > &my_charset_handler, > &my_collation_binary_handler > }; > + > + > +struct charset_info_st my_collation_contextually_typed_binary= > +{ > + 0,0,0, /* number */ I thought you will never need to resove the &my_collation_contextually_typed_binary pointer, that is, I thought the values in the my_collation_contextually_typed_binary struct should not matter as they'll never be used. Was that wrong? Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 2

Re: [Maria-developers] d8e915a88fc: MDEV-26009 Server crash when calling twice procedure using FOR-loop
by Sergei Golubchik 20 Mar '22

20 Mar '22

Hi, Oleksandr, ok to push also, see below On Mar 20, Oleksandr Byelkin wrote: > revision-id: d8e915a88fc (mariadb-10.3.33-32-gd8e915a88fc) > parent(s): 6a2d88c1322 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2022-03-18 11:13:09 +0100 > message: > > MDEV-26009 Server crash when calling twice procedure using FOR-loop > > The problem was that instructions sp_instr_cursor_copy_struct and > sp_instr_copen uses the same lex, adding and removing "tail" of > prelocked tables and forgetting that tail of all tables is kept in > LEX::query_tables_last. If the LEX used only by one instruction > or the query do not have prelocked tables it is not important. > But to work correctly in all cases LEX::query_tables_last should > be reset to make new tables added in the correct list (after last > table in the LEX instead after last table of the prelocking "tail" > which was cut). > > diff --git a/sql/sp_head.cc b/sql/sp_head.cc > index 57ab31d9edf..96dc78dcf49 100644 > --- a/sql/sp_head.cc > +++ b/sql/sp_head.cc > @@ -3486,6 +3486,11 @@ sp_lex_keeper::reset_lex_and_exec_core(THD *thd, uint *nextp, > lex_query_tables_own_last= m_lex->query_tables_own_last; > prelocking_tables= *lex_query_tables_own_last; > *lex_query_tables_own_last= NULL; > + /* > + Here we cut "tail" of prelocked tables, so have to return last table > + to the original position (wuthout the tail > + */ > + m_lex->query_tables_last= m_lex->query_tables_own_last; 1. the comment looks truncated 2. I don't even think this line needs a comment, this code block removes the list of prelocked tables from the query_tables list. Looking at how query_tables_last and query_tables_own_last are documented: /* Pointer to next_global member of last element in the previous list. */ TABLE_LIST **query_tables_last; /* If non-0 then indicates that query requires prelocking and points to next_global member of last own element in query table list (i.e. last table which was not added to it as part of preparation to prelocking). 0 - indicates that this query does not need prelocking. */ TABLE_LIST **query_tables_own_last; it is pretty clear that after the "tail" is removed, one has to set query_tables_last to point to the end of the list. The fact that it wasn't done was clearly an omission. That is, existing code and existing comments document it pretty well, your comment looks redundant and even confusing to me. So, please, remove the comment and ok to push > m_lex->mark_as_requiring_prelocking(NULL); > } > thd->rollback_item_tree_changes(); Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] Google Summer of Code 2022
by Vicențiu Ciorbaru 20 Mar '22

20 Mar '22

Hi Sergey! I'm happy to see that you have experience in MariaDB code base already. The MDEV-18827 is definitely up for grabs for GSoC 2022. The project is mostly a part-time project (requiring a commitment of around 175 hours), however depending on your availability we can increase the scope of the project and bump it up to a full time project. My suggestion for this task is to start looking at code parsing FRM files. Search for the function TABLE_SHARE::init_from_binary_frm_image to get an idea how an .frm is structured. After that, think how you would like to implement the feature described in MDEV-18827. Feel free to submit a draft proposal and we can iterate on it. Not 100% sure how the GSoC interface works this year, but last time you were able to share with potential mentors a Google Doc with the possibility to comment. Have a look at the proposal guidelines https://mariadb.org/gsoc2022-instructions/ and start writing it. When the program application opens you can submit it (4th of April). In the meantime, feel free to ask any questions you might have here. I encourage you to keep all discussions on the mailing list as much as possible so others can chime in. Vicențiu On Sat, 19 Mar 2022 at 09:54, Сергей Ваниславский <serzhmn(a)gmail.com> wrote: > Hello Vicentiu! > My name is Vanislavsky Sergey. Your colleague Nikita Malyavin > recommended contacting you about GSoC.I would like to do your task > MDEV-18827. I am a 3rd year student at FEFU. Here is my last pull request: > https://github.com/MariaDB/server/pull/2016 > Thank you for your consideration. >

1 0

[Maria-developers] Relevance of MDEV-19556
by Vladisav Kakurin 17 Mar '22

17 Mar '22

Hi, Vicențiu! With my mentor Nikita Malyavin we chose the topic of my bachelor's graduation work, and I was really interested in the MDEV-19556 <https://jira.mariadb.org/browse/MDEV-19556> ticket. Could you say some words about this, maybe the ticket ain’t actual, or something about work that was done by one of the developers to this direction?

1 0

Re: [Maria-developers] 2467eb2d314: MDEV-27743 Remove Lex::charset
by Sergei Golubchik 16 Mar '22

16 Mar '22

Hi, Alexander, I had only a couple of questions/comments, see below: On Mar 11, Alexander Barkov wrote: > revision-id: 2467eb2d314 (mariadb-10.6.1-330-g2467eb2d314) > parent(s): 61b72ed3dde > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2022-02-10 10:27:55 +0400 > message: > > MDEV-27743 Remove Lex::charset > > This patch also fixes: > MDEV-27690 Crash on `CHARACTER SET csname COLLATE DEFAULT` in column definition how do you plan to fix it in 10.2-10.8 ? > > diff --git a/mysql-test/main/ctype_collate_column.result b/mysql-test/main/ctype_collate_column.result > new file mode 100644 > index 00000000000..b20be5d3a48 > --- /dev/null > +++ b/mysql-test/main/ctype_collate_column.result > @@ -0,0 +1,450 @@ > +# > +# Start of 10.9 tests > +# > +# > +# MDEV-27743 Remove Lex::charset 1. Jira doesn't explain what user-visible changes one can expect. If there are none, it's not clear why you needed this big test case 2. the test is totally unreadable. Could you do a normal test instead? Like, put `select` instead of `execute immediate` and convert the output to a .test file? > +# > +CREATE TABLE cs (cs VARCHAR(64) NOT NULL); > +INSERT INTO cs (cs) VALUES > +(''), > +('CHARACTER SET latin1'), > +('CHARACTER SET utf8mb4'); > +CREATE TABLE cl0 (cl0 VARCHAR(64) NOT NULL); > +INSERT INTO cl0 (cl0) VALUES > +(''), > +('BINARY'), > +('COLLATE DEFAULT'), > +('COLLATE latin1_bin'), > +('COLLATE latin1_swedish_ci'), > +('COLLATE utf8mb4_bin'), > +('COLLATE utf8mb4_general_ci'); > +CREATE TABLE cl1 (cl1 VARCHAR(64) NOT NULL); > +INSERT INTO cl1 (cl1) VALUES > +(''), > +('COLLATE DEFAULT'), > +('COLLATE latin1_bin'), > +('COLLATE latin1_swedish_ci'), > +('COLLATE utf8mb4_bin'), > +('COLLATE utf8mb4_general_ci'); ... > diff --git a/mysql-test/main/ctype_latin1.result b/mysql-test/main/ctype_latin1.result > index 0b16d1cf6f2..70577f2c22c 100644 > --- a/mysql-test/main/ctype_latin1.result > +++ b/mysql-test/main/ctype_latin1.result > @@ -8889,6 +8889,38 @@ a b > 111 111 > DROP TABLE t1; > # > +# MDEV-27690 Crash on `CHARACTER SET csname COLLATE DEFAULT` in column definition > +# > +CREATE TABLE t1 (a CHAR(10) CHARACTER SET latin1 COLLATE DEFAULT); > +SHOW CREATE TABLE t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` char(10) DEFAULT NULL > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +DROP TABLE t1; > +SELECT CAST('a' AS CHAR(10) CHARACTER SET latin1 COLLATE DEFAULT); > +CAST('a' AS CHAR(10) CHARACTER SET latin1 COLLATE DEFAULT) > +a > +CREATE TABLE t1 AS > +SELECT CAST('a' AS CHAR(10) CHARACTER SET latin1 COLLATE DEFAULT) AS c1; > +SHOW CREATE TABLE t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `c1` varchar(10) DEFAULT NULL > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +DROP TABLE t1; > +SELECT COLUMN_GET(COLUMN_CREATE(0, 'string'),0 AS CHAR CHARACTER SET latin1 COLLATE DEFAULT) AS c1; > +c1 > +string > +CREATE TABLE t1 AS > +SELECT COLUMN_GET(COLUMN_CREATE(0, 'string'),0 AS CHAR CHARACTER SET latin1 COLLATE DEFAULT) AS c1; > +SHOW CREATE TABLE t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `c1` longtext DEFAULT NULL > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +DROP TABLE t1; 1. why is it in "10.2 tests" ? 2. shouldn't here be at least one test that COLLATE DEFAULT is not a no-op? Like CREATE TABLE (... COLLATE DEFAULT ...) COLLATE _non_default ? Or is it in your generated ctype_collate_column.test? > +# > # End of 10.2 tests > # > # > diff --git a/mysql-test/suite/roles/create_and_drop_role_invalid_user_table.test b/mysql-test/suite/roles/create_and_drop_role_invalid_user_table.test > index 9cf0d7b4aff..dac6eab21e9 100644 > --- a/mysql-test/suite/roles/create_and_drop_role_invalid_user_table.test > +++ b/mysql-test/suite/roles/create_and_drop_role_invalid_user_table.test > @@ -36,7 +36,7 @@ set default role test_role for root@localhost; > drop role test_role; > drop user test_user@localhost; > > -alter table user add column default_role char(80) binary default '' not null > +alter table user add column default_role char(80) default '' not null > COLLATE utf8_general_ci Heh, interesting that it worked before good catch > after is_role; > alter table user add max_statement_time decimal(12,6) default 0 not null > diff --git a/sql/structs.h b/sql/structs.h > index 9a5006e8b47..db065c49931 100644 > --- a/sql/structs.h > +++ b/sql/structs.h > @@ -599,11 +599,140 @@ class DDL_options: public DDL_options_st ... > +class Lex_collation: public Lex_collation_st > +{ > +public: > + Lex_collation(CHARSET_INFO *collation, Type type) > + { > + m_collation= collation; > + m_type= type; > + } > + static Lex_collation national(bool bin_mod) > + { > + if (bin_mod) > + return Lex_collation(&my_charset_utf8mb3_bin, TYPE_EXPLICIT); > + return Lex_collation(&my_charset_utf8mb3_general_ci, TYPE_IMPLICIT); utf8mb3? not utf8? > + } > +}; > + > + > struct Lex_length_and_dec_st > { > -private: > +protected: > uint32 m_length; > uint8 m_dec; > + uint8 m_collation_type:2; > bool m_has_explicit_length:1; > bool m_has_explicit_dec:1; > public: > diff --git a/sql/sql_lex.cc b/sql/sql_lex.cc > index ae6ae9a0cc0..6ca10267187 100644 > --- a/sql/sql_lex.cc > +++ b/sql/sql_lex.cc > @@ -11816,3 +11784,257 @@ bool SELECT_LEX_UNIT::explainable() const ... > +CHARSET_INFO * > +Lex_collation_st::resolved_to_character_set(CHARSET_INFO *def) const > +{ > + DBUG_ASSERT(def); > + if (m_type != TYPE_CONTEXTUALLY_TYPED) > + { > + if (!m_collation) > + return def; // Empty - not typed at all > + return m_collation; // Explicitly typed > + } > + > + // Contextually typed > + DBUG_ASSERT(m_collation); > + > + if (m_collation == &my_charset_bin) // CHAR(10) BINARY > + return find_bin_collation(def); > + > + if (m_collation == &my_charset_latin1) // CHAR(10) COLLATE DEFAULT > + return find_default_collation(def); Uhm. Could you create two dummy CHARSET_INFO variables for that instead? Like CHARSET_INFO binary_collaton, default_collation; no need to initialize them, but to have distinct values for binary/default and not reuse my_charset_bin and my_charset_latin1. > + > + /* > + Non-binary and non-default contextually typed collation. > + We don't have such yet - the parser cannot produce this. > + But will have soon, e.g. "uca1400_as_ci". > + */ > + DBUG_ASSERT(0); > + return NULL; > +} > + > + ... > diff --git a/sql/field.h b/sql/field.h > index 2ed02b37cfd..fe5ba435202 100644 > --- a/sql/field.h > +++ b/sql/field.h > @@ -5493,6 +5492,22 @@ class Column_definition: public Sql_alloc, > { return compression_method_ptr; } > > bool check_vcol_for_key(THD *thd) const; > + > + void set_lex_collation(const Lex_collation_st &lc) > + { > + charset= lc.collation(); > + if (lc.is_contextually_typed_collation()) > + flags|= BINCMP_FLAG; > + else > + flags&= ~BINCMP_FLAG; Isn't COLLATE DEFAULT also contextually typed? > + } > + Lex_collation lex_collation() const > + { > + return Lex_collation(charset, > + flags & BINCMP_FLAG ? > + Lex_collation_st::TYPE_CONTEXTUALLY_TYPED : > + Lex_collation_st::TYPE_IMPLICIT); > + } > }; > > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 2

Re: [Maria-developers] 75aa95ada6b: Tricky static asserts did not compile on some platforms. Commenting out.
by Sergei Golubchik 16 Mar '22

16 Mar '22

Hi, Alexander, On Mar 16, Alexander Barkov wrote: > revision-id: 75aa95ada6b (mariadb-10.6.1-338-g75aa95ada6b) > parent(s): e90062c20f1 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2022-03-09 13:07:35 +0400 > message: > > Tricky static asserts did not compile on some platforms. Commenting out. Details? What doesn't compile and where? I'd suggest to try to keep the assert in some form > diff --git a/sql/structs.h b/sql/structs.h > index abfafb635b1..23958490c7b 100644 > --- a/sql/structs.h > +++ b/sql/structs.h > @@ -602,13 +602,16 @@ class DDL_options: public DDL_options_st > > struct Lex_length_and_dec_st > { > + /* > + Does not compile on some platforms: > static constexpr uint collation_type_bits= 2; > static_assert(((1<<collation_type_bits)-1) >= > Lex_charset_collation_st::TYPE_LAST); > + */ > protected: > uint32 m_length; > uint8 m_dec; > - uint8 m_collation_type:collation_type_bits; > + uint8 m_collation_type:2; /*collation_type_bits;*/ > bool m_has_explicit_length:1; > bool m_has_explicit_dec:1; > public: Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] e90062c20f1: MDEV-27266 Improve UCA collation performance for utf8mb3 and utf8mb4
by Sergei Golubchik 16 Mar '22

16 Mar '22

Hi, Alexander, also ok On Mar 14, Alexander Barkov wrote: > revision-id: e90062c20f1 (mariadb-10.6.1-337-ge90062c20f1) > parent(s): 6a3201d3769 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2022-02-28 14:43:43 +0400 > message: > > MDEV-27266 Improve UCA collation performance for utf8mb3 and utf8mb4 > > Adding two levels of optimization: > > include/m_ctype.h | 53 ++++ > strings/ctype-uca-scanner_next.inl | 39 +++ > strings/ctype-uca.c | 548 ++++++++++++++++++++++++++++++++++++- > strings/ctype-uca.ic | 30 ++ I thought all *.ic files were renamed to *.inl ? > unittest/strings/strings-t.c | 2 +- > 5 files changed, 660 insertions(+), 12 deletions(-) Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 6a3201d3769: MDEV-27265 Improve contraction performance in UCA collations
by Sergei Golubchik 16 Mar '22

16 Mar '22

Hi, Alexander, that ok, up to you. I'm not looking into the strings/ctype*c code. On Mar 14, Alexander Barkov wrote: > revision-id: 6a3201d3769 (mariadb-10.6.1-336-g6a3201d3769) > parent(s): 49ecf935415 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2022-02-28 14:05:08 +0400 > message: > > MDEV-27265 Improve contraction performance in UCA collations > > Adding a hash table for contractions. > > The old code iterated through all items in MY_CONTRACTIONS, > and was much slower, especially for those contractions > in the end of the list. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] c67789f63c8: MDEV-27009 Add UCA-14.0.0 collations - adding version aware implicit weight handling
by Sergei Golubchik 14 Mar '22

14 Mar '22

Hi, Alexander, this is ok too, thanks On Mar 14, Alexander Barkov wrote: > revision-id: c67789f63c8 (mariadb-10.6.1-334-gc67789f63c8) > parent(s): c1010fdfac2 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2022-02-28 10:31:31 +0400 > message: > > MDEV-27009 Add UCA-14.0.0 collations - adding version aware implicit weight handling > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] c1010fdfac2: MDEV-27009 Add UCA-14.0.0 collations - dump logical positions and contractions
by Sergei Golubchik 14 Mar '22

14 Mar '22

Hi, Alexander, this is ok too On Mar 14, Alexander Barkov wrote: > revision-id: c1010fdfac2 (mariadb-10.6.1-333-gc1010fdfac2) > parent(s): ab62059bbd4 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2022-02-28 10:31:31 +0400 > message: > > MDEV-27009 Add UCA-14.0.0 collations - dump logical positions and contractions > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] ab62059bbd4: MDEV-27009 Add UCA-14.0.0 collations - Adding implicit weight handling for Unicode-14.0.0
by Sergei Golubchik 14 Mar '22

14 Mar '22

Hi, Alexander, I believe it's ok too, but see a couple of questions below On Mar 14, Alexander Barkov wrote: > revision-id: ab62059bbd4 (mariadb-10.6.1-332-gab62059bbd4) > parent(s): 5fe33342399 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2022-02-28 10:31:06 +0400 > message: > > MDEV-27009 Add UCA-14.0.0 collations - Adding implicit weight handling for Unicode-14.0.0 > > diff --git a/strings/ctype-uca.c b/strings/ctype-uca.c > index ad0b46ea296..eb6bc2eb9e6 100644 > --- a/strings/ctype-uca.c > +++ b/strings/ctype-uca.c > @@ -31694,7 +31694,7 @@ static inline void > my_uca_implicit_weight_put(uint16 *to, my_wc_t code, uint level) > { > MY_UCA_IMPLICIT_WEIGHT weight; > - weight= my_uca_520_implicit_weight_on_level(code, level); > + weight= my_uca_implicit_weight_on_level(520, code, level); will the version always be a constant in every invocation of my_uca_implicit_weight_put() ? > to[0]= weight.weight[0]; > to[1]= weight.weight[1]; > to[2]= 0; > diff --git a/strings/ctype-uca.h b/strings/ctype-uca.h > index 01f655b06cb..d439b8c578c 100644 > --- a/strings/ctype-uca.h > +++ b/strings/ctype-uca.h > @@ -136,11 +149,11 @@ my_uca_implicit_weight_quaternary() > > > static inline MY_UCA_IMPLICIT_WEIGHT > -my_uca_520_implicit_weight_on_level(my_wc_t code, uint level) > +my_uca_implicit_weight_on_level(uint version, my_wc_t code, uint level) > { > switch (level) { > case 0: > - return my_uca_520_implicit_weight_primary(code); > + return my_uca_implicit_weight_primary(version, code); > case 1: > return my_uca_implicit_weight_secondary(); > case 2: do you mean that secondary/tertiary/etc weights do not depend on the version? Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 5fe33342399: MDEV-27009 Add UCA-14.0.0 collations - adding uca-dump into build targets
by Sergei Golubchik 14 Mar '22

14 Mar '22

Hi, Alexander, this is ok (not ok to push as is just yet, please wait until we agree on commits in the branch) On Mar 14, Alexander Barkov wrote: > revision-id: 5fe33342399 (mariadb-10.6.1-331-g5fe33342399) > parent(s): 2467eb2d314 > author: Alexander Barkov > committer: Alexander Barkov > timestamp: 2022-02-10 11:53:47 +0400 > message: > > MDEV-27009 Add UCA-14.0.0 collations - adding uca-dump into build targets > > - Adding uca-dump into build targets > - Adding ctype-uca.h and moving implicit weight related routines there > - Reusing implicit weight routines in ctype-uca.c and uca-dump.c > - Adding handling of command line arguments to uca-dump > - Fixing some compile-time warnings in uca-dump.c > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 61b72ed3dde: MDEV-27712 Reduce the size of Lex_length_and_dec_st from 16 to 8
by Sergei Golubchik 11 Mar '22

11 Mar '22

Hi, Alexander, a question about error codes: On Mar 11, Alexander Barkov wrote: > commit 61b72ed3dde > Author: Alexander Barkov <bar(a)mariadb.com> > Date: Wed Feb 9 17:59:38 2022 +0400 > > diff --git a/mysql-test/main/type_blob.test b/mysql-test/main/type_blob.test > index c61ed124139..a5410244ed1 100644 > --- a/mysql-test/main/type_blob.test > +++ b/mysql-test/main/type_blob.test > @@ -517,7 +517,7 @@ CREATE TABLE b15776 (a char(2147483648)); > --error ER_TOO_BIG_FIELDLENGTH > CREATE TABLE b15776 (a char(4294967295)); > # Even BLOB won't hold > ---error ER_TOO_BIG_FIELDLENGTH > +--error ER_NOT_SUPPORTED_YET Why did you change the error? > CREATE TABLE b15776 (a char(4294967296)); > > > diff --git a/mysql-test/main/type_float.result b/mysql-test/main/type_float.result > index 5137a8229b6..9e216711ddd 100644 > --- a/mysql-test/main/type_float.result > +++ b/mysql-test/main/type_float.result > @@ -1162,3 +1162,36 @@ fdec 123.456.789,12345678900000000000000000000000000000 > # > # End of 10.4 tests > # > +# > +# Start of 10.9 tests > +# > +# > +# MDEV-27712 Reduce the size of Lex_length_and_dec_st from 16 to 8 > +# > +CREATE TABLE t1 (a DOUBLE(1000,1000)); > +ERROR 42000: This version of MariaDB doesn't yet support 'scale>255' > +CREATE TABLE t1 (a DOUBLE(1000,0)); > +ERROR 42000: Display width out of range for 'a' (max = 255) > +CREATE TABLE t1 (a DOUBLE(0,1000)); > +ERROR 42000: This version of MariaDB doesn't yet support 'scale>255' > +CREATE TABLE t1 (a DOUBLE(2147483647,2147483647)); > +ERROR 42000: This version of MariaDB doesn't yet support 'scale>255' > +CREATE TABLE t1 (a DOUBLE(2147483647,0)); > +ERROR 42000: Display width out of range for 'a' (max = 255) > +CREATE TABLE t1 (a DOUBLE(0,2147483647)); > +ERROR 42000: This version of MariaDB doesn't yet support 'scale>255' > +CREATE TABLE t1 (a DOUBLE(2147483648,2147483648)); > +ERROR 42000: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '2147483648,2147483648))' at line 1 This isn't technically a *syntax* error Can you use a different error here? > +CREATE TABLE t1 (a DOUBLE(2147483648,0)); > +ERROR 42000: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '2147483648,0))' at line 1 > +CREATE TABLE t1 (a DOUBLE(0,2147483648)); > +ERROR 42000: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '2147483648))' at line 1 > +CREATE TABLE t1 (a DOUBLE(999999999999999999999999999999,999999999999999999999999999999)); > +ERROR 42000: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '999999999999999999999999999999,999999999999999999999999999999))' at line 1 > +CREATE TABLE t1 (a DOUBLE(999999999999999999999999999999,0)); > +ERROR 42000: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '999999999999999999999999999999,0))' at line 1 > +CREATE TABLE t1 (a DOUBLE(0,999999999999999999999999999999)); > +ERROR 42000: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '999999999999999999999999999999))' at line 1 > +# > +# End of 10.9 tests > +# > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index 5bb03544762..fdc54f7df27 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -5975,11 +5977,10 @@ field_type_numeric: > | FLOAT_SYM float_options last_field_options > { > $$.set(&type_handler_float, $2); > - if ($2.length() && !$2.dec()) > + if ($2.has_explicit_length() && !$2.has_explicit_dec()) > { > - int err; > - ulonglong tmp_length= my_strtoll10($2.length(), NULL, &err); > - if (unlikely(err || tmp_length > PRECISION_FOR_DOUBLE)) > + ulonglong tmp_length= $2.length(); you don't really need tmp_length here anymore > + if (unlikely(tmp_length > PRECISION_FOR_DOUBLE)) > my_yyabort_error((ER_WRONG_FIELD_SPEC, MYF(0), > Lex->last_field->field_name.str)); > if (tmp_length > PRECISION_FOR_FLOAT) Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 0402f6dcb67: MDEV-25477 Auto-create breaks replication when triggering event was not replicated
by Sergei Golubchik 07 Mar '22

07 Mar '22

Hi, Aleksey, This was quite good, thanks! Many cases covered, not just ROLLBACK, but also LIMIT 0 and impossible WHERE. See a couple of comments below. Nothing big. On Feb 12, Aleksey Midenkov wrote: > commit 0402f6dcb67 > Author: Aleksey Midenkov <midenok(a)gmail.com> > Date: Tue Feb 8 22:54:03 2022 +0300 > > diff --git a/sql/sql_delete.cc b/sql/sql_delete.cc > index 0403d6e73c3..06068290247 100644 > --- a/sql/sql_delete.cc > +++ b/sql/sql_delete.cc > @@ -507,6 +508,18 @@ bool mysql_delete(THD *thd, TABLE_LIST *table_list, COND *conds, > if (thd->lex->describe || thd->lex->analyze_stmt) > goto produce_explain_and_leave; > > + if (thd->log_current_statement) > + { > + thd->reset_unsafe_warnings(); > + if (thd->binlog_query(THD::STMT_QUERY_TYPE, > + thd->query(), thd->query_length(), > + transactional_table, FALSE, FALSE, 0) > 0) > + { > + my_error(ER_ERROR_ON_WRITE, MYF(0), "binary log", -1); > + DBUG_RETURN(1); > + } > + } perhaps, move this block into a function? It's repeated four times here. by the way, you don't seem to have a test for a zero-rows insert. That is INSERT ... SELECT with an impossible where or limit 0. In fact, you can have a zero-row insert with a normal insert INSERT t1 VALUES (1); if this (1) is a unique key violation. I believe your code handles this case, thought, just a test case is missing. > + > my_ok(thd, 0); > DBUG_RETURN(0); > } > @@ -932,8 +958,8 @@ bool mysql_delete(THD *thd, TABLE_LIST *table_list, COND *conds, > else > errcode= query_error_code(thd, killed_status == NOT_KILLED); > > - ScopedStatementReplication scoped_stmt_rpl( > - table->versioned(VERS_TRX_ID) ? thd : NULL); > + StatementBinlog stmt_binlog(thd, table->versioned(VERS_TRX_ID) || > + thd->binlog_need_stmt_format(transactional_table)); I know that's not part of this patch, but still. Just trying to understand. Why did it change to statement-based if VERS_TRX_ID? > /* > [binlog]: If 'handler::delete_all_rows()' was called and the > storage engine does not inject the rows itself, we replicate Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 3

Re: [Maria-developers] 880e92a48ba: MDEV-27760 event may non stop replicate in circular semisync setup
by Sergei Golubchik 07 Mar '22

07 Mar '22

Hi, Andrei, I don't see a point in caching the value in memory that is very cheap to calculate per event. I've pushed f8ec4dd38d0 to 10.4-serg three hours ago and Jenkins said it's fine. So I'd suggest we go with it, because we really have to close this blocker and release ES now. And MDEV-27837 will be fixed as a separate bug. queue_event checks global_system_variables.server_id in multiple places, moving only do_accept_own_server_id into Master_info will not make it any better or worse against MDEV-27837. > > As far as I can see, you can calculate it for every event just the same. > > The per event computation is extraneous and not consistent with the > nature of the flag that belongs to the group of events. > > Also notice (or remember) a "FR" > MDEV-27837 disallow `set @@session.server_id` within transaction > so until this bug is fixed it's not 'just the same'. > > I am recomming the patch, assuming that you'll be fine with the Gtid time > only flag computation. > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 880e92a48ba: MDEV-27760 event may non stop replicate in circular semisync setup
by Sergei Golubchik 06 Mar '22

06 Mar '22

Hi, Andrei, On Mar 01, Andrei Elkin wrote: > > On Feb 28, Andrei wrote: > >> revision-id: 880e92a48ba (mariadb-10.6.6-16-g880e92a48ba) > >> parent(s): 4030a9fb2eb > >> author: Andrei > >> committer: Andrei > >> timestamp: 2022-02-21 14:58:13 +0200 > >> message: > >> > >> MDEV-27760 event may non stop replicate in circular semisync setup > >> > >> diff --git a/sql/slave.cc b/sql/slave.cc > >> index c0eef02ca7a..a1f4f4f0081 100644 > >> --- a/sql/slave.cc > >> +++ b/sql/slave.cc > >> @@ -6186,13 +6186,13 @@ static int queue_event(Master_info* mi, const uchar *buf, ulong event_len) > >> bool is_rows_event= false; > >> /* > >> The flag has replicate_same_server_id semantics and is raised to accept > >> - a same-server-id event on the semisync slave, for both the gtid and legacy > >> - connection modes. > >> - Such events can appear as result of this server recovery so the event > >> - was created there and replicated elsewhere right before the crash. At recovery > >> - it could be evicted from the server's binlog. > >> - */ > >> - bool do_accept_own_server_id= false; > >> + a same-server-id event group by the gtid strict mode semisync slave. > >> + Own server-id events can appear as result of this server crash-recovery: > >> + the transaction was created on this server then being master, got replicated > >> + elsewhere right before the crash before commit; > >> + finally at recovery the transaction gets evicted from the server's binlog. > >> + */ > >> + static bool do_accept_own_server_id= false; > > > > No, sorry, it cannot be static. If you need to preserve it between > > events, you can keep it, for example, in Master_info. > > The var is set by Gtid events to serve as a marker of the whole current Gtid-headed > group of event. > Only next Gtid event may change the current value. > > Why can't it? Why it cannot be static? Because it's not a property of the server, it's a property of a specific replication stream. Technically one can have a multi-source and do_accept_own_server_id could have different values for different masters. > > So, the question now is, why do you need to preserve the value of > > do_accept_own_server_id between queue_event invocations? > > The flag as said above applies to the whole group of events, so it's > preserved at least through so many > queue_event() calls as many event in the group (I count Gtid itself in). As far as I can see, you can calculate it for every event just the same. > >> @@ -6281,6 +6281,8 @@ static int queue_event(Master_info* mi, const uchar *buf, ulong event_len) > >> dbug_rows_event_count = 0; > >> };); > >> #endif > >> + s_id= uint4korr(buf + SERVER_ID_OFFSET); > >> + > >> mysql_mutex_lock(&mi->data_lock); > >> > >> switch (buf[EVENT_TYPE_OFFSET]) { > >> @@ -6722,6 +6724,11 @@ static int queue_event(Master_info* mi, const uchar *buf, ulong event_len) > >> > >> ++mi->events_queued_since_last_gtid; > >> inc_pos= event_len; > >> + > >> + do_accept_own_server_id= > >> + (s_id == global_system_variables.server_id && rpl_semi_sync_slave_enabled > >> + && mi->using_gtid != Master_info::USE_GTID_NO && opt_gtid_strict_mode) ? > >> + true : false; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] MDEV-27776 EXPLAIN is about to get new columns, EXPLAIN EXTENDED|PARTITIONS will be deprecated
by Oleg Smirnov 03 Mar '22

03 Mar '22

* EXPLAIN SELECT prints "partitions", "filtered" columns and produces the warning with rewritten query * EXPLAIN EXTENDED and EXPLAIN PARTITIONS produce the same output as EXPLAIN SELECT and two warnings, the second warning is about EXPLAIN EXTENDED| PARTITIONS being deprecated (same as in MySQL 5.7) -- Best regards, ------------------------- *Oleg Smirnov* Sr. Software Engineer MariaDB Server Team

1 0

Re: [Maria-developers] 5d8838a79d5: MDEV-24920: Merge "old" SQL variable to "old_mode" sql variable
by Sergei Golubchik 02 Mar '22

02 Mar '22

Hi, Rucha, On Mar 02, Rucha Deodhar wrote: > revision-id: 5d8838a79d5 (mariadb-10.6.1-352-g5d8838a79d5) > parent(s): f8b3c661231 > author: Rucha Deodhar > committer: Rucha Deodhar > timestamp: 2022-02-15 22:29:54 +0530 > message: > > MDEV-24920: Merge "old" SQL variable to "old_mode" sql variable > > Analysis: There are 2 server variables- "old_mode" and "old". "old" is no > longer needed as "old_mode" has replaced it (however still used in some places > in the code). "old_mode" and "old" has same purpose- emulate behavior from > previous MariaDB versions. So they can be merged to avoid confusion. > Fix: Deprecate "old" variable and create another mode for @@old_mode to mimic > behavior of previous "old" variable. Create specific modes for specifix task > that --old sql variable was doing earlier and use the new modes instead. > > diff --git a/mysql-test/main/mysqld--help.result b/mysql-test/main/mysqld--help.result > index 7cbfa52b846..261dbe2ffe0 100644 > --- a/mysql-test/main/mysqld--help.result > +++ b/mysql-test/main/mysqld--help.result > @@ -683,7 +683,8 @@ The following specify which files/extra groups are read (specified before remain > --old-mode=name Used to emulate old behavior from earlier MariaDB or > MySQL versions. Any combination of: > NO_DUP_KEY_WARNINGS_WITH_IGNORE, NO_PROGRESS_INFO, > - ZERO_DATE_TIME_CAST, UTF8_IS_UTF8MB3 > + ZERO_DATE_TIME_CAST, UTF8_IS_UTF8MB3, > + INDEX_HINT_MASK_JOIN, CHECKSUM_SLOW_NULLS I'm sorry, I still cannot understand what these new modes do :( I know that it was my idea to use "CHECKSUM_SLOW_NULLS", but now I'm, like, "what does it mean?". Let's discuss it on slack INDEX_HINT_MASK_JOIN - this seems to be easier, what about INGORE_INDEX_ONLY_FOR_JOIN ? That's a bit long, but shorter than NO_DUP_KEY_WARNINGS_WITH_IGNORE. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 880e92a48ba: MDEV-27760 event may non stop replicate in circular semisync setup
by Sergei Golubchik 01 Mar '22

01 Mar '22

Hi, Andrei, On Feb 28, Andrei wrote: > revision-id: 880e92a48ba (mariadb-10.6.6-16-g880e92a48ba) > parent(s): 4030a9fb2eb > author: Andrei > committer: Andrei > timestamp: 2022-02-21 14:58:13 +0200 > message: > > MDEV-27760 event may non stop replicate in circular semisync setup > > diff --git a/sql/slave.cc b/sql/slave.cc > index c0eef02ca7a..a1f4f4f0081 100644 > --- a/sql/slave.cc > +++ b/sql/slave.cc > @@ -6186,13 +6186,13 @@ static int queue_event(Master_info* mi, const uchar *buf, ulong event_len) > bool is_rows_event= false; > /* > The flag has replicate_same_server_id semantics and is raised to accept > - a same-server-id event on the semisync slave, for both the gtid and legacy > - connection modes. > - Such events can appear as result of this server recovery so the event > - was created there and replicated elsewhere right before the crash. At recovery > - it could be evicted from the server's binlog. > - */ > - bool do_accept_own_server_id= false; > + a same-server-id event group by the gtid strict mode semisync slave. > + Own server-id events can appear as result of this server crash-recovery: > + the transaction was created on this server then being master, got replicated > + elsewhere right before the crash before commit; > + finally at recovery the transaction gets evicted from the server's binlog. > + */ > + static bool do_accept_own_server_id= false; No, sorry, it cannot be static. If you need to preserve it between events, you can keep it, for example, in Master_info. yes, I remember, that's where it was and I asked why it's in Master_info. You could've answered, that you need to preserve the value between queue_event invocations - I didn't realize that at the time. So, the question now is, why do you need to preserve the value of do_accept_own_server_id between queue_event invocations? > /* > FD_q must have been prepared for the first R_a event > inside get_master_version_and_clock() > @@ -6281,6 +6281,8 @@ static int queue_event(Master_info* mi, const uchar *buf, ulong event_len) > dbug_rows_event_count = 0; > };); > #endif > + s_id= uint4korr(buf + SERVER_ID_OFFSET); > + > mysql_mutex_lock(&mi->data_lock); > > switch (buf[EVENT_TYPE_OFFSET]) { > @@ -6722,6 +6724,11 @@ static int queue_event(Master_info* mi, const uchar *buf, ulong event_len) > > ++mi->events_queued_since_last_gtid; > inc_pos= event_len; > + > + do_accept_own_server_id= > + (s_id == global_system_variables.server_id && rpl_semi_sync_slave_enabled > + && mi->using_gtid != Master_info::USE_GTID_NO && opt_gtid_strict_mode) ? > + true : false; this is a ridiculous pattern bool var = (bool expr) ? true : false; don't do that. > } > break; > /* > @@ -6909,7 +6916,6 @@ static int queue_event(Master_info* mi, const uchar *buf, ulong event_len) > */ > > mysql_mutex_lock(log_lock); > - s_id= uint4korr(buf + SERVER_ID_OFFSET); > /* > Write the event to the relay log, unless we reconnected in the middle > of an event group and now need to skip the initial part of the group that > @@ -6955,7 +6961,7 @@ static int queue_event(Master_info* mi, const uchar *buf, ulong event_len) > else > if ((s_id == global_system_variables.server_id && > !(mi->rli.replicate_same_server_id || > - (do_accept_own_server_id= rpl_semi_sync_slave_enabled))) || > + do_accept_own_server_id)) || > event_that_should_be_ignored(buf) || > /* > the following conjunction deals with IGNORE_SERVER_IDS, if set > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] MDEV-16329 ALTER ONLINE TABLE
by Sergei Golubchik 27 Feb '22

27 Feb '22

Hi, Nikita, this is a review of the combined diff of your branch. Of `git diff github/10.8...github/bb-10.8-online-alter`, that is of cf7cc376bac fix savepoints in myisam 4a128ca5458 fix main.delayed 7300151cc84 fix skipping rocksdb 19504dfdbcc add binlog/standalone combinations 55e993595aa add rocksdb combination 06645b17450 savepoints 9ffcd13b9fe rename tests 730926ae962 MDEV-16329 [5/5] ALTER ONLINE TABLE cf52ea33a70 MDEV-16329 [4/5] Refactor MYSQL_BIN_LOG: extract Event_log ancestor 0445c8f1ee9 MDEV-16329 [3/5] use binlog_cache_data directly in most places 3f3da0f65a3 MDEV-16329 [2/5] refactor binlog and cache_mngr dc79667ca73 MDEV-16329 [1/5] add THD::binlog_get_cache_mngr 6f9dd00ce2d rpl: repack table_def faf8680df8d Copy_field: add const to arguments It was mostly fine the last time already, so only few comments below: > diff --git a/mysql-test/include/have_log_bin_off.require b/mysql-test/include/have_log_bin_off.require > new file mode 100644 > index 00000000000..979dbe75f80 > --- /dev/null > +++ b/mysql-test/include/have_log_bin_off.require > @@ -0,0 +1,2 @@ > +Variable_name Value > +log_bin OFF better use `if (...) { skip }` pattern *.require is an obsolete 20-year old technique from before if- and skip-times > diff --git a/mysql-test/main/alter_table.test b/mysql-test/main/alter_table.test > index 31c69783248..91423ee8c1f 100644 > --- a/mysql-test/main/alter_table.test > +++ b/mysql-test/main/alter_table.test > @@ -1532,8 +1532,12 @@ ALTER TABLE t1 DROP INDEX i1, DROP INDEX i2, DROP INDEX i3, DROP INDEX i4; > ALTER TABLE t1 ADD INDEX i1(b), ALGORITHM= INPLACE, LOCK= NONE; > ALTER TABLE t1 ADD INDEX i2(b), ALGORITHM= INPLACE, LOCK= SHARED; > ALTER TABLE t1 ADD INDEX i3(b), ALGORITHM= INPLACE, LOCK= EXCLUSIVE; > ---error ER_ALTER_OPERATION_NOT_SUPPORTED_REASON > +--disable_info > +--disable_warnings > +--error 0,ER_ALTER_OPERATION_NOT_SUPPORTED_REASON why, because of embedded? > ALTER TABLE t1 ADD INDEX i4(b), ALGORITHM= COPY, LOCK= NONE; > +--enable_warnings > +--enable_info > ALTER TABLE t1 ADD INDEX i5(b), ALGORITHM= COPY, LOCK= SHARED; > ALTER TABLE t1 ADD INDEX i6(b), ALGORITHM= COPY, LOCK= EXCLUSIVE; > > diff --git a/mysql-test/main/alter_table_locknone.result b/mysql-test/main/alter_table_locknone.result > new file mode 100644 > index 00000000000..ea040925efe > --- /dev/null > +++ b/mysql-test/main/alter_table_locknone.result > @@ -0,0 +1,270 @@ > +create table t1 (a int not null primary key, b int, c varchar(80), e enum('a','b')) engine=myisam; > +insert into t1 (a) values (1),(2),(3); > +alter online table t1 modify b int default 5, alter c set default 'X'; > +alter online table t1 change b new_name int; > +alter online table t1 modify e enum('a','b','c'); > +alter online table t1 comment "new comment"; > +alter table t1 add constraint q check (a > 0); > +alter online table t1 drop constraint q; > +alter online table t1 algorithm=INPLACE, lock=NONE; > +alter online table t1; > +alter table t1 algorithm=INPLACE; > +alter table t1 lock=NONE; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` int(11) NOT NULL, > + `new_name` int(11) DEFAULT NULL, > + `c` varchar(80) DEFAULT 'X', > + `e` enum('a','b','c') DEFAULT NULL, > + PRIMARY KEY (`a`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 COMMENT='new comment' > +drop table t1; > +create temporary table t1 (a int not null primary key, b int, c varchar(80), e enum('a','b')); > +insert into t1 (a) values (1),(2),(3); > +alter online table t1 modify b int default 5, alter c set default 'X'; > +alter online table t1 change b new_name int; > +alter online table t1 modify e enum('a','b','c'); > +alter online table t1 comment "new comment"; > +alter online table t1 rename to t2; > +show create table t2; > +Table Create Table > +t2 CREATE TEMPORARY TABLE `t2` ( > + `a` int(11) NOT NULL, > + `new_name` int(11) DEFAULT NULL, > + `c` varchar(80) DEFAULT 'X', > + `e` enum('a','b','c') DEFAULT NULL, > + PRIMARY KEY (`a`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 COMMENT='new comment' > +drop table t2; > +create table t1 (a int not null primary key, b int, c varchar(80), e enum('a','b')) engine=aria; > +insert into t1 (a) values (1),(2),(3); > +alter online table t1 modify b int default 5; > +alter online table t1 change b new_name int; > +alter online table t1 modify e enum('a','b','c'); > +alter online table t1 comment "new comment"; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` int(11) NOT NULL, > + `new_name` int(11) DEFAULT NULL, > + `c` varchar(80) DEFAULT NULL, > + `e` enum('a','b','c') DEFAULT NULL, > + PRIMARY KEY (`a`) > +) ENGINE=Aria DEFAULT CHARSET=latin1 PAGE_CHECKSUM=1 COMMENT='new comment' > +alter online table t1 page_checksum=1; > +alter online table t1 page_checksum=0; > +ERROR 0A000: LOCK=NONE is not supported for this operation. Try LOCK=SHARED Why LOCK=NONE would be not supported now? Shouldn't your new feature allow just everything with LOCK=NONE? > +drop table t1; > +create table t1 (a int not null primary key, b int, c varchar(80), e enum('a','b')); > +insert into t1 (a) values (1),(2),(3); > +alter online table t1 drop column b, add b int; > +ERROR 0A000: LOCK=NONE is not supported for this operation. Try LOCK=SHARED ... > diff --git a/mysql-test/main/alter_table_online.combinations b/mysql-test/main/alter_table_online.combinations > new file mode 100644 > index 00000000000..ae144432c68 > --- /dev/null > +++ b/mysql-test/main/alter_table_online.combinations > @@ -0,0 +1,2 @@ > +[innodb] > +[rocksdb] what about non-trans tables? > diff --git a/mysql-test/main/alter_table_online.test b/mysql-test/main/alter_table_online.test > index 6ef9661c43b..815483d252d 100644 > --- a/mysql-test/main/alter_table_online.test > +++ b/mysql-test/main/alter_table_online.test > @@ -1,335 +1,596 @@ > -# > -# Test of ALTER ONLINE TABLE syntax > -# > - > ---source include/have_innodb.inc > ---source include/have_partition.inc > -# > -# Test of things that can be done online > -# > - > -create table t1 (a int not null primary key, b int, c varchar(80), e enum('a','b')) engine=myisam; > -insert into t1 (a) values (1),(2),(3); > - > -alter online table t1 modify b int default 5, alter c set default 'X'; > -alter online table t1 change b new_name int; > -alter online table t1 modify e enum('a','b','c'); > -alter online table t1 comment "new comment"; > -alter table t1 add constraint q check (a > 0); > -alter online table t1 drop constraint q; > - > -# No OPs > - > -alter online table t1 algorithm=INPLACE, lock=NONE; > -alter online table t1; > -alter table t1 algorithm=INPLACE; > -alter table t1 lock=NONE; > -show create table t1; > -drop table t1; > +-- source include/have_debug_sync.inc > +-- source include/not_embedded.inc > +-- source alter_table_online_binlog.inc > > -# > -# everything with temporary tables is "online", i.e. without locks > -# > -create temporary table t1 (a int not null primary key, b int, c varchar(80), e enum('a','b')); > -insert into t1 (a) values (1),(2),(3); > - > -alter online table t1 modify b int default 5, alter c set default 'X'; > -alter online table t1 change b new_name int; > -alter online table t1 modify e enum('a','b','c'); > -alter online table t1 comment "new comment"; > -alter online table t1 rename to t2; > -show create table t2; > -drop table t2; > +-- disable_query_log > +-- if ($MTR_COMBINATION_INNODB) { > +-- source include/have_innodb.inc > +set default_storage_engine= innodb; > +-- } > +-- if ($MTR_COMBINATION_ROCKSDB) { > +-- source include/have_rocksdb.inc > +set default_storage_engine= rocksdb; > +-- } > +-- enable_query_log > + > +-- let $default_engine= `select @@default_storage_engine` > + > +--connect (con2, localhost, root,,) > +--connection default > + > + > +--echo # > +--echo # Test insert > +--echo # > + > +--echo # Insert and add column > +create or replace table t1 (a int) engine=innodb; why do you specify the engine explicitly if you run it in two engine combinations? > +insert t1 values (5); > + > +--connection con2 > +--send > +set debug_sync= 'now WAIT_FOR ended'; > + > +--connection default > +set debug_sync= 'alter_table_copy_end SIGNAL ended WAIT_FOR end'; > + ... > + > +--connection default > +--reap > +--sorted_result > +select * from t1; > + > +--echo # > +--echo # MYISAM. Only Inserts can be tested. why? > +--echo # > + > +create or replace table t1 (a int) engine=myisam; > +insert t1 values (5); > + > +--connection con2 > +--send > +set debug_sync= 'now WAIT_FOR ended'; > + > +--connection default > +set debug_sync= 'alter_table_copy_end SIGNAL ended WAIT_FOR end'; > + > +--send > +alter table t1 add b int NULL, algorithm= copy, lock= none; > + ... > diff --git a/mysql-test/main/mdl_sync.test b/mysql-test/main/mdl_sync.test > index 3df19aca806..8c6bc76f5ca 100644 > --- a/mysql-test/main/mdl_sync.test > +++ b/mysql-test/main/mdl_sync.test > @@ -42,7 +42,7 @@ connection con1; > set debug_sync='mdl_upgrade_lock SIGNAL parked WAIT_FOR go'; > --send alter table t1 rename t3 > > -connection default; > + connection default; eh? (here and many other changes in this file) > --echo connection: default > set debug_sync= 'now WAIT_FOR parked'; > > diff --git a/mysql-test/suite/gcol/inc/gcol_select.inc b/mysql-test/suite/gcol/inc/gcol_select.inc > index 2386c55fdbc..8f6314233bd 100644 > --- a/mysql-test/suite/gcol/inc/gcol_select.inc > +++ b/mysql-test/suite/gcol/inc/gcol_select.inc > @@ -872,7 +872,9 @@ CREATE TABLE t1(a INT); > INSERT INTO t1 VALUES(2147483647); > ALTER TABLE t1 ADD COLUMN h INT AS (a) VIRTUAL; > ALTER TABLE t1 CHANGE h i INT AS (a) VIRTUAL, ALGORITHM=COPY; > +--error ER_WARN_DATA_OUT_OF_RANGE,ER_ALTER_OPERATION_NOT_SUPPORTED_REASON why two? embedded, again? > ALTER TABLE t1 ADD COLUMN b SMALLINT AS (a) VIRTUAL, ALGORITHM=COPY, LOCK=NONE; > +--error ER_WARN_DATA_OUT_OF_RANGE,ER_ALTER_OPERATION_NOT_SUPPORTED_REASON > ALTER TABLE t1 ADD COLUMN e SMALLINT AS (a) VIRTUAL, ALGORITHM=COPY, LOCK=NONE; > ALTER TABLE t1 ADD COLUMN f SMALLINT AS (a) VIRTUAL, ALGORITHM=COPY, LOCK=SHARED; > ALTER TABLE t1 ADD COLUMN g SMALLINT AS (a) VIRTUAL, ALGORITHM=COPY, LOCK=EXCLUSIVE; > diff --git a/sql/handler.h b/sql/handler.h > index fe61666bf20..5e46fa59d7e 100644 > --- a/sql/handler.h > +++ b/sql/handler.h > @@ -3538,6 +3543,7 @@ class handler :public Sql_alloc > /** to be actually called to get 'check()' functionality*/ > int ha_check(THD *thd, HA_CHECK_OPT *check_opt); > int ha_repair(THD* thd, HA_CHECK_OPT* check_opt); > + virtual void open_read_view(){} why did you change to that from start_consistent_snapshot()? > void ha_start_bulk_insert(ha_rows rows, uint flags= 0) > { > DBUG_ENTER("handler::ha_start_bulk_insert"); > diff --git a/sql/log.cc b/sql/log.cc > index 70ceecc66f8..9beab80773c 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -101,6 +102,9 @@ static int binlog_flush_cache(THD *thd, binlog_cache_mngr *cache_mngr, > Log_event *end_ev, bool all, bool using_stmt, > bool using_trx, bool is_ro_1pc); > > +int binlog_online_alter_commit(THD *thd, bool all); > +void binlog_online_alter_rollback(THD *thd, bool all); static? > + > static const LEX_CSTRING write_error_msg= > { STRING_WITH_LEN("error writing to the binary log") }; > > @@ -3565,6 +3613,91 @@ bool MYSQL_BIN_LOG::open_index_file(const char *index_file_name_arg, > } > > > +bool Event_log::open(const char *log_name, > + const char *new_name, ulong next_file_number, > + enum cache_type io_cache_type_arg) > +{ > + bool error= false; > + if (log_name || new_name) > + { > + error= MYSQL_LOG::open( > +#ifdef HAVE_PSI_INTERFACE > + 0, > +#endif > + log_name, LOG_NORMAL, new_name, next_file_number, io_cache_type_arg); > + } > + else > + { > +#ifdef HAVE_PSI_INTERFACE > + /* Keep the key for reopen */ > + m_log_file_key= 0; > +#endif > + error= init_io_cache(&log_file, -1, LOG_BIN_IO_SIZE, > + io_cache_type_arg, 0, 0, > + MYF(MY_WME | MY_NABP | MY_WAIT_IF_FULL)); > + > + log_state= LOG_OPENED; > + inited= true; > + } > + if (error) > + return error; > + > + longlong bytes_written= write_description_event( > + (enum_binlog_checksum_alg)binlog_checksum_options, > + encrypt_binlog, false); > + error= bytes_written < 0; > + return error; > +} > + > +longlong > +Event_log::write_description_event(enum_binlog_checksum_alg checksum_alg, > + bool encrypt, bool dont_set_created) > +{ > + Format_description_log_event s(BINLOG_VERSION); > + /* > + don't set LOG_EVENT_BINLOG_IN_USE_F for SEQ_READ_APPEND io_cache > + as we won't be able to reset it later > + */ > + if (io_cache_type == WRITE_CACHE) this was incorrectly rebased. in the previous patch you *moved* this block of code from MYSQL_BIN_LOG::open into a separate method. here you copied it. you forgot to delete it from MYSQL_BIN_LOG::open(). > + s.flags |= LOG_EVENT_BINLOG_IN_USE_F; > + s.checksum_alg= checksum_alg; > + > + crypto.scheme = 0; > + DBUG_ASSERT(s.checksum_alg != BINLOG_CHECKSUM_ALG_UNDEF); > + if (!s.is_valid()) > + return -1; > + s.dont_set_created= dont_set_created; > + if (write_event(&s, 0, &log_file)) > + return -1; > + > + if (encrypt) > + { > + uint key_version= encryption_key_get_latest_version(ENCRYPTION_KEY_SYSTEM_DATA); > + if (key_version == ENCRYPTION_KEY_VERSION_INVALID) > + { > + sql_print_error("Failed to enable encryption of binary logs"); > + return -1; > + } > + > + if (key_version != ENCRYPTION_KEY_NOT_ENCRYPTED) > + { > + if (my_random_bytes(crypto.nonce, sizeof(crypto.nonce))) > + return -1; > + > + Start_encryption_log_event sele(1, key_version, crypto.nonce); > + sele.checksum_alg= s.checksum_alg; > + if (write_event(&sele, 0, &log_file)) > + return -1; > + > + // Start_encryption_log_event is written, enable the encryption > + if (crypto.init(sele.crypto_scheme, key_version)) > + return -1; > + } > + } > + return (longlong)s.data_written; > +} > + > + > /** > Open a (new) binlog file. > > @@ -7252,6 +7477,160 @@ class CacheWriter: public Log_event_writer > bool first; > }; > > +int cache_copy(IO_CACHE *to, IO_CACHE *from) > +{ > + DBUG_ENTER("cache_copy"); > + if (reinit_io_cache(from, READ_CACHE, 0, 0, 0)) > + DBUG_RETURN(ER_ERROR_ON_WRITE); > + size_t bytes_in_cache= my_b_bytes_in_cache(from); > + > + do > + { > + my_b_write(to, from->read_pos, bytes_in_cache); > + > + from->read_pos += bytes_in_cache; > + bytes_in_cache= my_b_fill(from); > + if (from->error || to->error) > + DBUG_RETURN(ER_ERROR_ON_WRITE); > + } while (bytes_in_cache); > + > + DBUG_RETURN(0); > +} > + > +int binlog_online_alter_commit(THD *thd, bool all) > +{ > + DBUG_ENTER("online_alter_commit"); > + int error= 0; > +#ifdef HAVE_REPLICATION > + > + if (thd->online_alter_cache_list.empty()) > + DBUG_RETURN(0); > + > + bool is_ending_transaction= ending_trans(thd, all); > + > + for (auto &cache_mngr: thd->online_alter_cache_list) > + { > + auto *binlog= cache_mngr.share->online_alter_binlog; > + DBUG_ASSERT(binlog); > + > + error= binlog_flush_pending_rows_event(thd, > + /* > + do not set STMT_END for last event > + to leave table open in altering thd > + */ > + false, > + true, > + binlog, > + is_ending_transaction > + ? &cache_mngr.trx_cache > + : &cache_mngr.stmt_cache); > + if (is_ending_transaction) > + { > + mysql_mutex_lock(binlog->get_log_lock()); > + error= binlog->write_cache(thd, &cache_mngr.trx_cache.cache_log); > + > + mysql_mutex_unlock(binlog->get_log_lock()); > + } > + else > + { > + error= cache_copy(&cache_mngr.trx_cache.cache_log, > + &cache_mngr.stmt_cache.cache_log); Why do you need two caches here? It seems you could have just one trx_cache and if the statement fails you simply truncate it to the beginning of the statement. > + } > + > + if (error) > + { > + my_error(ER_ERROR_ON_WRITE, MYF(ME_ERROR_LOG), > + binlog->get_name(), errno); > + binlog_online_alter_cleanup(thd->online_alter_cache_list, > + is_ending_transaction); > + DBUG_RETURN(error); > + } > + } > + > + binlog_online_alter_cleanup(thd->online_alter_cache_list, > + is_ending_transaction); > + > + for (TABLE *table= thd->open_tables; table; table= table->next) > + { > + table->online_alter_cache= NULL; Why? > + } > +#endif // HAVE_REPLICATION > + DBUG_RETURN(error); > +} > + > +void binlog_online_alter_rollback(THD *thd, bool all) > +{ > +#ifdef HAVE_REPLICATION > + bool is_ending_trans= ending_trans(thd, all); > + > + /* > + This is a crucial moment that we are running through > + thd->online_alter_cache_list, and not through thd->open_tables to cleanup > + stmt cache, though both have it. The reason is that the tables can be closed > + to that moment in case of an error. > + The same reason applies to the fact we don't store cache_mngr in the table > + itself -- because it can happen to be not existing. > + Still in case if tables are left opened > + */ > + binlog_online_alter_cleanup(thd->online_alter_cache_list, is_ending_trans); > +#endif // HAVE_REPLICATION > +} > + > +SAVEPOINT** find_savepoint_in_list(THD *thd, LEX_CSTRING name, > + SAVEPOINT ** const list); > + > +SAVEPOINT* savepoint_add(THD *thd, LEX_CSTRING name, SAVEPOINT **list, > + int (*release_old)(THD*, SAVEPOINT*)); may be in transaction.h? > + > +int online_alter_savepoint_set(THD *thd, LEX_CSTRING name) > +{ > + > + DBUG_ENTER("binlog_online_alter_savepoint"); > +#ifdef HAVE_REPLICATION > + if (thd->online_alter_cache_list.empty()) > + DBUG_RETURN(0); > + > + if (savepoint_alloc_size < sizeof (SAVEPOINT) + sizeof(my_off_t)) > + savepoint_alloc_size= sizeof (SAVEPOINT) + sizeof(my_off_t); > + > + for (auto &cache: thd->online_alter_cache_list) > + { > + if (cache.share->db_type()->savepoint_set == NULL) > + continue; > + > + SAVEPOINT *sv= savepoint_add(thd, name, &cache.sv_list, NULL); wouldn't it be simpler to log SAVEPOINT and ROLLBACK/RELEASE as Query_log_event to the cache? Then you won't need to do any maintenance. > + if(unlikely(sv == NULL)) > + DBUG_RETURN(1); > + my_off_t *pos= (my_off_t*)(sv+1); > + *pos= cache.trx_cache.get_byte_position(); > + > + sv->prev= cache.sv_list; > + cache.sv_list= sv; > + } > +#endif > + DBUG_RETURN(0); > +} > + > +int online_alter_savepoint_rollback(THD *thd, LEX_CSTRING name) > +{ > + DBUG_ENTER("online_alter_savepoint_rollback"); > +#ifdef HAVE_REPLICATION > + for (auto &cache: thd->online_alter_cache_list) > + { > + if (cache.share->db_type()->savepoint_set == NULL) > + continue; > + > + SAVEPOINT **sv= find_savepoint_in_list(thd, name, &cache.sv_list); > + // sv is null if savepoint was set up before online table was modified > + my_off_t pos= *sv ? *(my_off_t*)(*sv+1) : 0; > + > + cache.trx_cache.restore_savepoint(pos); > + } > + > +#endif > + DBUG_RETURN(0); > +} > + > /* > Write the contents of a cache to the binary log. > > diff --git a/sql/rpl_rli.h b/sql/rpl_rli.h > index cc807852bf2..57757ecf3dd 100644 > --- a/sql/rpl_rli.h > +++ b/sql/rpl_rli.h > @@ -900,14 +900,20 @@ struct rpl_group_info > } > } > > - bool get_table_data(TABLE *table_arg, table_def **tabledef_var, TABLE **conv_table_var) const > + bool get_table_data(TABLE *table_arg, table_def **tabledef_var, > + TABLE **conv_table_var, > + const Copy_field *copy[], const Copy_field **copy_end) const > { > DBUG_ASSERT(tabledef_var && conv_table_var); > for (TABLE_LIST *ptr= tables_to_lock ; ptr != NULL ; ptr= ptr->next_global) > if (ptr->table == table_arg) > { > - *tabledef_var= &static_cast<RPL_TABLE_LIST*>(ptr)->m_tabledef; > - *conv_table_var= static_cast<RPL_TABLE_LIST*>(ptr)->m_conv_table; > + auto *rpl_table_list= static_cast<RPL_TABLE_LIST*>(ptr); > + if (rpl_table_list->m_tabledef_valid) when can it be false? > + *tabledef_var= &rpl_table_list->m_tabledef; > + *conv_table_var= rpl_table_list->m_conv_table; > + *copy= rpl_table_list->m_online_alter_copy_fields; > + *copy_end= rpl_table_list->m_online_alter_copy_fields_end; > DBUG_PRINT("debug", ("Fetching table data for table %s.%s:" > " tabledef: %p, conv_table: %p", > table_arg->s->db.str, table_arg->s->table_name.str, > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index b07efb29bba..9dfe2178667 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -9533,6 +9539,19 @@ bool mysql_alter_table(THD *thd, const LEX_CSTRING *new_db, > has been already processed. > */ > table_list->required_type= TABLE_TYPE_NORMAL; > + > + > + if (alter_info->requested_lock == Alter_info::ALTER_TABLE_LOCK_SHARED > + || alter_info->requested_lock == Alter_info::ALTER_TABLE_LOCK_EXCLUSIVE > + || thd->locked_tables_mode == LTM_LOCK_TABLES > + || thd->lex->sql_command == SQLCOM_OPTIMIZE > + || alter_info->algorithm(thd) == Alter_info::ALTER_TABLE_ALGORITHM_NOCOPY) why only NOCOPY? What about INPLACE and INSTANT? the rest of the condition looks fine. > + online= false; > + > + if (online) > + { > + table_list->lock_type= TL_READ; > + } > > DEBUG_SYNC(thd, "alter_table_before_open_tables"); > > @@ -10961,6 +10983,58 @@ bool mysql_trans_commit_alter_copy_data(THD *thd) > DBUG_RETURN(error); > } > > +#ifdef HAVE_REPLICATION > +static int online_alter_read_from_binlog(THD *thd, rpl_group_info *rgi, > + Cache_flip_event_log *log) > +{ > + MEM_ROOT event_mem_root; > + Query_arena backup_arena; > + Query_arena event_arena(&event_mem_root, Query_arena::STMT_INITIALIZED); > + init_sql_alloc(key_memory_gdl, &event_mem_root, > + MEM_ROOT_BLOCK_SIZE, 0, MYF(0)); > + > + int error= 0; > + > + IO_CACHE *log_file= log->flip(); > + > + thd_progress_report(thd, 0, my_b_write_tell(log_file)); > + > + Abort_on_warning_instant_set old_abort_on_warning(thd, 0); > + do > + { > + const auto *descr_event= rgi->rli->relay_log.description_event_for_exec; > + auto *ev= Log_event::read_log_event(log_file, descr_event, false); > + if (!ev) > + break; > + > + ev->thd= thd; > + thd->set_n_backup_active_arena(&event_arena, &backup_arena); > + error= ev->apply_event(rgi); > + thd->restore_active_arena(&event_arena, &backup_arena); > + > + event_arena.free_items(); > + free_root(&event_mem_root, MYF(MY_KEEP_PREALLOC)); > + if (ev != rgi->rli->relay_log.description_event_for_exec) > + delete ev; > + thd_progress_report(thd, my_b_tell(log_file), thd->progress.max_counter); > + DEBUG_SYNC(thd, "alter_table_online_progress"); > + } while(!error); > + > + return error; > +} > +#endif // HAVE_REPLICATION > + > +static void online_alter_cleanup_binlog(THD *thd, TABLE_SHARE *s) > +{ > +#ifdef HAVE_REPLICATION > + if (!s->online_alter_binlog) > + return; > + // s->online_alter_binlog->reset_logs(thd, false, NULL, 0, 0); forgot to delete? > + s->online_alter_binlog->cleanup(); > + s->online_alter_binlog->~Cache_flip_event_log(); > + s->online_alter_binlog= NULL; > +#endif > +} > > static int > copy_data_between_tables(THD *thd, TABLE *from, TABLE *to, > @@ -11306,6 +11431,76 @@ copy_data_between_tables(THD *thd, TABLE *from, TABLE *to, > cleanup_done= 1; > to->file->extra(HA_EXTRA_NO_IGNORE_DUP_KEY); > > +#ifdef HAVE_REPLICATION > + if (likely(online && error < 0)) > + { > + Ha_trx_info *trx_info_save= thd->transaction->all.ha_list; > + thd->transaction->all.ha_list = NULL; why? > + thd_progress_next_stage(thd); > + Table_map_log_event table_event(thd, from, from->s->table_map_id, > + from->file->has_transactions()); > + Relay_log_info rli(false); > + rpl_group_info rgi(&rli); > + RPL_TABLE_LIST rpl_table(to, TL_WRITE, from, table_event.get_table_def(), > + copy, copy_end); > + Cache_flip_event_log *binlog= from->s->online_alter_binlog; > + rgi.thd= thd; > + rgi.tables_to_lock= &rpl_table; > + > + rgi.m_table_map.set_table(from->s->table_map_id, to); > + > + DBUG_ASSERT(binlog->is_open()); > + > + rli.relay_log.description_event_for_exec= > + new Format_description_log_event(4); > + > + // We restore bitmaps, because update event is going to mess up with them. > + to->default_column_bitmaps(); > + > + error= online_alter_read_from_binlog(thd, &rgi, binlog); > + > + DEBUG_SYNC(thd, "alter_table_online_before_lock"); > + > + int lock_error= > + thd->mdl_context.upgrade_shared_lock(from->mdl_ticket, MDL_EXCLUSIVE, > + (double)thd->variables.lock_wait_timeout); > + if (!error) > + error= lock_error; > + > + if (!error) > + { > + thd_progress_next_stage(thd); > + error= online_alter_read_from_binlog(thd, &rgi, binlog); > + } > + > + thd->transaction->all.ha_list = trx_info_save; > + } > + else if (unlikely(online)) // error was on copy stage > + { > + /* > + We need to issue a barrier to clean up gracefully. > + Without this, following possible: > + T1: ALTER TABLE starts > + T2: INSERT starts > + T1: ALTER TABLE fails with error (i.e. ER_DUP_KEY) > + T1: from->s->online_alter_binlog sets to NULL > + T2: INSERT committs > + T2: thd->online_alter_cache_list is not empty > + T2: binlog_commit: DBUG_ASSERT(binlog); is issued. 1. do you have a test for that? 2. can it be fixed, like, on iterating thd->online_alter_cache_list thd notices that from->s->online_alter_cache_list is NULL, so it simply discards the cache? Then you won't need MDL_SHARED_NO_WRITE > + */ > + // Ignore the return result. We already have an error. > + thd->mdl_context.upgrade_shared_lock(from->mdl_ticket, > + MDL_SHARED_NO_WRITE, > + thd->variables.lock_wait_timeout); > + } > +#endif > + > + if (error > 0 && !from->s->tmp_table) > + { > + /* We are going to drop the temporary table */ > + to->file->extra(HA_EXTRA_PREPARE_FOR_DROP); > + } > + > DEBUG_SYNC(thd, "copy_data_between_tables_before_reset_backup_lock"); > if (backup_reset_alter_copy_lock(thd)) > error= 1; Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] MDEV-16329 ALTER ONLINE TABLE
by Sergei Golubchik 21 Feb '22

21 Feb '22

Hi, Nikita, On Dec 13, Nikita Malyavin wrote: > > > -class binlog_cache_mngr { > > > +class binlog_cache_mngr: public ilist_node<> { > > > public: > > > binlog_cache_mngr(my_off_t param_max_binlog_stmt_cache_size, > > > my_off_t param_max_binlog_cache_size, > > > ulong *param_ptr_binlog_stmt_cache_use, > > > ulong *param_ptr_binlog_stmt_cache_disk_use, > > > ulong *param_ptr_binlog_cache_use, > > > - ulong *param_ptr_binlog_cache_disk_use) > > > + ulong *param_ptr_binlog_cache_disk_use, > > > + TABLE_SHARE *share) > > > - : last_commit_pos_offset(0), using_xa(FALSE), xa_xid(0) > > > + : last_commit_pos_offset(0), using_xa(FALSE), xa_xid(0), share(share) > > > > does it work, can one do it? I mean share(share) > > in other places, as far as I remember, arguments always use a distinct > > name, > > like share(share_arg) or share(share_) or something. > > > Yes, this is a language feature for constructor initializers. Okay. We don't use it though and it will stop almost everyone looking at this code, as it's all too easy to assume it's a mistake. So I'd recommend to rename the argument to avoid misunderstanding. Like, to `share_`. But also feel free to keep it your way, it's a valid C++ after all. > > > { > > > stmt_cache.set_binlog_cache_info(param_max_binlog_stmt_cache_size, > > > param_ptr_binlog_stmt_cache_use, > > > @@ -2116,9 +2144,58 @@ int binlog_commit(THD *thd, bool all, bool ro_1pc) ... > > > + else > > > + { > > > + error= cache_copy(&cache_mngr.trx_cache.cache_log, > > > + &cache_mngr.stmt_cache.cache_log); > > > > Why do you need two caches here? It seems you could have just one trx_cache > > and if the statement fails you simply truncate it to the beginning of > > the statement. > > > Didn't think about that, but seems so. Why would binlog need two caches > then? I don't really know. But, for example, I can think of a multi-table update that modifies both InnoDB and MyISAM tables and then fails after updating some rows. It'll be reverted in InnoDB, but not in MyISAM. With two caches InnoDB row events can go into stmt_cache and MyISAM row events can go into trx_cache. And later stmt_cache will be discarded, but MyISAM changes will persist. This multi-table case does not apply to online alter. Are there other cases when a separate stmt_cache might be needed? > > > + binlog_cache_mngr *const cache_mngr= thd->binlog_get_cache_mngr(); > > > + /* > > > + cache_mngr can be NULL in case if binlog logging is disabled. > > > + */ > > > if (!cache_mngr) > > > { > > > DBUG_ASSERT(WSREP(thd) || Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0