- developers - lists.mariadb.org

Re: [Maria-developers] a2520676ba1: MDEV-16249 CHECKSUM TABLE for a spider table is not parallel and saves all data in memory in the spider head by default
by Sergei Golubchik 06 Jun '19

06 Jun '19

Hi, Kentoku! On Jun 05, Kentoku wrote: > revision-id: a2520676ba1 (mariadb-10.4.5-10-ga2520676ba1) > parent(s): 31fe70290c5 > author: Kentoku <kentokushiba(a)gmail.com> > committer: Kentoku <kentokushiba(a)gmail.com> > timestamp: 2019-05-23 14:36:18 +0900 > message: > > MDEV-16249 CHECKSUM TABLE for a spider table is not parallel and saves all data in memory in the spider head by default please remember to add an empty line after the first line in the commit comment, see https://git-scm.com/docs/git-commit#_discussion > add checksum_null for setting null value of checksum > > diff --git a/sql/ha_partition.cc b/sql/ha_partition.cc > index 4e7fccca3af..e47da94ab20 100644 > --- a/sql/ha_partition.cc > +++ b/sql/ha_partition.cc > @@ -8213,7 +8214,11 @@ int ha_partition::info(uint flag) > stats.delete_length+= file->stats.delete_length; > if (file->stats.check_time > stats.check_time) > stats.check_time= file->stats.check_time; > - stats.checksum+= file->stats.checksum; > + if (!file->stats.checksum_null) > + { > + stats.checksum+= file->stats.checksum; > + stats.checksum_null= FALSE; > + } I'm not sure about it. When a checksum can be NULL? I mean, normally, if you do A+B+C and one of the values is NULL, then the whole sum is NULL. But here you simply skip NULLs, like in aggregate SUM() function. To understand what semantics works better in this case, I need to understand when a checksum can be NULL in spider. > } > if (stats.records && stats.records < 2 && > !(m_file[0]->ha_table_flags() & HA_STATS_RECORDS_IS_EXACT)) All other server changes look good, thanks. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 2

Re: [Maria-developers] Fwd: [Commits] a602998: MDEV-19580 Unrelated JOINs corrupt usage of 'WHERE function() IN (subquery)'
by Sergey Petrunia 05 Jun '19

05 Jun '19

Hello Igor, On Sat, Jun 01, 2019 at 01:16:04PM -0700, Igor Babaev wrote: > Please review this patch for the bug that we discussed on slack a couple of > days ago. > As you can see the extra calls of make_cond_for_table() with > used_table=RAND_TABLE_BIT were not enough (it can be seen in debugger why). > It took me quite a lot of time to figure out that I could not trust > Item::marker==2 to check whether a conjunct was previously attached > or not. > First, I've found a testcase that stopped working after the patch. (As far as I understand, we should pass RAND_TABLE_BIT when constructing JOIN::outer_ref_cond): create table ten(a int); insert into ten values (0),(1),(2),(3),(4),(5),(6),(7),(8),(9); create table t1 (a int, b int, c int); insert into t1 select a,a,a from ten; create table t2 as select a,b,c from t1; mysql> explain format=json select (select max(c) from t1 where t1.a=t2.a and rand()=t2.b) from t2\G *************************** 1. row *************************** EXPLAIN: { "query_block": { "select_id": 1, "table": { "table_name": "t2", "access_type": "ALL", "rows": 10, "filtered": 100 }, "subqueries": [ { "query_block": { "select_id": 2, "table": { "table_name": "t1", "access_type": "ALL", "rows": 10, "filtered": 100, "attached_condition": "((t1.a = t2.a) and (rand() = t2.b))" } } } ] } } 1 row in set (0.00 sec) If I run the select, it hits a breakpoint in Item_func_rand::val_real. Now, after this patch (I applied it to 10.1): mysql> explain format=json select (select max(c) from t1 where t1.a=t2.a and rand()=t2.b) from t2\G *************************** 1. row *************************** EXPLAIN: { "query_block": { "select_id": 1, "table": { "table_name": "t2", "access_type": "ALL", "rows": 10, "filtered": 100 }, "subqueries": [ { "query_block": { "select_id": 2, "table": { "table_name": "t1", "access_type": "ALL", "rows": 10, "filtered": 100, "attached_condition": "(t1.a = t2.a)" } } } ] } } and running the SELECT doesn't hit the breakpoint in Item_func_rand::val_real A bit more of cosmetic feedback below, marked with 'psergey:' > > @@ -8968,6 +8963,20 @@ make_join_select(JOIN *join,SQL_SELECT *select,COND > *cond) > { > tmp= make_cond_for_table(thd, cond, used_tables, current_map, i, > FALSE, FALSE); > + if (tab == join->join_tab + last_top_base_tab_idx) > + { > + /* > + This pushes conjunctive conditions of WHERE condition such > that: > + - their used_tables() contain RAND_TABLE_BIT > + - the conditions does not refer to any fields > + (such like rand() > 0.5) > + */ > + table_map rand_table_bit= (table_map) RAND_TABLE_BIT; psergey: what is the purpose of rand_table_bit? If we can't use RAND_TABLE_BIT where a table bitmap is expected, we should fix that. The way it's done now it looks redundant and is confusing. (this also applies to other declarations if rand_table_bit) > + COND *rand_cond= make_cond_for_table(thd, cond, used_tables, > + rand_table_bit, -1, > + FALSE, FALSE); > + add_cond_and_fix(thd, &tmp, rand_cond); > + } > } > /* Add conditions added by add_not_null_conds(). */ > if (tab->select_cond) ... > @@ -18852,11 +18880,27 @@ make_cond_for_table_from_pred(THD *thd, Item > *root_cond, Item *cond, > Item *item; > while ((item=li++)) > { > + /* > + Special handling of top level conjuncts with RAND_TABLE_BIT: > + if such a conjunct contains a reference to a field then it is > pushed > + to the corresponding table by the same rule as all other > conjuncts. > + Otherwise, if the conjunct is used in WHERE is is pushed to the > last > + joined table, if is it is used in ON condition of an outer join > it > + is pushed into the last inner table of the outer join. Such > conjuncts > + are pushed in a call of make_cond_for_table_from_pred() with the > + parameter 'used_table' equal to RAND_TABLE_BIT. > + */ > + if (is_top_and_level && used_table == rand_table_bit && > + item->used_tables() != rand_table_bit) > + { > + /* The conjunct with RAND_TABLE_BIT has been allready pushed */ psergey: typo: allready > + continue; > + } > Item *fix=make_cond_for_table_from_pred(thd, root_cond, item, > tables, used_table, > - join_tab_idx_arg, > + join_tab_idx_arg, > exclude_expensive_cond, > - retain_ref_cond); > + retain_ref_cond, false); > if (fix) > new_cond->argument_list()->push_back(fix); BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Weekly Report
by Rucha Deodhar 03 Jun '19

03 Jun '19

Report for week 1: Hello! I made test cases and extended the parser for: INSERT...RETURNING * and INSERT...RETURNING <col_name/names> It is working fine for INSERT...ON DUPLICATE KEY UPDATE, INSERT...IGNORE, INSERT...SET, and simple insert statements. As I have never coded parser before, it was throwing errors during build and was also crashing the server sometimes. I managed to fix the errors eventually. Initially, I tried to extend <column_names> in RETURNING by referring how column names are parsed in INSERT statement, like using opt_fields production for insert_field_list, modified it a little and tried to see it if that would work. But it didn't. So I referred to DELETE...RETURNING as it was suggested earlier. I am still trying to extend it for INSERT...SELECT. It is not showing syntax error but shows error "column count doesn't match value count at row 1." I will add the test cases for this once it is working. Regards, Rucha

2 1

Re: [Maria-developers] [Commits] b3473961810: MDEV-11094: Blackhole table updates on slave fail when row annotation is enabled
by Sergey Vojtovich 29 May '19

29 May '19

Hi Sujatha, On Wed, May 29, 2019 at 04:37:27PM +0530, sujatha wrote: > revision-id: b347396181018cedc946450cb49891f1a0aa4575 (mariadb-10.1.39-48-gb3473961810) > parent(s): 8358c6f03edb941be488f009c2bd0eb9df47e8c5 > author: Sujatha > committer: Sujatha > timestamp: 2019-05-29 15:18:52 +0530 > message: > > MDEV-11094: Blackhole table updates on slave fail when row annotation is enabled ...skip... > diff --git a/storage/blackhole/ha_blackhole.cc b/storage/blackhole/ha_blackhole.cc > index 01aaa9ea15f..43bcdc541a1 100644 > --- a/storage/blackhole/ha_blackhole.cc > +++ b/storage/blackhole/ha_blackhole.cc > @@ -25,6 +25,16 @@ > #include "ha_blackhole.h" > #include "sql_class.h" // THD, SYSTEM_THREAD_SLAVE_SQL > > +static bool is_row_based_replication(THD *thd) > +{ > + /* > + A row event which has its thd->query() == NULL or a row event which has > + replicate_annotate_row_events enabled. In the later case the thd->query() > + will be pointing to the query, received through replicated annotate event > + from master. > + */ > + return ((thd->query() == NULL) || thd->variables.binlog_annotate_row_events); Any reason to have so many redundant parenthesis? Why code comment rather than function comment? May be move "thd->system_thread == SYSTEM_THREAD_SLAVE_SQL" inside as well? Regards, Sergey

1 0

[Maria-developers] Proposal for automatic code formatting (clang-format)
by Eugene Kosov 27 May '19

27 May '19

Hi. I'm proposing to add some tool that automatically formats source code. Main purpose is overall productivity. It's possible to save time on manual formatting. Also, reading a uniformly formatted code is easier. Some new languages has one correct coding style and code formatters. Examples are Go (https://golang.org/cmd/gofmt/) and (to be done) Rust (https://github.com/rust-dev-tools/fmt-rfcs/blob/master/guide/guide.md) I suggest to read 'Motivation - why use a formatting tool' from second link. There is no single correct coding style in C and C++ worlds: different projects use different styles if any at all. But there is a tool which does formatting and supports different styles: clang-format. It's a binary available for every major platform. It's configurable: just put .clang-format file with formatting options in source directory and formatter will use it for current and nested directories. I actually do not propose here to use exactly clang-format but as I know it's the best of its kind. Here is a demo https://youtu.be/uZI_Qla4pNA?t=1893 Tool is configurable but it will not act exactly as you might expect and this is a pay for uniformity. So, let's add .clang-format to root directory, start using and polishing style. Here is some info on how to add it to some text editors https://clang.llvm.org/docs/ClangFormat.html I do not(!) suggesting to reformat whole project. At least for now. Here is my PR to add .clang-format files to server and InnoDB: https://github.com/MariaDB/server/pull/1270 I think that eventually .clang-format usage will lead to reformatting everything and having some project-wide precommit hook which will properly format your diffs. Do you have any objections or questions on having such a tool? Let's discuss, please. -- Eugene

1 0

Re: [Maria-developers] Regarding approach used for Early NULL filtering in the range optimizer(MDEV-15777)
by Sergey Petrunia 27 May '19

27 May '19

On Fri, May 24, 2019 at 04:39:51PM -0700, Igor Babaev wrote: > On 05/24/2019 01:41 PM, Sergey Petrunia wrote: > > On Fri, May 24, 2019 at 10:08:04AM -0700, Igor Babaev wrote: > > > On 05/24/2019 04:03 AM, Varun Gupta wrote: > > > > Hi Igor, > > > > After discussing with Sergey, we came up with these conclusions as to > > > > why we used the approach of going through all the keyuses in the KEYUSE > > > > array > > > > > > > > Cases to consider: > > > > we have an index on column a > > > > > > > > 1) a OP const > > > > where OP can be (</<=/=/between/ etc on which we do range analysis) . > > > > This case is already handled by the range optimizer. We do not create a > > > > NULL rejecting predicate for such conditions. > > > > > > > > 2) eq_join conditions (tbl1.a = tbl2.col2) > > > > This is the specific case we have tried to implement in MDEV-15777, we > > > > create NULL rejecting predicates for the keyuses for each table and then > > > > feed these predicates to range optimizer > > > There is no logical explanation for checking null rejection of fields used > > > in equalities with the help of array keyuse. > > The logic here is as follows: > > > > We already collect the set of KEYUSE::null_rejecting attributes. Its > > meaning is very close to what the patch needs. > The meaning of this argument is to be used a key value for ref access by an > index i1. > You need range access by a different index i2. There is no logical > connection > between these two indexes. The usage of i1 can be blocked by IGNORE clause > and this case > there won't be any keyuses for it. But the equality from which the > null-rejection can be > deduced is still there. You need range scan to reduce the number of records > produced > by the left operand of a join operation. And you should not care weather it > should be the > argument of any ref access. > Here is the situation: > The field of interest for range access is t1.a and you have two conditions > t1.a=t2.a with index i1 on t2.a > t1.b=t3.b with index i2 on t3.b > The use prohibited to use i1. t1.a apparently remains null-rejecting. Yet it > cannot be not detected > as such by a look through array of keyuses. I don't follow the logic. What MDEV-15777 does is to basically look at KEYUSE object, which represent tbl.key_col = other_column and feed this into the range optimizer: tbl.key_col IS NOT NULL If the query CAN use a certain index: - we will have a KEYUSE object for it - we will feed the IS NOT NULL into the range optimizer. if the query CANNOT use a certain index (because of IGNORE INDEX), then - we won't have KEYUSE object for that index (and so won't construct an IS NOT NULL condition for it) - range optimizer will not build ranges for it (so it doesn't really matter if IS NOT NULL was constructed or not) Can you provide a full example? > > > > Because of this, the part of the patch for MDEV-15777 which analyzes the KEYUSE > > array is very small: 122 lines including the comment (I counted > > make_null_rejecting_conds() and add_cond()). > > > > I think, the CPU overhead is small, too. > > > > > > 3) a < func(b,c) > > > > we do not handle this case, because: > > > > > > > > a) It is harder to process. We would need to process the whole WHERE > > > > clause to infer > > > > non-nullability of the condition. Example: > > > > > > > > (t1.f2+1 < t1.f1+t1.f3) OR ... OR t1.f1 is null > > > > > > > > here the left part of the OR condition doesn't allow f1 to be NULL, but > > > > the > > > > right condition allows it. We would also need to take into account tables > > > > inside outer joins. We would need to go through Item classes and add code > > > > which say "Item_xxx() will compute to false when its argument Y is null" > > > > (there is item->not_null_tables() currently, but not not_null_columns()). > > > Have you actually looked at the implementations of the virtual function > > > not_null_tables()? > > > Do you need me to provide the implementation of the virtual function > > > not_null_columns()? > > It will require multiple implementations. At the moment we have: > > > > nm mysqld --demangle | grep 'Item.*::not_null_tables' | wc -l > > 21 > > > > Another question: do you intend to collect not_null_columns() for all columns, > > or just columns that are a part of some index? > > > > What would a good data structure to store a set of tablename.column_name ? > It will be just a bitmap created for each table that can be used for many > other purposes. > If you want to filter out those that are not part of any index it can be > easily done > (even once for any table share) My point was that constructing such a bitmap is a lot heavier than the not_null_table_map(). table.h declares a field bitmap class: typedef Bitmap<MAX_FIELDS> Field_map; but it's huge: (gdb) p sizeof(Field_map) $52 = 544 and note that we will not be able to just put one Field_map into the TABLE object: If we think about how to compute not_null_columns() for Item_cond_and or Item_cond_or: - Item_cond_and will compute a union of all its arguments' not_null_columns(). - Item_cond_or will compute an intersection of all its arguments' not_null_columns(). - Other items have different logic. that is, we will need to keep multiple "set of fields in tables" objects around. Just one bitmap attached to the table will not suffice. This is doable, but it's going to be more code and have more CPU overhead. And I'm still not convinced why we should do this. It is not clear if this will fit the scope of MDEV-15777 and once we got it done, what's the use of it? Is this something other query optimizers do (so the users expect it?)? > > > > > > b) the conditions in form of arbitrary functions are not as frequently > > > > used as > > > > ref access conditions. > > > > > > > > to sum up a) and b) - doable but will require a lot of effort. > > > > > > > > Do you have any specific practically relevant example in mind that we > > > > should > > > > handle? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] [Commits] 53d0e921794: MDEV-11094: Blackhole table updates on slave fail when row annotation is enabled
by andrei.elkin＠pp.inet.fi 27 May '19

27 May '19

Sujatha, howdy. Thank you for spelling it out for me! > Hello Andrei, ... >>> >>> Analysis: >>> ========= >>> Enabling "replicate_annotate_row_events" on slave, Tells the slave to write >>> annotate rows events received from the master to its own binary log. The >>> received annotate events are applied after the Gtid event as shown below. >>> thd->query() will be set to the actual query received from the master, through >>> annotate event. Annotate_rows event should not be deleted after the event is >>> applied as the thd->query will be used to generate new Annotate_rows event >>> during applying the subsequent Rows events. >> [ here ] >> The Table_map and Rows_log_event that follow do not touch thd->query(). > > The Table_map and Rows_log_event that follow "Annotate_rows" event do > make use of "thd->query()". Frankly I did not expect that. Perhaps it would have not been a bad idea to make Annotate_rows_log_event::do_apply_event() logging of itself, which still makes sense even if the slave applier will filter out its group's Rows_log_event:s so in this case the slave side event group would consists of mere Annotate. The above is a mere remark though, I have to accept what we have. >> >> I also suggest to check out mysqlbinlog run. There seems to be no test >> for that though, so one needs to be written. The goal would be to prove >> a created binlog output is as expected (e.g to include Annotate). > > > Sure, I will add a test case. Super! The patch is approved then. Cheers, Andrei

1 0

Re: [Maria-developers] [Commits] 42129dae63c: MDEV-18741: Optimizer trace: multi-part key ranges are printed incorrectly
by Sergey Petrunia 27 May '19

27 May '19

Hi Varun, One cosmetic comment below. Ok to push after it is addressed. On Fri, May 24, 2019 at 03:51:00PM +0530, Varun wrote: > revision-id: 42129dae63c4331b01f9a504c9f3ef1c0c1f857a (mariadb-10.4.4-126-g42129dae63c) > parent(s): 592fe954ef82be1bc08b29a8e54f7729eb1e1343 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2019-05-24 14:43:44 +0530 > message: > > MDEV-18741: Optimizer trace: multi-part key ranges are printed incorrectly > > Changed the function append_range_all_keyparts to use sel_arg_range_seq_init / sel_arg_range_seq_next to produce ranges. > Also adjusted to print format for the ranges, now the ranges are printed as: > (keypart1_min, keypart2_min,..) OP (keypart1_name,keypart2_name, ..) OP (keypart1_max,keypart2_max, ..) > > Also added more tests for range and index merge access for optimizer trace ... > diff --git a/sql/opt_range_mrr.cc b/sql/opt_range_mrr.cc > index 3a25da3edb2..5644e456dd4 100644 > --- a/sql/opt_range_mrr.cc > +++ b/sql/opt_range_mrr.cc > @@ -53,6 +53,8 @@ typedef struct st_sel_arg_range_seq > int i; /* Index of last used element in the above array */ > > bool at_start; /* TRUE <=> The traversal has just started */ > + /* TRUE if last checked tree->key can be used for ROR-scan */ > + bool is_ror_scan; > } SEL_ARG_RANGE_SEQ; > > This comment made sense at the previous location, but here it doesn't. Here the meaning is something like "Iteration functions will set this to FALSE" if ranges being traversed do not allow to construct a ROR-scan". BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] GSoC Introduction
by Rucha Deodhar 27 May '19

27 May '19

Hello, I am Rucha Deodhar. I am a GSoC student and will implement INSERT...RETURNING. In the first week I plan to extend the parser. Approximately the next 2 months, I plan to make changes the in the codes in insert.cc and other files related to the project, and remaining time for testing. As suggested, I am allotting 40% of the time for testing.This is my approximate plan, Regards, Rucha.

2 1

Re: [Maria-developers] Regarding approach used for Early NULL filtering in the range optimizer(MDEV-15777)
by Sergey Petrunia 24 May '19

24 May '19

On Fri, May 24, 2019 at 10:08:04AM -0700, Igor Babaev wrote: > On 05/24/2019 04:03 AM, Varun Gupta wrote: > > Hi Igor, > > After discussing with Sergey, we came up with these conclusions as to > > why we used the approach of going through all the keyuses in the KEYUSE > > array > > > > Cases to consider: > > we have an index on column a > > > > 1) a OP const > > where OP can be (</<=/=/between/ etc on which we do range analysis) . > > This case is already handled by the range optimizer. We do not create a > > NULL rejecting predicate for such conditions. > > > > 2) eq_join conditions (tbl1.a = tbl2.col2) > > This is the specific case we have tried to implement in MDEV-15777, we > > create NULL rejecting predicates for the keyuses for each table and then > > feed these predicates to range optimizer > There is no logical explanation for checking null rejection of fields used > in equalities with the help of array keyuse. The logic here is as follows: We already collect the set of KEYUSE::null_rejecting attributes. Its meaning is very close to what the patch needs. Because of this, the part of the patch for MDEV-15777 which analyzes the KEYUSE array is very small: 122 lines including the comment (I counted make_null_rejecting_conds() and add_cond()). I think, the CPU overhead is small, too. > > > > 3) a < func(b,c) > > we do not handle this case, because: > > > > a) It is harder to process. We would need to process the whole WHERE > > clause to infer > > non-nullability of the condition. Example: > > > > (t1.f2+1 < t1.f1+t1.f3) OR ... OR t1.f1 is null > > > > here the left part of the OR condition doesn't allow f1 to be NULL, but > > the > > right condition allows it. We would also need to take into account tables > > inside outer joins. We would need to go through Item classes and add code > > which say "Item_xxx() will compute to false when its argument Y is null" > > (there is item->not_null_tables() currently, but not not_null_columns()). > Have you actually looked at the implementations of the virtual function > not_null_tables()? > Do you need me to provide the implementation of the virtual function > not_null_columns()? It will require multiple implementations. At the moment we have: nm mysqld --demangle | grep 'Item.*::not_null_tables' | wc -l 21 Another question: do you intend to collect not_null_columns() for all columns, or just columns that are a part of some index? What would a good data structure to store a set of tablename.column_name ? > > > > b) the conditions in form of arbitrary functions are not as frequently > > used as > > ref access conditions. > > > > to sum up a) and b) - doable but will require a lot of effort. > > > > Do you have any specific practically relevant example in mind that we > > should > > handle? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Regarding approach used for Early NULL filtering in the range optimizer(MDEV-15777)
by Varun Gupta 24 May '19

24 May '19

Hi Igor, After discussing with Sergey, we came up with these conclusions as to why we used the approach of going through all the keyuses in the KEYUSE array Cases to consider: we have an index on column a 1) a OP const where OP can be (</<=/=/between/ etc on which we do range analysis) . This case is already handled by the range optimizer. We do not create a NULL rejecting predicate for such conditions. 2) eq_join conditions (tbl1.a = tbl2.col2) This is the specific case we have tried to implement in MDEV-15777, we create NULL rejecting predicates for the keyuses for each table and then feed these predicates to range optimizer 3) a < func(b,c) we do not handle this case, because: a) It is harder to process. We would need to process the whole WHERE clause to infer non-nullability of the condition. Example: (t1.f2+1 < t1.f1+t1.f3) OR ... OR t1.f1 is null here the left part of the OR condition doesn't allow f1 to be NULL, but the right condition allows it. We would also need to take into account tables inside outer joins. We would need to go through Item classes and add code which say "Item_xxx() will compute to false when its argument Y is null" (there is item->not_null_tables() currently, but not not_null_columns()). b) the conditions in form of arbitrary functions are not as frequently used as ref access conditions. to sum up a) and b) - doable but will require a lot of effort. Do you have any specific practically relevant example in mind that we should handle? Regards, Varun and Sergey

1 0

[Maria-developers] Review for: b4696ee97e684 : MDEV-15777: Support Early NULLs filtering-like restrictions in the range optimizer
by Sergey Petrunia 23 May '19

23 May '19

Hi Varun, Please find review input below. Only cosmetic changes are requested. I also assume that after this is pushed - we add a optimizer trace printout into 10.4 - the feature is enabled by default in 10.5 ? (do we need separate MDEVs for this?) One final question is the choice of name for @@optimizer_switch flag. Let me get back to you on this. > commit b4696ee97e6846ba49ef203cfc7189f50c1e53a7 > Author: Varun Gupta <varun.gupta(a)mariadb.com> > Date: Mon May 20 15:14:30 2019 +0530 > > MDEV-15777: Support Early NULLs filtering-like restrictions in the range optimizer > > For eqjoin conditions, we add a NOT NULL predicate so as to allow the range optimizer > to use the predicate and possibly create a range access on the given table. > > Example: > select * from t1,t2 where t1.a=t2.a; we have KEY(a) on t1 > we would inject a NOT NULL predicate t1.a IS NOT NULL for table t1 > this would allow the range optimizer to create ranges and we get a > range access on table t1, then we will be able > to do an early NULL filtering for ranges too. > > diff --git a/mysql-test/main/range.result b/mysql-test/main/range.result > index 32e0cf2868c..d75d6734277 100644 > --- a/mysql-test/main/range.result > +++ b/mysql-test/main/range.result > @@ -3024,3 +3024,151 @@ drop table t1; > # > # End of 10.2 tests > # > +# > +# MDEV-15777: Support Early NULLs filtering-like restrictions in the range optimizer > +# > +set @save_optimizer_switch= @@optimizer_switch; > +set @@optimizer_switch='null_rejecting_for_ranges=on'; > +create table ten(a int); > +insert into ten values (0),(1),(2),(3),(4),(5),(6),(7),(8),(9); > +create table one_k(a int); > +insert into one_k select A.a + B.a* 10 + C.a * 100 from ten A, ten B, ten C; Did we abandon the convention of naming the tables t1,t2,t3,... ? If not, please rename. > +create table t1 ( > +id int(10) unsigned NOT NULL, > +subset_id int(11) DEFAULT NULL, > +PRIMARY KEY (id), > +KEY t1_subset_id (subset_id)); ... > diff --git a/sql/opt_range.cc b/sql/opt_range.cc > index ec7b3dbbd7a..cb7800d1ba6 100644 > --- a/sql/opt_range.cc > +++ b/sql/opt_range.cc > @@ -2539,6 +2546,42 @@ int SQL_SELECT::test_quick_select(THD *thd, key_map keys_to_use, > TRP_GROUP_MIN_MAX *group_trp; > double best_read_time= read_time; > > + /* > + For the range optimizer, null_rejecting_conds are just an extra condition > + on which one can use index to create ranges. > + > + Also here we try to allow consider range access for a column involed in a > + null rejecting predicate only if it is the first keypart of an index. > + Let try to make it clear with examples: > + > + Query 1 > + select * from t1,t2 where t1.a=2 and t1.b=t2.b; > + and we have a key(a,b) > + So lets consider range access for table t1: > + Null rejecting predicate : t1.b IS NOT NULL > + but the column t1.b is NOT the first keypart in the index, > + so we don't consider it while building ranges > + > + Query 2 > + select * from t1,t2 where t1.a=t2.a and t1.b=t2.b; > + and we have a key (a,b) > + > + So lets consider range access for table t1: > + Null rejecting predicate : t1.a IS NOT NULL AND t1.b IS NOT NULL > + so here t1.a is the first keypart in the index, so we consider > + the predicate for building the ranges. > + > + The first keypart prerequisite is made as we try to reuse range > + estimates (because they are more accurate) at different places > + and we needed to make sure that correct/accurate estimates are used there. > + */ > + > + if (null_rejecting_conds) > + not_null_cond_tree= null_rejecting_conds->get_mm_tree(&param, > + &null_rejecting_conds); Please fix indentation and fit within 80 chars. ^ And also use curly braces as the code inside if doesn't fit on one line. > + if (not_null_cond_tree) > + remove_nonrange_trees(&param, not_null_cond_tree); > + > if (cond) > { > if ((tree= cond->get_mm_tree(&param, &cond))) > @@ -2557,6 +2600,7 @@ int SQL_SELECT::test_quick_select(THD *thd, key_map keys_to_use, > tree= NULL; > } > } > + tree= tree_and(&param, tree, not_null_cond_tree); > > /* > Try to construct a QUICK_GROUP_MIN_MAX_SELECT. > @@ -4422,6 +4466,128 @@ static bool create_partition_index_description(PART_PRUNE_PARAM *ppar) > return FALSE; > } > > +inline void add_cond(THD *thd, Item **e1, Item *e2) > +{ > + if (*e1) > + { > + if (!e2) > + return; > + Item *res; > + if ((res= new (thd->mem_root) Item_cond_and(thd, *e1, e2))) > + { > + res->fix_fields(thd, 0); > + res->update_used_tables(); > + *e1= res; > + } > + } > + else > + *e1= e2; > +} > + > +/* > + Create null rejecting conditions for a table, for all the equalites > + present in the WHERE clause of a query. > + > + SYNOPSIS > + make_null_rejecting_conds() > + @param TABLE - Keys of this table will participate in null > + rejecting conditions > + @param keyuse_array - array that has all the equalites of the > + WHERE clasuse The above doesn't match the function's definition. Out of date comment? > + > + DESCRIPTION > + This function creates null rejecting conditions for a table. These > + conditions are created to do range analysis on them , the conditions > + are of the form tbl.key.keypart IS NOT NULL. > + > + IMPLEMENTATION > + Lookup in the keyuse array to check if it has equalites that belong > + to the given table. If yes then find out if the conditions are null > + rejecting and accordingly create all the condition for the keys of a > + given table and AND them. > + > + > + RETURN > + NOT NULL - Found null rejecting conditions for the given table > + NULL - No null rejecting conditions for the given table The above doesn't match the function's definition. Out of date comment? > +*/ > + > +void make_null_rejecting_conds(THD *thd, JOIN_TAB *tab) > +{ > + KEY *keyinfo; > + Item *cond= NULL; > + KEYUSE* keyuse; > + TABLE *table= tab->table; > + key_map *const_keys= &tab->const_keys; ^ Does it make sense to introduce a variable if it is only used once? > + > + /* > + No need to add NOT NULL condition for materialized derived tables > + or materialized subqueries as we do not run the range optimizer > + on their conditions > + */ > + > + if (!optimizer_flag(thd, OPTIMIZER_SWITCH_NULL_REJECTING_FOR_RANGES)) > + return; > + > + if (tab->table->is_filled_at_execution() || > + (tab->table->pos_in_table_list->derived && > + tab->table->pos_in_table_list->is_materialized_derived())) > + return; > + > + Field_map not_null_keypart_map; > + > + /* > + The null rejecting conds added will be on the keypart of a key, so for > + that we need the table to atleast have a key. > + */ > + if (!table->s->keys || table->null_rejecting_conds || !tab->keyuse) > + return; > + > + for(keyuse= tab->keyuse; keyuse->table == table; keyuse++) > + { Please add space, "for ". > + /* > + No null rejecting conds for a hash key or full-text keys > + */ > + if (keyuse->key == MAX_KEY || keyuse->keypart == FT_KEYPART) > + continue; > + keyinfo= keyuse->table->key_info + keyuse->key; > + Field *field= keyinfo->key_part[keyuse->keypart].field; > + > + if (not_null_keypart_map.is_set(field->field_index)) > + continue; > + > + /* > + No need to add null-rejecting condition if we have a > + keyuse element as > + - table.key.keypart= const > + - (table.key.keypart= tbl.otherfield or table.key.keypart IS NULL) > + - table.key.keypart IS NOT NULLABLE > + */ > + > + if (keyuse->val->const_item() || > + !(keyuse->null_rejecting && field->maybe_null()) || > + keyuse->optimize & KEY_OPTIMIZE_REF_OR_NULL) > + continue; > + > + Item_field *field_item= new (thd->mem_root)Item_field(thd, field); > + Item* not_null_item= new (thd->mem_root)Item_func_isnotnull(thd, field_item); > + > + /* > + adding the key to const keys as we have the condition > + as key.keypart IS NOT NULL > + */ > + > + const_keys->set_bit(keyuse->key); > + not_null_item->fix_fields(thd, 0); ^^ please change 0 to &not_null_item. > + not_null_item->update_used_tables(); > + add_cond(thd, &cond, not_null_item); > + not_null_keypart_map.set_bit(field->field_index); > + } > + DBUG_EXECUTE("where", print_where(cond, "Early Null Filtering", > + QT_ORDINARY);); > + table->null_rejecting_conds= cond; > +} > + > > #ifndef DBUG_OFF > ... > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index 8cdcf3afc8b..140ceed5c2d 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -5510,15 +5512,30 @@ add_key_field(JOIN *join, > If the condition has form "tbl.keypart = othertbl.field" and > othertbl.field can be NULL, there will be no matches if othertbl.field > has NULL value. > - We use null_rejecting in add_not_null_conds() to add > - 'othertbl.field IS NOT NULL' to tab->select_cond. > + > + The field KEY_FIELD::null_rejecting is set to TRUE if either > + the left or the right hand side of the equality is NULLABLE > + > + null_rejecting is used > + - by add_not_null_conds(), to add "othertbl.field IS NOT NULL" to > + othertbl's tab->select_cond. (This is called "Early NULLs filtering") > + > + - by make_null_rejecting_conds(), to provide range optimizer with > + additional "tbl.keypart IS NOT NULL" condition. > */ > { > Item *real= (*value)->real_item(); > - if (((cond->functype() == Item_func::EQ_FUNC) || > - (cond->functype() == Item_func::MULT_EQUAL_FUNC)) && > - (real->type() == Item::FIELD_ITEM) && > - ((Item_field*)real)->field->maybe_null()) > + if (( > + (cond->functype() == Item_func::EQ_FUNC) || > + (cond->functype() == Item_func::MULT_EQUAL_FUNC) > + ) && > + ( > + ( > + (real->type() == Item::FIELD_ITEM) && > + ((Item_field*)real)->field->maybe_null() > + ) || > + (field->maybe_null()) > + )) > (*key_fields)->null_rejecting= true; > else > (*key_fields)->null_rejecting= false; > @@ -9982,8 +9999,12 @@ static bool create_ref_for_key(JOIN *join, JOIN_TAB *j, > uint maybe_null= MY_TEST(keyinfo->key_part[i].null_bit); > j->ref.items[i]=keyuse->val; // Save for cond removal > j->ref.cond_guards[i]= keyuse->cond_guard; > - if (keyuse->null_rejecting) > + Item *real= (keyuse->val)->real_item(); > + if (keyuse->null_rejecting && > + (real->type() == Item::FIELD_ITEM) && > + ((Item_field*)real)->field->maybe_null()) This code looks as if if keyuse->val->maybe_null() != keyuse->val->real_item()->maybe_null() do you have any specific case in mind or this is just to match the code in add_key_field? > j->ref.null_rejecting|= (key_part_map)1 << i; > + > keyuse_uses_no_tables= keyuse_uses_no_tables && !keyuse->used_tables; > /* > We don't want to compute heavy expressions in EXPLAIN, an example would BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

2 1

Re: [Maria-developers] [Commits] 53d0e921794: MDEV-11094: Blackhole table updates on slave fail when row annotation is enabled
by andrei.elkin＠pp.inet.fi 22 May '19

22 May '19

Sujatha, salute! Thanks for a detailed description! I would add up to it a minor piece > revision-id: 53d0e9217943719b806ef9fa1cac0a699df4839c (mariadb-10.1.39-31-g53d0e921794) > parent(s): 47637a3dd13d19e897a7cbfd1499f1bf3b2fdb2a > author: Sujatha > committer: Sujatha > timestamp: 2019-05-17 13:11:49 +0530 > message: > > MDEV-11094: Blackhole table updates on slave fail when row annotation is enabled > > Problem: > ======= > rpl_blackhole.test fails when executed with following options > mysqld=--binlog_annotate_row_events=1, mysqld=--replicate_annotate_row_events=1 > > Test output: > ------------ > worker[1] Using MTR_BUILD_THREAD 300, with reserved ports 16000..16019 > rpl.rpl_blackhole_bug 'mix' [ pass ] 791 > rpl.rpl_blackhole_bug 'row' [ fail ] > Replicate_Wild_Ignore_Table > Last_Errno 1032 > Last_Error Could not execute Update_rows_v1 event on table test.t1; Can't find > record in 't1', Error_code: 1032; handler error HA_ERR_END_OF_FILE; the event's > master log master-bin.000001, end_log_pos 1510 > > Analysis: > ========= > Enabling "replicate_annotate_row_events" on slave, Tells the slave to write > annotate rows events received from the master to its own binary log. The > received annotate events are applied after the Gtid event as shown below. > thd->query() will be set to the actual query received from the master, through > annotate event. Annotate_rows event should not be deleted after the event is > applied as the thd->query will be used to generate new Annotate_rows event > during applying the subsequent Rows events. [ here ] The Table_map and Rows_log_event that follow do not touch thd->query(). Notice that the being suggested sentence is sufficient to explain the not-NULL without the sentence below (fill free to remove then): > After the last Rows event has been > applied, the saved Annotate_rows event (if any) will be deleted. > > master-bin.000001 | 457 | Gtid | BEGIN GTID 0-1-2 > master-bin.000001 | 495 | Annotate_rows | insert into t values (10) > master-bin.000001 | 539 | Table_map | table_id: 19 (test.t) > master-bin.000001 | 579 | Write_rows_v1 | table_id: 19 flags: STMT_END_F > master-bin.000001 | 613 | Xid | COMMIT /* xid=7 */ > > In balckhole engine all the DML operations are noops as they donot store any > data. They simply return success without doing any operation. But the existing > strictly expects thd->query() to be 'NULL' to identify that row based > replication is in use. This assumption will fail when row annotations are > enabled as the query is not 'NULL'. Hence various row based operations like > 'update', 'delete', 'index lookup' will fail when row annotations are enabled. > > Fix: > === > Extend the row based replication check to include row annotations as well. > i.e Either the thd->query() is NULL or thd->query() points to query and row > annotations are in use. This works, but the check does not really refers to the Annotate. So in principle thd->query() may turn in not-NULL of different reasons. Besides the check is multiplied over few places (again hopefully covering all). We can do it simpler actually: Rows_log_event::do_apply_event() { ... do { /* in_use can have been set to NULL in close_tables_for_reopen */ THD* old_thd= table->in_use; if (!table->in_use) table->in_use= thd; --> add a check whether Annotate has been done and when so store/restore --> thd->query() like `old_thd` above/below, and pass it as NULL (thd->reset_query()) to the following: error= do_exec_row(rgi); if (error) DBUG_PRINT("info", ("error: %s", HA_ERR(error))); DBUG_ASSERT(error != HA_ERR_RECORD_DELETED); table->in_use = old_thd; I also suggest to check out mysqlbinlog run. There seems to be no test for that though, so one needs to be written. The goal would be to prove a created binlog output is as expected (e.g to include Annotate). Cheers, Andrei > > --- > mysql-test/suite/rpl/t/rpl_blackhole-master.opt | 1 + > mysql-test/suite/rpl/t/rpl_blackhole-slave.opt | 1 + > storage/blackhole/ha_blackhole.cc | 28 +++++++++++++++++++------ > 3 files changed, 24 insertions(+), 6 deletions(-) > > diff --git a/mysql-test/suite/rpl/t/rpl_blackhole-master.opt b/mysql-test/suite/rpl/t/rpl_blackhole-master.opt > new file mode 100644 > index 00000000000..1ad0b884c60 > --- /dev/null > +++ b/mysql-test/suite/rpl/t/rpl_blackhole-master.opt > @@ -0,0 +1 @@ > +--binlog_annotate_row_events > diff --git a/mysql-test/suite/rpl/t/rpl_blackhole-slave.opt b/mysql-test/suite/rpl/t/rpl_blackhole-slave.opt > new file mode 100644 > index 00000000000..7ac6a84faa7 > --- /dev/null > +++ b/mysql-test/suite/rpl/t/rpl_blackhole-slave.opt > @@ -0,0 +1 @@ > +--binlog_annotate_row_events --replicate_annotate_row_events > diff --git a/storage/blackhole/ha_blackhole.cc b/storage/blackhole/ha_blackhole.cc > index 01aaa9ea15f..43bcdc541a1 100644 > --- a/storage/blackhole/ha_blackhole.cc > +++ b/storage/blackhole/ha_blackhole.cc > @@ -25,6 +25,16 @@ > #include "ha_blackhole.h" > #include "sql_class.h" // THD, SYSTEM_THREAD_SLAVE_SQL > > +static bool is_row_based_replication(THD *thd) > +{ > + /* > + A row event which has its thd->query() == NULL or a row event which has > + replicate_annotate_row_events enabled. In the later case the thd->query() > + will be pointing to the query, received through replicated annotate event > + from master. > + */ > + return ((thd->query() == NULL) || thd->variables.binlog_annotate_row_events); > +} > /* Static declarations for handlerton */ > > static handler *blackhole_create_handler(handlerton *hton, > @@ -109,7 +119,8 @@ int ha_blackhole::update_row(const uchar *old_data, uchar *new_data) > { > DBUG_ENTER("ha_blackhole::update_row"); > THD *thd= ha_thd(); > - if (thd->system_thread == SYSTEM_THREAD_SLAVE_SQL && thd->query() == NULL) > + if (thd->system_thread == SYSTEM_THREAD_SLAVE_SQL && > + is_row_based_replication(thd)) > DBUG_RETURN(0); > DBUG_RETURN(HA_ERR_WRONG_COMMAND); > } > @@ -118,7 +129,8 @@ int ha_blackhole::delete_row(const uchar *buf) > { > DBUG_ENTER("ha_blackhole::delete_row"); > THD *thd= ha_thd(); > - if (thd->system_thread == SYSTEM_THREAD_SLAVE_SQL && thd->query() == NULL) > + if (thd->system_thread == SYSTEM_THREAD_SLAVE_SQL && > + is_row_based_replication(thd)) > DBUG_RETURN(0); > DBUG_RETURN(HA_ERR_WRONG_COMMAND); > } > @@ -135,7 +147,8 @@ int ha_blackhole::rnd_next(uchar *buf) > int rc; > DBUG_ENTER("ha_blackhole::rnd_next"); > THD *thd= ha_thd(); > - if (thd->system_thread == SYSTEM_THREAD_SLAVE_SQL && thd->query() == NULL) > + if (thd->system_thread == SYSTEM_THREAD_SLAVE_SQL && > + is_row_based_replication(thd)) > rc= 0; > else > rc= HA_ERR_END_OF_FILE; > @@ -220,7 +233,8 @@ int ha_blackhole::index_read_map(uchar * buf, const uchar * key, > int rc; > DBUG_ENTER("ha_blackhole::index_read"); > THD *thd= ha_thd(); > - if (thd->system_thread == SYSTEM_THREAD_SLAVE_SQL && thd->query() == NULL) > + if (thd->system_thread == SYSTEM_THREAD_SLAVE_SQL && > + is_row_based_replication(thd)) > rc= 0; > else > rc= HA_ERR_END_OF_FILE; > @@ -235,7 +249,8 @@ int ha_blackhole::index_read_idx_map(uchar * buf, uint idx, const uchar * key, > int rc; > DBUG_ENTER("ha_blackhole::index_read_idx"); > THD *thd= ha_thd(); > - if (thd->system_thread == SYSTEM_THREAD_SLAVE_SQL && thd->query() == NULL) > + if (thd->system_thread == SYSTEM_THREAD_SLAVE_SQL && > + is_row_based_replication(thd)) > rc= 0; > else > rc= HA_ERR_END_OF_FILE; > @@ -249,7 +264,8 @@ int ha_blackhole::index_read_last_map(uchar * buf, const uchar * key, > int rc; > DBUG_ENTER("ha_blackhole::index_read_last"); > THD *thd= ha_thd(); > - if (thd->system_thread == SYSTEM_THREAD_SLAVE_SQL && thd->query() == NULL) > + if (thd->system_thread == SYSTEM_THREAD_SLAVE_SQL && > + is_row_based_replication(thd)) > rc= 0; > else > rc= HA_ERR_END_OF_FILE; > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

Re: [Maria-developers] 9e176e81b72: MDEV-18531 : Use WolfSSL instead of YaSSL as "bundled" SSL/encryption library
by Sergei Golubchik 21 May '19

21 May '19

Hi, Vladislav! It's good to see how small the patch is, and that wolfssl is more OpenSSL compatible than YaSSL. Few comments below. On May 19, Vladislav Vaintroub wrote: > revision-id: 9e176e81b72 (mariadb-10.4.4-61-g9e176e81b72) > parent(s): ea679c88c32 > author: Vladislav Vaintroub <wlad(a)mariadb.com> > committer: Vladislav Vaintroub <wlad(a)mariadb.com> > timestamp: 2019-05-01 11:28:56 +0100 > message: > > MDEV-18531 : Use WolfSSL instead of YaSSL as "bundled" SSL/encryption library > > - Add new submodule for WolfSSL > - Build and use wolfssl and wolfcrypt instead of yassl/taocrypt > - Use HAVE_WOLFSSL instead of HAVE_YASSL > - Increase MY_AES_CTX_SIZE, to avoid compile time asserts in my_crypt.cc > (sizeof(EVP_CIPHER_CTX) is larger on WolfSSL) > > diff --git a/cmake/ssl.cmake b/cmake/ssl.cmake > index aa42333a5c8..79a531ec323 100644 > --- a/cmake/ssl.cmake > +++ b/cmake/ssl.cmake > @@ -48,29 +48,19 @@ MACRO (CHANGE_SSL_SETTINGS string) > ENDMACRO() > > MACRO (MYSQL_USE_BUNDLED_SSL) > - SET(INC_DIRS > - ${CMAKE_SOURCE_DIR}/extra/yassl/include > - ${CMAKE_SOURCE_DIR}/extra/yassl/taocrypt/include > + SET(INC_DIRS > + ${CMAKE_SOURCE_DIR}/extra/wolfssl/wolfssl > + ${CMAKE_SOURCE_DIR}/extra/wolfssl/wolfssl/wolfssl > + #${CMAKE_SOURCE_DIR}/extra/wolfssl/taocrypt/include don't you want to remove these (and many below) commented lines? > ) > - SET(SSL_LIBRARIES yassl taocrypt) > + SET(SSL_LIBRARIES wolfssl wolfcrypt) > SET(SSL_INCLUDE_DIRS ${INC_DIRS}) > - SET(SSL_INTERNAL_INCLUDE_DIRS ${CMAKE_SOURCE_DIR}/extra/yassl/taocrypt/mySTL) > + SET(SSL_DEFINES "-DHAVE_OPENSSL -DHAVE_WOLFSSL -DOPENSSL_ALL -DWOLFSSL_MYSQL_COMPATIBLE -DWC_NO_HARDEN") > - SET(SSL_DEFINES "-DHAVE_YASSL -DYASSL_PREFIX -DHAVE_OPENSSL -DMULTI_THREADED") > - SET(HAVE_ERR_remove_thread_state OFF CACHE INTERNAL "yassl doesn't have ERR_remove_thread_state") > - SET(HAVE_EncryptAes128Ctr OFF CACHE INTERNAL "yassl doesn't support AES-CTR") > - SET(HAVE_EncryptAes128Gcm OFF CACHE INTERNAL "yassl doesn't support AES-GCM") > + SET(HAVE_ERR_remove_thread_state ON CACHE INTERNAL "wolfssl doesn't have ERR_remove_thread_state") > + SET(HAVE_EncryptAes128Ctr ON CACHE INTERNAL "wolfssl does support AES-CTR") > + SET(HAVE_EncryptAes128Gcm OFF CACHE INTERNAL "wolfssl does not support AES-GCM") > CHANGE_SSL_SETTINGS("bundled") > - ADD_SUBDIRECTORY(extra/yassl) > - ADD_SUBDIRECTORY(extra/yassl/taocrypt) > - GET_TARGET_PROPERTY(src yassl SOURCES) > - FOREACH(file ${src}) > - SET(SSL_SOURCES ${SSL_SOURCES} ${CMAKE_SOURCE_DIR}/extra/yassl/${file}) > - ENDFOREACH() > - GET_TARGET_PROPERTY(src taocrypt SOURCES) > - FOREACH(file ${src}) > - SET(SSL_SOURCES ${SSL_SOURCES} > - ${CMAKE_SOURCE_DIR}/extra/yassl/taocrypt/${file}) > - ENDFOREACH() > + ADD_SUBDIRECTORY(extra/wolfssl) > MESSAGE_ONCE(SSL_LIBRARIES "SSL_LIBRARIES = ${SSL_LIBRARIES}") > ENDMACRO() > > diff --git a/extra/wolfssl/CMakeLists.txt b/extra/wolfssl/CMakeLists.txt > new file mode 100644 > index 00000000000..c1165ad6c72 > --- /dev/null > +++ b/extra/wolfssl/CMakeLists.txt > @@ -0,0 +1,119 @@ > +# CMakeLists.txt > +# > +# Copyright (C) 2006-2015 wolfSSL Inc. > +# > +# This file is part of wolfSSL. (formerly known as CyaSSL) > +# > +# wolfSSL is free software; you can redistribute it and/or modify > +# it under the terms of the GNU General Public License as published by > +# the Free Software Foundation; either version 2 of the License, or > +# (at your option) any later version. > +# > +# wolfSSL is distributed in the hope that it will be useful, > +# but WITHOUT ANY WARRANTY; without even the implied warranty of > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > +# GNU General Public License for more details. > +# > +# You should have received a copy of the GNU General Public License > +# along with this program; if not, write to the Free Software > +# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA > + > +SET(WOLFSSL_SRCDIR ${CMAKE_CURRENT_SOURCE_DIR}/wolfssl/src) > +ADD_DEFINITIONS(${SSL_DEFINES}) > +ADD_DEFINITIONS( > + -DWOLFSSL_MYSQL_COMPATIBLE > + -DHAVE_ECC > + -DECC_TIMING_RESISTANT > + -DBUILDING_WOLFSSL > + -DHAVE_HASHDRBG > + -DWOLFSSL_AES_DIRECT > + -DWOLFSSL_SHA384 > + -DWOLFSSL_SHA512 > + -DWOLFSSL_SHA224 > + -DSESSION_CERT > + -DKEEP_OUR_CERT > + -DWOLFSSL_STATIC_RSA > + -DWC_RSA_BLINDING > + -DHAVE_TLS_EXTENSIONS > + -DHAVE_AES_ECB > + -DWOLFSSL_AES_COUNTER > + -DNO_WOLFSSL_STUB) > + > +SET(WOLFSSL_SOURCES > + ${WOLFSSL_SRCDIR}/crl.c > + ${WOLFSSL_SRCDIR}/internal.c > + ${WOLFSSL_SRCDIR}/keys.c > + ${WOLFSSL_SRCDIR}/tls.c > + ${WOLFSSL_SRCDIR}/wolfio.c > + ${WOLFSSL_SRCDIR}/ocsp.c > + ${WOLFSSL_SRCDIR}/ssl.c) > +ADD_DEFINITIONS(-DWOLFSSL_LIB) > +INCLUDE_DIRECTORIES(BEFORE ${CMAKE_CURRENT_SOURCE_DIR}/wolfssl) > +IF(MSVC) > + # size_t to long truncation warning > + ADD_DEFINITIONS(-wd4267) > +ENDIF() > + > + > +ADD_CONVENIENCE_LIBRARY(wolfssl ${WOLFSSL_SOURCES}) > + > +# Workaround linker crash with older Ubuntu binutils > +# e.g aborting at ../../bfd/merge.c line 873 in _bfd_merged_section_offset > +IF((CMAKE_SYSTEM_NAME MATCHES "Linux") AND (CMAKE_LINKER STREQUAL "/usr/bin/ld")) > + EXECUTE_PROCESS(COMMAND ${CMAKE_LINKER} -v OUTPUT_VARIABLE LINKER_VERSION_STR) > + STRING(REGEX MATCH "[0-9]+\\.[0-9]+" LINKER_VERSION > + "${LINKER_VERSION_STR}") > + IF(LINKER_VERSION AND (LINKER_VERSION VERSION_LESS 2.25)) > + STRING(REPLACE "-g " "-g1 " CMAKE_C_FLAGS_RELWITHDEBINFO > + ${CMAKE_C_FLAGS_RELWITHDEBINFO}) > + STRING(REPLACE "-g " "-g1 " CMAKE_C_FLAGS_DEBUG > + ${CMAKE_C_FLAGS_DEBUG}) > + STRING(REPLACE "-ggdb3 " " " CMAKE_C_FLAGS_RELWITHDEBINFO > + ${CMAKE_C_FLAGS_RELWITHDEBINFO}) > + STRING(REPLACE "-ggdb3 " " " CMAKE_C_FLAGS_DEBUG > + ${CMAKE_C_FLAGS_DEBUG}) > + ENDIF() > +ENDIF() I would've just disabled -g\w+ unconditionally, for a simpler CMakeLists.txt We aren't going to debug wolfssl anyway (and if I'd need to debug it, for some weird reason, I know how to enable debugging temporarily) > + > + > +SET(WOLFCRYPT_SRCDIR ${CMAKE_CURRENT_SOURCE_DIR}/wolfssl/wolfcrypt/src) > +SET(WOLFCRYPT_SOURCES > +${WOLFCRYPT_SRCDIR}/aes.c > +${WOLFCRYPT_SRCDIR}/arc4.c > +${WOLFCRYPT_SRCDIR}/asn.c > +#${WOLFCRYPT_SRCDIR}/blake2b.c > +#${WOLFCRYPT_SRCDIR}/camellia.c > +#${WOLFCRYPT_SRCDIR}/chacha.c > +${WOLFCRYPT_SRCDIR}/coding.c > +#${WOLFCRYPT_SRCDIR}/compress.c > +${WOLFCRYPT_SRCDIR}/des3.c > +${WOLFCRYPT_SRCDIR}/dh.c > +${WOLFCRYPT_SRCDIR}/dsa.c > +${WOLFCRYPT_SRCDIR}/ecc.c > +${WOLFCRYPT_SRCDIR}/error.c > +#${WOLFCRYPT_SRCDIR}/hc128.c > +${WOLFCRYPT_SRCDIR}/hmac.c > +${WOLFCRYPT_SRCDIR}/integer.c > +${WOLFCRYPT_SRCDIR}/logging.c > +#${WOLFCRYPT_SRCDIR}/md2.c > +${WOLFCRYPT_SRCDIR}/md4.c > +${WOLFCRYPT_SRCDIR}/md5.c > +${WOLFCRYPT_SRCDIR}/memory.c > +#${WOLFCRYPT_SRCDIR}/pkcs7.c > +${WOLFCRYPT_SRCDIR}/pkcs12.c > +#${WOLFCRYPT_SRCDIR}/poly1305.c > +${WOLFCRYPT_SRCDIR}/pwdbased.c > +${WOLFCRYPT_SRCDIR}/rabbit.c > +${WOLFCRYPT_SRCDIR}/random.c > +#${WOLFCRYPT_SRCDIR}/ripemd.c > +${WOLFCRYPT_SRCDIR}/rsa.c > +${WOLFCRYPT_SRCDIR}/sha.c > +${WOLFCRYPT_SRCDIR}/sha256.c > +${WOLFCRYPT_SRCDIR}/sha512.c > +#${WOLFCRYPT_SRCDIR}/tfm.c > +${WOLFCRYPT_SRCDIR}/wc_port.c > +${WOLFCRYPT_SRCDIR}/wc_encrypt.c > +${WOLFCRYPT_SRCDIR}/hash.c > +${WOLFCRYPT_SRCDIR}/wolfmath.c) > +ADD_CONVENIENCE_LIBRARY(wolfcrypt ${WOLFCRYPT_SOURCES}) > + > diff --git a/extra/wolfssl/wolfssl b/extra/wolfssl/wolfssl > new file mode 160000 > index 00000000000..dc313ccf6ee > --- /dev/null > +++ b/extra/wolfssl/wolfssl > @@ -0,0 +1 @@ > +Subproject commit dc313ccf6ee0e99fb5a13793b82e412f396e11c6 I think we should only use wolfssl (any external project, actually) only at release points, not at intermediate commits. 4.0.0-stable should be ok, I guess. And, generaly, pefer to use system globally installed library, if available. Although in case of wolfssl (and libmarias3) I would not bother. > diff --git a/include/ssl_compat.h b/include/ssl_compat.h > index c94b9671d5f..105405efff2 100644 > --- a/include/ssl_compat.h > +++ b/include/ssl_compat.h > @@ -17,9 +17,9 @@ > #include <openssl/opensslv.h> > > /* OpenSSL version specific definitions */ > -#if !defined(HAVE_YASSL) && defined(OPENSSL_VERSION_NUMBER) > +#if defined(OPENSSL_VERSION_NUMBER) > > -#if OPENSSL_VERSION_NUMBER >= 0x10002000L && !defined(LIBRESSL_VERSION_NUMBER) > +#if (OPENSSL_VERSION_NUMBER >= 0x10002000L && !defined(LIBRESSL_VERSION_NUMBER)) || defined (HAVE_WOLFSSL) What's OPENSSL_VERSION_NUMBER in wolfssl? > #define HAVE_X509_check_host 1 May be move it to ssl.cmake? Where other such defines are. > #endif > > diff --git a/include/sslopt-case.h b/include/sslopt-case.h > index 4a8c65948cb..b129a97cc76 100644 > --- a/include/sslopt-case.h > +++ b/include/sslopt-case.h > @@ -29,8 +29,8 @@ > One can disable SSL later by using --skip-ssl or --ssl=0 > */ > opt_use_ssl= 1; > - /* crl has no effect in yaSSL */ > -#ifdef HAVE_YASSL > +#ifdef HAVE_WOLFSSL > + /* CRL does not work with WolfSSL */ Still? > opt_ssl_crl= NULL; > opt_ssl_crlpath= NULL; > #endif > diff --git a/mysys_ssl/my_crypt.cc b/mysys_ssl/my_crypt.cc > index 2d6f5188034..1ce0c6bbc03 100644 > --- a/mysys_ssl/my_crypt.cc > +++ b/mysys_ssl/my_crypt.cc > @@ -18,14 +18,10 @@ > #include <my_global.h> > #include <string.h> > > -#ifdef HAVE_YASSL > -#include "yassl.cc" delete yassl.cc file, perhaps? > -#else > #include <openssl/evp.h> > #include <openssl/aes.h> > #include <openssl/err.h> > #include <openssl/rand.h> > -#endif > > #include <my_crypt.h> > #include <ssl_compat.h> > @@ -70,8 +66,24 @@ class MyCTX > } > virtual int finish(uchar *dst, uint *dlen) > { > +#ifdef HAVE_WOLFSSL > + /* > + Bug in WolfSSL - sometimes EVP_CipherFinal_ex > + returns success without setting destination length > + when it should return error. > + We catch it by presetting invalid value for length, > + and checking if it has changed after the call. > + > + See https://github.com/wolfSSL/wolfssl/issues/2092 > + */ > + *dlen= UINT_MAX; > +#endif I suppose you can remove it. This is fixed as of 4.0.0-stable tag. > if (!EVP_CipherFinal_ex(ctx, dst, (int*)dlen)) > return MY_AES_BAD_DATA; > +#ifdef HAVE_WOLFSSL > + if (*dlen == UINT_MAX) > + return MY_AES_BAD_DATA; > +#endif > return MY_AES_OK; > } > }; > diff --git a/plugin/file_key_management/parser.cc b/plugin/file_key_management/parser.cc > index 13a9dfa0cb6..27277f62994 100644 > --- a/plugin/file_key_management/parser.cc > +++ b/plugin/file_key_management/parser.cc > @@ -103,7 +103,6 @@ openssl enc -aes-256-cbc -md sha1 -k "secret" -in keys.txt -out keys.enc > EVP_BytesToKey(EVP_aes_256_cbc(), EVP_sha1(), salt, > secret, strlen(secret), 1, key, iv); > > - but alas! we want to support yassl too does wolfssl have an equivalent of EVP_BytesToKey? if not, the comment still holds. > */ > > void Parser::bytes_to_key(const unsigned char *salt, const char *input, > diff --git a/sql/mysqld.cc b/sql/mysqld.cc > index bfa03aa57c1..8c3a59c55d1 100644 > --- a/sql/mysqld.cc > +++ b/sql/mysqld.cc > @@ -7317,13 +7315,13 @@ static int show_ssl_get_verify_mode(THD *thd, SHOW_VAR *var, char *buff, > { > var->type= SHOW_LONG; > var->value= buff; > -#ifndef HAVE_YASSL > +#ifndef HAVE_WOLFSSL > if( thd->net.vio && thd->net.vio->ssl_arg ) > *((long *)buff)= (long)SSL_get_verify_mode((SSL*)thd->net.vio->ssl_arg); no SSL_get_verify_mode or an analog in wolfssl? I'm asking, as I see it has gone a long way towards OpenSSL compatibility > else > *((long *)buff)= 0; > #else > - *((long *)buff) = 0; > + *((long *)buff)= 0; > #endif > return 0; > } Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 2

Re: [Maria-developers] ddec45aa50e: MDEV-16932: ASAN heap-use-after-free in my_charlen_utf8 / my_well_formed_char_length_utf8 on 2nd execution of SP with ALTER trying to add bad CHECK
by Sergei Golubchik 20 May '19

20 May '19

Hi, Oleksandr! On May 20, Oleksandr Byelkin wrote: > Alter table will trigger recompilation of the SP (as I remember all SPs) I've run this test, it complains about duplicate constraint name. > > What about this: > > > > CREATE TABLE t1 (a INT); > > CREATE PROCEDURE sp() ALTER TABLE t1 ADD CONSTRAINT CHECK (b > 0); > > --error ER_BAD_FIELD_ERROR > > CALL sp; > > alter table t1 add column b int, add constraint check (b < 10); > > CALL sp; > > > > ? > > second CALL will not regenerate the name of the constraint. > > Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] ddec45aa50e: MDEV-16932: ASAN heap-use-after-free in my_charlen_utf8 / my_well_formed_char_length_utf8 on 2nd execution of SP with ALTER trying to add bad CHECK
by Sergei Golubchik 20 May '19

20 May '19

Hi, Oleksandr! On May 20, Oleksandr Byelkin wrote: > On Mon, May 20, 2019 at 3:11 PM Sergei Golubchik <serg(a)mariadb.org> wrote: > > On May 20, Oleksandr Byelkin wrote: > > > revision-id: ddec45aa50e (mariadb-10.2.24-11-gddec45aa50e) > > > parent(s): 50999738eae > > > author: Oleksandr Byelkin <sanja(a)mariadb.com> > > > committer: Oleksandr Byelkin <sanja(a)mariadb.com> > > > timestamp: 2019-05-14 14:01:15 +0200 > > > message: > > > > > > MDEV-16932: ASAN heap-use-after-free in my_charlen_utf8 / > > > my_well_formed_char_length_utf8 on 2nd execution of SP with ALTER > > > trying to add bad CHECK > > > > > > In case of error the SP can be executed without re-comilation and so > > > will reuse constructed constaint name, so the name should be allocated > > > in the statement memory. > > > > Is that right? You're saying that the "unique constraint name" will be > > generated once and reused? > > > > But on the second SP invocation is might no longer be unique. I'd think > > that `ALTER TABLE t1 ADD CONSTRAINT CHECK (x > 0)` should add a new > > constraint, not "add a new contraint with the name that used to be > > unique when the SP was invoked for the first time". > > With the second call we have 2 possibility: > > 1) first execution failed and so we can reuse the name > 2) first execution succeed and we will re-compile SP or re-prepare PS and > so will generate new name (that is why we mostly had no problems) What about this: CREATE TABLE t1 (a INT); CREATE PROCEDURE sp() ALTER TABLE t1 ADD CONSTRAINT CHECK (b > 0); --error ER_BAD_FIELD_ERROR CALL sp; alter table t1 add column b int, add constraint check (b < 10); CALL sp; ? second CALL will not regenerate the name of the constraint. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] ddec45aa50e: MDEV-16932: ASAN heap-use-after-free in my_charlen_utf8 / my_well_formed_char_length_utf8 on 2nd execution of SP with ALTER trying to add bad CHECK
by Sergei Golubchik 20 May '19

20 May '19

Hi, Oleksandr! On May 20, Oleksandr Byelkin wrote: > revision-id: ddec45aa50e (mariadb-10.2.24-11-gddec45aa50e) > parent(s): 50999738eae > author: Oleksandr Byelkin <sanja(a)mariadb.com> > committer: Oleksandr Byelkin <sanja(a)mariadb.com> > timestamp: 2019-05-14 14:01:15 +0200 > message: > > MDEV-16932: ASAN heap-use-after-free in my_charlen_utf8 / > my_well_formed_char_length_utf8 on 2nd execution of SP with ALTER > trying to add bad CHECK > > In case of error the SP can be executed without re-comilation and so > will reuse constructed constaint name, so the name should be allocated > in the statement memory. Is that right? You're saying that the "unique constraint name" will be generated once and reused? But on the second SP invocation is might no longer be unique. I'd think that `ALTER TABLE t1 ADD CONSTRAINT CHECK (x > 0)` should add a new constraint, not "add a new contraint with the name that used to be unique when the SP was invoked for the first time". > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index 3df877792cc..ecd5de3a09d 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -4185,9 +4185,15 @@ mysql_prepare_create_table(THD *thd, HA_CREATE_INFO *create_info, > while ((check= c_it++)) > { > if (!check->name.length) > + { > + Query_arena backup; > + Query_arena *arena= thd->activate_stmt_arena_if_needed(&backup); > make_unique_constraint_name(thd, &check->name, > &alter_info->check_constraint_list, > &nr); > + if (arena) > + thd->restore_active_arena(arena, &backup); > + } > { > /* Check that there's no repeating constraint names. */ > List_iterator_fast<Virtual_column_info> Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] 66349551bf3: MDEV-16214: Incorrect plan taken by the optimizer , uses INDEX instead of ref access with ORDER BY
by Sergey Petrunia 19 May '19

19 May '19

Hi Varun, The patch is mostly fine. Please find some input on the comments / coding style below. Ok to push after it is addressed. On Fri, May 17, 2019 at 01:44:36PM +0530, Varun wrote: > revision-id: 66349551bf3a266fbefaebfe044abd05108723c2 (mariadb-10.3.12-205-g66349551bf3) > parent(s): 3d56adbfac394b2b3ffd22a89fe7c2978ed9a505 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2019-05-17 13:44:05 +0530 > message: > > MDEV-16214: Incorrect plan taken by the optimizer , uses INDEX instead of ref access with ORDER BY > > The issue in this case is that we take in account the estimates from quick keys instead of rec_per_key. > The estimates for quick keys are better than rec_per_key only if we have ref(const), so we need to check > that all keyparts in the ref key are of the type ref(const). > > --- > mysql-test/main/order_by.result | 57 +++++++++++++++++++++++++++++++++++++++++ > mysql-test/main/order_by.test | 37 ++++++++++++++++++++++++++ > sql/sql_select.cc | 42 ++++++++++++++++++++---------- > 3 files changed, 122 insertions(+), 14 deletions(-) ... > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index f36a68bc7ae..8cdcf3afc8b 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -9986,31 +9986,35 @@ static bool create_ref_for_key(JOIN *join, JOIN_TAB *j, > j->ref.null_rejecting|= (key_part_map)1 << i; > keyuse_uses_no_tables= keyuse_uses_no_tables && !keyuse->used_tables; > /* > - Todo: we should remove this check for thd->lex->describe on the next > - line. With SHOW EXPLAIN code, EXPLAIN printout code no longer depends > - on it. However, removing the check caused change in lots of query > - plans! Does the optimizer depend on the contents of > - table_ref->key_copy ? If yes, do we produce incorrect EXPLAINs? > + We don't want to compute heavy expressions in EXPLAIN, an example would > + select * from t1 where t1.key=(select thats very heavy); > + > + (select thats very heavy) => is a constant here > + eg: (select avg(order_cost) from orders) => constant but expensive > */ > if (!keyuse->val->used_tables() && !thd->lex->describe) > { // Compare against constant > - store_key_item tmp(thd, > + store_key_item tmp(thd, > keyinfo->key_part[i].field, > key_buff + maybe_null, > maybe_null ? key_buff : 0, > keyinfo->key_part[i].length, > keyuse->val, > FALSE); > - if (unlikely(thd->is_fatal_error)) > - DBUG_RETURN(TRUE); > - tmp.copy(); > + if (unlikely(thd->is_fatal_error)) > + DBUG_RETURN(TRUE); > + tmp.copy(); > j->ref.const_ref_part_map |= key_part_map(1) << i ; > } > else > - *ref_key++= get_store_key(thd, > - keyuse,join->const_table_map, > - &keyinfo->key_part[i], > - key_buff, maybe_null); > + { > + *ref_key++= get_store_key(thd, > + keyuse,join->const_table_map, > + &keyinfo->key_part[i], > + key_buff, maybe_null); > + if (!keyuse->val->used_tables()) > + j->ref.const_ref_part_map |= key_part_map(1) << i ; > + } > /* > Remember if we are going to use REF_OR_NULL > But only if field _really_ can be null i.e. we force JT_REF > @@ -25256,6 +25260,8 @@ bool JOIN_TAB::save_explain_data(Explain_table_access *eta, > { > if (!(eta->ref_list.append_str(thd->mem_root, "const"))) > return 1; > + if (thd->lex->describe) > + key_ref++; This is very cryptic. Please provide an explanation, something like: create_ref_for_key() handles keypart=const equalities as follows: - non-EXPLAIN execution will copy the "const" to lookup tuple immediately and will not add an element to ref.key_copy - EXPLAIN will put an element into ref.key_copy. Since we've just printed "const" for it, we should skip it here. > } > else > { > @@ -26921,7 +26927,15 @@ test_if_cheaper_ordering(const JOIN_TAB *tab, ORDER *order, TABLE *table, > */ > if (ref_key >= 0 && ref_key != MAX_KEY && tab->type == JT_REF) > { > - if (table->quick_keys.is_set(ref_key)) > + /* > + For all the parts of the ref key we check if all of them belong > + to the type ref(const). This is done because if all parts of the ref > + key are of type ref(const), then we are sure that the estimates > + provides by quick keys is better than that provide by rec_per_key. The above is hard to read and has typos. Here's a shorter variant: If ref access uses keypart=const for all its key parts, and quick select uses the same # of key parts, then they are equivalent. Reuse #rows estimate from quick select as it is more precise. > + */ > + if (tab->ref.const_ref_part_map == make_prev_keypart_map(tab->ref.key_parts) && > + table->quick_keys.is_set(ref_key) && > + table->quick_key_parts[ref_key] == tab->ref.key_parts) Please fix indetation; One of the lines is too long. > refkey_rows_estimate= table->quick_rows[ref_key]; > else > { BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] FYI: usage of F1-F12 keys in mysql client
by Aleksey Midenkov 17 May '19

17 May '19

By adding something like this in your .inputrc: $if MYSQL "\033OP": "use information_schema; show tables;\n" "\033OQ": "use test;\n" "\033OR": "select * from " "\033OS": "show columns from " "\033[15~": "\033[17~": "\033[18~": "\033[19~": "\033[20~": "\033[21~": "\033[23~": "\033[24~": $endif you may have F1-F12 keys utilized. Different terminals have different codes (.inputrc doesn't support capability names unfortunately), so you'll have to find out your own codes via tack utiity. With strong will and enough patience it is possible to add Ctrl+, Alt+, Shift+ and their combinations as well. This works only with MDEV-17609 patch. -- All the best, Aleksey Midenkov @midenok

1 0

Re: [Maria-developers] MDEV-19284-19285v3
by Eugene Kosov 14 May '19

14 May '19

Hi, Alexander! Patch looks good to me. 13.05.2019, 17:03, "Alexander Barkov" <bar(a)mariadb.com>: > Hi Eugene, > > Please review the next round of the patch. > > Are any changes in tests still needed? > > Thanks! -- Eugene

1 0

Re: [Maria-developers] 7198e7c71fc: MDEV-15458 Segfault in heap_scan() upon UPDATE after ADD SYSTEM VERSIONING
by Sergei Golubchik 13 May '19

13 May '19

Hi, Aleksey! On Dec 23, Aleksey Midenkov wrote: > revision-id: 7198e7c71fc (versioning-1.0.6-82-g7198e7c71fc) > parent(s): 6be155757b7 > author: Aleksey Midenkov <midenok(a)gmail.com> > committer: Aleksey Midenkov <midenok(a)gmail.com> > timestamp: 2018-12-20 13:25:00 +0300 > message: > > MDEV-15458 Segfault in heap_scan() upon UPDATE after ADD SYSTEM VERSIONING > > * HA_EXTRA_REMEMBER_POS, HA_EXTRA_RESTORE_POS for HEAP storage engine > * Versioning tests support As far as I can see, there are two approaches to fixing this bug. And if we go the way of HA_EXTRA_REMEMBER_POS/HA_EXTRA_RESTORE_POS then your fix is good. But I'd still like to understand whether write_row is supposed to disturb the scan. Why should heap_write() overwrite info->current_ptr=pos and info->current_hash_ptr? Inserting a row doesn't happen at the "current position", so I don't quite understand why it modifies current_ptr at all. There is no functionality like "continue the scan from the last inserted row", right? It just doesn't make sense to me. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 2

[Maria-developers] Welcome to GSoC 2019
by Vicențiu Ciorbaru 10 May '19

10 May '19

Hi! As you may have noticed a few days ago, you have received an email from Google that you have been accepted as part of Google Summer of Code. Congratulations on the effort to learn about the program, get involved and submit succesful proposals. The fun part begins now! If you have not done so already, please reach out to your mentor and seek further advice about your project. Each one of your projects can be implemented in multiple ways, some more complex than others. I strongly advise you to create a concrete plan based off of your proposal with your mentor and aim to stick to it. Your mentor can guide you to ensure you have a succesful GSoC. Remember that you are here to learn about Open Source, what it means to be part of an Open Source community, to interact with the community, all while becoming better programmers. Please take advantage of this opportunity and keep your technical discussions public, on Zulip https://mariadb.zulipchat.com, and on our developer mailing list https://launchpad.net/~maria-developers. By keeping your discussions public, you get the benefit of not only having your mentor see your questions but all of the MariaDB community. You never know when someone might lend a hand in case your mentor is not yet online. :) The mantra of our community should be that we are always welcoming to new people. Do not be afraid to ask questions. There are no bad or stupid or silly questions. The only key requirement is that you first make an effort to find the answer yourself and only afterwards seek outside help. If you can show a bit of involvement, the community will show you the same. I recommend that you agree with your mentor on some form of weekly progress reporting, via either a blog-post, an email, or something similar that can be publicly shared. Hope all of you will have a great GSoC! Vicențiu MariaDB Foundation GSoC 2019 Admin

1 0

Re: [Maria-developers] a4be420c920: MDEV-19235 MariaDB Server compiled for 128 Indexes crashes at startup
by Sergei Golubchik 09 May '19

09 May '19

Hi, Vladislav! Thanks. That's pretty much the solution I had in mind too :) A couple of comments on details below. I've only commented on sql_bitmap.h, everything else is fine. On May 08, Vladislav Vaintroub wrote: > revision-id: a4be420c920 (mariadb-10.4.4-25-ga4be420c920) > parent(s): ee4a2fef181 > author: Vladislav Vaintroub <wlad(a)mariadb.com> > committer: Vladislav Vaintroub <wlad(a)mariadb.com> > timestamp: 2019-04-17 17:17:06 +0100 > message: > > MDEV-19235 MariaDB Server compiled for 128 Indexes crashes at startup > > With MAX_INDEXIES=64(default), key_map=Bitmap<64> is just a wrapper around > ulonglong and thus "trivial" (can be bzero-ed, or memcpy-ed, and stays > valid still) > > With MAX_INDEXES=128, key_map = Bitmap<128> is not a "trivial" type > anymore. The implementation uses MY_BITMAP, and MY_BITMAP contains pointers > which make Bitmap invalid, when it is memcpy-ed/bzero-ed. > > The problem in 10.4 is that there are many new key_map members, inside TABLE > or KEY, and those are often memcopied and bzeroed > > The fix makes Bitmap "trivial", by inlining most of MY_BITMAP functionality. > pointers/heap allocations are not used anymore. > diff --git a/sql/sql_bitmap.h b/sql/sql_bitmap.h > index 705a8d169e1..74866336a1a 100644 > --- a/sql/sql_bitmap.h > +++ b/sql/sql_bitmap.h > @@ -25,73 +25,184 @@ > > #include <my_sys.h> > #include <my_bitmap.h> > +#include <my_bit.h> > > template <uint width> class Bitmap > { > - MY_BITMAP map; > +public: > uint32 buffer[(width+31)/32]; Why is it public? > public: > - Bitmap() { init(); } > + Bitmap():buffer() What does that do? > + { > + } > - Bitmap(const Bitmap& from) { *this=from; } > - explicit Bitmap(uint prefix_to_set) { init(prefix_to_set); } > + explicit Bitmap(uint prefix_to_set):buffer() > + { > + set_prefix(prefix_to_set); > + } > - void init() { my_bitmap_init(&map, buffer, width, 0); } > - void init(uint prefix_to_set) { init(); set_prefix(prefix_to_set); } > uint length() const { return width; } > - Bitmap& operator=(const Bitmap& map2) > - { > + void init() > + { > + clear_all(); > + } > + void init(uint prefix_to_set) > + { > init(); > - memcpy(buffer, map2.buffer, sizeof(buffer)); > - return *this; > + set_prefix(prefix_to_set); I wonder whether init() method is still needed. You've removed it from the constructor. And if there's any other code that uses it, I'd rather it'd be using clear_all(). > } > - void set_bit(uint n) { bitmap_set_bit(&map, n); } > + void set_bit(uint n) > + { > + DBUG_ASSERT(n < width); > + ((uchar*)buffer)[n / 8] |= (1 << (n & 7)); > + } > - void clear_bit(uint n) { bitmap_clear_bit(&map, n); } > + void clear_bit(uint n) > + { > + DBUG_ASSERT(n < width); > + ((uchar*)buffer)[n / 8] &= ~(1 << (n & 7)); > + } > - void set_prefix(uint n) { bitmap_set_prefix(&map, n); } > + void set_prefix(uint prefix_size) > + { > + set_if_smaller(prefix_size, width); > + uint prefix_bytes, prefix_bits, d; > + uchar* m = (uchar*)buffer; > + > + if ((prefix_bytes = prefix_size / 8)) > + memset(m, 0xff, prefix_bytes); > + m += prefix_bytes; > + if ((prefix_bits = prefix_size & 7)) > + { > + *(m++) = (1 << prefix_bits) - 1; > + // As the prefix bits are set, lets count this byte too as a prefix byte. > + prefix_bytes++; > + } > + if ((d = (width + 7) / 8 - prefix_bytes)) > + memset(m, 0, d); > + } > - void set_all() { bitmap_set_all(&map); } > + void set_all() > + { > + set_prefix(width); may be better to optimize it? memset(buffer, 0xff, sizeof(buffer) - 1); ((uchar*)buffer)[sizeof(buffer)-1] = last_byte_mask(width); > + } > - void clear_all() { bitmap_clear_all(&map); } > + void clear_all() > + { > + memset(buffer, 0x00, sizeof(buffer)); > + } > - void intersect(Bitmap& map2) { bitmap_intersect(&map, &map2.map); } > + void intersect(Bitmap & map2) > + { > + for (uint i = 0; i < array_elements(buffer); i++) > + buffer[i] &= map2.buffer[i]; > + } > - void intersect(ulonglong map2buff) > - { > - // Use a spearate temporary buffer, as bitmap_init() clears all the bits. > - ulonglong buf2; > - MY_BITMAP map2; > - > - my_bitmap_init(&map2, (uint32 *) &buf2, sizeof(ulonglong) * 8, 0); > - > - // Store the original bits. > - if (sizeof(ulonglong) >= 8) > - { > - int8store(const_cast<uchar *>(static_cast<uchar *> > - (static_cast<void *>(&buf2))), > - map2buff); > - } > - else > - { > - DBUG_ASSERT(sizeof(buffer) >= 4); > - int4store(const_cast<uchar *>(static_cast<uchar *> > - (static_cast<void *>(&buf2))), > - static_cast<uint32>(map2buff)); > - } > - bitmap_intersect(&map, &map2); > - } > + void intersect_and_pad(ulonglong map2buff, bool pad_with_ones) make it private? > + { > + compile_time_assert(sizeof(ulonglong) == 8); > + uint32 tmp[2]; > + int8store(tmp, map2buff); > + if (array_elements(buffer) > 2 && pad_with_ones) > + set_all(); really? I'd expect that if pad_with_ones==true, you simply don't need to do anything to the rest of the buffer. > + > + buffer[0] &= tmp[0]; > + if (array_elements(buffer) > 1) > + buffer[1] &= tmp[1]; > + > + if (array_elements(buffer) > 2 && !pad_with_ones) > + memset((char*)buffer + 8, 0 , sizeof(buffer) - 8); > + } > + void intersect(ulonglong map2buff) > + { > + intersect_and_pad(map2buff, 0); > + } > /* Use highest bit for all bits above sizeof(ulonglong)*8. */ > void intersect_extended(ulonglong map2buff) > { > - intersect(map2buff); > - if (map.n_bits > sizeof(ulonglong) * 8) > - bitmap_set_above(&map, sizeof(ulonglong), > - MY_TEST(map2buff & (1LL << (sizeof(ulonglong) * 8 - 1)))); > + intersect_and_pad(map2buff, (map2buff & (1ULL << 63))); > } > - void subtract(Bitmap& map2) { bitmap_subtract(&map, &map2.map); } > + void subtract(Bitmap & map2) > + { > + for (size_t i = 0; i < array_elements(buffer); i++) > + buffer[i] &= ~(map2.buffer[i]); > + } > - void merge(Bitmap& map2) { bitmap_union(&map, &map2.map); } > + void merge(Bitmap & map2) > + { > + for (size_t i = 0; i < array_elements(buffer); i++) > + buffer[i] |= map2.buffer[i]; > + } > - bool is_set(uint n) const { return bitmap_is_set(&map, n); } > + bool is_set(uint n) const > + { > + DBUG_ASSERT(n < width); > + return ((uchar*)buffer)[n / 8] & (1 << (n & 7)); > + } > - bool is_prefix(uint n) const { return bitmap_is_prefix(&map, n); } > + bool is_prefix(uint prefix_size) const > + { > + uint prefix_mask = last_byte_mask(prefix_size); > + uchar* m = (uchar*)buffer; > + uchar* end_prefix = m + (prefix_size - 1) / 8; > + uchar* end; > + DBUG_ASSERT(prefix_size <= width); > + > + /* Empty prefix is always true */ > + if (!prefix_size) > + return true; > + > + while (m < end_prefix) > + if (*m++ != 0xff) > + return false; > + > + end = ((uchar*)buffer) + (width + 7) / 8 - 1; > + if (m == end) > + return ((*m & last_byte_mask(width)) == prefix_mask); > + > + if (*m != prefix_mask) > + return false; > + > + while (++m < end) > + if (*m != 0) > + return false; > + return ((*m & last_byte_mask(width)) == 0); > + } > - bool is_clear_all() const { return bitmap_is_clear_all(&map); } > + bool is_clear_all() const > + { > + for (size_t i= 0; i < array_elements(buffer); i++) > + if (buffer[i]) > + return false; > + return true; > + } > - bool is_set_all() const { return bitmap_is_set_all(&map); } > + bool is_set_all() const > + { > + if (width == sizeof(buffer) * 8) > + { > + for (size_t i = 0; i < array_elements(buffer); i++) > + if (buffer[i] != 0xFFFFFFFFU) > + return false; > + return true; > + } > + else > + return is_prefix(width); this can also be optimized to not use complex is_prefix(), but I don't think is_set_all() is used as often as set_all(), so it shouldn't matter as much. > + } > - bool is_subset(const Bitmap& map2) const { return bitmap_is_subset(&map, &map2.map); } > + bool is_subset(const Bitmap & map2) const > + { > + for (size_t i= 0; i < array_elements(buffer); i++) > + if (buffer[i] & ~(map2.buffer[i])) > + return false; > + return true; > + } > - bool is_overlapping(const Bitmap& map2) const { return bitmap_is_overlapping(&map, &map2.map); } > + bool is_overlapping(const Bitmap & map2) const > + { > + for (size_t i = 0; i < array_elements(buffer); i++) > + if (buffer[i] & map2.buffer[i]) > + return true; > + return false; > + } > - bool operator==(const Bitmap& map2) const { return bitmap_cmp(&map, &map2.map); } > + bool operator==(const Bitmap & map2) const > + { > + return memcmp(buffer, map2.buffer, sizeof(buffer)) == 0; > + } > - bool operator!=(const Bitmap& map2) const { return !(*this == map2); } > + bool operator!=(const Bitmap & map2) const > + { > + return !(*this == map2); > + } > char *print(char *buf) const > { > char *s=buf; Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 2

Re: [Maria-developers] [Commits] 0ff8b73b60f: MDEV-15777:Support Early NULLs filtering-like restrictions in the range optimizer
by Sergey Petrunia 06 May '19

06 May '19

Hi Varun, Please finally find the review below. The patch is almost there I think, but there are some minor adjustments that still need to be made. > revision-id: 0ff8b73b60fae40556f91d3959f20bf3cf182b28 (mariadb-10.3.0-947-g0ff8b73b60f) > parent(s): 15419a558370aeed9521b498c34d50f20a8d47a5 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2018-05-15 13:53:11 +0530 > message: > > MDEV-15777:Support Early NULLs filtering-like restrictions in the range optimizer > > --- > mysql-test/main/mdev15777.result | 455 +++++++++++++++++++++++++++++++++++++++ > mysql-test/main/mdev15777.test | 138 ++++++++++++ > sql/opt_range.cc | 125 ++++++++++- > sql/opt_range.h | 2 + > sql/sql_select.cc | 22 +- > sql/table.cc | 1 + > sql/table.h | 6 + > 7 files changed, 745 insertions(+), 4 deletions(-) > > diff --git a/mysql-test/main/mdev15777.result b/mysql-test/main/mdev15777.result > new file mode 100644 > index 00000000000..549ac08b91f > --- /dev/null > +++ b/mysql-test/main/mdev15777.result > @@ -0,0 +1,455 @@ > +# Test added to check that null filtering is used by range optimizer > +create table ten(a int); > +insert into ten values (0),(1),(2),(3),(4),(5),(6),(7),(8),(9); > +create table one_k(a int); > +insert into one_k select A.a + B.a* 10 + C.a * 100 from ten A, ten B, ten C; > +create table one_m(a int); > +insert into one_m select A.a + B.a* 1000 from one_k A, one_k B; The testcases are extremely confusing. * Do you really need a table with one millon rows to demonstrate the issue? * Why use examples with subqueries (which are converted into join anyway)? - if we want to verify that this optimization works together with subquery-to-semi-join conversion, let's start with a join example, and then construct a subquery which is converted to the query. * The testcases produce a lot of EXPLAIN outputs, and there is no explanation provided what one should expect. Please create a testcase: - of reasonable size (I think tables of 10-20K rows should suffice) - start from a join query - Like in other testcases, tables should be named, t1,t2, ... - I assume the query plan should be: range access on the first table (constructed from a NOT NULL predicate), ref access for the second one. - The testcase should have '--echo #' comments explaining what is relevant in the query plan > +delete from one_m where a=0 limit 1; > +create table t1 ( > +id int(10) unsigned NOT NULL AUTO_INCREMENT, > +filler varchar(100), > +subset_id int(11) DEFAULT NULL, > +PRIMARY KEY (id), > +KEY t1_subset_id (subset_id) > +); > +create table t1_subsets ( ... > diff --git a/sql/opt_range.cc b/sql/opt_range.cc > index 8422917065f..ae06b899ac9 100644 > --- a/sql/opt_range.cc > +++ b/sql/opt_range.cc > @@ -155,6 +155,7 @@ class SEL_IMERGE; > #define CLONE_KEY1_MAYBE 1 > #define CLONE_KEY2_MAYBE 2 > #define swap_clone_flag(A) ((A & 1) << 1) | ((A & 2) >> 1) > +#define FT_KEYPART (MAX_FIELDS+10) > > > /* > @@ -2400,6 +2401,7 @@ int SQL_SELECT::test_quick_select(THD *thd, key_map keys_to_use, > { > uint idx; > double scan_time; > + Item *null_rejecting_conds= NULL; > DBUG_ENTER("SQL_SELECT::test_quick_select"); > DBUG_PRINT("enter",("keys_to_use: %lu prev_tables: %lu const_tables: %lu", > (ulong) keys_to_use.to_ulonglong(), (ulong) prev_tables, > @@ -2423,6 +2425,7 @@ int SQL_SELECT::test_quick_select(THD *thd, key_map keys_to_use, > read_time= (double) records + scan_time + 1; // Force to use index > > possible_keys.clear_all(); > + null_rejecting_conds= head->null_rejecting_conds; > > DBUG_PRINT("info",("Time to scan table: %g", read_time)); > > @@ -2431,7 +2434,7 @@ int SQL_SELECT::test_quick_select(THD *thd, key_map keys_to_use, > { > uchar buff[STACK_BUFF_ALLOC]; > MEM_ROOT alloc; > - SEL_TREE *tree= NULL; > + SEL_TREE *tree= NULL, *not_null_cond_tree= NULL; > KEY_PART *key_parts; > KEY *key_info; > PARAM param; > @@ -2540,6 +2543,12 @@ int SQL_SELECT::test_quick_select(THD *thd, key_map keys_to_use, > TRP_GROUP_MIN_MAX *group_trp; > double best_read_time= read_time; > Please add a comment saying that as far as range optimizer is concerned, null_rejecting_conds is just an extra condition on this table. (That is, we don't care if it has NOT NULL conditions or something else). > + if (null_rejecting_conds) > + { > + not_null_cond_tree= null_rejecting_conds->get_mm_tree(&param, > + &null_rejecting_conds); > + } > + > if (cond) > { > if ((tree= cond->get_mm_tree(&param, &cond))) > @@ -2558,6 +2567,13 @@ int SQL_SELECT::test_quick_select(THD *thd, key_map keys_to_use, > tree= NULL; > } > } > + if (not_null_cond_tree) > + { > + if (!tree) > + tree= not_null_cond_tree; > + else > + tree= tree_and(&param, tree, not_null_cond_tree); This if-else logic is redundant, as it's already present in tree_and(). Can just have this: tree= tree_and(tree, not_null_cond_tree); > + } > > /* > Try to construct a QUICK_GROUP_MIN_MAX_SELECT. > @@ -14643,6 +14659,113 @@ void QUICK_GROUP_MIN_MAX_SELECT::add_keys_and_lengths(String *key_names, > add_key_and_length(key_names, used_lengths, &first); > } > > +inline void add_cond(THD *thd, Item **e1, Item *e2) > +{ > + if (*e1) > + { > + if (!e2) > + return; > + Item *res; > + if ((res= new (thd->mem_root) Item_cond_and(thd, *e1, e2))) > + { > + res->fix_fields(thd, 0); > + res->update_used_tables(); > + *e1= res; > + } > + } > + else > + *e1= e2; > +} > + > +/* > + Create null rejecting conditions for a table, for all the equalites > + present in the WHERE clause of a query. > + > + SYNOPSIS > + make_null_rejecting_conds() > + @param TABLE - Keys of this table will participate in null > + rejecting conditions > + @param keyuse_array - array that has all the equalites of the > + WHERE clasuse > + > + DESCRIPTION > + This function creates null rejecting conditions for a table. These > + conditions are created to do range analysis on them , the conditions > + are of the form tbl.key.keypart IS NOT NULL. > + > + IMPLEMENTATION > + Lookup in the keyuse array to check if it has equalites that belong > + to the given table. If yes then find out if the conditions are null > + rejecting and accordingly create all the condition for the keys of a > + given table and AND them. > + > + > + RETURN > + NOT NULL - Found null rejecting conditions for the given table > + NULL - No null rejecting conditions for the given table > +*/ > + > +void make_null_rejecting_conds(THD *thd, TABLE *table, > + DYNAMIC_ARRAY *keyuse_array, key_map *const_keys) ^ Please fix identation. > +{ > + KEY *keyinfo; > + Item *cond= NULL; > + KEYUSE* keyuse; > + > + /* > + The null rejecting conds added will be on the keypart of a key, so for > + that we need the table to atleast have a key. > + */ please fix typos/wording in the above comment. > + if (!table->s->keys) > + return ; > + if (table->null_rejecting_conds) > + return; > + > + for(uint i=0; i < keyuse_array->elements; i++) > + { This loop does excessive work. Check out how best_access_path() examines the KEYUSE array. Key points - the array is sorted by table, index_nr, key_part_nr. - JOIN_TAB::keyuse points to the first element that refers to the table described by the JOIN_TAB - The array may have "duplicates", especially when multiple equalities are present: KEYUSE(t.keyXpartY = other_table.colX) KEYUSE(t.keyXpartY = other_table.colY) KEYUSE(t.keyXpartY = some_expr) ... Since the elements are sorted by keyXpartY, it is trival to avoid generating multiple "t.keyXpartY IS NOT NULL" for consecutive KEYUSEs. I would go even further: the same "t.keyXpartY" can be part of multiple indexes (and so its KEYUSE objects will not be adjacent). Can we use a table's column bitmap to avoid generating duplicate IS NOT NULL objects? > + keyuse= (KEYUSE*)dynamic_array_ptr(keyuse_array, i); > + if (keyuse->table == table) > + { > + /* > + No null rejecting conds for a hash key orr full-text keys > + */ please fix typos/wording in the above comment. > + if (keyuse->key == MAX_KEY || keyuse->keypart == FT_KEYPART) > + continue; > + keyinfo= keyuse->table->key_info+keyuse->key; > + Field *field= keyinfo->key_part[keyuse->keypart].field; > + > + /* > + No need to add null-rejecting condition if we have a > + keyuse element as > + - table.key.keypart= const > + - (table.key.keypart= tbl.otherfield or table.key.keypart IS NULL) > + - table.key.keypart IS NOT NULLABLE > + */ > + > + if (keyuse->val->const_item() > + || !(keyuse->null_rejecting && field->maybe_null()) Please remove the brackets: !(A && B) -> !A || !B > + || keyuse->optimize & KEY_OPTIMIZE_REF_OR_NULL) Please also put the '||' at the end of the lines, like it is done in the rest of the code. > + continue; > + > + Item_field *field_item= new (thd->mem_root)Item_field(thd, field); > + Item* not_null_item= new (thd->mem_root)Item_func_isnotnull(thd, > + field_item); Please add error handling (just return from the function if we failed to allocate any of the above) > + > + /* > + adding the key to const keys as we have the condition > + as key.keypart IS NOT NULL > + */ > + > + const_keys->set_bit(keyuse->key); > + not_null_item->fix_fields(thd, 0); > + not_null_item->update_used_tables(); > + add_cond(thd, &cond, not_null_item); > + } > + } > + table->null_rejecting_conds= cond; > + return; Please remove the above 'return;' as it is redundant and makes one suspect a merge error > +} > + > > #ifndef DBUG_OFF > > diff --git a/sql/opt_range.h b/sql/opt_range.h > index bd85a12d4a1..894d46a892c 100644 > --- a/sql/opt_range.h > +++ b/sql/opt_range.h > @@ -1728,6 +1728,8 @@ bool calculate_cond_selectivity_for_table(THD *thd, TABLE *table, Item **cond); > bool prune_partitions(THD *thd, TABLE *table, Item *pprune_cond); > #endif > void store_key_image_to_rec(Field *field, uchar *ptr, uint len); > +void make_null_rejecting_conds(THD *thd, TABLE *table, > + DYNAMIC_ARRAY *keyuse_array, key_map *const_keys); Please fix identation. > > extern String null_string; > > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index f658b78c33c..4aceb333340 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -4805,6 +4805,9 @@ make_join_statistics(JOIN *join, List<TABLE_LIST> &tables_list, > add_group_and_distinct_keys(join, s); > > s->table->cond_selectivity= 1.0; > + > + make_null_rejecting_conds(join->thd, s->table, > + keyuse_array, &s->const_keys); > > /* > Perform range analysis if there are keys it could use (1). > @@ -5352,15 +5356,24 @@ add_key_field(JOIN *join, > If the condition has form "tbl.keypart = othertbl.field" and > othertbl.field can be NULL, there will be no matches if othertbl.field > has NULL value. > + > + The field KEY_FIELD::null_rejecting is set to TRUE if we have both > + the left and right hand side of the equality are NULLABLE > + This is NOT what is happening. null_rejecting is set to true, if *either* left or right-hand side of the equality is nullable. Please fix the comment. Please also fix the comment in KEY_FIELD::null_rejecting to reflect that. > We use null_rejecting in add_not_null_conds() to add > 'othertbl.field IS NOT NULL' to tab->select_cond. > + > + We use null_rejecting in make_null_rejecting_conds() to add > + tbl.keypart IS NOT NULL so we can do range analysis on this condition > + Please join the above two sentences together: null_rejecting is used - by add_not_null_conds(), to add "othertbl.field IS NOT NULL" to othertbl's tab->select_cond. (This is called "Early NULLs filtering") - by make_null_rejecting_conds(), to provide range optimizer with additional "tbl.keypart IS NOT NULL" condition. > */ > { > Item *real= (*value)->real_item(); > if (((cond->functype() == Item_func::EQ_FUNC) || > (cond->functype() == Item_func::MULT_EQUAL_FUNC)) && > - (real->type() == Item::FIELD_ITEM) && > + (((real->type() == Item::FIELD_ITEM) && > ((Item_field*)real)->field->maybe_null()) > + ||(field->maybe_null()))) ^Please fix the identation. > (*key_fields)->null_rejecting= true; > else > (*key_fields)->null_rejecting= false; > @@ -9813,7 +9826,10 @@ static bool create_ref_for_key(JOIN *join, JOIN_TAB *j, > uint maybe_null= MY_TEST(keyinfo->key_part[i].null_bit); > j->ref.items[i]=keyuse->val; // Save for cond removal > j->ref.cond_guards[i]= keyuse->cond_guard; > - if (keyuse->null_rejecting) > + Item *real= (keyuse->val)->real_item(); > + if (keyuse->null_rejecting && > + (real->type() == Item::FIELD_ITEM) && > + ((Item_field*)real)->field->maybe_null()) ^ Please fix identation > j->ref.null_rejecting|= (key_part_map)1 << i; > keyuse_uses_no_tables= keyuse_uses_no_tables && !keyuse->used_tables; > /* > @@ -18538,7 +18554,7 @@ free_tmp_table(THD *thd, TABLE *entry) > DBUG_ASSERT(entry->pos_in_table_list->table == entry); > entry->pos_in_table_list->table= NULL; > } > - > + entry->null_rejecting_conds= NULL; What is this for? As far as I understand we are freeing the TABLE object > free_root(&own_root, MYF(0)); /* the table is allocated in its own root */ > thd_proc_info(thd, save_proc_info); > > diff --git a/sql/table.cc b/sql/table.cc > index a6e445d0d2e..097fee465de 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -4597,6 +4597,7 @@ void TABLE::init(THD *thd, TABLE_LIST *tl) > created= TRUE; > cond_selectivity= 1.0; > cond_selectivity_sampling_explain= NULL; > + null_rejecting_conds= NULL; > #ifdef HAVE_REPLICATION > /* used in RBR Triggers */ > master_had_triggers= 0; > diff --git a/sql/table.h b/sql/table.h > index 6fd3f219914..50d47899b49 100644 > --- a/sql/table.h > +++ b/sql/table.h > @@ -1356,6 +1356,12 @@ struct TABLE > SplM_opt_info *spl_opt_info; > key_map keys_usable_for_splitting; > > + /* > + Null rejecting conds added for all tables so we can do range analysis > + on these conditions > + */ "for all tables"? Please change the comment to something like: "NULL-rejecting conditions for this table. They are created from eq_ref access, and used by the range optimizer". > + Item* null_rejecting_conds; > + > void init(THD *thd, TABLE_LIST *tl); > bool fill_item_list(List<Item> *item_list) const; > void reset_item_list(List<Item> *item_list, uint skip) const; BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] [MDEV-18684] [5.5] memory leak
by Michal Schorm 04 May '19

04 May '19

Hello, can somebody please take a brief look to atleast evaulate, how hard it would be to backoport the patch form the latest library versions? https://jira.mariadb.org/browse/MDEV-18684 Thank you -- Michal Schorm Software Engineer Core Services - Databases Team Red Hat --

1 0

Re: [Maria-developers] [Commits] 5e23f561984: MDEV-18741: Optimizer trace: multi-part key ranges are printed incorrectly
by Sergey Petrunia 03 May '19

03 May '19

Hi Varun, Please find my review comments below. On Fri, Apr 05, 2019 at 02:59:17PM +0530, Varun wrote: > revision-id: 5e23f56198493ed19ac270759963f615324b2c49 (mariadb-10.4.3-162-g5e23f561984) > parent(s): 02d9b048a2ab549a3227a81e15ff2f8c45562a65 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2019-04-05 14:58:39 +0530 > message: > > MDEV-18741: Optimizer trace: multi-part key ranges are printed incorrectly > > Changed the function append_range_all_keyparts to use sel_arg_range_seq_init / sel_arg_range_seq_next to produce ranges. > Also adjusted to print format for the ranges, now the ranges are printed as: > (keypart1_min, keypart2_min,..) OP (keypart1_name,keypart2_name, ..) OP (keypart1_max,keypart2_max, ..) > > Also added more tests for range and index merge access for optimizer trace > > --- > mysql-test/main/opt_trace.result | 276 ++++++++++- > mysql-test/main/opt_trace.test | 57 +++ > mysql-test/main/opt_trace_index_merge.result | 509 ++++++++++++++++++++- > mysql-test/main/opt_trace_index_merge.test | 112 +++++ > .../main/opt_trace_index_merge_innodb.result | 6 +- > sql/opt_range.cc | 281 ++++++------ > sql/opt_range.h | 7 +- > 7 files changed, 1079 insertions(+), 169 deletions(-) ... > +explain select * from t1 force index(a_b_c) where a between 1 and 4 and b < 50; > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 range a_b_c a_b_c 8 NULL 4 Using index condition > +select JSON_DETAILED(JSON_EXTRACT(trace, '$**.analyzing_range_alternatives')) from INFORMATION_SCHEMA.OPTIMIZER_TRACE; > +JSON_DETAILED(JSON_EXTRACT(trace, '$**.analyzing_range_alternatives')) > +[ > + > + { > + "range_scan_alternatives": > + [ > + > + { > + "index": "a_b_c", > + "ranges": > + [ > + "(1) <= (a) < (4,50)" In the example above, only the name of the first key part is printed, even if the second endpoint uses two. As far as I understand the code, just using range->start_key.keypart_map | range->end_key.keypart_map in print_keyparts_name will be sufficient to fix this. > + ], > + "rowid_ordered": false, > + "using_mrr": false, > + "index_only": false, > + "rows": 4, > + "cost": 6.2648, > + "chosen": true > + } > + ], > + "analyzing_roworder_intersect": > + { > + "cause": "too few roworder scans" > + }, > + "analyzing_index_merge_union": > + [ > + ] > + } > +] > diff --git a/sql/opt_range.cc b/sql/opt_range.cc > index 5ab3d70214d..1827e2fcaf2 100644 > --- a/sql/opt_range.cc > +++ b/sql/opt_range.cc > @@ -431,16 +431,18 @@ static int and_range_trees(RANGE_OPT_PARAM *param, > static bool remove_nonrange_trees(RANGE_OPT_PARAM *param, SEL_TREE *tree); > > static void print_key_value(String *out, const KEY_PART_INFO *key_part, > - const uchar *key); > + const uchar* key, uint length); > +static void print_keyparts_name(String *out, const KEY_PART_INFO *key_part, > + uint n_keypart, key_part_map keypart_map); > > static void append_range_all_keyparts(Json_writer_array *range_trace, > - String *range_string, > - String *range_so_far, const SEL_ARG *keypart, > + PARAM param, uint idx, > + SEL_ARG *keypart, > const KEY_PART_INFO *key_parts); > > static > -void append_range(String *out, const KEY_PART_INFO *key_parts, > - const uchar *min_key, const uchar *max_key, const uint flag); > +void append_range(String *out, const KEY_PART_INFO *key_part, > + KEY_MULTI_RANGE *range, uint n_key_parts); Why is this function called append_() while other functions that print stuff into strings are called print_key_value, print_keyparts_name ? The name "append_range_all_keyparts" is also confusing. The function prints a sequence of ranges. One can say it "appends" them to the trace, but that verb is not used to refer to this action anywhere else. Functions that print something into trace are called trace_XXX(), e.g. TRP_RANGE::trace_basic_info(), trace_plan_prefix(), etc. I think trace_ranges() would be a better name here. What do you think? > > > /* > @@ -2273,10 +2275,7 @@ void TRP_RANGE::trace_basic_info(const PARAM *param, > // TRP_RANGE should not be created if there are no range intervals > DBUG_ASSERT(key); > > - String range_info; > - range_info.length(0); > - range_info.set_charset(system_charset_info); > - append_range_all_keyparts(&trace_range, NULL, &range_info, key, key_part); > + append_range_all_keyparts(&trace_range, *param, key_idx, key, key_part); > } > > > @@ -2489,10 +2488,8 @@ void TRP_GROUP_MIN_MAX::trace_basic_info(const PARAM *param, > // can have group quick without ranges > if (index_tree) > { > - String range_info; > - range_info.set_charset(system_charset_info); > - append_range_all_keyparts(&trace_range, NULL, &range_info, index_tree, > - key_part); > + append_range_all_keyparts(&trace_range, *param, param_idx, > + index_tree, key_part); > } > } > > @@ -6398,20 +6395,9 @@ void TRP_ROR_INTERSECT::trace_basic_info(const PARAM *param, > trace_isect_idx.add("rows", (*cur_scan)->records); > > Json_writer_array trace_range(thd, "ranges"); > - for (const SEL_ARG *current= (*cur_scan)->sel_arg->first(); current; > - current= current->next) > - { > - String range_info; > - range_info.set_charset(system_charset_info); > - for (const SEL_ARG *part= current; part; > - part= part->next_key_part ? part->next_key_part : nullptr) > - { > - const KEY_PART_INFO *cur_key_part= key_part + part->part; > - append_range(&range_info, cur_key_part, part->min_value, > - part->max_value, part->min_flag | part->max_flag); > - } > - trace_range.add(range_info.ptr(), range_info.length()); > - } > + > + append_range_all_keyparts(&trace_range, *param, (*cur_scan)->idx, > + (*cur_scan)->sel_arg->first(), key_part); What is this for? I think this is not needed (at best). Range enumeration should start from the tree root, not from the leftmost element. > } > } > ... > @@ -15775,60 +15763,39 @@ void append_range(String *out, const KEY_PART_INFO *key_part, > > Add ranges to the trace > For ex: > - query: select * from t1 where a=2 ; > - and we have an index on a , so we create a range > - 2 <= a <= 2 > + lets say we have an index a_b(a,b) > + query: select * from t1 where a=2 and b=4 ; > + so we create a range: > + (2,4) <= (a,b) <= (2,4) > this is added to the trace > */ > > static void append_range_all_keyparts(Json_writer_array *range_trace, > - String *range_string, > - String *range_so_far, const SEL_ARG *keypart, > + PARAM param, uint idx, Is it intentional that PARAM is passed by value? What for? > + SEL_ARG *keypart, > const KEY_PART_INFO *key_parts) > { > - > - DBUG_ASSERT(keypart); > - DBUG_ASSERT(keypart && keypart != &null_element); > - > - // Navigate to first interval in red-black tree > + SEL_ARG_RANGE_SEQ seq; > + KEY_MULTI_RANGE range; > + range_seq_t seq_it; > + uint flags= 0; > + RANGE_SEQ_IF seq_if = {NULL, sel_arg_range_seq_init, > + sel_arg_range_seq_next, 0, 0}; > + KEY *keyinfo= param.table->key_info + param.real_keynr[idx]; > + uint n_key_parts= param.table->actual_n_key_parts(keyinfo); > + seq.keyno= idx; > + seq.real_keyno= param.real_keynr[idx]; > + seq.param= &param; > + seq.start= keypart; > const KEY_PART_INFO *cur_key_part= key_parts + keypart->part; > - const SEL_ARG *keypart_range= keypart->first(); > - const size_t save_range_so_far_length= range_so_far->length(); > - > + seq_it= seq_if.init((void *) &seq, 0, flags); > > - while (keypart_range) > + while (!seq_if.next(seq_it, &range)) > { > - // Append the current range predicate to the range String > - switch (keypart->type) > - { > - case SEL_ARG::Type::KEY_RANGE: > - append_range(range_so_far, cur_key_part, keypart_range->min_value, > - keypart_range->max_value, > - keypart_range->min_flag | keypart_range->max_flag); > - break; > - case SEL_ARG::Type::MAYBE_KEY: > - range_so_far->append("MAYBE_KEY"); > - break; > - case SEL_ARG::Type::IMPOSSIBLE: > - range_so_far->append("IMPOSSIBLE"); > - break; > - default: > - DBUG_ASSERT(false); > - break; > - } > - > - if (keypart_range->next_key_part && > - keypart_range->next_key_part->part == > - keypart_range->part + 1 && > - keypart_range->is_singlepoint()) > - { > - append_range_all_keyparts(range_trace, range_string, range_so_far, > - keypart_range->next_key_part, key_parts); > - } > - else > - range_trace->add(range_so_far->c_ptr_safe(), range_so_far->length()); > - keypart_range= keypart_range->next; > - range_so_far->length(save_range_so_far_length); > + String range_info; > + range_info.set_charset(system_charset_info); Can you use "StringBuffer<128> tmp(system_charset_info);" like you're already doing in other places of this patch? > + append_range(&range_info, cur_key_part, &range, n_key_parts); > + range_trace->add(range_info.c_ptr_safe(), range_info.length()); > } > } > BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] [Commits] bf951046e8d: MDEV-15296: wrong result with window function inside a subquery
by Oleksandr Byelkin 30 Apr '19

30 Apr '19

Hi, Varun! This bug shows us whole class of the same problem in the code which slip of our scope during window function development. so please check and fix them (one or separate change-set as you wish). See comment below. Am 29.04.19 um 16:40 schrieb Varun: > revision-id: bf951046e8df13702ee56e4e269ce5076e15407f (mariadb-10.2.23-99-gbf951046e8d) > parent(s): 092602ac9b650f921ec5380866d17d740f0eedb4 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2019-04-29 20:07:47 +0530 > message: > > MDEV-15296: wrong result with window function inside a subquery > > Window Functions were treated as a constant in a dependent tables less subquery. > Made sure that the behaviour of window functions is same as the aggregate function > for dependent tables less subquery. > > --- > mysql-test/r/win.result | 17 +++++++++++++++++ > mysql-test/t/win.test | 10 ++++++++++ > sql/item_subselect.cc | 1 + > 3 files changed, 28 insertions(+) > > diff --git a/mysql-test/r/win.result b/mysql-test/r/win.result > index 0ddffc551dc..b86f5b1fc16 100644 > --- a/mysql-test/r/win.result > +++ b/mysql-test/r/win.result > @@ -3518,5 +3518,22 @@ rank() OVER (ORDER BY 1) ROW_NUMBER() OVER (ORDER BY (EXPORT_SET(5,'Y','N',',',4 > 1 3 > drop table t1; > # > +# MDEV-15296: wrong result with window function inside a subquery > +# > +CREATE TABLE t1(i INT); > +INSERT INTO t1 VALUES (1), (2); > +EXPLAIN EXTENDED SELECT (SELECT SUM(i) OVER (partition BY i)) FROM t1; > +id select_type table type possible_keys key key_len ref rows filtered Extra > +1 PRIMARY t1 ALL NULL NULL NULL NULL 2 100.00 > +2 DEPENDENT SUBQUERY NULL NULL NULL NULL NULL NULL NULL NULL No tables used > +Warnings: > +Note 1276 Field or reference 'test.t1.i' of SELECT #2 was resolved in SELECT #1 > +Note 1003 select <expr_cache><`test`.`t1`.`i`>((select sum(`test`.`t1`.`i`) over ( partition by `test`.`t1`.`i`))) AS `(SELECT SUM(i) OVER (partition BY i))` from `test`.`t1` > +SELECT (SELECT SUM(i) OVER (partition BY i)) FROM t1; > +(SELECT SUM(i) OVER (partition BY i)) > +1 > +2 > +drop table t1; > +# > # End of 10.2 tests > # > diff --git a/mysql-test/t/win.test b/mysql-test/t/win.test > index fd31e9d4bd9..6cc681cace4 100644 > --- a/mysql-test/t/win.test > +++ b/mysql-test/t/win.test > @@ -2265,6 +2265,16 @@ insert into t1 values (1),(2),(3); > SELECT rank() OVER (ORDER BY 1), ROW_NUMBER() OVER (ORDER BY (EXPORT_SET(5,'Y','N',',',4))) FROM t1; > drop table t1; > > +--echo # > +--echo # MDEV-15296: wrong result with window function inside a subquery > +--echo # > + > +CREATE TABLE t1(i INT); > +INSERT INTO t1 VALUES (1), (2); > +EXPLAIN EXTENDED SELECT (SELECT SUM(i) OVER (partition BY i)) FROM t1; > +SELECT (SELECT SUM(i) OVER (partition BY i)) FROM t1; > +drop table t1; > + > --echo # > --echo # End of 10.2 tests > --echo # > diff --git a/sql/item_subselect.cc b/sql/item_subselect.cc > index 8cff8f3a5c4..95482364225 100644 > --- a/sql/item_subselect.cc > +++ b/sql/item_subselect.cc > @@ -1138,6 +1138,7 @@ Item_singlerow_subselect::select_transformer(JOIN *join) > !select_lex->table_list.elements && > select_lex->item_list.elements == 1 && > !select_lex->item_list.head()->with_sum_func && > + !select_lex->item_list.head()->with_window_func && This case is definitely OK, but I also asked you to check other cases with with_sum_func, I do not see traces of it here so I did and it is my findings: Item_cache_wrapper::Item_cache_wrapper do not transfer with_window_func, is it impossible to have it here? (then better to put ASSERT) (same Item_cache_wrapper::get_tmp_table_item) Item_in_optimizer::fix_left, Item_in_optimizer::fix_fields, Item_func_interval::fix_length_and_dec do not transfer the flag also. ASSERT in Item_equal::fix_fields Item_func::get_tmp_table_item, Item_subselect::get_tmp_table_item impossible? than better put an ASSERT. Item_func constructors do not transfer the flag. Item_allany_subselect::cleanup reset. Item_singlerow_subselect::select_transformer, Item_in_subselect::single_value_transformer, Item_allany_subselect::transform_into_max_min, Item_exists_subselect::exists2in_processor, check_and_do_in_subquery_rewrites I doubts that window function allows the transformation is aggregate prohibit it. Item_in_subselect::create_single_in_to_exists_cond probably the same as aggregate. Item_in_subselect::create_row_in_to_exists_cond not sure if it is possible to have window functions hare but should be checked. st_select_lex::check_unrestricted_recursive, pushdown_cond_for_derived It looks like here also should be checked. > /* > We cant change name of Item_field or Item_ref, because it will > prevent it's correct resolving, but we should save name of > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

Re: [Maria-developers] [Commits] 88a80e92dc4: MDEV-9959: A serious MariaDB server performance bug
by Sergey Petrunia 27 Apr '19

27 Apr '19

Hi Varun, On Wed, Apr 24, 2019 at 01:31:38PM +0530, Varun wrote: > revision-id: 88a80e92dc444ce30718f3e08d3ab66fb02bcea4 (mariadb-10.3.10-284-g88a80e92dc4) > parent(s): 3032cd8e91f1e1ead8b6f941e75cd29e473e7eaa > author: Varun Gupta > committer: Varun Gupta > timestamp: 2019-04-24 13:31:24 +0530 > message: > > MDEV-9959: A serious MariaDB server performance bug > > Step #2: If any field in the select list of the derived tables is present in the group by list also , then we are again guaranteed > that ref access to the derived table would always produce one row per key. ... > diff --git a/sql/sql_lex.cc b/sql/sql_lex.cc > index c52005e7683..aa645ac00ca 100644 > --- a/sql/sql_lex.cc > +++ b/sql/sql_lex.cc > @@ -7617,6 +7617,68 @@ Item *st_select_lex::build_cond_for_grouping_fields(THD *thd, Item *cond, > } > > > +/** > + Check if any any item in the group by list is also present in the select_list > + @retval true: All elements common between select and group by list > +*/ > + > +void st_select_lex::is_group_by_prefix(KEY *keyinfo) I think the name of the function is poor - "is_something()" hints at hints at boolean return value, and I would say it doesn't imply any side effects. Something like 'mark_derived_index_stats()' would be better, I think. > +{ > + uint key_parts= keyinfo->usable_key_parts; > + KEY_PART_INFO *key_part_info= keyinfo->key_part; > + bool found= FALSE; > + > + if (key_parts < group_list.elements) > + return; > + > + uint matched_fields=0, i, j; > + Item *item; > + > + for (i= 0; i < key_parts; key_part_info++, i++) > + { > + uint fld_idx= key_part_info->fieldnr - 1; > + item= join->fields_list.elem(fld_idx); > + for (ORDER *order= group_list.first; order; order= order->next) > + { > + Item *ord_item= order->item[0]->real_item(); > + Item_equal *item_equal= ord_item->get_item_equal(); > + > + if (item_equal) > + { > + Item_equal_fields_iterator it(*item_equal); > + Item *equal_item; > + while ((equal_item= it++)) > + { > + if (equal_item->eq(item, 0)) > + { > + matched_fields++; > + found= TRUE; > + break; > + } > + } > + } > + else > + { > + if (item->eq(ord_item, 0)) > + { > + matched_fields++; > + found= TRUE; > + } > + } > + if (found) > + break; > + } > + > + if (matched_fields == group_list.elements) > + { > + for (j=matched_fields - 1; j < key_parts; j++) > + keyinfo->rec_per_key[j]= 1; This doesn't always work. An example: create table t1 (a int, b int, c int, d int); insert into t1 select a,a,a,a from one_k; create table t2 as select * from t1; # Note the "a, a AS b" in the select list. This is intentional. explain select * from t2, (select a, a as b from t1 group by a,b) TBL where t2.a=TBL.a and t2.b=TBL.b; Here, the select output is not guaranteed to be unique (as it contains only one of two GROUP BY columns). But I see the execution to reac the above line, assigning keyinfo->rec_per_key[j]= 1. > + return; > + } > + found= FALSE; > + } > +} > + > int set_statement_var_if_exists(THD *thd, const char *var_name, > size_t var_name_length, ulonglong value) > { A general comment: do you think we should expose this in optimizer trace? At the moment, Derived-with-keys optimization doesn't print anything there (neither in MariaDB nor in MySQL). One can see the generated KEYUSEs when they are printed. But perhaps we could add a node that would print that a derived table was added, with this cardinality, and index with these keypart cardinalities. This would make debugging easier. Any thoughts? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] [Commits] 3032cd8e91f: step #1: if a derived table has SELECT DISTINCT, provide index statistics for it so that the join optimizer in the
by Sergey Petrunia 26 Apr '19

26 Apr '19

Hi Varun, Please mention the MDEV# in the commit comment. Otherwise, the patch is ok to push. On Mon, Apr 22, 2019 at 07:09:45PM +0530, Varun wrote: > revision-id: 3032cd8e91f1e1ead8b6f941e75cd29e473e7eaa (mariadb-10.3.10-283-g3032cd8e91f) > parent(s): f4019f5b3544a18f3ddf32df2c5214c3f8dabdce > author: Varun Gupta > committer: Varun Gupta > timestamp: 2019-04-22 18:19:25 +0530 > message: > > step #1: if a derived table has SELECT DISTINCT, provide index statistics for it so that the join optimizer in the > upper select knows that ref access to the table will produce one row. > > --- > mysql-test/main/cte_nonrecursive.result | 8 ++-- > mysql-test/main/derived.result | 54 ++++++++++++++++++++++ > mysql-test/main/derived.test | 30 ++++++++++++ > mysql-test/main/derived_view.result | 2 +- > mysql-test/main/subselect_extra.result | 2 +- > mysql-test/main/subselect_extra_no_semijoin.result | 2 +- > sql/sql_lex.h | 1 + > sql/sql_union.cc | 40 ++++++++++++++++ > sql/table.cc | 20 ++++++++ > 9 files changed, 152 insertions(+), 7 deletions(-) > > diff --git a/mysql-test/main/cte_nonrecursive.result b/mysql-test/main/cte_nonrecursive.result > index b846ec2d8ac..d80d34ecc7f 100644 > --- a/mysql-test/main/cte_nonrecursive.result > +++ b/mysql-test/main/cte_nonrecursive.result > @@ -244,7 +244,7 @@ with t as (select distinct a from t1 where b >= 'c') > select * from t as r1, t as r2 where r1.a=r2.a; > id select_type table type possible_keys key key_len ref rows Extra > 1 PRIMARY <derived2> ALL NULL NULL NULL NULL 8 Using where > -1 PRIMARY <derived3> ref key0 key0 5 r1.a 2 > +1 PRIMARY <derived3> ref key0 key0 5 r1.a 1 > 3 DERIVED t1 ALL NULL NULL NULL NULL 8 Using where; Using temporary > 2 DERIVED t1 ALL NULL NULL NULL NULL 8 Using where; Using temporary > explain > @@ -253,7 +253,7 @@ select * from (select distinct a from t1 where b >= 'c') as r1, > where r1.a=r2.a; > id select_type table type possible_keys key key_len ref rows Extra > 1 PRIMARY <derived2> ALL NULL NULL NULL NULL 8 Using where > -1 PRIMARY <derived3> ref key0 key0 5 r1.a 2 > +1 PRIMARY <derived3> ref key0 key0 5 r1.a 1 > 3 DERIVED t1 ALL NULL NULL NULL NULL 8 Using where; Using temporary > 2 DERIVED t1 ALL NULL NULL NULL NULL 8 Using where; Using temporary > # two references to t specified by a query > @@ -369,7 +369,7 @@ select c as a from t2 where c < 4) > select * from t2,t where t2.c=t.a; > id select_type table type possible_keys key key_len ref rows Extra > 1 PRIMARY t2 ALL NULL NULL NULL NULL 4 Using where > -1 PRIMARY <derived2> ref key0 key0 5 test.t2.c 2 > +1 PRIMARY <derived2> ref key0 key0 5 test.t2.c 1 > 2 DERIVED t1 ALL NULL NULL NULL NULL 8 Using where > 3 UNION t2 ALL NULL NULL NULL NULL 4 Using where > NULL UNION RESULT <union2,3> ALL NULL NULL NULL NULL NULL > @@ -381,7 +381,7 @@ select c as a from t2 where c < 4) as t > where t2.c=t.a; > id select_type table type possible_keys key key_len ref rows Extra > 1 PRIMARY t2 ALL NULL NULL NULL NULL 4 Using where > -1 PRIMARY <derived2> ref key0 key0 5 test.t2.c 2 > +1 PRIMARY <derived2> ref key0 key0 5 test.t2.c 1 > 2 DERIVED t1 ALL NULL NULL NULL NULL 8 Using where > 3 UNION t2 ALL NULL NULL NULL NULL 4 Using where > NULL UNION RESULT <union2,3> ALL NULL NULL NULL NULL NULL > diff --git a/mysql-test/main/derived.result b/mysql-test/main/derived.result > index f0d0289c1ce..857246d68b4 100644 > --- a/mysql-test/main/derived.result > +++ b/mysql-test/main/derived.result > @@ -1195,3 +1195,57 @@ drop table t1,t2,t3; > # > # End of 10.2 tests > # > +# > +# MDEV-9959: A serious MariaDB server performance bug > +# > +create table t1(a int); > +insert into t1 values (1),(2),(3),(4),(5),(6); > +create table t2(a int, b int,c int); > +insert into t2(a,b,c) values (1,1,2),(2,2,3),(3,1,4),(4,2,2),(5,1,1),(6,2,5); > +create table t3(a int, b int); > +insert into t3(a,b) values (1,1),(2,2),(2,1),(1,2),(5,1),(9,2); > +table "<derived2>" should have type=ref and rows=1 > +one select in derived table > +with distinct > +analyze select * from t1 , ((select distinct t2.a from t2 order by c))q where t1.a=q.a; > +id select_type table type possible_keys key key_len ref rows r_rows filtered r_filtered Extra > +1 PRIMARY t1 ALL NULL NULL NULL NULL 6 6.00 100.00 100.00 Using where > +1 PRIMARY <derived2> ref key0 key0 5 test.t1.a 1 1.00 100.00 100.00 > +2 DERIVED t2 ALL NULL NULL NULL NULL 6 6.00 100.00 100.00 Using temporary; Using filesort > +analyze select * from t1 , ((select distinct t2.a, t2.b from t2 order by c))q where t1.a=q.a; > +id select_type table type possible_keys key key_len ref rows r_rows filtered r_filtered Extra > +1 PRIMARY t1 ALL NULL NULL NULL NULL 6 6.00 100.00 100.00 Using where > +1 PRIMARY <derived2> ref key0 key0 5 test.t1.a 2 1.00 100.00 100.00 > +2 DERIVED t2 ALL NULL NULL NULL NULL 6 6.00 100.00 100.00 Using temporary; Using filesort > +# multiple selects in derived table > +# NO UNION ALL > +analyze select * from t1 , ( (select t2.a from t2 order by c) union (select t2.a from t2 order by c))q where t1.a=q.a; > +id select_type table type possible_keys key key_len ref rows r_rows filtered r_filtered Extra > +1 PRIMARY t1 ALL NULL NULL NULL NULL 6 6.00 100.00 100.00 Using where > +1 PRIMARY <derived2> ref key0 key0 5 test.t1.a 1 1.00 100.00 100.00 > +2 DERIVED t2 ALL NULL NULL NULL NULL 6 6.00 100.00 100.00 > +3 UNION t2 ALL NULL NULL NULL NULL 6 6.00 100.00 100.00 > +NULL UNION RESULT <union2,3> ALL NULL NULL NULL NULL NULL 6.00 NULL NULL > +select * from t1 , ( (select t2.a from t2 order by c) union (select t2.a from t2 order by c))q where t1.a=q.a; > +a a > +1 1 > +2 2 > +3 3 > +4 4 > +5 5 > +6 6 > +# UNION ALL and EXCEPT > +analyze select * from t1 , ( (select t2.a from t2 order by c) union all (select t2.a from t2 order by c) except(select t3.a from t3 order by b))q where t1.a=q.a; > +id select_type table type possible_keys key key_len ref rows r_rows filtered r_filtered Extra > +1 PRIMARY t1 ALL NULL NULL NULL NULL 6 6.00 100.00 100.00 Using where > +1 PRIMARY <derived2> ref key0 key0 5 test.t1.a 1 0.50 100.00 100.00 > +2 DERIVED t2 ALL NULL NULL NULL NULL 6 6.00 100.00 100.00 > +3 UNION t2 ALL NULL NULL NULL NULL 6 6.00 100.00 100.00 > +4 EXCEPT t3 ALL NULL NULL NULL NULL 6 6.00 100.00 100.00 > +NULL UNIT RESULT <unit2,3,4> ALL NULL NULL NULL NULL NULL 3.00 NULL NULL > +select * from t1 , ( (select t2.a from t2 order by c) union all (select t2.a from t2 order by c) except(select t3.a from t3 order by b))q where t1.a=q.a; > +a a > +3 3 > +4 4 > +6 6 > +drop table t1,t2,t3; > diff --git a/mysql-test/main/derived.test b/mysql-test/main/derived.test > index 6c51f23c51e..990f955450a 100644 > --- a/mysql-test/main/derived.test > +++ b/mysql-test/main/derived.test > @@ -1032,3 +1032,33 @@ drop table t1,t2,t3; > --echo # > --echo # End of 10.2 tests > --echo # > + > +--echo # > +--echo # MDEV-9959: A serious MariaDB server performance bug > +--echo # > + > +create table t1(a int); > +insert into t1 values (1),(2),(3),(4),(5),(6); > +create table t2(a int, b int,c int); > +insert into t2(a,b,c) values (1,1,2),(2,2,3),(3,1,4),(4,2,2),(5,1,1),(6,2,5); > +create table t3(a int, b int); > +insert into t3(a,b) values (1,1),(2,2),(2,1),(1,2),(5,1),(9,2); > + > +--echo table "<derived2>" should have type=ref and rows=1 > +--echo one select in derived table > + > +--echo with distinct > +analyze select * from t1 , ((select distinct t2.a from t2 order by c))q where t1.a=q.a; > +analyze select * from t1 , ((select distinct t2.a, t2.b from t2 order by c))q where t1.a=q.a; > + > +--echo # multiple selects in derived table > +--echo # NO UNION ALL > +analyze select * from t1 , ( (select t2.a from t2 order by c) union (select t2.a from t2 order by c))q where t1.a=q.a; > +select * from t1 , ( (select t2.a from t2 order by c) union (select t2.a from t2 order by c))q where t1.a=q.a; > + > +--echo # UNION ALL and EXCEPT > +analyze select * from t1 , ( (select t2.a from t2 order by c) union all (select t2.a from t2 order by c) except(select t3.a from t3 order by b))q where t1.a=q.a; > + > +select * from t1 , ( (select t2.a from t2 order by c) union all (select t2.a from t2 order by c) except(select t3.a from t3 order by b))q where t1.a=q.a; > + > +drop table t1,t2,t3; > diff --git a/mysql-test/main/derived_view.result b/mysql-test/main/derived_view.result > index 86dd73f5733..30831e75341 100644 > --- a/mysql-test/main/derived_view.result > +++ b/mysql-test/main/derived_view.result > @@ -1525,7 +1525,7 @@ EXPLAIN > SELECT a FROM t1 WHERE (a,b) IN (SELECT * FROM v2); > id select_type table type possible_keys key key_len ref rows Extra > 1 PRIMARY t1 ALL NULL NULL NULL NULL 2 Using where > -1 PRIMARY <derived3> ref key0 key0 10 test.t1.a,test.t1.b 2 FirstMatch(t1) > +1 PRIMARY <derived3> ref key0 key0 10 test.t1.a,test.t1.b 1 FirstMatch(t1) > 3 DERIVED t2 ALL NULL NULL NULL NULL 6 > 4 UNION t3 ALL NULL NULL NULL NULL 4 > NULL UNION RESULT <union3,4> ALL NULL NULL NULL NULL NULL > diff --git a/mysql-test/main/subselect_extra.result b/mysql-test/main/subselect_extra.result > index a3a0f1f9a15..dbcf00268c2 100644 > --- a/mysql-test/main/subselect_extra.result > +++ b/mysql-test/main/subselect_extra.result > @@ -409,7 +409,7 @@ EXPLAIN > SELECT a FROM t1 WHERE (a,b) IN (SELECT * FROM v2); > id select_type table type possible_keys key key_len ref rows Extra > 1 PRIMARY t1 ALL NULL NULL NULL NULL 2 Using where > -1 PRIMARY <derived3> ref key0 key0 10 test.t1.a,test.t1.b 2 FirstMatch(t1) > +1 PRIMARY <derived3> ref key0 key0 10 test.t1.a,test.t1.b 1 FirstMatch(t1) > 3 DERIVED t2 ALL NULL NULL NULL NULL 6 > 4 UNION t3 ALL NULL NULL NULL NULL 4 > NULL UNION RESULT <union3,4> ALL NULL NULL NULL NULL NULL > diff --git a/mysql-test/main/subselect_extra_no_semijoin.result b/mysql-test/main/subselect_extra_no_semijoin.result > index ec9ddb0452e..49a1431eb9b 100644 > --- a/mysql-test/main/subselect_extra_no_semijoin.result > +++ b/mysql-test/main/subselect_extra_no_semijoin.result > @@ -411,7 +411,7 @@ EXPLAIN > SELECT a FROM t1 WHERE (a,b) IN (SELECT * FROM v2); > id select_type table type possible_keys key key_len ref rows Extra > 1 PRIMARY t1 ALL NULL NULL NULL NULL 2 Using where > -2 DEPENDENT SUBQUERY <derived3> index_subquery key0 key0 10 func,func 2 Using where > +2 DEPENDENT SUBQUERY <derived3> index_subquery key0 key0 10 func,func 1 Using where > 3 DERIVED t2 ALL NULL NULL NULL NULL 6 > 4 UNION t3 ALL NULL NULL NULL NULL 4 > NULL UNION RESULT <union3,4> ALL NULL NULL NULL NULL NULL > diff --git a/sql/sql_lex.h b/sql/sql_lex.h > index 72ca4ac0b43..4eaec7d062b 100644 > --- a/sql/sql_lex.h > +++ b/sql/sql_lex.h > @@ -896,6 +896,7 @@ class st_select_lex_unit: public st_select_lex_node { > bool union_needs_tmp_table(); > > void set_unique_exclude(); > + bool check_distinct_in_union(); > > friend struct LEX; > friend int subselect_union_engine::exec(); > diff --git a/sql/sql_union.cc b/sql/sql_union.cc > index 87fbbebe4ba..3fb5552c77a 100644 > --- a/sql/sql_union.cc > +++ b/sql/sql_union.cc > @@ -2049,3 +2049,43 @@ void st_select_lex_unit::set_unique_exclude() > } > } > } > + > +/** > + @brief > + Check if the derived table is guaranteed to have distinct rows because of > + UNION operations used to populate it. > + > + @detail > + UNION operation removes duplicate rows from its output. That is, a query like > + > + select * from t1 UNION select * from t2 > + > + will not produce duplicate rows in its output, even if table t1 (and/or t2) > + contain duplicate rows. EXCEPT and INTERSECT operations also have this > + property. > + > + On the other hand, UNION ALL operation doesn't remove duplicates. (The SQL > + standard also defines EXCEPT ALL and INTERSECT ALL, but we don't support > + them). > + > + st_select_lex_unit computes its value left to right. That is, if there is > + a st_select_lex_unit object describing > + > + (select #1) OP1 (select #2) OP2 (select #3) > + > + then ((select #1) OP1 (select #2)) is computed first, and OP2 is computed > + second. > + > + How can one tell if st_select_lex_unit is guaranteed to have distinct > + output rows? This depends on whether the last operation was duplicate- > + removing or not: > + - UNION ALL is not duplicate-removing > + - all other operations are duplicate-removing > +*/ > + > +bool st_select_lex_unit::check_distinct_in_union() > +{ > + if (union_distinct && !union_distinct->next_select()) > + return true; > + return false; > +} > diff --git a/sql/table.cc b/sql/table.cc > index 80995abc1f9..c4494c9ae4b 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -7269,6 +7269,26 @@ bool TABLE::add_tmp_key(uint key, uint key_parts, > key_part_info++; > } > > + /* > + For the case when there is a derived table that would give distinct rows, > + the index statistics are passed to the join optimizer to tell that a ref > + access to all the fields of the derived table will produce only one row. > + */ > + > + st_select_lex_unit* derived= pos_in_table_list ? > + pos_in_table_list->derived: NULL; > + if (derived) > + { > + st_select_lex* first= derived->first_select(); > + uint select_list_items= first->get_item_list()->elements; > + if (key_parts == select_list_items) > + { > + if ((!first->is_part_of_union() && (first->options & SELECT_DISTINCT)) || > + derived->check_distinct_in_union()) > + keyinfo->rec_per_key[key_parts - 1]= 1; > + } > + } > + > set_if_bigger(s->max_key_length, keyinfo->key_length); > s->keys++; > return FALSE; > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Big endian issue in libmariadb: test: main.mysql_client_test fails
by REIX, Tony 26 Apr '19

26 Apr '19

Hi, We are building MariaDB on AIX. The test: main.mysql_client_test fails (after time-out). This is due to little endian code being used though HAVE_BIGENDIAN is correctly set. This is due to: libmariadb/libmariadb/ma_dtoa.c : #if defined(WORDS_BIGENDIAN) || (defined(__FLOAT_WORD_ORDER) && \ (__FLOAT_WORD_ORDER == __BIG_ENDIAN)) #define word0(x) (x)->L[0] #define word1(x) (x)->L[1] #else #define word0(x) (x)->L[1] #define word1(x) (x)->L[0] #endif which results in the #else part (Little Endian) since none of the tested macros are defined. File libmariadb/libmariadb/ma_dtoa.c includes ma_global.h which has: #define HAVE_BIGENDIAN 1 but nothing defined about WORDS_BIGENDIAN . However, probably that some changes were made in the past in ma_global.h, since I see: #ifdef HAVE_BIGENDIAN #endif /* WORDS_BIGENDIAN */ 2 times in this file. I do not master MariaDB, so I do not have a clear idea about how to fix this. I see 3 solutions: a) libmariadb/CMakeLists.txt : Add: SET(HAVE_WORDS_BIGENDIAN TRUE ...) after : TEST_BIG_ENDIAN(HAVE_BIGENDIAN) if it is BigEndian. b) libmariadb/include/ma_global.h : Add: #ifdef HAVE_BIGENDIAN #define WORDS_BIGENDIAN 1 #endif c) libmariadb/libmariadb/ma_dtoa.c : Change: #if defined(WORDS_BIGENDIAN) || (defined(__FLOAT_WORD_ORDER) && \ (__FLOAT_WORD_ORDER == __BIG_ENDIAN)) by: #if defined(HAVE_BIGENDIAN) || defined(WORDS_BIGENDIAN) || (defined(__FLOAT_WORD_ORDER) && \ (__FLOAT_WORD_ORDER == __BIG_ENDIAN)) Please let me know which seems appropriate. Regards, Tony ATOS / Bull SAS ATOS Expert IBM Coop Architect & Technical Leader Office : +33 (0) 4 76 29 72 67 1 rue de Provence - 38432 Échirolles - France www.atos.net<https://mail.ad.bull.net/owa/redir.aspx?C=PvphmPvCZkGrAgHVnWGsdMcDKgzl_dEIs…>

1 0

Re: [Maria-developers] [Commits] 647c22692ad: MDEV-17588 replicate-do filters cause errors when creating filtered-out tables on master with syntax unsupported on slave
by andrei.elkin＠pp.inet.fi 25 Apr '19

25 Apr '19

Sachin, howdy. > revision-id: 647c22692ad1fe466eb14b749520cc520807d4fa (mariadb-10.1.38-114-g647c22692ad) > parent(s): 6c5e4c9bc0d9ac30f7ec7ee334630bacb58687ba > author: Sachin > committer: Sachin > timestamp: 2019-04-23 17:58:03 +0530 > message: > > MDEV-17588 replicate-do filters cause errors when creating filtered-out tables on master with syntax unsupported on slave > > Prototype Commit. In next commit could you please brief about the problem and attempted solution. > > Delay throwing of error in parsing stage untill we know the result from rpl > filter. I think you were on the right track implementing this deferred error/warning issuing. Thanks for attempting it! There's a new THD::"flag" for that purpose. I suggest we place it inside struct LEX_MASTER_INFO to set pretty similarly to how the patch does. Secondly, let blocks in sql_yacc.yy that call error/warning functions that gets deferred (because of the slave thread) be invoked (if they will be) in mysql_execute_command() in verbatim. That means to turn the blocks into functions and call them at once or later. Your patch presents two such blocks, I point to them below. For each block I suggest we introduce a flag. The block flag is set when the block gets visited (but not executed 'cos of slave). The "flag" therefore is going to be an array (or enum?). > > --- > mysql-test/suite/rpl/r/rpl_mdev_17588.result | 26 ++++++++++++++++++++++ > mysql-test/suite/rpl/t/rpl_mdev_17588-slave.opt | 1 + > mysql-test/suite/rpl/t/rpl_mdev_17588.test | 29 +++++++++++++++++++++++++ > sql/sql_class.h | 1 + > sql/sql_parse.cc | 7 ++++++ > sql/sql_yacc.yy | 12 +++++++--- > 6 files changed, 73 insertions(+), 3 deletions(-) > > diff --git a/mysql-test/suite/rpl/r/rpl_mdev_17588.result b/mysql-test/suite/rpl/r/rpl_mdev_17588.result > new file mode 100644 > index 00000000000..976efde2bdd > --- /dev/null > +++ b/mysql-test/suite/rpl/r/rpl_mdev_17588.result > @@ -0,0 +1,26 @@ > +include/master-slave.inc > +[connection master] > +set sql_log_bin= 0; > +install soname 'ha_tokudb'; > +set sql_log_bin= 1; > +create table t1 (a int) engine=TokuDB; > +create table t2 (a int); > +create table t3 (a int) engine=TokuDB; > +include/wait_for_slave_sql_error.inc [errno=1286] > +show create table t1; > +ERROR 42S02: Table 'test.t1' doesn't exist > +show create table t2; > +Table Create Table > +t2 CREATE TABLE `t2` ( > + `a` int(11) DEFAULT NULL > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show create table t3; > +ERROR 42S02: Table 'test.t3' doesn't exist > +SET GLOBAL SQL_SLAVE_SKIP_COUNTER=1; > +START SLAVE; > +drop table t1, t2, t3; > +set sql_log_bin= 0; > +uninstall soname 'ha_tokudb'; > +set sql_log_bin= 1; > +CALL mtr.add_suppression('Slave: Unknown storage engine .* Error_code: 1286'); > +include/rpl_end.inc > diff --git a/mysql-test/suite/rpl/t/rpl_mdev_17588-slave.opt b/mysql-test/suite/rpl/t/rpl_mdev_17588-slave.opt > new file mode 100644 > index 00000000000..19497afd22a > --- /dev/null > +++ b/mysql-test/suite/rpl/t/rpl_mdev_17588-slave.opt > @@ -0,0 +1 @@ > +--replicate-do-table=test.t2 --replicate-do-table=test.t3 --sql-mode='NO_ENGINE_SUBSTITUTION' > diff --git a/mysql-test/suite/rpl/t/rpl_mdev_17588.test b/mysql-test/suite/rpl/t/rpl_mdev_17588.test > new file mode 100644 > index 00000000000..c636077e4d1 > --- /dev/null > +++ b/mysql-test/suite/rpl/t/rpl_mdev_17588.test > @@ -0,0 +1,29 @@ > +--source include/master-slave.inc > + > +--connection master > +set sql_log_bin= 0; > +install soname 'ha_tokudb'; > +set sql_log_bin= 1; > + > +create table t1 (a int) engine=TokuDB; > +create table t2 (a int); > +create table t3 (a int) engine=TokuDB; > + > +--connection slave > +let $slave_sql_errno= 1286; > +source include/wait_for_slave_sql_error.inc; > +--error ER_NO_SUCH_TABLE > +show create table t1; > +show create table t2; > +--error ER_NO_SUCH_TABLE > +show create table t3; > +SET GLOBAL SQL_SLAVE_SKIP_COUNTER=1; > +START SLAVE; > +--connection master > +drop table t1, t2, t3; > +set sql_log_bin= 0; > +uninstall soname 'ha_tokudb'; > +set sql_log_bin= 1; > +--sync_slave_with_master > +CALL mtr.add_suppression('Slave: Unknown storage engine .* Error_code: 1286'); > +--source include/rpl_end.inc > diff --git a/sql/sql_class.h b/sql/sql_class.h > index 6640e02147a..4361700ee6c 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -2802,6 +2802,7 @@ class THD :public Statement, > uint8 password; /* 0, 1 or 2 */ > uint8 failed_com_change_user; > bool slave_thread; > + char *rpl_unavailable_storage_plugin= NULL; > bool extra_port; /* If extra connection */ > > bool no_errors; > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > index 6649c60f827..5f8c833c76f 100644 > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -2656,6 +2656,13 @@ mysql_execute_command(THD *thd) > /* we warn the slave SQL thread */ > my_message(ER_SLAVE_IGNORED_TABLE, ER_THD(thd, ER_SLAVE_IGNORED_TABLE), > MYF(0)); > + thd->rpl_unavailable_storage_plugin= NULL; > + DBUG_RETURN(0); > + } > + if (thd->rpl_unavailable_storage_plugin) > + { > + my_error(ER_UNKNOWN_STORAGE_ENGINE, MYF(0), thd->rpl_unavailable_storage_plugin); > + thd->rpl_unavailable_storage_plugin= NULL; Here the check would go for each thd->lex.mi.flag[i] when set call "flag_func[i]"; I index flag[] but I don't insist on implementing it like that. Here is the function's body { if (thd->variables.sql_mode & MODE_NO_ENGINE_SUBSTITUTION) my_yyabort_error((ER_UNKNOWN_STORAGE_ENGINE, MYF(0), $1.str)); $$= 0; push_warning_printf(thd, Sql_condition::WARN_LEVEL_WARN, ER_UNKNOWN_STORAGE_ENGINE, ER_THD(thd, ER_UNKNOWN_STORAGE_ENGINE), $1.str); } and a flag function also might need some parameters like $1.str in above. > DBUG_RETURN(0); > } > /* > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index 5111f0690ab..0756028c34a 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -2476,7 +2476,7 @@ create: > LEX *lex= thd->lex; > lex->current_select= &lex->select_lex; > if ((lex->create_info.used_fields & HA_CREATE_USED_ENGINE) && > - !lex->create_info.db_type) > + !lex->create_info.db_type && !thd->rpl_unavailable_storage_plugin) > { > lex->create_info.use_default_db_type(thd); This is the 2nd block which depends on the 1st one. if ((lex->create_info.used_fields & HA_CREATE_USED_ENGINE) && !lex->create_info.db_type) { lex->create_info.use_default_db_type(thd); if (!thd->slave_thread || !lex->mi.flag[0]) print_warn_using_other_handler(...) } where print_warn_using_other_handler() is { push_warning_printf(thd, Sql_condition::WARN_LEVEL_WARN, ER_WARN_USING_OTHER_HANDLER, ER_THD(thd, ER_WARN_USING_OTHER_HANDLER), hton_name(lex->create_info.db_type)->str, $5->table.str); } To call it deferredly we seem need two args to memorize (reserve fields too) in Lex->mi. I hope my suggestions make sense and feasible. Cheers, Andrei > push_warning_printf(thd, Sql_condition::WARN_LEVEL_WARN, > @@ -5791,9 +5791,15 @@ storage_engines: > else > { > if (thd->variables.sql_mode & MODE_NO_ENGINE_SUBSTITUTION) > - my_yyabort_error((ER_UNKNOWN_STORAGE_ENGINE, MYF(0), $1.str)); > + { > + if (!thd->slave_thread) > + my_yyabort_error((ER_UNKNOWN_STORAGE_ENGINE, MYF(0), $1.str)); > + else > + thd->rpl_unavailable_storage_plugin= $1.str; > + } > $$= 0; > - push_warning_printf(thd, Sql_condition::WARN_LEVEL_WARN, > + if (!thd->rpl_unavailable_storage_plugin) > + push_warning_printf(thd, Sql_condition::WARN_LEVEL_WARN, > ER_UNKNOWN_STORAGE_ENGINE, > ER_THD(thd, ER_UNKNOWN_STORAGE_ENGINE), > $1.str); > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

Re: [Maria-developers] [Commits] 8ae7a5eb037: MDEV-17260: Memory leaks in mysqlbinlog
by andrei.elkin＠pp.inet.fi 24 Apr '19

24 Apr '19

Sujatha, hello. The patch looks good, I only have a suggestion to the test for better readability, within the quoted patch below. Thanks for a good piece of work! Cheers, Andrei >commit 8ae7a5eb03725441393c353e4e203f80b666c035 >Author: Sujatha Sivakumar <sujatha.sivakumar(a)mariadb.com> >Date: Tue Apr 23 19:47:59 2019 +0530 > > MDEV-17260: Memory leaks in mysqlbinlog > > Problem: > ======== > The mysqlbinlog tool is leaking memory, causing failures in various tests when > compiling and testing with AddressSanitizer or LeakSanitizer like this: > > cmake -DCMAKE_BUILD_TYPE=Debug -DWITH_ASAN:BOOL=ON /path/to/source > make -j$(nproc) > cd mysql-test > ASAN_OPTIONS=abort_on_error=1 ./mtr --parallel=auto > > Analysis: > ========= > Two types of leaks were observed during above execution. > > 1) Leak in Log_event::read_log_event(char const*, unsigned int, char const**, > Format_description_log_event const*, char) > File: sql/log_event.cc:2150 > > For all row based replication events the memory which is allocated during > read_log_event is not freed after the event is processed. The event specific > memory has to be retained only when flashback option is enabled with > mysqlbinlog tool. In this case all the events are retained till the end > statement is received and they are processed in reverse order and they are > destroyed. But in the existing code all events are retained irrespective of > flashback mode. Hence the memory leaks are observed. > > 2) read_remote_annotate_event(unsigned char*, unsigned long, char const**) > File: client/mysqlbinlog.cc:194 > > In general the Annotate event is not printed immediately because all > subsequent rbr-events can be filtered away. Instead it will be printed > together with the first not filtered away Table map or the last rbr will be > processed. While reading remote annotate events memory is allocated for event > buffer and event's temp_buf is made to point to the allocated buffer as shown > below. The TRUE flag is used for doing proper cleanup using free_temp_buf(). > i.e at the time of deletion of annotate event its destructor takes care of > clearing the temp_buf. > > /* > Ensure the event->temp_buf is pointing to the allocated buffer. > (TRUE = free temp_buf on the event deletion) > */ > event->register_temp_buf((char*)event_buf, TRUE); > > But existing code does the following when it receives a remote annotate_event. > > if (remote_opt) > ev->temp_buf= 0; > > That is code immediately sets temp_buf=0, because of which free_temp_buf() > call will return empty handed as it has lost the reference to the allocated > temporary buffer. This results in memory leak > > Fix: > ==== > 1) If not in flashback mode, destroy the memory for events once they are > processed. > > 2) Remove the ev->temp_buf=0 code for remote option. Let the proper cleanup to > be done as part of free_temp_buf(). > >diff --git a/client/mysqlbinlog.cc b/client/mysqlbinlog.cc >index cfc05cbf794..3dad4fef67b 100644 >--- a/client/mysqlbinlog.cc >+++ b/client/mysqlbinlog.cc >@@ -227,8 +227,7 @@ void print_annotate_event(PRINT_EVENT_INFO *print_event_info) > if (annotate_event) > { > annotate_event->print(result_file, print_event_info); >- delete annotate_event; // the event should not be printed more than once >- annotate_event= 0; >+ free_annotate_event(); > } > } > >@@ -1465,7 +1464,7 @@ Exit_status process_event(PRINT_EVENT_INFO *print_event_info, Log_event *ev, > if (print_row_event(print_event_info, ev, e->get_table_id(), > e->get_flags(Rows_log_event::STMT_END_F))) > goto err; >- if (!is_stmt_end) >+ if (opt_flashback && !is_stmt_end) > destroy_evt= FALSE; > break; > } >@@ -1478,7 +1477,7 @@ Exit_status process_event(PRINT_EVENT_INFO *print_event_info, Log_event *ev, > if (print_row_event(print_event_info, ev, e->get_table_id(), > e->get_flags(Old_rows_log_event::STMT_END_F))) > goto err; >- if (!is_stmt_end) >+ if (opt_flashback && !is_stmt_end) > destroy_evt= FALSE; > break; > } >@@ -1539,8 +1538,6 @@ Exit_status process_event(PRINT_EVENT_INFO *print_event_info, Log_event *ev, > } > } > >- if (remote_opt) >- ev->temp_buf= 0; > if (destroy_evt) /* destroy it later if not set (ignored table map) */ > delete ev; > } >diff --git a/mysql-test/suite/binlog/r/flashback.result b/mysql-test/suite/binlog/r/flashback.result >index c96eaebe838..047c032b5df 100644 >--- a/mysql-test/suite/binlog/r/flashback.result >+++ b/mysql-test/suite/binlog/r/flashback.result >@@ -674,6 +674,28 @@ world.city 563256876 > DROP TABLE test.test; > DROP TABLE world.city; > DROP DATABASE world; >+# < CASE 7 > >+# Test Case for MDEV-17260 >+# >+RESET MASTER; >+CREATE TABLE t1 ( f INT PRIMARY KEY ) ENGINE=innodb; >+INSERT INTO t1 VALUES (1),(2),(3),(4),(5),(6); >+# 6- Rows must be present >+SELECT COUNT(*) FROM t1; >+COUNT(*) >+6 >+FLUSH LOGS; >+DELETE FROM t1; >+FLUSH LOGS; >+# 0- Rows must be present >+SELECT COUNT(*) FROM t1; >+COUNT(*) >+0 >+# 6- Rows must be present upon restoring from flashback >+SELECT COUNT(*) FROM t1; >+COUNT(*) >+6 >+DROP TABLE t1; > SET binlog_format=statement; > Warnings: > Warning 1105 MariaDB Galera and flashback do not support binlog format: STATEMENT >diff --git a/mysql-test/suite/binlog/t/flashback.test b/mysql-test/suite/binlog/t/flashback.test >index 3fc8c44c60c..fb80669ca51 100644 >--- a/mysql-test/suite/binlog/t/flashback.test >+++ b/mysql-test/suite/binlog/t/flashback.test >@@ -335,8 +335,29 @@ DROP TABLE test.test; > DROP TABLE world.city; > DROP DATABASE world; > >-## Clear >+--echo # < CASE 7 > >+--echo # Test Case for MDEV-17260 >+--echo # >+RESET MASTER; >+ >+CREATE TABLE t1 ( f INT PRIMARY KEY ) ENGINE=innodb; >+INSERT INTO t1 VALUES (1),(2),(3),(4),(5),(6); >+--echo # 6- Rows must be present >+SELECT COUNT(*) FROM t1; SELECT COUNT(*) as six FROM t1; would read better though it's far from being the optimal automation. (The optimal one clearly must also count 'six' or whatever number itself, as well as bail out with an error when count(*) won't match the expected # - sure not for the current patch). >+FLUSH LOGS; >+DELETE FROM t1; >+FLUSH LOGS; >+--echo # 0- Rows must be present >+SELECT COUNT(*) FROM t1; >+--exec $MYSQL_BINLOG -vv -B --read-from-remote-server --user=root --host=127.0.0.1 --port=$MASTER_MYPORT master-bin.000002> $MYSQLTEST_VARDIR/tmp/mysqlbinlog_row_flashback_7.sql >+--exec $MYSQL -e "SET binlog_format= ROW; source $MYSQLTEST_VARDIR/tmp/mysqlbinlog_row_flashback_7.sql;" >+ >+--echo # 6- Rows must be present upon restoring from flashback >+SELECT COUNT(*) FROM t1; > >+DROP TABLE t1; >+ >+## Clear > SET binlog_format=statement; > --error ER_FLASHBACK_NOT_SUPPORTED > SET GLOBAL binlog_format=statement; >

1 0

Re: [Maria-developers] 70c5ab5658d: Move XID_STATE::rm_error to XID_cache_element
by Sergei Golubchik 24 Apr '19

24 Apr '19

Hi, Sergey! On Apr 24, Sergey Vojtovich wrote: > revision-id: 70c5ab5658d (mariadb-10.4.4-29-g70c5ab5658d) > parent(s): c5b1799435b > author: Sergey Vojtovich <svoj(a)mariadb.org> > committer: Sergey Vojtovich <svoj(a)mariadb.org> > timestamp: 2019-04-20 14:45:16 +0400 > message: > > Move XID_STATE::rm_error to XID_cache_element It's somewhat confusing. What if it's an internal 2PC and some engine's rollback returns an error? The error won't be stored in xid_state.rm_error anymore. Basically, you're changing the semantics of rm_error from "an error that happened during rollback" to "an error that happened during rollback or an external XA transaction". Mention this change explicitly in the commit comment, please. > Also removed redundant xid reset from THD::init_for_queries(). Must've > been done already either by THD::transaction constructor or by > THD::cleanup(). > > Part of MDEV-7974 - backport fix for mysql bug#12161 (XA and binlog) > > diff --git a/sql/sql_class.h b/sql/sql_class.h > index c68628be65d..e7584a0d825 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -2606,13 +2606,8 @@ class THD: public THD_count, /* this must be first */ > DBUG_ENTER("thd::cleanup"); rename to "THD::st_transaction::cleanup" please > changed_tables= 0; > savepoints= 0; > - /* > - If rm_error is raised, it means that this piece of a distributed > - transaction has failed and must be rolled back. But the user must > - rollback it explicitly, so don't start a new distributed XA until > - then. > - */ > - if (!xid_state.rm_error) > + /* xid_cache_delete() resets xid of explicitly started XA transaction */ > + if (!xid_state.is_explicit_XA()) > xid_state.xid.null(); > free_root(&mem_root,MYF(MY_KEEP_PREALLOC)); > DBUG_VOID_RETURN; Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] C API prepared statements parameters for IN (x, x, x, ...)
by Simon Walter 24 Apr '19

24 Apr '19

I want to add support for this or sponsor it - if it is really true that this is not possible with the current c API. I have been told to do: SELECT x FROM x WHERE id IN (?, ?); Where the content of the parenthesis are not dynamic. Rather than going at it brute force on my own, which is my tendency, I want to hear what other developers think and if someone, perhaps a core dev, wants to, I would like to sponsor it. Sorry for the short email. It's late here in Japan. I have been thinking about this for months. Until now I have been using all sorts of work-arounds, but I think that dealing with the core issue will be a better use of my time/resources. More experienced developers may state reasons why this is not possible. Though as you may be able to tell, I can see how it could work. Of course I may be not seeing how everyone else is doing it. From what I can tell (lots code grepping, asking on mailing lists), noone in doing it. In my opinion, that is a shame. Because an, AFAICT, an IN clause means one disk read. The application I am working on make extensive use of IN clauses for reading many records in one call. Some may call this an micro-optimization. If that is really so, please show me that thousands of individual calls to a prepared statement like: SELECT x FROM x WHERE id = x; is as efficient as one call to: SELECT x FROM x WHERE id IN (x, x, x, ...); Background: I am using APR Util and have asked a few people about this and it boils down to the underlying drivers not supporting it. I would ultimately like to code a solution for pgsql as well. Mariadb is where I would like to start however. Then add support for it in APR. Suggesting I use sprintf to create dynamic queries means that I need to write my own sprintf, because as you know, va_arg etc. An array is actually dynamic - not va_arg - reading from file etc. Please feel free to tell me I am stupid, suggest efficient alternatives, tell me that it is actually supported (point me to the docs), etc. SSDs and such are great. Though, I still think it is a shame to be that lazy. No, I do not think the world will overheat because of our lazy programming. I have to throw that in there with the current political climate. No pun intended. ;) Thanks, Simon PS I do not work for any stock trading companies. The project I am working on is opensource. I just want to engineer something very nice.

1 1

[Maria-developers] mysql-proxy to maxscale migration
by Witold Filipczyk 24 Apr '19

24 Apr '19

Hi, I have simple lua script: cdb = require("cdb") function read_auth() local db = cdb.open("/etc/mysql/proxy/data.cdb") local value = db:get(proxy.connection.client.username) db:close() if value == nil then proxy.response.type = proxy.MYSQLD_PACKET_ERR proxy.response.errmsg = "Access denied" return proxy.PROXY_SEND_RESULT end end Format of data: username1 1 username2 1 and so on. Unfortunately after upgrade from mariadb-10.1 to mariadb-10.3 on buster mysql-proxy is defunct. Cannot get "mysql prompt". I tried to replace mysql-proxy by maxscale-2.3.6, but without success. Failed to load users from server 'server1': Lost connection to MySQL server during query Aborted connection 8578832 to db: 'mysql' user: 'debian-sys-maint' host: 'localhost' (Got an error writing communication packets) Note that there is a few hundreds thousands users in mysql.user, but in data.cdb only a few hundreds. Could you point me to the right place, where to find info how to fix mysql-proxy, or configure maxscale, or some alternative to both? Witold Filipczyk

2 1

[Maria-developers] Please review a patch for MDEV-19284 and MDEV-19285 (instant ALTER)
by Alexander Barkov 23 Apr '19

23 Apr '19

Hi Marko and Eugene, please review a patch fixing: - MDEV-19284 INSTANT ALTER with ucs2-to-utf16 conversion produces bad data - MDEV-19285 INSTANT ALTER from ascii_general_ci to latin1_general_ci produces currupt data Thanks!

3 8

[Maria-developers] SELECT INTO OUTFILE + LOAD DATA for a mixture of character and binary data
by Alexander Barkov 22 Apr '19

22 Apr '19

Hi Kenneth, I just closed this issue as "not a bug": MDEV-13361 Regression: LOAD DATA INFILE utf8 enconding issues Can you please document that when doing "SELECT INTO OUTFILE" followed by "LOAD DATA INFILE" for a mixture of text and binary data, one should use "CHARACTER SET binary" in both statements. This is needed to avoid escape sequences to be interpreted in a wrong way at "LOAD DATA" type. Thanks.

1 0

Re: [Maria-developers] [Commits] 167a96b5157: MDEV-9959: A serious MariaDB server performance bug
by Sergey Petrunia 20 Apr '19

20 Apr '19

On Tue, Mar 27, 2018 at 04:48:10PM +0530, Varun wrote: > revision-id: 167a96b5157049408a6ad4bca7abcd376af93fb5 (mariadb-10.3.0-644-g167a96b5157) > parent(s): 4359c6b4806605c78987e50cab3a6b42016b7603 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2018-03-27 16:45:46 +0530 > message: > > MDEV-9959: A serious MariaDB server performance bug > > step#1: if a derived table has SELECT DISTINCT, provide index statistics for it so that the join optimizer in the > upper select knows that ref access to the table will produce one row. > > Added handling for multiple selects in the derived table > > --- > mysql-test/r/cte_nonrecursive.result | 8 +-- > mysql-test/r/cte_recursive.result | 6 +- > mysql-test/r/derived_cond_pushdown.result | 82 ++++++++++++------------- > mysql-test/r/derived_view.result | 2 +- > mysql-test/r/join_cache.result | 6 +- > mysql-test/r/mdev9959.result | 46 ++++++++++++++ > mysql-test/r/subselect_extra.result | 2 +- > mysql-test/r/subselect_extra_no_semijoin.result | 2 +- > mysql-test/t/mdev9959.test | 25 ++++++++ > sql/sql_lex.h | 2 + > sql/sql_union.cc | 60 ++++++++++++++++++ > sql/sql_yacc.yy | 3 + > sql/table.cc | 19 ++++++ > 13 files changed, 209 insertions(+), 54 deletions(-) > > diff --git a/mysql-test/r/derived_cond_pushdown.result b/mysql-test/r/derived_cond_pushdown.result > index 32d3c88cc8d..3723e25a494 100644 > --- a/mysql-test/r/derived_cond_pushdown.result > +++ b/mysql-test/r/derived_cond_pushdown.result > @@ -5187,7 +5187,7 @@ explain select * from v2_union as v,t2 where > ((v.a=6) or (v.a=8)) and (v.c>200) and (v.a=t2.a); > id select_type table type possible_keys key key_len ref rows Extra > 1 PRIMARY t2 ALL NULL NULL NULL NULL 9 Using where > -1 PRIMARY <derived2> ref key0 key0 5 test.t2.a 6 Using where > +1 PRIMARY <derived2> ref key0 key0 5 test.t2.a 1 Using where As agreed on the call: need to check what is the cause of this. Here, the temp table has distinct rows, that is, {a,b,c} are distinct. But ref access only uses the first component, where does rows=1 come from? (if this is how best_access_path computes an estimate for prefix when it only has the estimate for the full key ... fine) > 2 DERIVED t1 ALL NULL NULL NULL NULL 20 Using where; Using temporary; Using filesort > 3 UNION t1 ALL NULL NULL NULL NULL 20 Using where; Using temporary; Using filesort > 4 UNION t1 ALL NULL NULL NULL NULL 20 Using where; Using temporary; Using filesort > @@ -5213,7 +5213,7 @@ EXPLAIN > "key_length": "5", > "used_key_parts": ["a"], > "ref": ["test.t2.a"], > - "rows": 6, > + "rows": 1, > "filtered": 100, > "attached_condition": "v.c > 200", > "materialized": { > @@ -5358,7 +5358,7 @@ a b c a b c d > explain select * from v3_union as v,t2 where (v.a=t2.a) and (v.c>6); > id select_type table type possible_keys key key_len ref rows Extra > 1 PRIMARY t2 ALL NULL NULL NULL NULL 9 Using where > -1 PRIMARY <derived2> ref key0 key0 5 test.t2.a 4 Using where > +1 PRIMARY <derived2> ref key0 key0 5 test.t2.a 1 Using where > 2 DERIVED t1 ALL NULL NULL NULL NULL 20 Using where > 3 UNION t1 ALL NULL NULL NULL NULL 20 Using where > NULL UNION RESULT <union2,3> ALL NULL NULL NULL NULL NULL > @@ -5382,7 +5382,7 @@ EXPLAIN > "key_length": "5", > "used_key_parts": ["a"], > "ref": ["test.t2.a"], > - "rows": 4, > + "rows": 1, > "filtered": 100, > "attached_condition": "v.c > 6", > "materialized": { > @@ -5476,7 +5476,7 @@ a b c a b c d > explain select * from v3_union as v,t2 where (v.a=t2.a) and ((t2.a>1) or (v.b<20)); > id select_type table type possible_keys key key_len ref rows Extra > 1 PRIMARY t2 ALL NULL NULL NULL NULL 9 Using where > -1 PRIMARY <derived2> ref key0 key0 5 test.t2.a 4 Using where > +1 PRIMARY <derived2> ref key0 key0 5 test.t2.a 1 Using where > 2 DERIVED t1 ALL NULL NULL NULL NULL 20 Using where > 3 UNION t1 ALL NULL NULL NULL NULL 20 Using where > NULL UNION RESULT <union2,3> ALL NULL NULL NULL NULL NULL > @@ -5500,7 +5500,7 @@ EXPLAIN > "key_length": "5", > "used_key_parts": ["a"], > "ref": ["test.t2.a"], > - "rows": 4, > + "rows": 1, > "filtered": 100, > "attached_condition": "t2.a > 1 or v.b < 20", > "materialized": { > @@ -5561,7 +5561,7 @@ explain select * from v3_union as v,t2 where > (v.a=t2.a) and ((v.b=19) or (v.b=21)) and ((v.c<3) or (v.c>600)); > id select_type table type possible_keys key key_len ref rows Extra > 1 PRIMARY t2 ALL NULL NULL NULL NULL 9 Using where > -1 PRIMARY <derived2> ref key0 key0 5 test.t2.a 4 Using where > +1 PRIMARY <derived2> ref key0 key0 5 test.t2.a 1 Using where > 2 DERIVED t1 ALL NULL NULL NULL NULL 20 Using where > 3 UNION t1 ALL NULL NULL NULL NULL 20 Using where > NULL UNION RESULT <union2,3> ALL NULL NULL NULL NULL NULL > @@ -5586,7 +5586,7 @@ EXPLAIN > "key_length": "5", > "used_key_parts": ["a"], > "ref": ["test.t2.a"], > - "rows": 4, > + "rows": 1, > "filtered": 100, > "attached_condition": "(v.b = 19 or v.b = 21) and (v.c < 3 or v.c > 600)", > "materialized": { > @@ -5645,7 +5645,7 @@ a b c a b c d > explain select * from v4_union as v,t2 where (v.a=t2.a) and (v.b<20); > id select_type table type possible_keys key key_len ref rows Extra > 1 PRIMARY t2 ALL NULL NULL NULL NULL 9 Using where > -1 PRIMARY <derived2> ref key0 key0 5 test.t2.a 4 Using where > +1 PRIMARY <derived2> ref key0 key0 5 test.t2.a 1 Using where > 2 DERIVED t1 ALL NULL NULL NULL NULL 20 Using where; Using temporary; Using filesort > 3 UNION t1 ALL NULL NULL NULL NULL 20 Using where > NULL UNION RESULT <union2,3> ALL NULL NULL NULL NULL NULL > @@ -5669,7 +5669,7 @@ EXPLAIN > "key_length": "5", > "used_key_parts": ["a"], > "ref": ["test.t2.a"], > - "rows": 4, > + "rows": 1, > "filtered": 100, > "attached_condition": "v.b < 20", > "materialized": { > @@ -5752,7 +5752,7 @@ explain select * from v4_union as v,t2 where > (v.a=t2.a) and ((t2.a<3) or (v.b<40)) and (v.c>500); > id select_type table type possible_keys key key_len ref rows Extra > 1 PRIMARY t2 ALL NULL NULL NULL NULL 9 Using where > -1 PRIMARY <derived2> ref key0 key0 5 test.t2.a 4 Using where > +1 PRIMARY <derived2> ref key0 key0 5 test.t2.a 1 Using where > 2 DERIVED t1 ALL NULL NULL NULL NULL 20 Using where; Using temporary; Using filesort > 3 UNION t1 ALL NULL NULL NULL NULL 20 Using where > NULL UNION RESULT <union2,3> ALL NULL NULL NULL NULL NULL > @@ -5777,7 +5777,7 @@ EXPLAIN > "key_length": "5", > "used_key_parts": ["a"], > "ref": ["test.t2.a"], > - "rows": 4, > + "rows": 1, > "filtered": 100, > "attached_condition": "(t2.a < 3 or v.b < 40) and v.c > 500", > "materialized": { > @@ -7952,7 +7952,7 @@ a b c a b c > explain select * from v1,t2 where (v1.a=t2.a) and (v1.a<5); > id select_type table type possible_keys key key_len ref rows Extra > 1 PRIMARY t2 ALL NULL NULL NULL NULL 9 Using where > -1 PRIMARY <derived2> ref key0 key0 5 test.t2.a 2 > +1 PRIMARY <derived2> ref key0 key0 5 test.t2.a 1 > 2 DERIVED t1 ALL NULL NULL NULL NULL 18 Using where; Using temporary; Using filesort > 3 EXCEPT t1 ALL NULL NULL NULL NULL 18 Using where; Using temporary; Using filesort > NULL EXCEPT RESULT <except2,3> ALL NULL NULL NULL NULL NULL > @@ -7976,7 +7976,7 @@ EXPLAIN > "key_length": "5", > "used_key_parts": ["a"], > "ref": ["test.t2.a"], > - "rows": 2, > + "rows": 1, > "filtered": 100, > "materialized": { > "query_block": { > @@ -8226,7 +8226,7 @@ a b c a b c > explain select * from v1,t2 where (v1.a=t2.a) and (v1.c>500); > id select_type table type possible_keys key key_len ref rows Extra > 1 PRIMARY t2 ALL NULL NULL NULL NULL 9 Using where > -1 PRIMARY <derived2> ref key0 key0 5 test.t2.a 2 Using where > +1 PRIMARY <derived2> ref key0 key0 5 test.t2.a 1 Using where > 2 DERIVED t1 ALL NULL NULL NULL NULL 18 Using where; Using temporary; Using filesort > 3 EXCEPT t1 ALL NULL NULL NULL NULL 18 Using where; Using temporary; Using filesort > NULL EXCEPT RESULT <except2,3> ALL NULL NULL NULL NULL NULL > @@ -8250,7 +8250,7 @@ EXPLAIN > "key_length": "5", > "used_key_parts": ["a"], > "ref": ["test.t2.a"], > - "rows": 2, > + "rows": 1, > "filtered": 100, > "attached_condition": "v1.c > 500", > "materialized": { > @@ -8317,7 +8317,7 @@ a b c a b c > explain select * from v1,t2 where (v1.a=t2.a) and (v1.a<5) and (v1.c>500); > id select_type table type possible_keys key key_len ref rows Extra > 1 PRIMARY t2 ALL NULL NULL NULL NULL 9 Using where > -1 PRIMARY <derived2> ref key0 key0 5 test.t2.a 2 Using where > +1 PRIMARY <derived2> ref key0 key0 5 test.t2.a 1 Using where > 2 DERIVED t1 ALL NULL NULL NULL NULL 18 Using where; Using temporary; Using filesort > 3 EXCEPT t1 ALL NULL NULL NULL NULL 18 Using where; Using temporary; Using filesort > NULL EXCEPT RESULT <except2,3> ALL NULL NULL NULL NULL NULL > @@ -8341,7 +8341,7 @@ EXPLAIN > "key_length": "5", > "used_key_parts": ["a"], > "ref": ["test.t2.a"], > - "rows": 2, > + "rows": 1, > "filtered": 100, > "attached_condition": "v1.c > 500", > "materialized": { > @@ -8411,7 +8411,7 @@ a b c a b c > explain select * from v1,t2 where (v1.a=t2.a) and ((v1.b>27) or (v1.b<19)); > id select_type table type possible_keys key key_len ref rows Extra > 1 PRIMARY t2 ALL NULL NULL NULL NULL 9 Using where > -1 PRIMARY <derived2> ref key0 key0 5 test.t2.a 2 Using where > +1 PRIMARY <derived2> ref key0 key0 5 test.t2.a 1 Using where > 2 DERIVED t1 ALL NULL NULL NULL NULL 18 Using where; Using temporary; Using filesort > 3 EXCEPT t1 ALL NULL NULL NULL NULL 18 Using where; Using temporary; Using filesort > NULL EXCEPT RESULT <except2,3> ALL NULL NULL NULL NULL NULL > @@ -8435,7 +8435,7 @@ EXPLAIN > "key_length": "5", > "used_key_parts": ["a"], > "ref": ["test.t2.a"], > - "rows": 2, > + "rows": 1, > "filtered": 100, > "attached_condition": "v1.b > 27 or v1.b < 19", > "materialized": { > @@ -8508,7 +8508,7 @@ explain select * from v1,t2 where > (v1.a=t2.a) and ((v1.c<400) or (v1.c>800)); > id select_type table type possible_keys key key_len ref rows Extra > 1 PRIMARY t2 ALL NULL NULL NULL NULL 9 Using where > -1 PRIMARY <derived2> ref key0 key0 5 test.t2.a 2 Using where > +1 PRIMARY <derived2> ref key0 key0 5 test.t2.a 1 Using where > 2 DERIVED t1 ALL NULL NULL NULL NULL 18 Using where; Using temporary; Using filesort > 3 EXCEPT t1 ALL NULL NULL NULL NULL 18 Using where; Using temporary; Using filesort > NULL EXCEPT RESULT <except2,3> ALL NULL NULL NULL NULL NULL > @@ -8533,7 +8533,7 @@ EXPLAIN > "key_length": "5", > "used_key_parts": ["a"], > "ref": ["test.t2.a"], > - "rows": 2, > + "rows": 1, > "filtered": 100, > "attached_condition": "v1.c < 400 or v1.c > 800", > "materialized": { > @@ -8762,7 +8762,7 @@ where > ((d1.a>4) and (d1.c>500))); > id select_type table type possible_keys key key_len ref rows Extra > 1 PRIMARY t2 ALL NULL NULL NULL NULL 9 Using where > -1 PRIMARY <derived2> ref key0 key0 5 test.t2.b 2 Using where > +1 PRIMARY <derived2> ref key0 key0 5 test.t2.b 1 Using where > 2 DERIVED t1 ALL NULL NULL NULL NULL 18 Using where; Using temporary; Using filesort > 3 EXCEPT t1 ALL NULL NULL NULL NULL 18 Using where; Using temporary; Using filesort > NULL EXCEPT RESULT <except2,3> ALL NULL NULL NULL NULL NULL > @@ -8796,7 +8796,7 @@ EXPLAIN > "key_length": "5", > "used_key_parts": ["b"], > "ref": ["test.t2.b"], > - "rows": 2, > + "rows": 1, > "filtered": 100, > "attached_condition": "t2.c = 988 and t2.b > 13 or d1.a > 4 and d1.c > 500", > "materialized": { > @@ -8872,7 +8872,7 @@ a b c a b c > explain select * from v1,t2 where (v1.a=t2.a) and (v1.a>5) and (v1.c>200); > id select_type table type possible_keys key key_len ref rows Extra > 1 PRIMARY t2 ALL NULL NULL NULL NULL 9 Using where > -1 PRIMARY <derived2> ref key0 key0 5 test.t2.a 3 Using where > +1 PRIMARY <derived2> ref key0 key0 5 test.t2.a 1 Using where > 2 DERIVED t1 ALL NULL NULL NULL NULL 18 Using where; Using temporary; Using filesort > 3 UNION <derived4> ALL NULL NULL NULL NULL 18 Using where > 4 DERIVED t1 ALL NULL NULL NULL NULL 18 Using where; Using temporary; Using filesort > @@ -8899,7 +8899,7 @@ EXPLAIN > "key_length": "5", > "used_key_parts": ["a"], > "ref": ["test.t2.a"], > - "rows": 3, > + "rows": 1, > "filtered": 100, > "attached_condition": "v1.c > 200", > "materialized": { > @@ -9138,7 +9138,7 @@ a b c a b c > explain select * from v1,t2 where (v1.a=t2.a) and (v1.a>5) and (v1.c>200); > id select_type table type possible_keys key key_len ref rows Extra > 1 PRIMARY t2 ALL NULL NULL NULL NULL 9 Using where > -1 PRIMARY <derived2> ref key0 key0 5 test.t2.a 3 Using where > +1 PRIMARY <derived2> ref key0 key0 5 test.t2.a 1 Using where > 2 DERIVED t1 ALL NULL NULL NULL NULL 18 Using where; Using temporary; Using filesort > 3 UNION t1 ALL NULL NULL NULL NULL 18 Using where; Using temporary; Using filesort > 4 EXCEPT t1 ALL NULL NULL NULL NULL 18 Using where; Using temporary; Using filesort > @@ -9163,7 +9163,7 @@ EXPLAIN > "key_length": "5", > "used_key_parts": ["a"], > "ref": ["test.t2.a"], > - "rows": 3, > + "rows": 1, > "filtered": 100, > "attached_condition": "v1.c > 200", > "materialized": { > @@ -9260,7 +9260,7 @@ a b c a b c > explain select * from v1,t2 where (v1.a=t2.a) and (v1.a>4) and (v1.c<200); > id select_type table type possible_keys key key_len ref rows Extra > 1 PRIMARY t2 ALL NULL NULL NULL NULL 9 Using where > -1 PRIMARY <derived2> ref key0 key0 5 test.t2.a 3 Using where > +1 PRIMARY <derived2> ref key0 key0 5 test.t2.a 1 Using where > 2 DERIVED t1 ALL NULL NULL NULL NULL 18 Using where; Using temporary; Using filesort > 3 EXCEPT t1 ALL NULL NULL NULL NULL 18 Using where; Using temporary; Using filesort > 4 UNION t1 ALL NULL NULL NULL NULL 18 Using where; Using temporary; Using filesort > @@ -9285,7 +9285,7 @@ EXPLAIN > "key_length": "5", > "used_key_parts": ["a"], > "ref": ["test.t2.a"], > - "rows": 3, > + "rows": 1, > "filtered": 100, > "attached_condition": "v1.c < 200", > "materialized": { > @@ -9498,7 +9498,7 @@ a b c a b c > explain select * from v1,t2 where (v1.a=t2.a) and (v1.a>4) and (v1.c<130); > id select_type table type possible_keys key key_len ref rows Extra > 1 PRIMARY t2 ALL NULL NULL NULL NULL 9 Using where > -1 PRIMARY <derived2> ref key0 key0 5 test.t2.a 2 Using where > +1 PRIMARY <derived2> ref key0 key0 5 test.t2.a 1 Using where > 2 DERIVED t1 ALL NULL NULL NULL NULL 18 Using where; Using temporary; Using filesort > 3 EXCEPT <derived4> ALL NULL NULL NULL NULL 18 Using where > 4 DERIVED t1 ALL NULL NULL NULL NULL 18 Using where; Using temporary; Using filesort > @@ -9525,7 +9525,7 @@ EXPLAIN > "key_length": "5", > "used_key_parts": ["a"], > "ref": ["test.t2.a"], > - "rows": 2, > + "rows": 1, > "filtered": 100, > "attached_condition": "v1.c < 130", > "materialized": { > @@ -9643,7 +9643,7 @@ a b c a b c > explain select * from v1,t2 where (v1.a=t2.a) and (v1.a>4) and (v1.c<130); > id select_type table type possible_keys key key_len ref rows Extra > 1 PRIMARY t2 ALL NULL NULL NULL NULL 9 Using where > -1 PRIMARY <derived2> ref key0 key0 5 test.t2.a 3 Using where > +1 PRIMARY <derived2> ref key0 key0 5 test.t2.a 1 Using where > 2 DERIVED t1 ALL NULL NULL NULL NULL 18 Using where; Using temporary; Using filesort > 3 EXCEPT <derived4> ALL NULL NULL NULL NULL 18 Using where > 4 DERIVED t1 ALL NULL NULL NULL NULL 18 Using where; Using temporary; Using filesort > @@ -9671,7 +9671,7 @@ EXPLAIN > "key_length": "5", > "used_key_parts": ["a"], > "ref": ["test.t2.a"], > - "rows": 3, > + "rows": 1, > "filtered": 100, > "attached_condition": "v1.c < 130", > "materialized": { > @@ -10143,7 +10143,7 @@ a b c a b c > explain select * from v1,t2 where (v1.a=t2.a) and (v1.a<2) and (v1.b<30) and (v1.c>450); > id select_type table type possible_keys key key_len ref rows Extra > 1 PRIMARY t2 ALL NULL NULL NULL NULL 9 Using where > -1 PRIMARY <derived2> ref key0 key0 5 test.t2.a 2 Using where > +1 PRIMARY <derived2> ref key0 key0 5 test.t2.a 1 Using where > 2 DERIVED t1 ALL NULL NULL NULL NULL 18 Using where; Using temporary; Using filesort > 3 EXCEPT t1 ALL NULL NULL NULL NULL 18 Using where; Using temporary; Using filesort > NULL EXCEPT RESULT <except2,3> ALL NULL NULL NULL NULL NULL > @@ -10167,7 +10167,7 @@ EXPLAIN > "key_length": "5", > "used_key_parts": ["a"], > "ref": ["test.t2.a"], > - "rows": 2, > + "rows": 1, > "filtered": 100, > "attached_condition": "v1.b < 30 and v1.c > 450", > "materialized": { > @@ -10243,7 +10243,7 @@ a b c a b c > explain select * from v1,t2 where (v1.a=t2.a) and ((v1.a<2) or (v1.a<5)) and (v1.c>450); > id select_type table type possible_keys key key_len ref rows Extra > 1 PRIMARY t2 ALL NULL NULL NULL NULL 9 Using where > -1 PRIMARY <derived2> ref key0 key0 5 test.t2.a 2 Using where > +1 PRIMARY <derived2> ref key0 key0 5 test.t2.a 1 Using where > 2 DERIVED t1 ALL NULL NULL NULL NULL 18 Using where; Using temporary; Using filesort > 3 EXCEPT t1 ALL NULL NULL NULL NULL 18 Using where; Using temporary; Using filesort > NULL EXCEPT RESULT <except2,3> ALL NULL NULL NULL NULL NULL > @@ -10267,7 +10267,7 @@ EXPLAIN > "key_length": "5", > "used_key_parts": ["a"], > "ref": ["test.t2.a"], > - "rows": 2, > + "rows": 1, > "filtered": 100, > "attached_condition": "v1.c > 450", > "materialized": { > @@ -10636,7 +10636,7 @@ a b c a b c > explain select * from v1,t2 where (v1.b=t2.b) and (v1.a<3); > id select_type table type possible_keys key key_len ref rows Extra > 1 PRIMARY t2 ALL NULL NULL NULL NULL 9 Using where > -1 PRIMARY <derived2> ref key0 key0 5 test.t2.b 2 Using where > +1 PRIMARY <derived2> ref key0 key0 5 test.t2.b 1 Using where > 2 DERIVED t3 range i1 i1 5 NULL 1 Using index condition > 3 UNION t3 ALL NULL NULL NULL NULL 20 Using where; Using temporary; Using filesort > NULL UNION RESULT <union2,3> ALL NULL NULL NULL NULL NULL > @@ -10660,7 +10660,7 @@ EXPLAIN > "key_length": "5", > "used_key_parts": ["b"], > "ref": ["test.t2.b"], > - "rows": 2, > + "rows": 1, > "filtered": 100, > "attached_condition": "v1.a < 3", > "materialized": { > @@ -12333,7 +12333,7 @@ EXPLAIN > "key_length": "5", > "used_key_parts": ["b"], > "ref": ["test.t1.a"], > - "rows": 2, > + "rows": 1, > "filtered": 100, > "materialized": { > "query_block": { > diff --git a/mysql-test/r/derived_view.result b/mysql-test/r/derived_view.result > index 85e56ff176e..3a0fde7b053 100644 > --- a/mysql-test/r/derived_view.result > +++ b/mysql-test/r/derived_view.result > @@ -1525,7 +1525,7 @@ EXPLAIN > SELECT a FROM t1 WHERE (a,b) IN (SELECT * FROM v2); > id select_type table type possible_keys key key_len ref rows Extra > 1 PRIMARY t1 ALL NULL NULL NULL NULL 2 Using where > -1 PRIMARY <derived3> ref key0 key0 10 test.t1.a,test.t1.b 2 FirstMatch(t1) > +1 PRIMARY <derived3> ref key0 key0 10 test.t1.a,test.t1.b 1 FirstMatch(t1) Interesting.. here, we potentially could infer that this semi-join can be converted to an inner join (as the subquery may only have one match). But I think this is outside of the scope of this MDEV. > 3 DERIVED t2 ALL NULL NULL NULL NULL 6 > 4 UNION t3 ALL NULL NULL NULL NULL 4 > NULL UNION RESULT <union3,4> ALL NULL NULL NULL NULL NULL > diff --git a/mysql-test/r/join_cache.result b/mysql-test/r/join_cache.result > index eea397402ad..cc185b640bc 100644 > --- a/mysql-test/r/join_cache.result > +++ b/mysql-test/r/join_cache.result > @@ -5197,7 +5197,7 @@ SELECT * FROM (SELECT DISTINCT * FROM t1) t > WHERE t.a IN (SELECT t2.a FROM t2); > id select_type table type possible_keys key key_len ref rows Extra > 1 PRIMARY t2 ALL NULL NULL NULL NULL 4 Using where; Start temporary > -1 PRIMARY <derived2> ref key0 key0 5 test.t2.a 2 End temporary > +1 PRIMARY <derived2> ref key0 key0 5 test.t2.a 1 End temporary > 2 DERIVED t1 ALL NULL NULL NULL NULL 3 Using temporary > SELECT * FROM (SELECT DISTINCT * FROM t1) t > WHERE t.a IN (SELECT t2.a FROM t2); > @@ -5208,8 +5208,8 @@ EXPLAIN > SELECT * FROM (SELECT DISTINCT * FROM t1) t > WHERE t.a IN (SELECT t2.a FROM t2); > id select_type table type possible_keys key key_len ref rows Extra > -1 PRIMARY <derived2> ALL NULL NULL NULL NULL 3 > -1 PRIMARY t2 ALL NULL NULL NULL NULL 4 Using where; Start temporary; End temporary; Using join buffer (flat, BNL join) > +1 PRIMARY t2 ALL NULL NULL NULL NULL 4 Using where; Start temporary > +1 PRIMARY <derived2> ref key0 key0 5 test.t2.a 1 End temporary > 2 DERIVED t1 ALL NULL NULL NULL NULL 3 Using temporary > SELECT * FROM (SELECT DISTINCT * FROM t1) t > WHERE t.a IN (SELECT t2.a FROM t2); > diff --git a/mysql-test/r/mdev9959.result b/mysql-test/r/mdev9959.result > new file mode 100644 > index 00000000000..049e0350cca > --- /dev/null > +++ b/mysql-test/r/mdev9959.result > @@ -0,0 +1,46 @@ > +create table t1(a int); > +insert into t1 values (1),(2),(3),(4),(5),(6); > +create table t2(a int, b int,c int); > +insert into t2(a,b,c) values (1,1,2),(2,2,3),(3,1,4),(4,2,2),(5,1,1),(6,2,5); > +create table t3(a int, b int); > +insert into t3(a,b) values (1,1),(2,2),(2,1),(1,2),(5,1),(9,2); > +table "<derived2>" should have type=ref and rows=1 > +one select in derived table > +with distinct > +analyze select * from t1 , ((select distinct t2.a from t2 order by c))q where t1.a=q.a; > +id select_type table type possible_keys key key_len ref rows r_rows filtered r_filtered Extra > +1 PRIMARY t1 ALL NULL NULL NULL NULL 6 6.00 100.00 100.00 Using where > +1 PRIMARY <derived2> ref key0 key0 5 test.t1.a 1 1.00 100.00 100.00 > +2 DERIVED t2 ALL NULL NULL NULL NULL 6 6.00 100.00 100.00 Using temporary; Using filesort > +# multiple selects in derived table > +# NO UNION ALL > +analyze select * from t1 , ( (select t2.a,t2.b from t2 order by c) union (select t2.a,t2.b from t2 order by c))q where t1.a=q.a; > +id select_type table type possible_keys key key_len ref rows r_rows filtered r_filtered Extra > +1 PRIMARY t1 ALL NULL NULL NULL NULL 6 6.00 100.00 100.00 Using where > +1 PRIMARY <derived2> ref key0 key0 5 test.t1.a 1 1.00 100.00 100.00 > +2 DERIVED t2 ALL NULL NULL NULL NULL 6 6.00 100.00 100.00 > +3 UNION t2 ALL NULL NULL NULL NULL 6 6.00 100.00 100.00 > +NULL UNION RESULT <union2,3> ALL NULL NULL NULL NULL NULL 6.00 NULL NULL > +select * from t1 , ( (select t2.a from t2 order by c) union (select t2.a from t2 order by c))q where t1.a=q.a; > +a a > +1 1 > +2 2 > +3 3 > +4 4 > +5 5 > +6 6 > +# UNION ALL and EXCEPT > +analyze select * from t1 , ( (select t2.a,t2.b from t2 order by c) union all (select t2.a,t2.b from t2 order by c) except(select t3.a, t3.b from t3 order by b))q where t1.a=q.a; > +id select_type table type possible_keys key key_len ref rows r_rows filtered r_filtered Extra > +1 PRIMARY t1 ALL NULL NULL NULL NULL 6 6.00 100.00 100.00 Using where > +1 PRIMARY <derived2> ref key0 key0 5 test.t1.a 1 0.50 100.00 100.00 > +2 DERIVED t2 ALL NULL NULL NULL NULL 6 6.00 100.00 100.00 > +3 UNION t2 ALL NULL NULL NULL NULL 6 6.00 100.00 100.00 > +4 EXCEPT t3 ALL NULL NULL NULL NULL 6 6.00 100.00 100.00 > +NULL UNIT RESULT <unit2,3,4> ALL NULL NULL NULL NULL NULL 3.00 NULL NULL > +select * from t1 , ( (select t2.a,t2.b from t2 order by c) union all (select t2.a,t2.b from t2 order by c) except(select t3.a, t3.b from t3 order by b))q where t1.a=q.a; > +a a b > +3 3 1 > +4 4 2 > +6 6 2 > +drop table t1,t2,t3; > diff --git a/mysql-test/t/mdev9959.test b/mysql-test/t/mdev9959.test > new file mode 100644 > index 00000000000..36f364261c9 > --- /dev/null > +++ b/mysql-test/t/mdev9959.test > @@ -0,0 +1,25 @@ > +create table t1(a int); > +insert into t1 values (1),(2),(3),(4),(5),(6); > +create table t2(a int, b int,c int); > +insert into t2(a,b,c) values (1,1,2),(2,2,3),(3,1,4),(4,2,2),(5,1,1),(6,2,5); > +create table t3(a int, b int); > +insert into t3(a,b) values (1,1),(2,2),(2,1),(1,2),(5,1),(9,2); > + > +--echo table "<derived2>" should have type=ref and rows=1 > +--echo one select in derived table > + > +--echo with distinct > +analyze select * from t1 , ((select distinct t2.a from t2 order by c))q where t1.a=q.a; > + > +--echo # multiple selects in derived table > +--echo # NO UNION ALL > +analyze select * from t1 , ( (select t2.a,t2.b from t2 order by c) union (select t2.a,t2.b from t2 order by c))q where t1.a=q.a; > +select * from t1 , ( (select t2.a from t2 order by c) union (select t2.a from t2 order by c))q where t1.a=q.a; > + > +--echo # UNION ALL and EXCEPT > +analyze select * from t1 , ( (select t2.a,t2.b from t2 order by c) union all (select t2.a,t2.b from t2 order by c) except(select t3.a, t3.b from t3 order by b))q where t1.a=q.a; > + > +select * from t1 , ( (select t2.a,t2.b from t2 order by c) union all (select t2.a,t2.b from t2 order by c) except(select t3.a, t3.b from t3 order by b))q where t1.a=q.a; > + > +drop table t1,t2,t3; > + > diff --git a/sql/sql_lex.h b/sql/sql_lex.h > index f0241a32acf..16d5e55d251 100644 > --- a/sql/sql_lex.h > +++ b/sql/sql_lex.h > @@ -761,6 +761,7 @@ class st_select_lex_unit: public st_select_lex_node { > Procedure *last_procedure; /* Pointer to procedure, if such exists */ > > bool columns_are_renamed; > + bool union_all; /* TRUE if there is a UNION ALL operation */ This is not initialized properly. The testcase: create table t10 (a int , b int); insert into t10 values (1,1),(1,1); create table t11 (a int , b int); create table t12 (a int , b int); insert into t11 values (2,2),(2,2); create table t13 (a int , b int); insert into t12 values (3,3),(3,3); create table ten(a int); insert into ten values (0),(1),(2),(3),(4),(5),(6),(7),(8),(9); explain select * from ten A, (select * from t10 union select * from t11 union select * from t12) T where T.a=A.a; +------+--------------+--------------+------+---------------+------+---------+----------+------+-------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +------+--------------+--------------+------+---------------+------+---------+----------+------+-------------+ | 1 | PRIMARY | A | ALL | NULL | NULL | NULL | NULL | 10 | Using where | | 1 | PRIMARY | <derived2> | ref | key0 | key0 | 5 | test.A.a | 1 | | | 2 | DERIVED | t10 | ALL | NULL | NULL | NULL | NULL | 2 | | | 3 | UNION | t11 | ALL | NULL | NULL | NULL | NULL | 2 | | | 4 | UNION | t12 | ALL | NULL | NULL | NULL | NULL | 2 | | | NULL | UNION RESULT | <union2,3,4> | ALL | NULL | NULL | NULL | NULL | NULL | | +------+--------------+--------------+------+---------------+------+---------+----------+------+-------------+ (gdb) wher #0 st_select_lex_unit::check_distinct_in_union (this=0x7fff700149e0) at /home/psergey/dev-git/10.3-cp/sql/sql_union.cc:2100 #1 0x0000555555d2a693 in TABLE::add_tmp_key (this=0x7fff70062908, key=0, key_parts=1, next_field_no=0x555555c6bf47 <get_next_field_for_derived_key(uchar*)>, arg=0x7fffd98e9230 "X\207\006p\377\177", unique=false) at /home/psergey/dev-git/10.3-cp/sql/table.cc:7270 #2 0x0000555555c6c214 in generate_derived_keys_for_table (keyuse=0x7fff700687b0, count=1, keys=1) at /home/psergey/dev-git/10.3-cp/sql/sql_select.cc:11091 #3 0x0000555555c6c42a in generate_derived_keys (keyuse_array=0x7fff70018968) at /home/psergey/dev-git/10.3-cp/sql/sql_select.cc:11166 #4 0x0000555555c60267 in sort_and_filter_keyuse (thd=0x7fff70000d60, keyuse=0x7fff70018968, skip_unprefixed_keyparts=true) at /home/psergey/dev-git/10.3-cp/sql/sql_select.cc:6403 #5 0x0000555555c5a910 in make_join_statistics (join=0x7fff70018678, tables_list=..., keyuse_array=0x7fff70018968) at /home/psergey/dev-git/10.3-cp/sql/sql_select.cc:4622 #6 0x0000555555c513d4 in JOIN::optimize_inner (this=0x7fff70018678) at /home/psergey/dev-git/10.3-cp/sql/sql_select.cc:1888 #7 0x0000555555c4fa73 in JOIN::optimize (this=0x7fff70018678) at /home/psergey/dev-git/10.3-cp/sql/sql_select.cc:1451 #8 0x0000555555c59638 in mysql_select (thd=0x7fff70000d60, tables=0x7fff70013f68, wild_num=1, fields=..., conds=0x7fff70017d98, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=2147748612, result=0x7fff70017ff0, unit=0x7fff70004c28, select_lex=0x7fff70005398) at /home/psergey/dev-git/10.3-cp/sql/sql_select.cc:4225 #9 0x0000555555c91d6f in mysql_explain_union (thd=0x7fff70000d60, unit=0x7fff70004c28, result=0x7fff70017ff0) at /home/psergey/dev-git/10.3-cp/sql/sql_select.cc:25866 #10 0x0000555555c15da6 in execute_sqlcom_select (thd=0x7fff70000d60, all_tables=0x7fff70013f68) at /home/psergey/dev-git/10.3-cp/sql/sql_parse.cc:6487 #11 0x0000555555c0c7a5 in mysql_execute_command (thd=0x7fff70000d60) at /home/psergey/dev-git/10.3-cp/sql/sql_parse.cc:3821 #12 0x0000555555c1a0fd in mysql_parse (thd=0x7fff70000d60, rawbuf=0x7fff70013ca8 "explain select * from ten A, (select * from t10 union select * from t11 union select * from t12) T where T.a=A.a", length=112, parser_state=0x7fffd98ea5d0, is_com_multi=false, is_next_command=false) at /home/psergey/dev-git/10.3-cp/sql/sql_parse.cc:8091 (gdb) p union_all $36 = 165 > > void init_query(); > st_select_lex* outer_select(); > @@ -800,6 +801,7 @@ class st_select_lex_unit: public st_select_lex_node { > bool union_needs_tmp_table(); > > void set_unique_exclude(); > + bool check_distinct_in_union(); can this be made 'const'? if yes, please make it > > friend struct LEX; > friend int subselect_union_engine::exec(); > diff --git a/sql/sql_union.cc b/sql/sql_union.cc > index 857c9a117f5..6dac0cb0458 100644 > --- a/sql/sql_union.cc > +++ b/sql/sql_union.cc > @@ -1960,3 +1960,63 @@ void st_select_lex_unit::set_unique_exclude() > } > } > } > + > +/* > + Check if the selects in the derived table can give distinct rows irrespective > + of the data given for the tables. > + > + for example: > + select * from > + ((select t1.a from t1) op (select t2.a from t2) op (select t3.a from t3)); > + the op here being UNION/INTERSECT/EXCEPT > + > + so this function would check if the derived table like the case above > + can give distinct rows or not irrespective of the data in the tables. I think the above is difficult to read. Readability is a matter of opinion, but I would suggest another variant: /* @brief Check if the derived table is guaranteed to have distinct rows because of UNION operations used to populate it. @detail UNION operation removes duplicate rows from its output. That is, a query like select * from t1 UNION select * from t2 will not produce duplicate rows in its output, even if table t1 (and/or t2) contain duplicate rows. EXCEPT and INTERSECT operations also have this property. On the other hand, UNION ALL operation doesn't remove duplicates. (The SQL standard also defines EXCEPT ALL and INTERSECT ALL, but we don't support them). st_select_lex_unit computes its value left to right. That is, if there is a st_select_lex_unit object describing (select #1) OP1 (select #2) OP2 (select #3) then ((select #1) OP1 (select #2)) is computed first, and OP2 is computed second. How can one tell if st_select_lex_unit is guaranteed to have distinct output rows? This depends on whether the last operation was duplicate- removing or not: - UNION ALL is not duplicate-removing - all other operations are duplicate-removing */ Is the above correct? > + > + So what the function is handling: > + - If there is no UNION ALL, we are guarenteed distinct rows. > + > + Example > + ((select t1.a from t1) UNION (select t2.a from t2); > + > + - If there is UNION ALL, we can still guarantee distinct rows if > + the last operation of the selects does not involve UNION ALL or > + INTERSECT is not there . > + > + Example: > + ((select t1.a from t1) UNION ALL (select t2.a from t2) UNION/EXCEPT > + (select t3.a from t3)): guarantees distinct rows, UNION and EXCEPT > + have the same precedence. > + > + Example: > + ((select t1.a from t1) UNION ALL (select t2.a from t2) INTERSECT > + (select t3.a from t3)): does not guarantee distinct rows because > + INTERSECT has higher precedence than UNION so we would evaluate > + the INTERSECT part first and then do UNION ALL, so there we can > + end up with duplicates, so distinct rows are not guaranteed. > + > + @retval false Distinct rows are not guaranteed > + @retval true Distinct rows are guanranteed irrespective of the data > + in the tables > + > +*/ > + > +bool st_select_lex_unit::check_distinct_in_union() > +{ > + bool is_intersect_present=FALSE; > + st_select_lex* first= first_select(); > + for(st_select_lex *sl=first; sl ; sl=sl->next_select()) > + is_intersect_present|= (sl->linkage == INTERSECT_TYPE); > + > + if (!union_all) > + return true; > + else > + { > + if (union_distinct) > + { > + if (!is_intersect_present && !union_distinct->next_select()) > + return true; > + } I'm unable to undestand what kind of check we are making here. According to my variant of the comment above, we should just check whether the last operation (operation made with the output of the last select) was duplicate-removing or not. We seem to be doing something more complex? > + } > + return false; > +} > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index 35ec2d29d21..112873e8362 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -571,7 +571,10 @@ bool LEX::add_select_to_union_list(bool is_union_distinct, > current_select; > } > else > + { > DBUG_ASSERT(type == UNION_TYPE); > + current_select->master_unit()->union_all= true; > + } > return FALSE; > } > > diff --git a/sql/table.cc b/sql/table.cc > index 4f90d429ce5..d2938a29ea6 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -7113,6 +7113,25 @@ bool TABLE::add_tmp_key(uint key, uint key_parts, > if (!keyinfo->rec_per_key) > return TRUE; > bzero(keyinfo->rec_per_key, sizeof(ulong)*key_parts); > + /* > + For the case when there is a derived table that would give distinct rows, > + the index statistics are passed to the join optimizer to tell that > + a ref access to the derived table will produce only one row. > + */ > + > + st_select_lex_unit* derived= pos_in_table_list ? pos_in_table_list->derived: NULL; > + if (derived) > + { > + /* > + This handles the case when we have a single select in the derived table > + */ Does it anymore? Now it is handling the UNION case, too. > + st_select_lex* first= derived->first_select(); > + if ((first && !first->is_part_of_union() && > + first->options & SELECT_DISTINCT) || please put brackets around the condition on the line above as some too-smart compilers now complain about a possible typo. > + derived->check_distinct_in_union()) > + keyinfo->rec_per_key[key_parts-1]=1; > + } > + > keyinfo->read_stats= NULL; > keyinfo->collected_stats= NULL; > BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] [Commits] 167a96b5157: MDEV-9959: A serious MariaDB server performance bug
by Sergey Petrunia 18 Apr '19

18 Apr '19

On Tue, Mar 27, 2018 at 04:48:10PM +0530, Varun wrote: > revision-id: 167a96b5157049408a6ad4bca7abcd376af93fb5 (mariadb-10.3.0-644-g167a96b5157) > parent(s): 4359c6b4806605c78987e50cab3a6b42016b7603 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2018-03-27 16:45:46 +0530 > message: > > MDEV-9959: A serious MariaDB server performance bug > > step#1: if a derived table has SELECT DISTINCT, provide index statistics for it so that the join optimizer in the > upper select knows that ref access to the table will produce one row. > > Added handling for multiple selects in the derived table ... > diff --git a/mysql-test/r/mdev9959.result b/mysql-test/r/mdev9959.result > new file mode 100644 > index 00000000000..049e0350cca > --- /dev/null > +++ b/mysql-test/r/mdev9959.result > @@ -0,0 +1,46 @@ > +create table t1(a int); > +insert into t1 values (1),(2),(3),(4),(5),(6); > +create table t2(a int, b int,c int); > +insert into t2(a,b,c) values (1,1,2),(2,2,3),(3,1,4),(4,2,2),(5,1,1),(6,2,5); > +create table t3(a int, b int); > +insert into t3(a,b) values (1,1),(2,2),(2,1),(1,2),(5,1),(9,2); > +table "<derived2>" should have type=ref and rows=1 > +one select in derived table > +with distinct > +analyze select * from t1 , ((select distinct t2.a from t2 order by c))q where t1.a=q.a; > +id select_type table type possible_keys key key_len ref rows r_rows filtered r_filtered Extra > +1 PRIMARY t1 ALL NULL NULL NULL NULL 6 6.00 100.00 100.00 Using where > +1 PRIMARY <derived2> ref key0 key0 5 test.t1.a 1 1.00 100.00 100.00 > +2 DERIVED t2 ALL NULL NULL NULL NULL 6 6.00 100.00 100.00 Using temporary; Using filesort > +# multiple selects in derived table > +# NO UNION ALL > +analyze select * from t1 , ( (select t2.a,t2.b from t2 order by c) union (select t2.a,t2.b from t2 order by c))q where t1.a=q.a; > +id select_type table type possible_keys key key_len ref rows r_rows filtered r_filtered Extra > +1 PRIMARY t1 ALL NULL NULL NULL NULL 6 6.00 100.00 100.00 Using where > +1 PRIMARY <derived2> ref key0 key0 5 test.t1.a 1 1.00 100.00 100.00 > +2 DERIVED t2 ALL NULL NULL NULL NULL 6 6.00 100.00 100.00 > +3 UNION t2 ALL NULL NULL NULL NULL 6 6.00 100.00 100.00 > +NULL UNION RESULT <union2,3> ALL NULL NULL NULL NULL NULL 6.00 NULL NULL How would you explain the above? The derived table is a union of two selects, each of which produces 6 rows. Neither of the selects have DISTINCT attribute. Why does the optimization fire and set ref=1 in this case? (if both parts of UNION had distinct, this would make sense. But if neither does, I don't see any logic) BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] MariaDB 10.4 on AIX. Issue with: HAVE_POOL_OF_THREADS
by REIX, Tony 18 Apr '19

18 Apr '19

Hi, In my environment (AIX & MariaDB v10.4.3 or 10.4.4), I have an issue with HAVE_POOL_OF_THREADS. When building, I have the following trace and error (full line of g++ compilation at end of email) : [ 44%] Linking CXX executable explain_filename-t /opt/freeware/bin/cmake -E cmake_link_script CMakeFiles/explain_filename-t.dir/link.txt --verbose=1 /opt/freeware/bin/g++ ........... -o explain_filename-t .... ld: 0711-317 ERROR: Undefined symbol: .TP_pool_generic::TP_pool_generic() ld: 0711-317 ERROR: Undefined symbol: .TP_pool_generic::init() This seems to be due to the following lines of the file: sql/CMakeLists.txt : IF (CMAKE_SYSTEM_NAME MATCHES "Linux" OR CMAKE_SYSTEM_NAME MATCHES "Windows" OR CMAKE_SYSTEM_NAME MATCHES "SunOS" OR HAVE_KQUEUE) ADD_DEFINITIONS(-DHAVE_POOL_OF_THREADS) IF(WIN32) SET(SQL_SOURCE ${SQL_SOURCE} threadpool_win.cc) ENDIF() SET(SQL_SOURCE ${SQL_SOURCE} threadpool_generic.cc) ENDIF() In case the Operating System is not Linux, Windows, or SunOS, then there is NO -DHAVE_POOL_OF_THREADS and the file: threadpool_generic.cc is NOT compiled. However, the C++ code of MariaDB still looks at least for 2 routines of threadpool_generic.cc : TP_pool_generic::TP_pool_generic() TP_pool_generic::init() but these routines are not available since threadpool_generic.cc was not compiled. So, in my opinion, there are some #ifdef HAVE_POOL_OF_THREADS lines that are missing in common threadpool code that would prevent to use threadpool generic routines. However, I do not master this code. Why not implement on AIX the generic pool of thread and thus add AIX to the above IF( ) test of sql/CMakeLists.txt ?? The reason if that, out of the 4 possibilities listed in sql/threadpool_generic.cc : epoll, KQUEUE, port, and IOCP, none of them are available on AIX with the EXACT required behavior. We have to implement something miming epoll(), we think. Before fixing this performance improvement, we'd like to be able to build & test MariaDB for all its other features. sql/threadpool_generic.cc : #ifdef __linux__ #include <sys/epoll.h> typedef struct epoll_event native_event; #elif defined(HAVE_KQUEUE) #include <sys/event.h> typedef struct kevent native_event; #elif defined (__sun) #include <port.h> typedef port_event_t native_event; #elif defined (HAVE_IOCP) typedef OVERLAPPED_ENTRY native_event; #else #error threadpool is not available on this platform <<---- AIX #endif Regards, Tony /opt/freeware/bin/g++ -O2 -g -L/opt/freeware/lib64 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -maix64 -Wl,-bbigtoc -fPIC -O2 -g -L/opt/freeware/lib64 -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -maix64 -Wl,-bbigtoc -fPIC -pie -fPIC -fno-rtti -O3 -g -static-libgcc -fno-omit-frame-pointer -fno-strict-aliasing -Wno-uninitialized -D_FORTIFY_SOURCE=2 -DDBUG_OFF -Wl,-blibpath:/opt/freeware/lib:/opt/freeware/lib64:/usr/lib:/lib -lpgport CMakeFiles/explain_filename-t.dir/explain_filename-t.cc.o -o explain_filename-t -Wl,-bnoipath -Wl,-blibpath:/opt/freeware/lib64:/opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/8.2.0/ppc64:/opt/freeware/lib/ppc64:/opt/freeware/lib/gcc/powerpc-ibm-aix7.2.0.0/8.2.0:/opt/freeware/lib:/usr/lib:/lib -lpthreads ../../sql/libsql.a ../mytap/libmytap.a ../../storage/archive/libarchive.a ../../storage/blackhole/libblackhole.a ../../storage/csv/libcsv.a ../../storage/federatedx/libfederatedx.a ../../storage/heap/libheap.a ../../storage/innobase/libinnobase.a -llz4 ../../storage/maria/libaria.a ../../storage/myisam/libmyisam.a ../../storage/myisammrg/libmyisammrg.a ../../storage/sequence/libsequence.a ../../plugin/feedback/libfeedback.a ../../plugin/user_variables/libuser_variables.a ../../plugin/userstat/libuserstat.a ../../sql/libpartition.a ../../sql/libsql_sequence.a ../../mysys/libmysys.a ../../mysys_ssl/libmysys_ssl.a ../../dbug/libdbug.a ../../mysys/libmysys.a ../../mysys_ssl/libmysys_ssl.a ../../dbug/libdbug.a /usr/lib64/libz.so -lm ../../extra/yassl/libyassl.a ../../extra/yassl/taocrypt/libtaocrypt.a ../../strings/libstrings.a ../../vio/libvio.a ../../pcre/libpcre.a ../../wsrep-lib/src/libwsrep-lib.a -lpthread -ldl ../../wsrep-lib/wsrep-API/libwsrep_api_v26.a -lpthreads

2 5

Re: [Maria-developers] [JIRA] (MDEV-17260) Memory leaks in mysqlbinlog
by andrei.elkin＠pp.inet.fi 17 Apr '19

17 Apr '19

Sujatha, hello. The patches look good though I have a concern in .. > [ https://jira.mariadb.org/browse/MDEV-17260?page=com.atlassian.jira.plugin.s… ] > > Sujatha Sivakumar commented on MDEV-17260: > ------------------------------------------ > > In the above report there are three different kinds of leaks were observed. > > In 5.5: > ===== > CURRENT_TEST: rpl.rpl_row_mysqlbinlog > ================================================================ > ==8988==ERROR: LeakSanitizer: detected memory leaks > Direct leak of 112 byte(s) in 1 object(s) allocated from: > #0 0x4eff87 in __interceptor_malloc (/dev/shm/5.5/client/mysqlbinlog+0x4eff87) > #1 0x60eaab in my_malloc /mariadb/5.5/mysys/my_malloc.c:41:10 > #2 0x5300dd in Log_event::read_log_event(char const*, unsigned int, char const**, Format_description_log_event const*, char) /mariadb/5.5/sql/log_event.cc:1568:12 > #3 0x564a9c in dump_remote_log_entries(st_print_event_info*, char const*) /mariadb/5.5/client/mysqlbinlog.cc:1978:17 > #4 0x536408 in dump_log_entries(char const*) /mariadb/5.5/client/mysqlbinlog.cc:1794:21 > #5 0x535e4b in main /mariadb/5.5/client/mysqlbinlog.cc:2505:18 > #6 0x7f71ea0a1b16 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x22b16) > > This issue exists only in 5.5. It is specific to fake rotate events while reading binary log from a remote server. > This issues is already fixed in MariaDB 10.0.23 and higher versions as part of > commit c3018b0ff4fb02c029787d03867adf0530607bab > > 10.2 and above: > ============= > 1) Leak in Log_event::read_log_event(char const*, unsigned int, char const**, > Format_description_log_event const*, char) > File: sql/log_event.cc:2150 .. in 2). > 2) read_remote_annotate_event(unsigned char*, unsigned long, char const**) > File: client/mysqlbinlog.cc:194 > What would you think could be the reason of the being deleted two-liner: @@ -1539,10 +1539,11 @@ Exit_status process_event(PRINT_EVENT_INFO *print_event_info, Log_event *ev, } } - if (remote_opt) - ev->temp_buf= 0; I see there were done as a part of the flashback feature. Do we have the flashback and remote combination in any test? Cheers, Andrei > These were caused because of flashback feature and its related bug fixes. > > To reproduce the failures use: > > ASAN_OPTIONS=abort_on_error=1 ./mtr --parallel=auto binlog.binlog_row_mysqlbinlog_verbose binlog_encryption.binlog_row_annotate binlog.binlog_row_mysqlbinlog_options binlog.binlog_row_annotate encryption.tempfiles binlog.binlog_mysqlbinlog_row_myisam binlog.binlog_mysqlbinlog_row binlog.binlog_mysqlbinlog_base64 binlog_encryption.rpl_skip_replication binlog.binlog_mysqlbinlog_row_innodb binlog.flashback > >> Memory leaks in mysqlbinlog >> --------------------------- >> >> Key: MDEV-17260 >> URL: https://jira.mariadb.org/browse/MDEV-17260 >> Project: MariaDB Server >> Issue Type: Bug >> Components: Scripts & Clients >> Affects Versions: 10.0, 10.1, 10.2, 10.3, 10.4, 5.5.62 >> Reporter: Marko Mäkelä >> Assignee: Sujatha Sivakumar >> Priority: Major >> Labels: ASAN, Memory_leak, valgrind >> Fix For: 10.1, 10.2, 10.3, 10.4 >> >> Attachments: mdev-17260-wip.patch >> >> Time Spent: 5h >> Remaining Estimate: 0d >> >> The {{mysqlbinlog}} tool is leaking memory, causing failures in various tests when compiling and testing with AddressSanitizer or LeakSanitizer like this: >> {code:sh} >> cmake -DCMAKE_BUILD_TYPE=Debug -DWITH_ASAN:BOOL=ON /path/to/source >> make -j$(nproc) >> cd mysql-test >> ASAN_OPTIONS=abort_on_error=1 ./mtr --parallel=auto >> {code} >> {noformat:title=5.5 e07118946a82af60c7cc3804c321d3fd9a49f128} >> CURRENT_TEST: rpl.rpl_row_mysqlbinlog >> ================================================================= >> ==8988==ERROR: LeakSanitizer: detected memory leaks >> Direct leak of 112 byte(s) in 1 object(s) allocated from: >> #0 0x4eff87 in __interceptor_malloc (/dev/shm/5.5/client/mysqlbinlog+0x4eff87) >> #1 0x60eaab in my_malloc /mariadb/5.5/mysys/my_malloc.c:41:10 >> #2 0x5300dd in Log_event::read_log_event(char const*, unsigned int, char const**, Format_description_log_event const*, char) /mariadb/5.5/sql/log_event.cc:1568:12 >> #3 0x564a9c in dump_remote_log_entries(st_print_event_info*, char const*) /mariadb/5.5/client/mysqlbinlog.cc:1978:17 >> #4 0x536408 in dump_log_entries(char const*) /mariadb/5.5/client/mysqlbinlog.cc:1794:21 >> #5 0x535e4b in main /mariadb/5.5/client/mysqlbinlog.cc:2505:18 >> #6 0x7f71ea0a1b16 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x22b16) >> {noformat} > > > > -- > This message was sent by Atlassian Jira > (v7.13.1#713001)

1 0

[Maria-developers] MariaDB 10.4 on AIX. Issue with: sql/wsrep_mysqld.cc : ip_len
by REIX, Tony 16 Apr '19

16 Apr '19

Hi, In my environment (AIX & MariaDB v10.4.3 or 10.4.4), the following line of sql/wsrep_mysqld.cc : size_t const ip_len= wsrep_host_len(node_addr.c_str(), node_addr.size()); is preprocessed (gcc -E) as: size_t const ip_ff.ip_vhltl.ip_x.ip_xlen = wsrep_host_len(node_addr.c_str(), node_addr.size()); where "ip_len" becomes: ip_ff.ip_vhltl.ip_x.ip_xlen , which is wrong and breaks. This is due to the following macro in AIX /usr/include/netinet/ip.h : #define ip_len ip_ff.ip_flen The attached patch (replace ip_len by ip_len_mdb) is an easy work-around for fixing the issue. However, it may not be the perfect appropriate change. Since I do not master MariaDB, I'd like someone to propose a fully correct fix. Thanks/Regards, Tony Reix

1 0

[Maria-developers] MariaDB 10.4 on AIX. Issue with: #define SI_KERNEL in include/my_pthread.h
by REIX, Tony 16 Apr '19

16 Apr '19

Hi, In my environment (AIX & MariaDB v10.4.3 or 10.4.4), I have: include/config.h: #define HAVE_SIGWAITINFO 1 and thus in the file : include/my_pthread.h : static inline int my_sigwait(sigset_t *set, int *sig, int *code) { #ifdef HAVE_SIGWAITINFO siginfo_t siginfo; *sig= sigwaitinfo(set, &siginfo); *code= siginfo.si_code; return *sig < 0 ? errno : 0; #else #define SI_KERNEL 128 *code= 0; return sigwait(set, sig); #endif } the SI_KERNEL macro is NOT defined. Then, I get the following error at compilation: mariadb-10.4.3/sql/mysqld.cc:3272:36: error: 'SI_KERNEL' was not declared in this scope if (!abort_loop && origin != SI_KERNEL) The attached patch is an easy work-around for fixing the issue. However, it may not be the perfect appropriate change. Since I do not master MariaDB, I'd like someone to propose a fully correct fix. Thanks/Regards, Tony Reix tony.reix(a)atos.net ATOS / Bull SAS ATOS Expert IBM Coop Architect & Technical Leader Office : +33 (0) 4 76 29 72 67 1 rue de Provence - 38432 Échirolles - France www.atos.net

1 0

Re: [Maria-developers] 404 galera-arbitrator-3_25.3.26-xenial_amd64.deb
by Jongmin Kim 12 Apr '19

12 Apr '19

Ping to the Cc'ed folks, and add more Cc: our new mailing list (harukasan-ftp(a)lists.launchpad.net) On Thu, Apr 11, 2019 at 11:33:33PM +0900, Jongmin Kim wrote: > Hello, > > Thanks for reporting your failure! > > On Thu, Apr 11, 2019 at 03:51:17PM +0300, Sergei Shvarts wrote: > > Hi guys! > > > > looks like there is no galera-arbitraro-3 for xenial > > > > ERROR: unable to update: download errors: > > HTTP code 404 while fetching http://ftp.harukasan.org/mariadb/repo/10.3/ubuntu/pool/main/g/galera-3/gale… > > HTTP code 404 while fetching http://ftp.harukasan.org/mariadb/repo/10.3/ubuntu/pool/main/g/galera-3/gale… > > > > We digged some, and we found that: > - the package with 'Filename' is provided in Xenial 'Packages' [1], > - but the file is not exist at 'Filename' [2]. > We don't have idea about this issue. An official upstream mirror does have same issue [3,4]. > > [1] http://ftp.harukasan.org/mariadb/repo/10.3/ubuntu/dists/xenial/main/binary-… > [2] http://ftp.harukasan.org/mariadb/repo/10.3/ubuntu/pool/main/g/galera-3/ > [3] http://ftp.osuosl.org/pub/mariadb/repo/10.3/ubuntu/dists/xenial/main/binary… > [4] http://ftp.osuosl.org/pub/mariadb/repo/10.3/ubuntu/pool/main/g/galera-3/ > > So I'm forwarding this error to the package maintainer of 'galera-arbitrator-3': > - Codership Oy <info(a)codership.com> > > Also forwarding to the repository master of MariaDB repository: > - MariaDB Developers <maria-developers(a)lists.launchpad.net> > > Dear the package maintainer and MariaDB Developers, would you please dig this problem? > > Thank you! -- Jongmin Kim OpenPGP key located at https://jongmin.dev/pgp OpenPGP fingerprint: 012E 4A06 79E1 4EFC DAAE 9472 D39D 8D29 BAF3 6DF8

2 3

[Maria-developers] GSoC 2019 proposal
by Rajiv nayan 09 Apr '19

09 Apr '19

Hello. My name is Rajiv Nayan. I would like to contribute to mariaDB during GSoC 2019. More information can be found in my draft of proposal on the attached google doc. Thanks Proposal for "Implement UPDATE with result set, and Implement multiple-table UPDATE/DELETE returning a result set” for GSoC’19 https://docs.google.com/document/d/1vb9ixQlhOcHj4RpmaAuhMNNHw4jE_j1wjnzKuNW…

1 0

[Maria-developers] (no subject)
by Rajiv nayan 09 Apr '19

09 Apr '19

Hello. My name is Rajiv Nayan. I would like to contribute to mariaDB during GSoC 2019. More information can be found in my draft of proposal on the attached google doc. Thanks Proposal for "Implement UPDATE with result set, and Implement multiple-table UPDATE/DELETE returning a result set” for GSoC’19 https://docs.google.com/document/d/1vb9ixQlhOcHj4RpmaAuhMNNHw4jE_j1wjnzKuNW…

1 0

[Maria-developers] set_time() on slave and unversioned -> versioned replication
by Aleksey Midenkov 08 Apr '19

08 Apr '19

Hello, Sergei! In unversioned -> versioned scenario in the code below it first gets to Set time 4, creates some records (on slave) and then seconds on slave increased (X+1) while on master seconds are not yet increased (X). Then we get to Set time 3 and reset time on slave to X.0 therefore we get back in time and all stored records with timestamp X.n will be in future. 'n' came from ++system_time.sec_part in Set time 4. Why did you decided to use such logic of getting seconds from master and microseconds from slave? Since microseconds sooner or later reset to 0 it's not better than just assigning some random number. What sending microseconds from master conditionally only is good for? And for RBR since you don't send microseconds at all I see no good solution for this. inline void set_time(my_time_t t, ulong sec_part) { if (opt_secure_timestamp > (slave_thread ? SECTIME_REPL : SECTIME_SUPER)) { set_time(); // note that BINLOG itself requires SUPER DBUG_EXECUTE("time", print_start_time("Set time 1");); } else { if (sec_part <= TIME_MAX_SECOND_PART) { start_time= system_time.sec= t; start_time_sec_part= system_time.sec_part= sec_part; DBUG_EXECUTE("time", print_start_time("Set time 2");); } else if (t != system_time.sec) { DBUG_EXECUTE("time", print_system_time("System time"););; start_time= system_time.sec= t; start_time_sec_part= system_time.sec_part= 0; DBUG_EXECUTE("time", print_start_time("Set time 3");); } else { start_time= t; start_time_sec_part= ++system_time.sec_part; DBUG_EXECUTE("time", print_start_time("Set time 4");); } user_time.val= hrtime_from_time(start_time) + start_time_sec_part; PSI_CALL_set_thread_start_time(start_time); start_utime= utime_after_lock= microsecond_interval_timer(); } } -- All the best, Aleksey Midenkov @midenok

2 7

[Maria-developers] GSoC'19 proposal
by Rucha Deodhar 05 Apr '19

05 Apr '19

Hello. My name is Rucha Deodhar. I would like to contribute to mariaDB during GSoC 2019. More information can be found in my draft of proposal on the attached google doc. Any feedback would be appreciated. Thanks Proposal for “ADD RETURNING TO INSERT” for GSoC’19 <https://docs.google.com/document/d/1vE52kqZI-bhbRKbGBxeIrQKss8xXaGUU-0_PJHq…>

1 0

[Maria-developers] MDEV-18822: possible slowdown with optimizer trace
by Sergey Petrunia 04 Apr '19

04 Apr '19

Hi Varun, So I was thinking of the impact of the optimizer trace, and decided to look at how the most common check "if optimizer trace is enabled" is done. The code at SQL layer does this kind of check if (thd->trace_started()) (it is wrapped somewhere with "unlikely" but this is not consistent) THD::trace_started is defined as: bool THD::trace_started() { return opt_trace.is_started(); } which opt_trace here is Opt_trace_context. looking there: class Opt_trace_context { ... bool is_started() { return current_trace && is_enabled(); } following: bool Opt_trace_context::is_enabled() { if (current_trace) return current_trace->is_enabled(); return false; } current_trace is Opt_trace_stmt, looking there: bool Opt_trace_stmt::is_enabled() { return I_S_disabled == 0; } This is quite convoluted, and could probably made shorter. Disassembling sql_select.cc, I can see that the calls to thd->trace_started() are not inlined: if (unlikely(thd->trace_started())) 10ac9: 4c 89 ff mov rdi,r15 10acc: e8 00 00 00 00 call 10ad1 <_ZL32best_extension_by_limited_searchP4JOINyjddjjj+0x1a1> 10ad1: 84 c0 test al,al 10ad3: 0f 85 fc 02 00 00 jne 10dd5 <_ZL32best_extension_by_limited_searchP4JOINyjddjjj+0x4a5> if (unlikely(thd->trace_started())) 1197b: 4c 89 f7 mov rdi,r14 1197e: e8 00 00 00 00 call 11983 <_Z11choose_planP4JOINy+0x8a3> 11983: 84 c0 test al,al 11985: 0f 85 2f 01 00 00 jne 11aba <_Z11choose_planP4JOINy+0x9da> if (thd->trace_started()) 2d6f7: 48 8b bd b0 fc ff ff mov rdi,QWORD PTR [rbp-0x350] 2d6fe: e8 00 00 00 00 call 2d703 <_ZN4JOIN14optimize_innerEv+0x1963> 2d703: 84 c0 test al,al 2d705: 0f 85 6c 09 00 00 jne 2e077 <_ZN4JOIN14optimize_innerEv+0x22d7> if (thd->trace_started()) 2d7da: 48 8b bd b0 fc ff ff mov rdi,QWORD PTR [rbp-0x350] 2d7e1: e8 00 00 00 00 call 2d7e6 <_ZN4JOIN14optimize_innerEv+0x1a46> 2d7e6: 84 c0 test al,al 2d7e8: 0f 85 ff 1a 00 00 jne 2f2ed <_ZN4JOIN14optimize_innerEv+0x354d> But does it all matter? I think the action plan should be: 1. Reproduce the slowdown as it was reported in MDEV-18822. 2. If #1 succeeds in several re-runs, declare THD::trace_started as: class THD { public: bool trace_started; // initialize this to false in constructor and never touch it inline bool trace_started() const { return trace_started; } }; and then use search-replace to change "thd->trace_started()" to "unlikely(thd->trace_started)" and then re-run the benchmark. Any thoughts? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] [Commits] a1929d001c7: MDEV-18741: Optimizer trace: multi-part key ranges are printed incorrectly
by Sergey Petrunia 04 Apr '19

04 Apr '19

Hi Varun, In general, the patch moves in the right direction, but it is not yet there. Consider this example: create table ten(a int); insert into ten values (0),(1),(2),(3),(4),(5),(6),(7),(8),(9); create table t1 ( a int not null, b int not null, c int not null, d int not null, key(a,b,c) ); insert into t1 select a,a, a,a from ten; set optimizer_trace=1; explain select * from t1 where a between 1 and 4 and b < 50; mysql> select -> JSON_DETAILED(JSON_EXTRACT(trace, '$**.analyzing_range_alternatives')) -> from INFORMATION_SCHEMA.OPTIMIZER_TRACE; { "range_scan_alternatives": [ { "index": "a", "ranges": [ "(a) < (4,50)" ], "rowid_ordered": false, "using_mrr": false, "index_only": false, "rows": 1, "cost": 2.5002, "chosen": true } ], "analyzing_roworder_intersect": { "cause": "too few roworder scans" }, "analyzing_index_merge_union": [ ] } ] | while print_quick produces: T@18 : | | | | | | | | | | >print_quick quick range select, key a, length: 8 1 <= X < 4/50 other_keys: 0x0: T@18 : | | | | | | | | | | <print_quick Please investigate why this happens. On Wed, Apr 03, 2019 at 01:16:56PM +0530, Varun wrote: > revision-id: a1929d001c7b25e26d178834a196020bade819b8 (mariadb-10.3.6-318-ga1929d001c7) Any idea why does the commit trigger show "mariadb-10.3.." ? The patch is against 10.4, isnt it? > parent(s): 0dc442ac61ac7ff1a1d6bd7d6090c0f2251fb558 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2019-04-03 13:10:07 +0530 > message: > > MDEV-18741: Optimizer trace: multi-part key ranges are printed incorrectly > > Changed the function append_range_all_keyparts to use sel_arg_range_seq_init / sel_arg_range_seq_next to produce ranges. > Also adjusted to print format for the ranges, now the ranges are printed as: > (keypart1_min, keypart2_min,..) OP (keypart1_name,keypart2_name, ..) OP (keypart1_max,keypart2_max, ..) > > Also added more tests for range and index merge access for optimizer trace ... > diff --git a/sql/opt_range.cc b/sql/opt_range.cc > index 5ab3d70214d..f53efc55158 100644 > --- a/sql/opt_range.cc > +++ b/sql/opt_range.cc > @@ -431,16 +431,18 @@ static int and_range_trees(RANGE_OPT_PARAM *param, > static bool remove_nonrange_trees(RANGE_OPT_PARAM *param, SEL_TREE *tree); > > static void print_key_value(String *out, const KEY_PART_INFO *key_part, > - const uchar *key); > + const uchar* key, uint length); > +static void print_keyparts_name(String *out, const KEY_PART_INFO *key_part, > + uint n_keypart, key_part_map keypart_map); Please fix indentation ^ > > static void append_range_all_keyparts(Json_writer_array *range_trace, > - String *range_string, > - String *range_so_far, const SEL_ARG *keypart, > + PARAM param, uint idx, > + SEL_ARG *keypart, > const KEY_PART_INFO *key_parts); > > static > -void append_range(String *out, const KEY_PART_INFO *key_parts, > - const uchar *min_key, const uchar *max_key, const uint flag); > +void append_range(String *out, const KEY_PART_INFO *key_part, > + KEY_MULTI_RANGE *range, uint n_key_parts); Please fix indentation ^ > > > /* > @@ -2273,10 +2275,7 @@ void TRP_RANGE::trace_basic_info(const PARAM *param, > // TRP_RANGE should not be created if there are no range intervals > DBUG_ASSERT(key); > > - String range_info; > - range_info.length(0); > - range_info.set_charset(system_charset_info); > - append_range_all_keyparts(&trace_range, NULL, &range_info, key, key_part); > + append_range_all_keyparts(&trace_range, *param, key_idx, key, key_part); > } > > > @@ -2489,10 +2488,8 @@ void TRP_GROUP_MIN_MAX::trace_basic_info(const PARAM *param, > // can have group quick without ranges > if (index_tree) > { > - String range_info; > - range_info.set_charset(system_charset_info); > - append_range_all_keyparts(&trace_range, NULL, &range_info, index_tree, > - key_part); > + append_range_all_keyparts(&trace_range, *param, param_idx, > + index_tree, key_part); > } > } > > @@ -6398,20 +6395,9 @@ void TRP_ROR_INTERSECT::trace_basic_info(const PARAM *param, > trace_isect_idx.add("rows", (*cur_scan)->records); > > Json_writer_array trace_range(thd, "ranges"); > - for (const SEL_ARG *current= (*cur_scan)->sel_arg->first(); current; > - current= current->next) > - { > - String range_info; > - range_info.set_charset(system_charset_info); > - for (const SEL_ARG *part= current; part; > - part= part->next_key_part ? part->next_key_part : nullptr) > - { > - const KEY_PART_INFO *cur_key_part= key_part + part->part; > - append_range(&range_info, cur_key_part, part->min_value, > - part->max_value, part->min_flag | part->max_flag); > - } > - trace_range.add(range_info.ptr(), range_info.length()); > - } > + > + append_range_all_keyparts(&trace_range, *param, (*cur_scan)->idx, > + (*cur_scan)->sel_arg->first(), key_part); Please fix indentation ^ > } > } > > @@ -7389,9 +7375,6 @@ static TRP_RANGE *get_key_scans_params(PARAM *param, SEL_TREE *tree, > const KEY &cur_key= param->table->key_info[keynr]; > const KEY_PART_INFO *key_part= cur_key.key_part; > > - String range_info; > - range_info.set_charset(system_charset_info); > - > index_scan->idx= idx; > index_scan->keynr= keynr; > index_scan->key_info= &param->table->key_info[keynr]; > @@ -7402,8 +7385,8 @@ static TRP_RANGE *get_key_scans_params(PARAM *param, SEL_TREE *tree, > *tree->index_scans_end++= index_scan; > > if (unlikely(thd->trace_started())) > - append_range_all_keyparts(&trace_range, NULL, &range_info, key, > - key_part); > + append_range_all_keyparts(&trace_range, (*param), idx, > + key, key_part); Please fix indentation ^ > trace_range.end(); > > trace_idx.add("rowid_ordered", param->is_ror_scan) > @@ -13554,11 +13537,8 @@ get_best_group_min_max(PARAM *param, SEL_TREE *tree, double read_time) > Json_writer_array trace_range(thd, "ranges"); > > const KEY_PART_INFO *key_part= cur_index_info->key_part; > - > - String range_info; > - range_info.set_charset(system_charset_info); > - append_range_all_keyparts(&trace_range, NULL, &range_info, > - cur_index_tree, key_part); > + append_range_all_keyparts(&trace_range, *param, cur_param_idx, > + cur_index_tree, key_part); Please fix indentation ^ > } > } > cost_group_min_max(table, cur_index_info, cur_used_key_parts, > @@ -15730,12 +15710,17 @@ void QUICK_GROUP_MIN_MAX_SELECT::dbug_dump(int indent, bool verbose) > } > > #endif /* !DBUG_OFF */ > + > static > void append_range(String *out, const KEY_PART_INFO *key_part, > - const uchar *min_key, const uchar *max_key, const uint flag) > + KEY_MULTI_RANGE *range, uint n_key_parts) > { > - if (out->length() > 0) > - out->append(STRING_WITH_LEN(" AND ")); > + uint flag= range->range_flag; > + String key_name; > + key_name.set_charset(system_charset_info); > + key_part_map keypart_map= range->start_key.length ? > + range->start_key.keypart_map : > + range->end_key.keypart_map; > > if (flag & GEOM_FLAG) > { > @@ -15744,22 +15729,24 @@ void append_range(String *out, const KEY_PART_INFO *key_part, > range types, so printing "col < some_geom" doesn't make sense. > Just print the column name, not operator. > */ > - out->append(key_part->field->field_name); > + print_keyparts_name(out, key_part, n_key_parts, keypart_map); > out->append(STRING_WITH_LEN(" ")); > - print_key_value(out, key_part, min_key); > + print_key_value(out, key_part, range->start_key.key, > + range->start_key.length); > return; > } > > if (!(flag & NO_MIN_RANGE)) > { > - print_key_value(out, key_part, min_key); > + print_key_value(out, key_part, range->start_key.key, > + range->start_key.length); > if (flag & NEAR_MIN) > out->append(STRING_WITH_LEN(" < ")); > else > out->append(STRING_WITH_LEN(" <= ")); > } > > - out->append(key_part->field->field_name); > + print_keyparts_name(out, key_part, n_key_parts, keypart_map); > > if (!(flag & NO_MAX_RANGE)) > { > @@ -15767,7 +15754,8 @@ void append_range(String *out, const KEY_PART_INFO *key_part, > out->append(STRING_WITH_LEN(" < ")); > else > out->append(STRING_WITH_LEN(" <= ")); > - print_key_value(out, key_part, max_key); > + print_key_value(out, key_part, range->end_key.key, > + range->end_key.length); > } > } > > @@ -15775,60 +15763,39 @@ void append_range(String *out, const KEY_PART_INFO *key_part, > > Add ranges to the trace > For ex: > - query: select * from t1 where a=2 ; > - and we have an index on a , so we create a range > - 2 <= a <= 2 > + lets say we have an index a_b(a,b) > + query: select * from t1 where a=2 and b=4 ; > + so we create a range: > + (2,4) <= (a,b) <= (2,4) > this is added to the trace > */ > > static void append_range_all_keyparts(Json_writer_array *range_trace, > - String *range_string, > - String *range_so_far, const SEL_ARG *keypart, > + PARAM param, uint idx, > + SEL_ARG *keypart, > const KEY_PART_INFO *key_parts) > { > - > - DBUG_ASSERT(keypart); > - DBUG_ASSERT(keypart && keypart != &null_element); > - > - // Navigate to first interval in red-black tree > + SEL_ARG_RANGE_SEQ seq; > + KEY_MULTI_RANGE range; > + range_seq_t seq_it; > + uint flags= 0; > + RANGE_SEQ_IF seq_if = {NULL, sel_arg_range_seq_init, > + sel_arg_range_seq_next, 0, 0}; > + KEY *keyinfo= param.table->key_info + param.real_keynr[idx]; > + uint n_key_parts= param.table->actual_n_key_parts(keyinfo); > + seq.keyno= idx; > + seq.real_keyno= param.real_keynr[idx]; > + seq.param= &param; > + seq.start= keypart; > const KEY_PART_INFO *cur_key_part= key_parts + keypart->part; > - const SEL_ARG *keypart_range= keypart->first(); > - const size_t save_range_so_far_length= range_so_far->length(); > - > + seq_it= seq_if.init((void *) &seq, 0, flags); > > - while (keypart_range) > + while (!seq_if.next(seq_it, &range)) > { > - // Append the current range predicate to the range String > - switch (keypart->type) > - { > - case SEL_ARG::Type::KEY_RANGE: > - append_range(range_so_far, cur_key_part, keypart_range->min_value, > - keypart_range->max_value, > - keypart_range->min_flag | keypart_range->max_flag); > - break; > - case SEL_ARG::Type::MAYBE_KEY: > - range_so_far->append("MAYBE_KEY"); > - break; > - case SEL_ARG::Type::IMPOSSIBLE: > - range_so_far->append("IMPOSSIBLE"); > - break; > - default: > - DBUG_ASSERT(false); > - break; > - } > - > - if (keypart_range->next_key_part && > - keypart_range->next_key_part->part == > - keypart_range->part + 1 && > - keypart_range->is_singlepoint()) > - { > - append_range_all_keyparts(range_trace, range_string, range_so_far, > - keypart_range->next_key_part, key_parts); > - } > - else > - range_trace->add(range_so_far->c_ptr_safe(), range_so_far->length()); > - keypart_range= keypart_range->next; > - range_so_far->length(save_range_so_far_length); > + String range_info; > + range_info.set_charset(system_charset_info); > + append_range(&range_info, cur_key_part, &range, n_key_parts); > + range_trace->add(range_info.c_ptr_safe(), range_info.length()); > } > } > > @@ -15838,70 +15805,110 @@ static void append_range_all_keyparts(Json_writer_array *range_trace, > @param[out] out String the key is appended to > @param[in] key_part Index components description > @param[in] key Key tuple > + @param[in] used_length length of the key tuple > */ > + > static void print_key_value(String *out, const KEY_PART_INFO *key_part, > - const uchar *key) > + const uchar* key, uint used_length) > { > + out->append(STRING_WITH_LEN("(")); > Field *field= key_part->field; > + StringBuffer<128> tmp(system_charset_info); > + TABLE *table= field->table; > + uint store_length; > + my_bitmap_map *old_sets[2]; > + dbug_tmp_use_all_columns(table, old_sets, table->read_set, table->write_set); > + const uchar *key_end= key+used_length; > > - if (field->flags & BLOB_FLAG) > + for (; key < key_end; key+=store_length, key_part++) > { > - // Byte 0 of a nullable key is the null-byte. If set, key is NULL. > - if (field->real_maybe_null() && *key) > - out->append(STRING_WITH_LEN("NULL")); > - else > - (field->type() == MYSQL_TYPE_GEOMETRY) > - ? out->append(STRING_WITH_LEN("unprintable_geometry_value")) > - : out->append(STRING_WITH_LEN("unprintable_blob_value")); > - return; > - } > + field= key_part->field; > + store_length= key_part->store_length; > + if (field->flags & BLOB_FLAG) > + { > + // Byte 0 of a nullable key is the null-byte. If set, key is NULL. > + if (field->real_maybe_null() && *key) > + out->append(STRING_WITH_LEN("NULL")); > + else > + (field->type() == MYSQL_TYPE_GEOMETRY) > + ? out->append(STRING_WITH_LEN("unprintable_geometry_value")) > + : out->append(STRING_WITH_LEN("unprintable_blob_value")); > + goto next; > + } > > - uint store_length= key_part->store_length; > + if (field->real_maybe_null()) > + { > + /* > + Byte 0 of key is the null-byte. If set, key is NULL. > + Otherwise, print the key value starting immediately after the > + null-byte > + */ > + if (*key) > + { > + out->append(STRING_WITH_LEN("NULL")); > + goto next; > + } > + key++; // Skip null byte > + store_length--; > + } > > - if (field->real_maybe_null()) > - { > /* > - Byte 0 of key is the null-byte. If set, key is NULL. > - Otherwise, print the key value starting immediately after the > - null-byte > + Binary data cannot be converted to UTF8 which is what the > + optimizer trace expects. If the column is binary, the hex > + representation is printed to the trace instead. > */ > - if (*key) > + if (field->flags & BINARY_FLAG) > { > - out->append(STRING_WITH_LEN("NULL")); > - return; > + out->append("0x"); > + for (uint i = 0; i < store_length; i++) > + { > + out->append(_dig_vec_lower[*(key + i) >> 4]); > + out->append(_dig_vec_lower[*(key + i) & 0x0F]); > + } > + goto next; > } > - key++; // Skip null byte > - store_length--; > + > + field->set_key_image(key, key_part->length); > + if (field->type() == MYSQL_TYPE_BIT) > + (void)field->val_int_as_str(&tmp, 1); // may change tmp's charset > + else > + field->val_str(&tmp); // may change tmp's charset > + out->append(tmp.ptr(), tmp.length(), tmp.charset()); > + > + next: > + if (key + store_length < key_end) > + out->append(STRING_WITH_LEN(",")); > } > + dbug_tmp_restore_column_maps(table->read_set, table->write_set, old_sets); > + out->append(STRING_WITH_LEN(")")); > +} > > - /* > - Binary data cannot be converted to UTF8 which is what the > - optimizer trace expects. If the column is binary, the hex > - representation is printed to the trace instead. > - */ > - if (field->flags & BINARY_FLAG) > +/** > + Print key parts involed in a range > + @param[out] out String the key is appended to > + @param[in] key_part Index components description > + @param[in] n_keypart Number of keyparts in index > + @param[in] keypart_map map for keyparts involved in the range > +*/ > + > +void print_keyparts_name(String *out, const KEY_PART_INFO *key_part, > + uint n_keypart, key_part_map keypart_map) > +{ > + uint i; > + out->append(STRING_WITH_LEN("(")); > + bool first_keypart= TRUE; > + for (i=0; i < n_keypart; key_part++, i++) > { > - out->append("0x"); > - for (uint i = 0; i < store_length; i++) > + if (keypart_map & (1 << i)) > { > - out->append(_dig_vec_lower[*(key + i) >> 4]); > - out->append(_dig_vec_lower[*(key + i) & 0x0F]); > + if (first_keypart) > + first_keypart= FALSE; > + else > + out->append(STRING_WITH_LEN(",")); > + out->append(key_part->field->field_name); > } > - return; > + else > + break; > } > - > - StringBuffer<128> tmp(system_charset_info); > - TABLE *table= field->table; > - my_bitmap_map *old_sets[2]; > - > - dbug_tmp_use_all_columns(table, old_sets, table->read_set, table->write_set); > - > - field->set_key_image(key, key_part->length); > - if (field->type() == MYSQL_TYPE_BIT) > - (void)field->val_int_as_str(&tmp, 1); // may change tmp's charset > - else > - field->val_str(&tmp); // may change tmp's charset > - out->append(tmp.ptr(), tmp.length(), tmp.charset()); > - > - dbug_tmp_restore_column_maps(table->read_set, table->write_set, old_sets); > -} > + out->append(STRING_WITH_LEN(")")); > +} BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] 1581f65be57: MDEV-13301 Optimize DROP INDEX, ADD INDEX into RENAME INDEX
by Sergei Golubchik 02 Apr '19

02 Apr '19

Hi, Eugene! See the review below. This looked pretty much ok, a couple of style comments. And why did you not implement the new ALTER TABLE .. .RENAME INDEX syntax? I agree that a pair of DROP/ADD should act as RENAME. But a dedicated RENAME would be also good to have, at least for compatibility reasons. On Apr 01, Eugene Kosov wrote: > revision-id: 1581f65be57 (mariadb-10.4.3-69-g1581f65be57) > parent(s): e012d266807 > author: Eugene Kosov <claprix(a)yandex.ru> > committer: Eugene Kosov <claprix(a)yandex.ru> > timestamp: 2019-03-15 18:44:05 +0300 > message: > > MDEV-13301 Optimize DROP INDEX, ADD INDEX into RENAME INDEX > > Just rename index in data dictionary and in InnoDB cache when it's possible. > Introduce ALTER_INDEX_RENAME for that purpose so that engines can optimize > such operation. > > Unused code between macro MYSQL_RENAME_INDEX was removed. > > compare_keys_but_name(): compare index definitions except for index names > > Alter_inplace_info::rename_keys: > ha_innobase_inplace_ctx::rename_keys: vector of rename indexes > > fill_alter_inplace_info():: fills Alter_inplace_info::rename_keys > > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index e1337a0f710..e1131422117 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -7062,6 +7061,43 @@ static bool fill_alter_inplace_info(THD *thd, TABLE *table, bool varchar, > new_key->option_struct; > } > > + uint &add_count= ha_alter_info->index_add_count; Coding style: no modifiable references please. write ha_alter_info->index_add_count++ same below > + for (uint i= 0; i < add_count; i++) > + { > + uint *&add_buffer= ha_alter_info->index_add_buffer; > + const KEY *new_key= ha_alter_info->key_info_buffer + add_buffer[i]; > + > + uint &drop_count= ha_alter_info->index_drop_count; > + for (uint j= 0; j < drop_count; j++) > + { > + KEY **&drop_buffer= ha_alter_info->index_drop_buffer; > + const KEY *old_key= drop_buffer[j]; > + > + if (!lex_string_cmp(system_charset_info, &old_key->name, &new_key->name)) > + continue; Why? I'd think and if it's Compare_keys::Equal and the name is the same too, you can simply remove add/drop elements without adding a rename element. > + > + if (Compare_keys::Equal != compare_keys_but_name(old_key, new_key, > + alter_info, table, > + new_pk, old_pk)) Coding style: don't invert comparisons > + { > + continue; > + } > + > + ha_alter_info->handler_flags|= ALTER_RENAME_INDEX; > + ha_alter_info->rename_keys.push_back( > + Alter_inplace_info::Rename_key_pair(old_key, new_key)); > + > + memcpy(add_buffer + i, add_buffer + i + 1, > + sizeof(add_buffer[0]) * (add_count - i - 1)); > + memcpy(drop_buffer + j, drop_buffer + j + 1, > + sizeof(drop_buffer[0]) * (drop_count - j - 1)); > + --add_count; > + --drop_count; > + --i; // this index once again > + break; > + } > + } > + Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 3

Re: [Maria-developers] 126a9b6db25: MDEV-18875 Assertion `thd->transaction.stmt.ha_list == __null || trans == &thd->transaction.stmt' failed or bogus ER_DUP_ENTRY upon ALTER TABLE with versioning
by Sergei Golubchik 02 Apr '19

02 Apr '19

Hi, Nikita! On Apr 02, Nikita Malyavin wrote: > revision-id: 126a9b6db25 (mariadb-10.3.12-101-g126a9b6db25) > parent(s): 1c60f40868f > author: Nikita Malyavin <nikitamalyavin(a)gmail.com> > committer: Nikita Malyavin <nikitamalyavin(a)gmail.com> > timestamp: 2019-03-23 03:37:10 +1000 > message: > > MDEV-18875 Assertion `thd->transaction.stmt.ha_list == __null || trans == &thd->transaction.stmt' failed or bogus ER_DUP_ENTRY upon ALTER TABLE with versioning > > diff --git a/sql/handler.cc b/sql/handler.cc > index e01392d236c..9d295afb631 100644 > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -1420,7 +1420,8 @@ int ha_commit_trans(THD *thd, bool all) > > #if 1 // FIXME: This should be done in ha_prepare(). > if (rw_trans || (thd->lex->sql_command == SQLCOM_ALTER_TABLE && > - thd->lex->alter_info.flags & ALTER_ADD_SYSTEM_VERSIONING)) > + thd->lex->alter_info.flags & ALTER_ADD_SYSTEM_VERSIONING && > + is_real_trans)) Why there's a special check for SQLCOM_ALTER_TABLE + ALTER_ADD_SYSTEM_VERSIONING in the first place? Does it mean that ALTER TABLE ... ADD SYSTEM VERSIONING does not marks a transaction as read/write? That's be weird. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 2

Re: [Maria-developers] 5469d88e7b4: MDEV-19049 Server crashes in check_duplicate_long_entry_key, ASAN stack-buffer-overflow in Field_blob::get_key_image
by Sergei Golubchik 02 Apr '19

02 Apr '19

Hi, Sachin! On Apr 02, Sachin Setiya wrote: > revision-id: 5469d88e7b4 (mariadb-10.4.3-104-g5469d88e7b4) > parent(s): 0bc42602266 > author: Sachin <sachin.setiya(a)mariadb.com> > committer: Sachin <sachin.setiya(a)mariadb.com> > timestamp: 2019-03-28 11:29:25 +0530 > message: > > MDEV-19049 Server crashes in check_duplicate_long_entry_key, ASAN stack-buffer-overflow in Field_blob::get_key_image > > Long Unique keys should always be last unique key. Why? > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index ad62ecc1103..c755a74e174 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -2796,6 +2797,14 @@ static int sort_keys(KEY *a, KEY *b) > { > if (!(b_flags & HA_NOSAME)) > return -1; > + /* > + Long Unique keys should always be last unique key. > + Before this patch they used to change order wrt to partial keys (MDEV-19049) don't write the code history "before this patch it worked like that" in the code itself. When you look at the code there is no "patch", just a comment. The concepts of "patch" and "before it used to be" belongs to the history, and logically should be in the commit comment, not in a code comment. > + */ > + if (a->algorithm == HA_KEY_ALG_LONG_HASH) > + return 1; > + if (b->algorithm == HA_KEY_ALG_LONG_HASH) > + return -1; > if ((a_flags ^ b_flags) & HA_NULL_PART_KEY) > { > /* Sort NOT NULL keys before other keys */ > Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] GSOC 2019 proposal draft
by Алексей Могилёвкин 02 Apr '19

02 Apr '19

Hello. My name is Alexey Mogilyovkin. I would like to contribute to mariadb during GSOC 2019. More information can be found in my draft of proposal on the attached google doc. Any feedback would be appreciated. MariaDB proposal <https://docs.google.com/document/d/12utv0vUoV2FfPMSH2fOR3lrYXT_mS5er_S1XVuM…>

3 2

[Maria-developers] Buildbot image update request
by Otto Kekäläinen 01 Apr '19

01 Apr '19

Hello! Can you please update the image used by test kvm-deb-jessie-ppc64le so that the 'apt-get update' is fresh and package lists are recent? It would fix the error in http://buildbot.askmonty.org/buildbot/builders/kvm-deb-jessie-ppc64le/build… E: Failed to fetch http://ftp.us.debian.org/debian/pool/main/m/mpfr4/libmpfr4_3.1.2-2_ppc64el.… 404 Not Found [IP: 64.50.233.100 80] E: Failed to fetch http://ftp.us.debian.org/debian/pool/main/g/gcc-4.9/libatomic1_4.9.2-10_ppc… 404 Not Found [IP: 64.50.233.100 80] E: Failed to fetch http://ftp.us.debian.org/debian/pool/main/r/rsync/rsync_3.1.1-3_ppc64el.deb 404 Not Found [IP: 64.50.233.100 80] E: Failed to fetch http://ftp.us.debian.org/debian/pool/main/s/socat/socat_1.7.2.4-2_ppc64el.d… 404 Not Found [IP: 64.50.233.100 80] E: Failed to fetch http://ftp.us.debian.org/debian/pool/main/libd/libdbd-mysql-perl/libdbd-mys… 404 Not Found [IP: 64.50.233.100 80] Also, can you please run 'apt-get install libzstd-dev' in the image used in arm64-xenial-deb? It would fix http://buildbot.askmonty.org/buildbot/builders/arm64-xenial-deb/builds/2973… dpkg-checkbuilddeps: error: Unmet build dependencies: libzstd-dev dpkg-buildpackage: warning: build dependencies/conflicts unsatisfied; aborting Thanks!

2 1

Re: [Maria-developers] required mysql_upgrade; MDEV-14637
by Michal Schorm 01 Apr '19

01 Apr '19

On Thu, Mar 28, 2019 at 11:16 AM Michal Schorm <mschorm(a)redhat.com> wrote: > > Hello, > I have a questions regarding the warnings showing up at 10.2 and 10.3 releases: > > When upgrading from MariaDB 10.2.16 or earlier to MariaDB 10.2.17 or higher, > running mysql_upgrade is required due to changes introduced in MDEV-14637. > > When upgrading from MariaDB 10.3.8 or earlier to MariaDB 10.3.9 or higher, > running mysql_upgrade is required due to changes introduced in MDEV-14637. > > -- > > 1) The same applies for upgrading <=10.2.16 to >= 10.3.9, right? > > 2) Does it mean, it is impossible to use "dump and restore" technique, > or it only means I need to run the mysql_upgrade afterwards? > > 3) The problem is not triggered when upgrading from 5.5, 10.0 and 10.1? > 4) The problem is not triggered when upgrading to 10.4 ? I found a reply in the mailing list archive which haven't reached my inbox: From: Reindl Harald Date: Thu, 28 Mar 2019 11:59:50 +0100 > just run "mysql_upgrade" after each update and you are done > our production datadirs date back to 2002, originally on windows then > moved to MacOS and in 2008 to Fedora and i did not dump/restore a single > time in my whole life, this is not postgresql My question is focused on the clarification in our downstream documentation. So while usually I'd be fine with personal experience, in this case, I'd prefer more in-depth answer covering. I'm forwarding this mail to devel mailing list. -- Michal Schorm Software Engineer Core Services - Databases Team Red Hat --

1 0

Re: [Maria-developers] 9976699033c: Test fixes (versioning.replace, versioning.foreign)
by Sergei Golubchik 31 Mar '19

31 Mar '19

Hi, Aleksey! On Mar 27, Aleksey Midenkov wrote: > revision-id: 9976699033c (mariadb-10.3.7-30-g9976699033c) > parent(s): c0f97710582 > author: Aleksey Midenkov <midenok(a)gmail.com> > committer: Aleksey Midenkov <midenok(a)gmail.com> > timestamp: 2018-05-30 13:12:47 +0300 > message: > > Test fixes (versioning.replace, versioning.foreign) > > * replace.test: unwrapped create_table procedure; > * foreign.test: use key_type combinations. > > diff --git a/mysql-test/suite/versioning/key_type.combinations b/mysql-test/suite/versioning/key_type.combinations > index 1929aee9a84..fb3888d3d0f 100644 > --- a/mysql-test/suite/versioning/key_type.combinations > +++ b/mysql-test/suite/versioning/key_type.combinations > @@ -1,2 +1,3 @@ > [unique] > -[pk] > +[primary] > +[secondary] generally prefer shorter combination names, otherwise mtr output becomes garbled and difficult to read. > diff --git a/mysql-test/suite/versioning/primary_or_unique.inc b/mysql-test/suite/versioning/primary_or_unique.inc > new file mode 100644 > index 00000000000..199d8bab942 > --- /dev/null > +++ b/mysql-test/suite/versioning/primary_or_unique.inc > @@ -0,0 +1,5 @@ > +--source suite/versioning/key_type.inc > +--require suite/versioning/primary_or_unique.require don't do .require, please, it's a technique from 1998, before mysqltest had the "skip" command. Use if (...) { skip; } > +disable_query_log; > +eval select "$KEY_TYPE" != "key" as pk_or_unique; > +enable_query_log; > diff --git a/mysql-test/suite/versioning/t/engines.combinations b/mysql-test/suite/versioning/t/engines.combinations > deleted file mode 100644 > index 561c5656929..00000000000 > --- a/mysql-test/suite/versioning/t/engines.combinations > +++ /dev/null > @@ -1,8 +0,0 @@ > -[timestamp] > -default-storage-engine=innodb > - > -[trx_id] > -default-storage-engine=innodb > - > -[myisam] > -default-storage-engine=myisam Is that intentional? I thought many tests use it to run in different versioning modes. > diff --git a/mysql-test/suite/versioning/t/foreign.test b/mysql-test/suite/versioning/t/foreign.test > index 566d481c2a8..72e5b047cbb 100644 > --- a/mysql-test/suite/versioning/t/foreign.test > +++ b/mysql-test/suite/versioning/t/foreign.test > @@ -1,11 +1,14 @@ > +--source suite/versioning/primary_only.inc > --source suite/versioning/common.inc what's the point of this? if you're only allowing primary key here, you can just as well write manually "primary key" in all tests, without $KEY_TYPE and replace_result. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 1

[Maria-developers] EITS member variables in both TABLE and TABLE_SHARE?
by Sergey Petrunia 29 Mar '19

29 Mar '19

Hello Igor, TABLE_SHARE has TABLE_STATISTICS_CB, which has these members: struct TABLE_STATISTICS_CB { ... bool stats_is_read; /* Statistical data for table has been read from statistical tables */ bool histograms_are_read; } At the same time, TABLE has: struct TABLE { ... bool stats_is_read; /* Persistent statistics is read for the table */ bool histograms_are_read; I would like to understand why both structures need these variables? Is there any difference in meaning between them? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] 74b2eba1ca6: MDEV-15458 Segfault in heap_scan() upon UPDATE after ADD SYSTEM VERSIONING
by Sergei Golubchik 29 Mar '19

29 Mar '19

Hi, Aleksey! On Mar 26, Aleksey Midenkov wrote: > revision-id: 74b2eba1ca6 (mariadb-10.3.12-115-g74b2eba1ca6) > parent(s): 2c0901d808b > author: Aleksey Midenkov <midenok(a)gmail.com> > committer: Sergei Golubchik <serg(a)mariadb.com> > timestamp: 2019-03-26 17:43:38 +0100 > message: > > MDEV-15458 Segfault in heap_scan() upon UPDATE after ADD SYSTEM VERSIONING > > * HA_EXTRA_REMEMBER_POS, HA_EXTRA_RESTORE_POS for HEAP storage engine > * Versioning tests support > > diff --git a/sql/sql_insert.cc b/sql/sql_insert.cc > index abe4254db3e..1d51947cc86 100644 > --- a/sql/sql_insert.cc > +++ b/sql/sql_insert.cc > @@ -1643,7 +1643,12 @@ int vers_insert_history_row(TABLE *table) > if (row_start->cmp(row_start->ptr, row_end->ptr) >= 0) > return 0; > > - return table->file->ha_write_row(table->record[0]); > + int res; > + if ((res= table->file->extra(HA_EXTRA_REMEMBER_POS))) > + return res; > + if ((res= table->file->ha_write_row(table->record[0]))) > + return res; > + return table->file->extra(HA_EXTRA_RESTORE_POS); > } Frankly, I don't like it. I mean, this particular fix is fine, but here's the problem: We now have (at least) three features where the server might need to do something (insert/update/search) during the index/rnd scan. And three different solutions: you use HA_EXTRA_REMEMBER_POS in system versioning, Nikita simply sets delete_while_scanning=false in application time period tables, and Sachin uses handler::clone() to create a second handler that can be used to check for hash collisions without disrupting the scan on the primary handler. I think it's getting somewhat out of hands. Can we please reduce the number of approaches to fix the same issue? Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

4 4

Re: [Maria-developers] ab6ef429a84: MDEV-16370 row-based binlog events (updates by primary key) can not be applied multiple times to system versioned tables
by Sergei Golubchik 29 Mar '19

29 Mar '19

Hi, Aleksey! On Jan 31, Aleksey Midenkov wrote: > revision-id: ab6ef429a84 (versioning-1.0.7-2-gab6ef429a84) > parent(s): 14c2f90ad08 > author: Aleksey Midenkov <midenok(a)gmail.com> > committer: Aleksey Midenkov <midenok(a)gmail.com> > timestamp: 2018-11-26 12:46:37 +0300 > message: > > MDEV-16370 row-based binlog events (updates by primary key) can not be applied multiple times to system versioned tables > > Case 2: > > * Idempotent BINLOG updates: duplicate key on history row is skipped; > * Fix row start and warn when applying BINLOG from the past. As far as I can see now you cannot just arbitrarily adjust row_start. Think of it in term or application time period tables, a primary key is (kind of) defined WITHOUT OVERLAPS. There is no explicit constraint, it happens automatically when the history is valid. Now, here's your test case: timestamp=100; insert (1,1) -> the table has (1,1,100,MAX); timestamp=300; update to (1,3) -> (1,1,100,300),(1,3,300,MAX); timestamp=200; update to (1,2) -> here the hisory row should've been (1,3,300,200) but you adjust it to get 1,1,100,300 1,3,199,200 1,2,200,MAX and if you select AS OF 199, you'll get two rows for the pk=1 So, let's * ignore HA_ERR_FOUND_DUPP_KEY (please check if it's needed for insert or delete too) * add TIMESTAMP=xxx to mysqldumps. can vers_insert_history_row see row_start>row_end in a normal use case? no replication or any other way of adjusting the timestamp. equality can surely happen, but greater than ? Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 1

[Maria-developers] THD::set_time()
by Aleksey Midenkov 28 Mar '19

28 Mar '19

What if system_time.sec_part == TIME_MAX_SECOND_PART here: else { start_time= t; start_time_sec_part= ++system_time.sec_part; } ? -- All the best, Aleksey Midenkov @midenok

1 0

[Maria-developers] (no subject)
by Samyak Jain 28 Mar '19

28 Mar '19

I want to contribute for full outer join gsoc project of mariaDB. Please provide the technical requirements for this and how I start contributing on this

1 0

Re: [Maria-developers] c1bf5766aed: MDEV-16490: It's possible to make a system versioned table without any versioning field
by Sergei Golubchik 27 Mar '19

27 Mar '19

Hi, Nikita! On Mar 27, Nikita Malyavin wrote: > revision-id: c1bf5766aed (versioning-1.0.5-41-gc1bf5766aed) > parent(s): 141a5b24843 > author: Nikita Malyavin <nikitamalyavin(a)gmail.com> > committer: Nikita Malyavin <nikitamalyavin(a)gmail.com> > timestamp: 2018-07-20 23:37:26 +1000 > message: > > MDEV-16490: It's possible to make a system versioned table without any versioning field > > * do not allow versioned table to be without versioned (non-system) fields > * prohibit changing field versioning, when removing table versioning > > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index dbf400b53d7..ee4cf457f45 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -8473,11 +8473,21 @@ mysql_prepare_alter_table(THD *thd, TABLE *table, > } > } > > - if (table->versioned() && !(alter_info->flags & ALTER_DROP_SYSTEM_VERSIONING) && > - new_create_list.elements == VERSIONING_FIELDS) > + if (table->versioned()) > { > - my_error(ER_VERS_TABLE_MUST_HAVE_COLUMNS, MYF(0), table->s->table_name.str); > - goto err; > + uint versioned_fields= 0; > + field_it.rewind(); > + while (Create_field *f= field_it++) > + { > + if (!(f->flags & VERS_UPDATE_UNVERSIONED_FLAG)) > + versioned_fields++; > + } > + if (!(alter_info->flags & ALTER_DROP_SYSTEM_VERSIONING) && > + versioned_fields == VERSIONING_FIELDS) > + { > + my_error(ER_VERS_TABLE_MUST_HAVE_COLUMNS, MYF(0), table->s->table_name.str); > + goto err; > + } wouldn't that fit naturally into Vers_parse_info::fix_alter_info() ? Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] e0fa1993692: MDEV-18899: Server crashes in Field::set_warning_truncated_wrong_value
by Sergey Petrunia 27 Mar '19

27 Mar '19

Hi Varun, First, Some input on the test: On Wed, Mar 27, 2019 at 01:55:16PM +0530, Varun wrote: > revision-id: e0fa1993692a3552feddae56cbee3078d4da2121 (mariadb-10.2.22-91-ge0fa1993692) > parent(s): 50a8fc52988d13a5164a1a542b9d7a85e3ecc1c1 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2019-03-27 13:48:44 +0530 > message: > > MDEV-18899: Server crashes in Field::set_warning_truncated_wrong_value > > To fix the crash there we need to make sure that the > server while storing the statistical values in statistical tables should do it > in a multi-byte safe way. > Also there is no need to throw warnings if there is truncation while storing > values from statistical fields. > ... > --- a/mysql-test/r/stat_tables_innodb.result > +++ b/mysql-test/r/stat_tables_innodb.result > @@ -651,6 +651,63 @@ SELECT MAX(pk) FROM t1; > MAX(pk) > NULL > DROP TABLE t1; > +# > +# MDEV-18899: Server crashes in Field::set_warning_truncated_wrong_value > +# > +set names utf8; > +set @save_optimizer_use_condition_selectivity=@@optimizer_use_condition_selectivity; > +set optimizer_use_condition_selectivity=4; > +set use_stat_tables=preferably; > +set @save_histogram_size= @@histogram_size; > +set histogram_size=255; > +create table t1 ( a varchar(255) character set utf8); > +insert into t1 values (REPEAT('ӥ',255)), (REPEAT('ç',255)); > +analyze table t1; > +Table Op Msg_type Msg_text > +test.t1 analyze status Engine-independent statistics collected > +test.t1 analyze status OK > +select HEX(RIGHT(min_value, 1)), length(min_value) from mysql.column_stats; > +HEX(RIGHT(min_value, 1)) length(min_value) > +A7 254 > +select HEX(RIGHT(max_value, 1)), length(max_value) from mysql.column_stats; > +HEX(RIGHT(max_value, 1)) length(max_value) > +A5 254 > +analyze select * from t1 where a >= 'ӥ'; > +id select_type table type possible_keys key key_len ref rows r_rows filtered r_filtered Extra > +1 SIMPLE t1 ALL NULL NULL NULL NULL 2 2.00 50.00 50.00 Using where > +set @save_sql_mode= @@sql_mode; > +set sql_mode='ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION'; > +update mysql.column_stats set min_value= REPEAT('ӥ',255); This looks scary, what if there were some other data in these tables? Can you change add a WHERE db_name=database() and table_name='t1' ? > +Warnings: > +Warning 1265 Data truncated for column 'min_value' at row 1 > +select HEX(RIGHT(min_value, 1)), length(min_value) from mysql.column_stats; > +HEX(RIGHT(min_value, 1)) length(min_value) > +D3 255 > +analyze select * from t1 where a >= 'ӥ'; > +id select_type table type possible_keys key key_len ref rows r_rows filtered r_filtered Extra > +1 SIMPLE t1 ALL NULL NULL NULL NULL 2 2.00 50.00 50.00 Using where > +set names latin1; > +drop table t1; > +CREATE TABLE t1 (col1 date); > +INSERT INTO t1 VALUES('2004-01-01'),('2004-02-29'); > +INSERT INTO t1 VALUES('0000-10-31'); > +analyze table t1; > +Table Op Msg_type Msg_text > +test.t1 analyze status Engine-independent statistics collected > +test.t1 analyze status OK > +update mysql.column_stats set min_value='2004-0-31123'; The same as above. > +select min_value from mysql.column_stats; > +min_value > +2004-0-31123 This case also makes one wonder whether not producing a warning was a good idea after all. A varchar string that is 255-byte long, and after 255 bytes it gets truncated in the middle of a character is a rare case. An incorrect datetime value... not as rare? > +select * from t1; Is the above supposed to read the EITS data? It doesn't as the query doesn't use the WHERE clause. > +col1 > +2004-01-01 > +2004-02-29 > +0000-10-31 > +drop table t1; > +set @@sql_mode= @save_sql_mode; > set use_stat_tables=@save_use_stat_tables; > +set @@histogram_size= @save_histogram_size; > +set @@optimizer_use_condition_selectivity=@save_optimizer_use_condition_selectivity; > set optimizer_switch=@save_optimizer_switch_for_stat_tables_test; > SET SESSION STORAGE_ENGINE=DEFAULT; > diff --git a/mysql-test/t/stat_tables.test b/mysql-test/t/stat_tables.test > index b89ab2bbd2d..114c5c97a7e 100644 > --- a/mysql-test/t/stat_tables.test > +++ b/mysql-test/t/stat_tables.test > @@ -401,4 +401,44 @@ SELECT MAX(pk) FROM t1; > > DROP TABLE t1; > > +--echo # > +--echo # MDEV-18899: Server crashes in Field::set_warning_truncated_wrong_value > +--echo # > + > +set names utf8; > +set @save_optimizer_use_condition_selectivity=@@optimizer_use_condition_selectivity; > +set optimizer_use_condition_selectivity=4; > +set use_stat_tables=preferably; > +set @save_histogram_size= @@histogram_size; > +set histogram_size=255; > + > +create table t1 ( a varchar(255) character set utf8); > +insert into t1 values (REPEAT('ӥ',255)), (REPEAT('ç',255)); > + > +analyze table t1; > +select HEX(RIGHT(min_value, 1)), length(min_value) from mysql.column_stats; > +select HEX(RIGHT(max_value, 1)), length(max_value) from mysql.column_stats; > +analyze select * from t1 where a >= 'ӥ'; > + > +set @save_sql_mode= @@sql_mode; > +set sql_mode='ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION'; > +update mysql.column_stats set min_value= REPEAT('ӥ',255); > +select HEX(RIGHT(min_value, 1)), length(min_value) from mysql.column_stats; > +analyze select * from t1 where a >= 'ӥ'; > + > +set names latin1; > +drop table t1; > + > +CREATE TABLE t1 (col1 date); > +INSERT INTO t1 VALUES('2004-01-01'),('2004-02-29'); > +INSERT INTO t1 VALUES('0000-10-31'); > +analyze table t1; > +update mysql.column_stats set min_value='2004-0-31123'; > +select min_value from mysql.column_stats; > +select * from t1; > +drop table t1; > + > +set @@sql_mode= @save_sql_mode; > set use_stat_tables=@save_use_stat_tables; > +set @@histogram_size= @save_histogram_size; > +set @@optimizer_use_condition_selectivity=@save_optimizer_use_condition_selectivity; > diff --git a/sql/field.cc b/sql/field.cc > index 080cf34c76d..c26195a3650 100644 > --- a/sql/field.cc > +++ b/sql/field.cc > @@ -2219,7 +2219,6 @@ bool Field_str::can_be_substituted_to_equal_item(const Context &ctx, > return false; > } > > - > void Field_num::make_field(Send_field *field) > { > Field::make_field(field); > @@ -7023,12 +7022,16 @@ Field_longstr::check_string_copy_error(const String_copier *copier, > { > const char *pos; > char tmp[32]; > + THD *thd= get_thd(); > > if (!(pos= copier->most_important_error_pos())) > return FALSE; > > - convert_to_printable(tmp, sizeof(tmp), pos, (end - pos), cs, 6); > - set_warning_truncated_wrong_value("string", tmp); > + if (thd->count_cuted_fields) > + { > + convert_to_printable(tmp, sizeof(tmp), pos, (end - pos), cs, 6); > + set_warning_truncated_wrong_value("string", tmp); > + } > return TRUE; > } > > diff --git a/sql/sql_statistics.cc b/sql/sql_statistics.cc > index b5811c683e8..0960a9a1ec5 100644 > --- a/sql/sql_statistics.cc > +++ b/sql/sql_statistics.cc > @@ -1044,6 +1044,7 @@ class Column_stat: public Stat_table > { > char buff[MAX_FIELD_WIDTH]; > String val(buff, sizeof(buff), &my_charset_bin); > + uint32 length= 0; > > for (uint i= COLUMN_STAT_MIN_VALUE; i <= COLUMN_STAT_HISTOGRAM; i++) > { > @@ -1060,7 +1061,9 @@ class Column_stat: public Stat_table > else > { > table_field->collected_stats->min_value->val_str(&val); > - stat_field->store(val.ptr(), val.length(), &my_charset_bin); > + length= Well_formed_prefix(val.charset(), val.ptr(), > + MY_MIN(val.length(), stat_field->field_length)).length(); > + stat_field->store(val.ptr(), length, &my_charset_bin); > } > break; > case COLUMN_STAT_MAX_VALUE: > @@ -1069,7 +1072,9 @@ class Column_stat: public Stat_table > else > { > table_field->collected_stats->max_value->val_str(&val); > - stat_field->store(val.ptr(), val.length(), &my_charset_bin); > + length= Well_formed_prefix(val.charset(), val.ptr(), > + MY_MIN(val.length(), stat_field->field_length)).length(); > + stat_field->store(val.ptr(), length, &my_charset_bin); > } > break; > case COLUMN_STAT_NULLS_RATIO: > @@ -2934,7 +2939,6 @@ int update_statistics_for_table(THD *thd, TABLE *table) > DBUG_RETURN(rc); > } > > - > /** > @brief > Read statistics for a table from the persistent statistical tables > @@ -2980,6 +2984,7 @@ int read_statistics_for_table(THD *thd, TABLE *table, TABLE_LIST *stat_tables) > Table_statistics *read_stats= table_share->stats_cb.table_stats; > > DBUG_ENTER("read_statistics_for_table"); > + DBUG_ASSERT(thd->count_cuted_fields == CHECK_FIELD_IGNORE); > > /* Read statistics from the statistical table table_stats */ > stat_table= stat_tables[TABLE_STAT].table; > @@ -3059,7 +3064,7 @@ int read_statistics_for_table(THD *thd, TABLE *table, TABLE_LIST *stat_tables) > } > } > } > - > + > table->stats_is_read= TRUE; > > DBUG_RETURN(0); BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] 494c0448c87: MDEV-15951 system versioning by trx id doesn't work with partitioning
by Sergei Golubchik 26 Mar '19

26 Mar '19

Hi, Nikita! That's good, thanks! It's always nice to see a fix that removes more code than it adds :) But I'll reformulate the error message a bit, "Transactional system versioned tables" also means InnoDB system versioned by timestamp. I'll clarify (somehow) that it means versioned by trx_id, and push. On Mar 26, Nikita Malyavin wrote: > revision-id: 494c0448c87 (mariadb-10.3.12-65-g494c0448c87) > parent(s): 4946eb7b543 > author: Nikita Malyavin <nikitamalyavin(a)gmail.com> > committer: Nikita Malyavin <nikitamalyavin(a)gmail.com> > timestamp: 2019-02-28 19:29:12 +1000 > message: > > MDEV-15951 system versioning by trx id doesn't work with partitioning > > Fix partitioning for trx_id-versioned tables. > `partition by hash`, `range` and others now work. > `partition by system_time` is forbidden. > Currently we cannot use row_start and row_end in `partition by`, because > insertion of versioned field is done by engine's handler, as well as > row_start/row_end's value set up, which is a transaction id -- so it's > also forbidden. > > The drawback is that it's now impossible to use `partition by key()` > without parameters for such tables, because it references row_start and > row_end implicitly. > > * add handler::vers_can_native() > * drop Table_scope_and_contents_source_st::vers_native() > * drop partition_element::find_engine_flag as unused > * forbid versioning partitioning for trx_id as not supported > * adopt vers tests for trx_id partitioning > * forbid any row_end referencing in `partition by` clauses, > including implicit `by key()` > > diff --git a/sql/share/errmsg-utf8.txt b/sql/share/errmsg-utf8.txt > index 9112795b754..dff12ef0a97 100644 > --- a/sql/share/errmsg-utf8.txt > +++ b/sql/share/errmsg-utf8.txt > @@ -7907,8 +7907,8 @@ ER_UNUSED_24 > ER_VERS_TEMPORARY > eng "TEMPORARY tables do not support system versioning" > > -ER_VERS_NOT_SUPPORTED > - eng "%s is not supported for %s system-versioned tables" > +ER_VERS_TRX_PART_HISTORIC_ROW_NOT_SUPPORTED > + eng "Transactional system versioned tables do not support partitioning by ROW START or ROW END" > ER_INDEX_FILE_FULL > eng "The index file for table '%-.192s' is full" Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] 842ef80b029: MDEV-18464: Port kill_one_trx fixes from 10.4 to 10.1
by Marko Mäkelä 26 Mar '19

26 Mar '19

Hi Jan, On Fri, Mar 22, 2019 at 1:59 PM jan <jan.lindstrom(a)mariadb.com> wrote: > > revision-id: 842ef80b029c975b7f7f7ec25fe55817ec626f02 (mariadb-10.1.38-68-g842ef80b029) Was this pushed to buildbot as well? I did not find the commit ID. First, some general comments: * The functions trx_start_if_not_started_low() and trx_start_for_ddl_low() differ between XtraDB and InnoDB with regard to #ifdef WITH_WSREP. Why? Are the debug assignments truly necessary? I see that the difference was introduced 5 years ago in MDEV-6247. * The files storage/{innobase,xtradb}/trx/trx0roll.cc differ from each other after the change. It looks like the innobase version is better. Please remove the unnecessary lines from xtradb: --- storage/xtradb/trx/trx0roll.cc 2019-03-26 10:20:31.767971537 +0200 +++ storage/innobase/trx/trx0roll.cc 2019-03-26 10:20:31.759971465 +0200 @@ -33,8 +33,6 @@ #include "trx0roll.ic" #endif -#include <mysql/service_wsrep.h> - #include "fsp0fsp.h" #include "mach0data.h" #include "trx0rseg.h" @@ -51,9 +49,6 @@ #include "pars0pars.h" #include "srv0mon.h" #include "trx0sys.h" -#ifdef WITH_WSREP -#include "ha_prototypes.h" -#endif /* WITH_WSREP */ /** This many pages must be undone before a truncate is tried within rollback */ @@ -1075,12 +1070,6 @@ if (trx->update_undo) { trx_undo_truncate_end(trx, trx->update_undo, limit); } - -#ifdef WITH_WSREP_OUT - if (wsrep_on(trx->mysql_thd)) { - trx->lock.was_chosen_as_deadlock_victim = FALSE; - } -#endif /* WITH_WSREP */ } /***********************************************************************//** On to the detailed review: > diff --git a/storage/innobase/include/trx0trx.h b/storage/innobase/include/trx0trx.h > index fe16b8272b8..b83734b154a 100644 > --- a/storage/innobase/include/trx0trx.h > +++ b/storage/innobase/include/trx0trx.h > @@ -1,7 +1,7 @@ > /***************************************************************************** > > Copyright (c) 1996, 2016, Oracle and/or its affiliates. All Rights Reserved. > -Copyright (c) 2015, 2018, MariaDB Corporation. > +Copyright (c) 2015, 2019, MariaDB Corporation. > > This program is free software; you can redistribute it and/or modify it under > the terms of the GNU General Public License as published by the Free Software > @@ -613,6 +613,9 @@ struct trx_lock_t { > transaction as a victim in deadlock > resolution, it sets this to TRUE. > Protected by trx->mutex. */ > + bool was_chosen_as_replication_victim; > + /* replication has marked this > + trx to abort */ > time_t wait_started; /*!< lock wait started at this time, > protected only by lock_sys->mutex */ > > @@ -624,6 +627,12 @@ struct trx_lock_t { > only be modified by the thread that is > serving the running transaction. */ > > +#ifdef WITH_WSREP > + bool was_chosen_as_wsrep_victim; > + /*!< high priority wsrep thread has > + marked this trx to abort */ > +#endif /* WITH_WSREP */ > + > mem_heap_t* lock_heap; /*!< memory heap for trx_locks; > protected by lock_sys->mutex */ > > @@ -695,14 +704,6 @@ lock_rec_convert_impl_to_expl()) will access transactions associated > to other connections. The locks of transactions are protected by > lock_sys->mutex and sometimes by trx->mutex. */ > > -enum trx_abort_t { > - TRX_SERVER_ABORT = 0, > -#ifdef WITH_WSREP > - TRX_WSREP_ABORT, > -#endif > - TRX_REPLICATION_ABORT > -}; > - > struct trx_t{ > ulint magic_n; > > @@ -880,8 +881,6 @@ struct trx_t{ > /*------------------------------*/ > THD* mysql_thd; /*!< MySQL thread handle corresponding > to this trx, or NULL */ > - trx_abort_t abort_type; /*!< Transaction abort type*/ > - > const char* mysql_log_file_name; > /*!< if MySQL binlog is used, this field > contains a pointer to the latest file Why are you replacing trx_t::abort_type with two fields in trx_lock_t that are located far apart from each other? To avoid changing the performance too much due to caching, I would suggest to introduce the two fields in trx_t, replacing the former location of abort_type. Do we really need such long names? Would trx_t::replication_victim and trx_t::wsrep_victim suffice? Please document how concurrent access to the fields is being protected. Apparently for the WSREP field it is both trx->mutex and lock_sys->mutex. > diff --git a/storage/innobase/lock/lock0lock.cc b/storage/innobase/lock/lock0lock.cc > index 3970a559a4c..984ac6cacbc 100644 > --- a/storage/innobase/lock/lock0lock.cc > +++ b/storage/innobase/lock/lock0lock.cc > @@ -4785,9 +4783,8 @@ lock_report_waiters_to_mysql( > innobase_kill_query. We mark this by setting > current_lock_mutex_owner, so we can avoid trying > to recursively take lock_sys->mutex. */ > - w_trx->abort_type = TRX_REPLICATION_ABORT; > + w_trx->lock.was_chosen_as_replication_victim = true; > thd_report_wait_for(mysql_thd, w_trx->mysql_thd); > - w_trx->abort_type = TRX_SERVER_ABORT; > } > ++i; > } Here, we seem to be only holding lock_sys->mutex, not w_trx->mutex. > @@ -7993,6 +7981,37 @@ lock_trx_handle_wait( > return DB_LOCK_WAIT; > } > > +/*********************************************************************//** > +Check whether the transaction has already been rolled back because it > +was selected as a deadlock victim, or if it has to wait then cancel > +the wait lock. > +@return DB_DEADLOCK, DB_LOCK_WAIT or DB_SUCCESS */ > +UNIV_INTERN > +dberr_t > +lock_trx_handle_wait( > +/*=================*/ > + trx_t* trx) /*!< in/out: trx lock state */ > +{ > + bool lock_mutex_taken = false; Do we really need this variable? > +#ifdef WITH_WSREP > + /* We already own mutexes */ > + if (trx->lock.was_chosen_as_wsrep_victim) { > + return lock_trx_handle_wait_low(trx); > + } > +#endif /* WITH_WSREP */ > + if (!trx->lock.was_chosen_as_replication_victim) { > + lock_mutex_enter(); > + lock_mutex_taken = true; > + } > + trx_mutex_enter(trx); > + dberr_t err = lock_trx_handle_wait_low(trx); > + if (lock_mutex_taken) { > + lock_mutex_exit(); > + } > + trx_mutex_exit(trx); > + return err; > +} I would expand the second part of the code and remove the condition before releasing the 2 mutexes: if (!trx->lock.was_chosen_as_replication_victim) { lock_mutex_enter(); trx_mutex_enter(trx); dberr_t err = lock_trx_handle_wait_low(trx); lock_mutex_exit(); trx_mutex_exit(trx); return err; } else { trx_mutex_enter(trx); dberr_t err = lock_trx_handle_wait_low(trx); trx_mutex_exit(trx); return err; } Also, I think that it would be better to declare lock_trx_handle_wait_low() only static but not inline, to reduce unnecessary code expansion. Best regards, Marko

1 0

Re: [Maria-developers] 0fa832dc426: Revert THD::THD(skip_global_sys_var_lock) argument
by Sergei Golubchik 25 Mar '19

25 Mar '19

Hi, Sergey! ok to push On Mar 25, Sergey Vojtovich wrote: > revision-id: 0fa832dc426 (mariadb-10.3.12-97-g0fa832dc426) > parent(s): 6f14489ad31 > author: Sergey Vojtovich <svoj(a)mariadb.org> > committer: Sergey Vojtovich <svoj(a)mariadb.org> > timestamp: 2019-03-21 17:17:26 +0400 > message: > > Revert THD::THD(skip_global_sys_var_lock) argument > > Originally introduced by e972125f1 to avoid harmless wait for > LOCK_global_system_variables in a newly created thread, which creation was > initiated by system variable update. > > At the same time it opens dangerous hole, when system variable update > thread already released LOCK_global_system_variables and ack_receiver > thread haven't yet completed new THD construction. In this case THD > constructor goes completely unprotected. > > Since ack_receiver.stop() waits for the thread to go down, we have to > temporarily release LOCK_global_system_variables so that it doesn't > deadlock with ack_receiver.run(). Unfortunately it breaks atomicity > of rpl_semi_sync_master_enabled updates and makes them not serialized. > > LOCK_rpl_semi_sync_master_enabled was introduced to workaround the above. > TODO: move ack_receiver start/stop into repl_semisync_master > enable_master/disable_master under LOCK_binlog protection? > > Part of MDEV-14984 - regression in connect performance > Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] 6f14489ad31: Adieu find_sys_var_ex()
by Sergei Golubchik 25 Mar '19

25 Mar '19

Hi, Sergey! ok to push On Mar 25, Sergey Vojtovich wrote: > revision-id: 6f14489ad31 (mariadb-10.3.12-96-g6f14489ad31) > parent(s): 48af6851da2 > author: Sergey Vojtovich <svoj(a)mariadb.org> > committer: Sergey Vojtovich <svoj(a)mariadb.org> > timestamp: 2019-03-21 01:32:04 +0400 > message: > > Adieu find_sys_var_ex() > > Only take LOCK_plugin for plugin system variables. > > Reverted optimisation that was originally done for session tracker: it > makes much less sense now. Specifically only if connections would want to > track plugin session variables changes and these changes would actually > happen frequently. If this ever becomes an issue, there're much better > ways to optimise this workload. > > Part of MDEV-14984 - regression in connect performance > Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] 48af6851da2: Make connect speed great again
by Sergei Golubchik 25 Mar '19

25 Mar '19

Hi, Sergey! On Mar 25, Sergey Vojtovich wrote: > revision-id: 48af6851da2 (mariadb-10.3.12-95-g48af6851da2) > parent(s): a83d6ee812b > author: Sergey Vojtovich <svoj(a)mariadb.org> > committer: Sergey Vojtovich <svoj(a)mariadb.org> > timestamp: 2019-03-20 01:58:42 +0400 > message: > > Make connect speed great again > > Rather than parsing session_track_system_variables when thread starts, do > it when first trackable event occurs. > > Benchmarked on a 2socket/20core/40threads Broadwell system using sysbench > connect brencmark @40 threads (with select 1 disabled): > 101379.77 -> 143016.68 CPS, whereas 10.2 is currently at 137766.31 CPS. > > Part of MDEV-14984 - regression in connect performance > > diff --git a/sql/session_tracker.cc b/sql/session_tracker.cc > index 5d98e58046d..c493c790596 100644 > --- a/sql/session_tracker.cc > +++ b/sql/session_tracker.cc > @@ -378,16 +378,9 @@ bool Session_sysvars_tracker::configure() > > bool Session_sysvars_tracker::enable(THD *thd) > { > - LEX_STRING tmp= { session_track_system_variables, > - safe_strlen(session_track_system_variables) }; > orig_list.reinit(); > - if (orig_list.parse_var_list(thd, tmp, true, thd->charset()) == true) > - { > - orig_list.reinit(); > - m_enabled= false; > - return true; > - } > - m_enabled= true; > + m_parsed= false; > + m_enabled= session_track_system_variables && *session_track_system_variables; wouldn't it be simpler not to allow both NULL and "" as valid values for session_track_system_variables ? Either always store the empty string as NULL or as "", why would you want both? > return false; > } > > @@ -530,6 +524,19 @@ void Session_sysvars_tracker::mark_as_changed(THD *thd, > { > sysvar_node_st *node; > sys_var *svar= (sys_var *)var; > + > + if (!m_parsed) > + { > + LEX_STRING tmp= { session_track_system_variables, > + safe_strlen(session_track_system_variables) }; > + if (orig_list.parse_var_list(thd, tmp, true, thd->charset())) > + { > + orig_list.reinit(); > + return; > + } > + m_parsed= true; This is of somewhat dubious value. The default value for the @@session_track_system_variables is DEFAULT("autocommit,character_set_client,character_set_connection," "character_set_results,time_zone"), and these are variables that are typically set by connectors automatically on every new connection. Only a pure C/C will benefit from it, that's what sysbench shows. If you'd tested with java, you, probably, wouldn't get any speedup. > + } > + > /* > Check if the specified system variable is being tracked, if so > mark it as changed and also set the class's m_changed flag. > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index 7194bbd03ef..53203d15f0b 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -7320,8 +7320,13 @@ void THD::set_last_commit_gtid(rpl_gtid &gtid) > #ifndef EMBEDDED_LIBRARY > if (changed_gtid && session_tracker.session_sysvars_tracker.is_enabled()) > { > + THD *orig_thd= current_thd; > + if (orig_thd != this) When can it be true? > + set_current_thd(this); > session_tracker.session_sysvars_tracker. > mark_as_changed(this, (LEX_CSTRING*)Sys_last_gtid_ptr); > + if (orig_thd != this) > + set_current_thd(orig_thd); > } > #endif > } > Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] a83d6ee812b: Safe session_track_system_variables snapshot
by Sergei Golubchik 25 Mar '19

25 Mar '19

Hi, Sergey! Just one question. Why do you need Session_sysvars_tracker::session_track_system_variables if you have THD::system_variables::session_track_system_variables ? On Mar 25, Sergey Vojtovich wrote: > revision-id: a83d6ee812b (mariadb-10.3.12-94-ga83d6ee812b) > parent(s): 3c6f17a130b > author: Sergey Vojtovich <svoj(a)mariadb.org> > committer: Sergey Vojtovich <svoj(a)mariadb.org> > timestamp: 2019-03-19 23:40:08 +0400 > message: > > Safe session_track_system_variables snapshot > > When enabling system variables tracker, make a copy of > global_system_variables.session_track_system_variables > under LOCK_global_system_variables. This protects from concurrent variable > updates and potential freed memory access, as well as from variable > reconstruction (which was previously protected by LOCK_plugin). > > We can also use this copy as a session variable pointer, so that we don't > have to allocate memory and reconstruct it every time it is referenced. > > For this very reason we don't need buffer_length stuff anymore. > > As well as don't we need to take LOCK_plugin early in ::enable(). > Unified ::parse_var_list() to acquire LOCK_plugin unconditionally. > For no apparent reason it wasn't previously acquired for global > variable update. > > No reason to access session_sysvars_tracker via get_tracker(), so access > it directly instead. > > Part of MDEV-14984 - regression in connect performance Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] 3c6f17a130b: Allocate Session_sysvars_tracker statically
by Sergei Golubchik 25 Mar '19

25 Mar '19

Hi, Sergey! Please, see few questions below On Mar 25, Sergey Vojtovich wrote: > revision-id: 3c6f17a130b (mariadb-10.3.12-93-g3c6f17a130b) > parent(s): db9854b7d48 > author: Sergey Vojtovich <svoj(a)mariadb.org> > committer: Sergey Vojtovich <svoj(a)mariadb.org> > timestamp: 2019-03-19 23:40:07 +0400 > message: > > Allocate Session_sysvars_tracker statically > > One less new/delete per connection. > > Removed m_mem_flag since most allocs are thread specific. The only > exception are allocs performed during initialization. > > Removed State_tracker and Session_tracker constructors as they don't make > sense anymore. > > Rather than having session_tracker.deinit(), fixed memory accounting so > that it gets initialized before any other constructor and deinitialized > after all destructors done. Cons: makes THD a few pointers larger :( > > Part of MDEV-14984 - regression in connect performance > > diff --git a/sql/sql_class.h b/sql/sql_class.h > index 69e6f58f854..64d772e5d17 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -2145,13 +2145,34 @@ struct wait_for_commit > > extern "C" void my_message_sql(uint error, const char *str, myf MyFlags); > > + > +/** > + A wrapper around thread_count. > + > + It must be specified as a first base class of THD, so that increment is > + done before any other THD constructors and decrement - after any other THD > + destructors. > + > + Destructor unblocks close_conneciton() if there are no more THD's left. > +*/ > +class THD_count > +{ > + THD *orig_thd; > + THD_count(); > + ~THD_count(); > + friend class THD; > +}; 1. Why a separate class, what's wrong with having it in THD? No downcasts, no risk that someone would try to use it separately. 2. orig_thd looks very suspicious. You preserve the old THD* pointer for the whole duration of this THD lifetime and restore it at the end. How can you know that the "orig_thd" is still valid? Why do you need to do it anyway? > + > + > /** > @class THD > For each client connection we create a separate thread with THD serving as > a thread/connection descriptor > */ > > -class THD :public Statement, > +class THD: public THD_count, /* this must be first */ an assert would be better than "/* must be first */". Like, assert(&orig_thd == this) or something. > + public Statement, > + > /* > This is to track items changed during execution of a prepared > statement/stored procedure. It's created by > @@ -2176,19 +2197,6 @@ class THD :public Statement, > inline bool is_conventional() const > { DBUG_ASSERT(0); return Statement::is_conventional(); } > > - void dec_thread_count(void) > - { > - DBUG_ASSERT(thread_count > 0); > - thread_safe_decrement32(&thread_count); > - signal_thd_deleted(); > - } > - > - > - void inc_thread_count(void) > - { > - thread_safe_increment32(&thread_count); > - } > - > public: > MDL_context mdl_context; > > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index 9c9fa228e2a..afe4123a52c 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -597,8 +597,46 @@ extern "C" void thd_kill_timeout(THD* thd) > thd->awake(KILL_TIMEOUT); > } > > -THD::THD(my_thread_id id, bool is_wsrep_applier, bool skip_global_sys_var_lock) > - :Statement(&main_lex, &main_mem_root, STMT_CONVENTIONAL_EXECUTION, > + > +THD_count::THD_count() > +{ > + THD *thd= static_cast<THD*>(this); > + thread_safe_increment32(&thread_count); > + /* > + We set THR_THD to temporally point to this THD to register all the > + variables that allocates memory for this THD > + */ > + orig_thd= current_thd; > + set_current_thd(thd); > + thd->status_var.local_memory_used= sizeof(THD); > + thd->status_var.max_local_memory_used= thd->status_var.local_memory_used; > + thd->status_var.global_memory_used= 0; > + thd->variables.max_mem_used= global_system_variables.max_mem_used; > +} > + > + > +THD_count::~THD_count() > +{ > + THD *thd= static_cast<THD*>(this); > + if (thd->status_var.local_memory_used != 0) > + { > + DBUG_PRINT("error", ("memory_used: %lld", > + thd->status_var.local_memory_used)); > + SAFEMALLOC_REPORT_MEMORY(thd->thread_id); > + DBUG_ASSERT(thd->status_var.local_memory_used == 0 || > + !debug_assert_on_not_freed_memory); > + } > + update_global_memory_status(thd->status_var.global_memory_used); > + set_current_thd(orig_thd); > + DBUG_ASSERT(thread_count > 0); > + thread_safe_decrement32(&thread_count); > + signal_thd_deleted(); > +} > + > + > +THD::THD(my_thread_id id, bool is_wsrep_applier, bool skip_global_sys_var_lock): > + THD_count(), > + Statement(&main_lex, &main_mem_root, STMT_CONVENTIONAL_EXECUTION, > /* statement id */ 0), > rli_fake(0), rgi_fake(0), rgi_slave(NULL), > protocol_text(this), protocol_binary(this), > @@ -654,15 +692,6 @@ THD::THD(my_thread_id id, bool is_wsrep_applier, bool skip_global_sys_var_lock) > ulong tmp; > bzero(&variables, sizeof(variables)); > > - /* > - We set THR_THD to temporally point to this THD to register all the > - variables that allocates memory for this THD > - */ > - THD *old_THR_THD= current_thd; > - set_current_thd(this); > - status_var.local_memory_used= sizeof(THD); > - status_var.max_local_memory_used= status_var.local_memory_used; > - status_var.global_memory_used= 0; > variables.pseudo_thread_id= thread_id; > variables.max_mem_used= global_system_variables.max_mem_used; > main_da.init(); > @@ -849,8 +878,7 @@ THD::THD(my_thread_id id, bool is_wsrep_applier, bool skip_global_sys_var_lock) > save_prep_leaf_list= FALSE; > org_charset= 0; > /* Restore THR_THD */ > - set_current_thd(old_THR_THD); > - inc_thread_count(); > + set_current_thd(orig_thd); > } > > > @@ -1589,7 +1617,6 @@ void THD::reset_for_reuse() > > THD::~THD() > { > - THD *orig_thd= current_thd; > THD_CHECK_SENTRY(this); > DBUG_ENTER("~THD()"); > /* Check that we have already called thd->unlink() */ > @@ -1601,7 +1628,11 @@ THD::~THD() > In error cases, thd may not be current thd. We have to fix this so > that memory allocation counting is done correctly > */ > - set_current_thd(this); > + orig_thd= current_thd; > + if (orig_thd == this) > + orig_thd= 0; > + else > + set_current_thd(this); > if (!status_in_global) > add_status_to_global(); > > @@ -1655,22 +1686,6 @@ THD::~THD() > lf_hash_put_pins(xid_hash_pins); > /* Ensure everything is freed */ > status_var.local_memory_used-= sizeof(THD); > - > - /* trick to make happy memory accounting system */ > -#ifndef EMBEDDED_LIBRARY > - session_tracker.deinit(); > -#endif //EMBEDDED_LIBRARY > - > - if (status_var.local_memory_used != 0) > - { > - DBUG_PRINT("error", ("memory_used: %lld", status_var.local_memory_used)); > - SAFEMALLOC_REPORT_MEMORY(thread_id); > - DBUG_ASSERT(status_var.local_memory_used == 0 || > - !debug_assert_on_not_freed_memory); > - } > - update_global_memory_status(status_var.global_memory_used); > - set_current_thd(orig_thd == this ? 0 : orig_thd); > - dec_thread_count(); > DBUG_VOID_RETURN; > } Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] 72e4e57a52a: Cleanup session tracker API
by Sergei Golubchik 25 Mar '19

25 Mar '19

Hi, Sergey! ok to push On Mar 25, Sergey Vojtovich wrote: > revision-id: 72e4e57a52a (mariadb-10.3.12-88-g72e4e57a52a) > parent(s): 68abb07298b > author: Sergey Vojtovich <svoj(a)mariadb.org> > committer: Sergey Vojtovich <svoj(a)mariadb.org> > timestamp: 2019-03-19 17:11:16 +0400 > message: > > Cleanup session tracker API > > - Session_sysvars_tracker::server_init_check() -> sysvartrack_validate_value() > - Session_sysvars_tracker::check_var_list() -> sysvartrack_validate_value() > > - Session_sysvars_tracker::server_init_process() -> sysvartrack_global_update() > - sysvartrack_reprint_value() -> sysvartrack_global_update() > > - sysvartrack_value_len() -> sysvartrack_session_value_ptr() > - sysvartrack_value_construct() -> sysvartrack_session_value_ptr() > > - sysvartrack_update() -> Session_sysvars_tracker::update() > - Session_tracker::server_boot_verify() -> session_tracker_init() > - sysvar_tracker() -> /dev/null > > Part of MDEV-14984 - regression in connect performance Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] 62d62228aa6: Static current schema and state change trackers
by Sergei Golubchik 25 Mar '19

25 Mar '19

Hi, Sergey! Why two trackers are static and the other two aren't? Please, explain it in a comment. On Mar 25, Sergey Vojtovich wrote: > revision-id: 62d62228aa6 (mariadb-10.3.12-91-g62d62228aa6) > parent(s): 61b9bffbedc > author: Sergey Vojtovich <svoj(a)mariadb.org> > committer: Sergey Vojtovich <svoj(a)mariadb.org> > timestamp: 2019-03-19 23:40:07 +0400 > message: > > Static current schema and state change trackers > > Saves 2 new/delete per connection. > > Part of MDEV-14984 - regression in connect performance Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 1

Re: [Maria-developers] db9854b7d48: Allocate Transaction_state_tracker statically
by Sergei Golubchik 25 Mar '19

25 Mar '19

Hi, Sergey! ok to push On Mar 25, Sergey Vojtovich wrote: > revision-id: db9854b7d48 (mariadb-10.3.12-92-gdb9854b7d48) > parent(s): 62d62228aa6 > author: Sergey Vojtovich <svoj(a)mariadb.org> > committer: Sergey Vojtovich <svoj(a)mariadb.org> > timestamp: 2019-03-19 23:40:07 +0400 > message: > > Allocate Transaction_state_tracker statically > > One less new/delete per connection. > > Part of MDEV-14984 - regression in connect performance > Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] 61b9bffbedc: Allocate orig_list statically
by Sergei Golubchik 25 Mar '19

25 Mar '19

Hi, Sergey! ok to push On Mar 25, Sergey Vojtovich wrote: > revision-id: 61b9bffbedc (mariadb-10.3.12-90-g61b9bffbedc) > parent(s): 9dd4fcb3f7c > author: Sergey Vojtovich <svoj(a)mariadb.org> > committer: Sergey Vojtovich <svoj(a)mariadb.org> > timestamp: 2019-03-19 23:39:19 +0400 > message: > > Allocate orig_list statically > > tool_list is a temporary list needed only for SET SESSION > session_track_system_variables, so allocate it on stack instead. > > Sane reinit() calls. > > Saves 2 new/delete per connection. > > Part of MDEV-14984 - regression in connect performance > Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] 9dd4fcb3f7c: Get rid of not implemented SESSION_GTIDS_TRACKER
by Sergei Golubchik 25 Mar '19

25 Mar '19

Hi, Sergey! ok to push On Mar 25, Sergey Vojtovich wrote: > revision-id: 9dd4fcb3f7c (mariadb-10.3.12-89-g9dd4fcb3f7c) > parent(s): 72e4e57a52a > author: Sergey Vojtovich <svoj(a)mariadb.org> > committer: Sergey Vojtovich <svoj(a)mariadb.org> > timestamp: 2019-03-19 17:28:09 +0400 > message: > > Get rid of not implemented SESSION_GTIDS_TRACKER > > One less new/delete per connection. > > Part of MDEV-14984 - regression in connect performance > Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] 4c0eb412ff5: A proper State_tracker::m_changed enacpsulation
by Sergei Golubchik 25 Mar '19

25 Mar '19

Hi, Sergey! ok to push On Mar 25, Sergey Vojtovich wrote: > revision-id: 4c0eb412ff5 (mariadb-10.3.12-85-g4c0eb412ff5) > parent(s): 69b33fca8cc > author: Sergey Vojtovich <svoj(a)mariadb.org> > committer: Sergey Vojtovich <svoj(a)mariadb.org> > timestamp: 2019-03-18 22:51:45 +0400 > message: > > A proper State_tracker::m_changed enacpsulation > > Note: preserved original behaviour, where remaining trackers are not reset > on error from store(). This effectively means subsequent statements will > start tracking from unclean state. > > Part of MDEV-14984 - regression in connect performance > Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] 68abb07298b: Cleanup session tracker redundnacy
by Sergei Golubchik 25 Mar '19

25 Mar '19

Hi, Sergey! ok to push On Mar 25, Sergey Vojtovich wrote: > revision-id: 68abb07298b (mariadb-10.3.12-87-g68abb07298b) > parent(s): facc973189a > author: Sergey Vojtovich <svoj(a)mariadb.org> > committer: Sergey Vojtovich <svoj(a)mariadb.org> > timestamp: 2019-03-18 22:51:58 +0400 > message: > > Cleanup session tracker redundnacy > > - m_enabled is initialised by the base class constructor > - removed unused schema_track_inited > - moved Transaction_state_tracker constructor to declaration > - common enable() > - removed unused Session_sysvars_tracker::check_str() > - classes are "private" by default > - don't even try to compile for embedded > > Part of MDEV-14984 - regression in connect performance > Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] facc973189a: Session_sysvars_tracker::vars_list cleanups
by Sergei Golubchik 25 Mar '19

25 Mar '19

Hi, Sergey! On Mar 25, Sergey Vojtovich wrote: > revision-id: facc973189a (mariadb-10.3.12-86-gfacc973189a) > parent(s): 4c0eb412ff5 > author: Sergey Vojtovich <svoj(a)mariadb.org> > committer: Sergey Vojtovich <svoj(a)mariadb.org> > timestamp: 2019-03-18 22:51:58 +0400 > message: > > Session_sysvars_tracker::vars_list cleanups > > - return proper type > - removed useless node argument > - removed useless mem_flag > - classes are "private" by default > - simplified iterators > > Part of MDEV-14984 - regression in connect performance ok to push Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] MDEV-18734 ASAN heap-use-after-free in my_strnxfrm_simple_internal upon update on versioned partitioned table
by Aleksey Midenkov 25 Mar '19

25 Mar '19

Hi, Sergei! ha_partition::handle_ordered_index_scan() stores records in m_ordered_rec_buffer. Then TABLE::update_virtual_fields() updates blob buffer and frees the old one. Then ha_partition::return_top_record() returns record from m_ordered_rec_buffer with stale blob pointer. What should we do with this? I propose to duplicate blob buffer when record gets into m_ordered_rec_buffer. More details can be found here: https://github.com/tempesta-tech/mariadb/issues/581 -- All the best, Aleksey Midenkov @midenok

2 4

[Maria-developers] question about running latest build.
by Luuk 24 Mar '19

24 Mar '19

After noticing that a bug i reported was set to 'solved' i wanted to see/check that. So, i downloaded/compiled Now i seem to be able to run the server, but i am unable to connect using 'mysql.exe' Luuk@ZES MINGW64 /d/TEMP/server (master) $ bld/client/RelWithDebInfo/mysqladmin -h localhost -u root version D:\TEMP\server\bld\client\RelWithDebInfo\mysqladmin.exe Ver 9.1 Distrib 10.4.4-MariaDB, for Win32 on AMD64 Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Server version 10.4.4-MariaDB Protocol version 10 Connection localhost via TCP/IP TCP port 3306 Uptime: 23 min 15 sec Threads: 7 Questions: 4 Slow queries: 0 Opens: 16 Flush tables: 1 Open tables: 10 Queries per second avg: 0.002 Luuk@ZES MINGW64 /d/TEMP/server (master) $ bld/client/RelWithDebInfo/mysql -h localhost -u root #### no reaction, killin 'mysql.exe' just add another 'Aborted connection' to the server-log.... QUESTION: What am i missing here? The log from the server (Windows 10): Luuk@ZES MINGW64 /d/TEMP/server/bld (master) $ sql/RelWithDebInfo/mysqld.exe --console 2019-03-24 16:20:01 0 [Note] D:\TEMP\server\bld\sql\RelWithDebInfo\mysqld.exe (mysqld 10.4.4-MariaDB) starting as process 2024 ... 2019-03-24 16:20:01 0 [Note] InnoDB: Mutexes and rw_locks use Windows interlocked functions 2019-03-24 16:20:01 0 [Note] InnoDB: Uses event mutexes 2019-03-24 16:20:01 0 [Note] InnoDB: Compressed tables use zlib 1.2.11 2019-03-24 16:20:01 0 [Note] InnoDB: Number of pools: 1 2019-03-24 16:20:01 0 [Note] InnoDB: Using SSE2 crc32 instructions 2019-03-24 16:20:01 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M 2019-03-24 16:20:01 0 [Note] InnoDB: Completed initialization of buffer pool 2019-03-24 16:20:01 0 [Note] InnoDB: 128 out of 128 rollback segments are active. 2019-03-24 16:20:01 0 [Note] InnoDB: Creating shared tablespace for temporary tables 2019-03-24 16:20:01 0 [Note] InnoDB: Setting file '.\ibtmp1' size to 12 MB. Physically writing the file full; Please wait ... 2019-03-24 16:20:01 0 [Note] InnoDB: File '.\ibtmp1' size is now 12 MB. 2019-03-24 16:20:01 0 [Note] InnoDB: Waiting for purge to start 2019-03-24 16:20:01 0 [Note] InnoDB: 10.4.4 started; log sequence number 139881; transaction id 21 2019-03-24 16:20:01 0 [Note] InnoDB: Loading buffer pool(s) from D:\TEMP\server\bld\sql\data\ib_buffer_pool 2019-03-24 16:20:01 0 [Note] Plugin 'FEEDBACK' is disabled. 2019-03-24 16:20:01 0 [Note] InnoDB: Buffer pool(s) load completed at 190324 16:20:01 2019-03-24 16:20:01 0 [Note] Server socket created on IP: '::'. 2019-03-24 16:20:01 0 [Note] Reading of all Master_info entries succeded 2019-03-24 16:20:01 0 [Note] Added new Master_info '' to hash table 2019-03-24 16:20:01 0 [Note] D:\TEMP\server\bld\sql\RelWithDebInfo\mysqld.exe: ready for connections. Version: '10.4.4-MariaDB' socket: '' port: 3306 Source distribution 2019-03-24 16:21:24 8 [Warning] Aborted connection 8 to db: 'unconnected' user: 'root' host: 'localhost' (Got an error reading communication packets) 2019-03-24 16:21:49 9 [Warning] Aborted connection 9 to db: 'unconnected' user: 'root' host: 'localhost' (Got an error reading communication packets)

1 0

Re: [Maria-developers] d260871fccc: MDEV-18967 Load data in system version with long unique does not work
by Sergei Golubchik 20 Mar '19

20 Mar '19

Hi, Sachin! ok to push! On Mar 20, Sachin Setiya wrote: > revision-id: d260871fccc (mariadb-10.4.3-87-gd260871fccc) > parent(s): f6166101e0f > author: Sachin <sachin.setiya(a)mariadb.com> > committer: Sachin <sachin.setiya(a)mariadb.com> > timestamp: 2019-03-19 16:43:43 +0530 > message: > > MDEV-18967 Load data in system version with long unique does not work > > Update system versioning fields before generaled columns for left out > fill_record > Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] f6166101e0f: MDEV-18904 Assertion `m_part_spec.start_part >= m_part_spec.end_part' failed in ha_partition::index_read_idx_map
by Sergei Golubchik 20 Mar '19

20 Mar '19

Hi, Sachin! On Mar 20, Sachin Setiya wrote: > revision-id: f6166101e0f (mariadb-10.4.3-86-gf6166101e0f) > parent(s): 8076c594ce2 > author: sachinsetia1001(a)gmail.com <sachin.setiya(a)mariadb.com> > committer: Sachin <sachin.setiya(a)mariadb.com> > timestamp: 2019-03-19 16:43:43 +0530 > message: > > MDEV-18904 Assertion `m_part_spec.start_part >= m_part_spec.end_part' failed in ha_partition::index_read_idx_map > > We do not support long unique index in partition. Why not? I would think that the assert is wrong, there is no reason why ha_partition::index_read_idx_map() would always hit just one partition. ha_partition::index_read_map() doesn't have this assert. And if you check the history of the assert you'll find that the comment originally was /* How can it be more than one partition with the current use? */ So, apparently now you've introduced a new "use" and there can be more than one partition, and removing the assert looks justified. > +CREATE TABLE t1 (a int , unique(a) using hash) PARTITION BY HASH (a) PARTITIONS 2; > +ERROR HY000: Long unique index is not supported in partition Also - for the future - there are no "long unique" indexes from the user point of view. This is only an internal name. The SQL syntax does use "long unique" concept, documentation doesn't, error messages don't. That is, users simply won't understand what you mean here. If we'll need an error here, I'd suggest, for example, HASH UNIQUE indexes cannot be used in partitioned tables. but it's not strictly correct, if some engine would implement hash indexes natively. The more precise message could be HASH UNIQUE indexes cannot be used in par titioned MyISAM tables. or Partitioned MyISAM tables do not support HASH UNIQUE indexes. (where "MyISAM" is a parameter, of course). But, again, I am not yet convinced that we need this limitation at all. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] 500d909: MDEV-18741: Optimizer trace: multi-part key ranges are printed incorrectly
by Sergey Petrunia 18 Mar '19

18 Mar '19

> revision-id: 500d909a795022255fb6f6c44cd310636afc0b5a (mariadb-10.3.6-226-g500d909) > parent(s): 88d89ee0bae24b71416c2af4f4c2f2be7b6a033a > author: Varun Gupta > committer: Varun Gupta > timestamp: 2019-03-11 20:20:35 +0530 > message: > > MDEV-18741: Optimizer trace: multi-part key ranges are printed incorrectly > > Keys with multi-key parts were not being printed correctly, only the first key part > was getting printed. > Fixed it by making sure append_range_all_keyparts function is called for the remaining keyparts. > ... > diff --git a/mysql-test/main/opt_trace.result b/mysql-test/main/opt_trace.result > index 4c3e2b3..a4c2b86 100644 > --- a/mysql-test/main/opt_trace.result > +++ b/mysql-test/main/opt_trace.result > @@ -6022,3 +6022,428 @@ t_outer_2.a in (select t_inner_3.a from t2 t_inner_3, t1 t_inner_4) { > set @@optimizer_switch= @save_optimizer_switch; > drop table t1,t2; > set optimizer_trace='enabled=off'; > +# > +# MDEV-18741: Optimizer trace: multi-part key ranges are printed incorrectly > +# > +create table t0(a int); > +insert into t0 values (0),(1),(2),(3),(4),(5),(6),(7),(8),(9); > +create table one_k (a int); > +insert into one_k select A.a + B.a*10 + C.a*100 from t0 A, t0 B, t0 C; > +create table t1 ( a int, b int, c int, d int, key (a,b), key(c,d)); > +insert into t1 select a,a,a,a from one_k; > +set optimizer_trace=1; > +explain format=json select * from t1 where a > 10 and b < 10 and c=0 and d=1; ... > +select * from INFORMATION_SCHEMA.OPTIMIZER_TRACe; This produces a lot of irrelevant output. Can you use something like select JSON_DETAILED(JSON_EXTRACT(a, '$**.range_analysis')) from optimizer_trace to select on the part of the trace that we need? > + "analyzing_range_alternatives": { > + "range_scan_alternatives": [ > + { > + "index": "a", > + "ranges": ["10 < a AND NULL < b < 10"], This is not a valid range. If one forces this quick select to be picked for the query plan, they can look into .trace and see the ranges: explain format=json select * from t1 force index (a) where a > 10 and b < 10; T@19 : | | | | | | | | | | >print_quick quick range select, key a, length: 5 10 < X other_keys: 0x0: T@19 : | | | | | | | | | | <print_quick (you can also try with a >=10 to see a different kind of range being produced). (One could argue that what is printed is a representation of the SEL_ARG graph, but in that case it is extremely confusing. I think, we should start with showing ranges, not SEL_ARG graphs). I think, we should not try to intrepret the SEL_ARG graph ourselves but rather use sel_arg_range_seq_init / sel_arg_range_seq_next to produce ranges that can be printed. Any objections to this? > + "rowid_ordered": false, > + "using_mrr": false, > + "index_only": false, > + "rows": 989, > + "cost": 1272.8, > + "chosen": false, > + "cause": "cost" > + }, > + { > + "index": "c", > + "ranges": ["0 <= c <= 0 AND 1 <= d <= 1"], > + "rowid_ordered": true, > + "using_mrr": false, > + "index_only": false, > + "rows": 1, > + "cost": 2.3783, > + "chosen": true > + } > + ], BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] ASAN segfaults on vcol case
by Aleksey Midenkov 18 Mar '19

18 Mar '19

This case segfaults in ASAN code: delimiter ~~ create or replace procedure t1() begin create or replace table t1 ( b tinytext, v text as (b) virtual ); end~~ delimiter ; call t1; Both in 10.3 and 10.4 in different places. Does someone know about such ASAN faults? -- All the best, Aleksey Midenkov @midenok

2 2

Re: [Maria-developers] 21a15ea673c: MDEV-16240: Assertion `0' failed in row_sel_convert_mysql_key_to_innobase
by Sergei Golubchik 18 Mar '19

18 Mar '19

Hi, Oleksandr! On Mar 18, Oleksandr Byelkin wrote: > revision-id: 21a15ea673c (mariadb-10.2.22-72-g21a15ea673c) > parent(s): bb8c82c66ab > author: Oleksandr Byelkin <sanja(a)mariadb.com> > committer: Oleksandr Byelkin <sanja(a)mariadb.com> > timestamp: 2019-03-14 15:12:47 +0100 > message: > > MDEV-16240: Assertion `0' failed in row_sel_convert_mysql_key_to_innobase > > Set table in row ID position mode before using this function. > > diff --git a/mysql-test/r/multi_update_innodb.result b/mysql-test/r/multi_update_innodb.result > index 5890fd24f5f..294ebfcebdf 100644 > --- a/mysql-test/r/multi_update_innodb.result > +++ b/mysql-test/r/multi_update_innodb.result > @@ -151,3 +151,43 @@ create table t2 like t1; > insert into t2 select * from t1; > delete t1,t2 from t2,t1 where t1.a<'B' and t2.b=t1.b; > drop table t1,t2; > +# > +# MDEV-16240: Assertion `0' failed in > +# row_sel_convert_mysql_key_to_innobase > +# > +SET @save_sql_mode=@@sql_mode; > +set sql_mode=''; > +CREATE TABLE `t3` ( > +`f1` TIMESTAMP NOT NULL DEFAULT '0000-00-00 00:00:00' ON UPDATE current_timestamp(), > +`f2` datetime DEFAULT '2000-01-01 00:00:00' ON UPDATE current_timestamp(), > +`f3` TIMESTAMP NULL DEFAULT '2000-01-01 00:00:00', > +`pk` datetime NOT NULL DEFAULT '0000-00-00 00:00:00', > +`f4` datetime DEFAULT current_timestamp(), > +PRIMARY KEY (`pk`), > +UNIQUE KEY `f2k` (`f2`), > +KEY `f4k` (`f4`) > +) ENGINE=InnoDB; > +INSERT INTO `t3` VALUES ('2018-05-18 15:08:07','2018-05-18 17:08:07','0000-00-00 00:00:00','0000-00-00 00:00:00','0000-00-00 00:00:00'),('0000-00-00 00:00:00','0000-00-00 00:00:00','1999-12-31 23:00:00','2002-07-03 23:04:40','0000-00-00 00:00:00'); > +CREATE VIEW `v1` AS > +SELECT `t3`.`pk` AS `pk`, > +`t3`.`f3` AS `f3`, > +`t3`.`f4` AS `f4`, > +`t3`.`f2` AS `f2`, > +`t3`.`f1` AS `f1` > +FROM `t3`; > +CREATE TABLE `t4` ( > +`f1` datetime DEFAULT current_timestamp() ON UPDATE current_timestamp(), > +`f3` timestamp NULL DEFAULT NULL, > +`f2` timestamp NULL DEFAULT '1999-12-31 23:00:00' ON UPDATE current_timestamp(), > +`pk` int(11) NOT NULL, > +`f4` timestamp NOT NULL DEFAULT current_timestamp() ON UPDATE current_timestamp(), > +PRIMARY KEY (`pk`) > +) ENGINE=InnoDB; > +INSERT INTO `t4` VALUES ('2018-05-18 17:08:06','0000-00-00 00:00:00',NULL,1,'2018-05-18 15:08:06'),('2018-05-18 17:08:06','0000-00-00 00:00:00',NULL,2,'2018-05-18 15:08:06'),('2018-05-18 17:08:06','0000-00-00 00:00:00',NULL,3,'2018-05-18 15:08:06'),('0000-00-00 00:00:00','0000-00-00 00:00:00',NULL,1976,'0000-00-00 00:00:00'),('2018-05-18 17:08:06','0000-00-00 00:00:00',NULL,2000,'2018-05-18 15:08:06'),('2018-05-18 17:08:06','0000-00-00 00:00:00',NULL,2001,'2018-05-18 15:08:06'),('2018-05-18 17:08:06','0000-00-00 00:00:00',NULL,2002,'2018-05-18 15:08:06'),('2018-05-18 17:08:06','0000-00-00 00:00:00',NULL,2003,'2018-05-18 15:08:06'),('2018-05-18 17:08:06','0000-00-00 00:00:00',NULL,2004,'2018-05-18 15:08:06'),('2018-05-18 17:08:06','0000-00-00 00:00:00','2018-05-18 15:08:06',2005,'2018-05-18 15:08:06'),('2018-05-18 17:08:06','0000-00-00 00:00:00','2018-05-18 15:08:06',2018,'2018-05-18 15:08:06'),('2018-05-18 17:08:06','0000-00-00 00:00:00','2018-05-18 15:08:06',2019,'2018-05-18 15:08:06'),('2018-05-18 17:08:06','0000-00-00 00:00:00','2018-05-18 15:08:06',2024,'2018-05-18 15:08:06'),('2018-05-18 17:08:06','0000-00-00 00:00:00','1999-12-31 23:00:00',2025,'2018-05-18 15:08:06'),('0000-00-00 00:00:00',NULL,'2018-05-18 15:08:06',2026,'2018-05-18 15:08:06'),('2018-05-18 17:08:07','0000-00-00 00:00:00','0000-00-00 00:00:00',2027,'0000-00-00 00:00:00'); > +UPDATE `v1` t1, `t4` t2 > +SET t1.`f2` = 6452736 WHERE t1.`f4` = 6272000; > +ERROR 23000: Duplicate entry '0000-00-00 00:00:00' for key 'f2k' actually, when I wrote "please, rename all columns in the test to have unique names" I meant _all_ columns, that is, in the t3 columnd could've been f1-f5, in v1 they could be called f6-f10, in t4 - f11-f15. but don't bother renaming them now, apparently the most important name was 'f2k' and it's unique now, so it's sufficiently clear, thanks. Still, for the future, it's generally helpful to have all names different to be able to understand what the error message refers to. > +DROP VIEW v1; > +DROP TABLE t3,t4; > +SET @@sql_mode=@save_sql_mode; > +# End of 10.2 tests > diff --git a/sql/handler.cc b/sql/handler.cc > index ae63640ae5c..d307433cf5d 100644 > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -2648,8 +2648,7 @@ int handler::ha_rnd_pos(uchar *buf, uchar *pos) > DBUG_ENTER("handler::ha_rnd_pos"); > DBUG_ASSERT(table_share->tmp_table != NO_TMP_TABLE || > m_lock_type != F_UNLCK); > - /* TODO: Find out how to solve ha_rnd_pos when finding duplicate update. */ > - /* DBUG_ASSERT(inited == RND); */ > + DBUG_ASSERT(inited == RND); > > TABLE_IO_WAIT(tracker, m_psi, PSI_TABLE_FETCH_ROW, MAX_KEY, 0, > { result= rnd_pos(buf, pos); }) > @@ -3307,6 +3306,10 @@ void handler::get_auto_increment(ulonglong offset, ulonglong increment, > ulonglong nr; > int error; > MY_BITMAP *old_read_set; > + bool rnd_inited= (inited == RND); > + when is that true? > diff --git a/sql/item_subselect.cc b/sql/item_subselect.cc > index 8a9dd083911..cc1649b3c92 100644 > --- a/sql/item_subselect.cc > +++ b/sql/item_subselect.cc > @@ -5855,12 +5855,17 @@ Ordered_key::cmp_keys_by_row_data_and_rownum(Ordered_key *key, > } > > > -void Ordered_key::sort_keys() > +bool Ordered_key::sort_keys() > { > + /* Prepare table for random positioning (importent for innodb) */ 1) typo: "important" but 2) I don't think you need a comment here at all now. There's an assert in ha_rnd_pos(), it's rather more clear and more reliable anyway. > + if (tbl->file->ha_rnd_init(0)) > + return TRUE; > my_qsort2(key_buff, (size_t) key_buff_elements, sizeof(rownum_t), > (qsort2_cmp) &cmp_keys_by_row_data_and_rownum, (void*) this); > /* Invalidate the current row position. */ > cur_key_idx= HA_POS_ERROR; > + tbl->file->ha_rnd_end(); > + return FALSE; > } > > > diff --git a/sql/sql_insert.cc b/sql/sql_insert.cc > index 307473cecbd..5593580dbca 100644 > --- a/sql/sql_insert.cc > +++ b/sql/sql_insert.cc > @@ -3177,6 +3193,10 @@ bool Delayed_insert::handle_inserts(void) > max_rows= ULONG_MAX; // Do as much as possible > } > > + /* Prepare table for random positioning (importent for innodb) */ same > + if (table->file->ha_rnd_init(0)) > + goto err; what's that for? duplicate key error message? > + > /* > We can't use row caching when using the binary log because if > we get a crash, then binary log will contain rows that are not yet > @@ -3649,7 +3671,12 @@ select_insert::prepare(List<Item> &values, SELECT_LEX_UNIT *u) > > thd->cuted_fields=0; > if (info.ignore || info.handle_duplicates != DUP_ERROR) > + { > table->file->extra(HA_EXTRA_IGNORE_DUP_KEY); > + if (table->file->ha_table_flags() & HA_DUPLICATE_POS && > + table->file->ha_rnd_init(0)) > + DBUG_RETURN(1); here and in other places, you don't do my_error(). Is there some catch-all my_error somewhere up the stack or it'll be the usual assert in Diagnostic_area? > + } > if (info.handle_duplicates == DUP_REPLACE && > (!table->triggers || !table->triggers->has_delete_triggers())) > table->file->extra(HA_EXTRA_WRITE_CAN_REPLACE); > @@ -3929,6 +3959,11 @@ void select_insert::abort_result_set() { > if (thd->locked_tables_mode <= LTM_LOCK_TABLES) > table->file->ha_end_bulk_insert(); > > + if (table->file->inited) why not "if (table->file->ha_table_flags() & HA_DUPLICATE_POS)" ? > + table->file->ha_rnd_end(); > + table->file->extra(HA_EXTRA_NO_IGNORE_DUP_KEY); > + table->file->extra(HA_EXTRA_WRITE_CANNOT_REPLACE); > + > /* > If at least one row has been inserted/modified and will stay in > the table (the table doesn't have transactions) we must write to > diff --git a/sql/sql_load.cc b/sql/sql_load.cc > index 8c2f17dac3f..3283b962f65 100644 > --- a/sql/sql_load.cc > +++ b/sql/sql_load.cc > @@ -638,6 +638,13 @@ int mysql_load(THD *thd,sql_exchange *ex,TABLE_LIST *table_list, > > thd->abort_on_warning= !ignore && thd->is_strict_mode(); > > + /* Prepare table for random positioning (importent for innodb) */ same > + if ((table_list->table->file->ha_table_flags() & HA_DUPLICATE_POS) && > + table_list->table->file->ha_rnd_init(0)) > + { > + error= 1; > + goto err; > + } > thd_progress_init(thd, 2); > if (table_list->table->validate_default_values_of_unset_fields(thd)) > { > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index 91a6445c870..4c401ea9011 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -20317,6 +20317,12 @@ end_unique_update(JOIN *join, JOIN_TAB *join_tab __attribute__((unused)), > table->file->print_error(error,MYF(0)); /* purecov: inspected */ > DBUG_RETURN(NESTED_LOOP_ERROR); /* purecov: inspected */ > } > + /* Prepare table for random positioning */ > + bool rnd_inited= (table->file->inited == handler::RND); when is that true? and when false? > + if (!rnd_inited && > + ((error= table->file->ha_index_end()) || > + (error= table->file->ha_rnd_init(0)))) > + DBUG_RETURN(NESTED_LOOP_ERROR); > if (table->file->ha_rnd_pos(table->record[1],table->file->dup_ref)) > { > table->file->print_error(error,MYF(0)); /* purecov: inspected */ > diff --git a/sql/sql_update.cc b/sql/sql_update.cc > index 11ffa684216..6d4c11d494a 100644 > --- a/sql/sql_update.cc > +++ b/sql/sql_update.cc > @@ -211,6 +211,10 @@ static void prepare_record_for_error_message(int error, TABLE *table) > bitmap_union(table->read_set, &unique_map); > /* Tell the engine about the new set. */ > table->file->column_bitmaps_signal(); > + /* Prepare table for random positioning (importent for innodb) */ same > + if (table->file->ha_index_or_rnd_end() || > + table->file->ha_rnd_init(0)) > + DBUG_VOID_RETURN; > /* Read record that is identified by table->file->ref. */ > (void) table->file->ha_rnd_pos(table->record[1], table->file->ref); > /* Copy the newly read columns into the new record. */ > Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] 34098adf32b: Get rid of rea_create_table()
by Sergei Golubchik 14 Mar '19

14 Mar '19

Hi, Sergey! On Mar 13, Sergey Vojtovich wrote: > revision-id: 34098adf32b (mariadb-10.4.1-138-g34098adf32b) > parent(s): d953dc02d9a > author: Sergey Vojtovich <svoj(a)mariadb.org> > committer: Sergey Vojtovich <svoj(a)mariadb.org> > timestamp: 2019-02-07 20:04:51 +0400 > message: > > Get rid of rea_create_table() > > Moved rea_create_table() to the sole caller. > > Visible difference: regular ALTER TABLE doesn't create .frm for temporary > and discoverable tables anymore. > > Also ha_create_partitioning_metadata(CHF_CREATE_FLAG) does cleanup on > error now. > > Part of MDEV-17805 - Remove InnoDB cache for temporary tables. ok to push Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] d953dc02d9a: Fix inplace ALTER TABLE to not register tmp table
by Sergei Golubchik 14 Mar '19

14 Mar '19

Hi, Sergey! On Mar 13, Sergey Vojtovich wrote: > revision-id: d953dc02d9a (mariadb-10.4.1-137-gd953dc02d9a) > parent(s): 9496394002f > author: Sergey Vojtovich <svoj(a)mariadb.org> > committer: Sergey Vojtovich <svoj(a)mariadb.org> > timestamp: 2019-02-07 20:04:38 +0400 > message: > > Fix inplace ALTER TABLE to not register tmp table > > Do not register intermediate tables created by inplace ALTER TABLE in > THD::temporary_tables. > > For inplace ALTER TABLE moved .frm creation to > create_table_for_inplace_alter(). For regular ALTER TABLE it is (and was > redundantly) created by ha_create_table(). > > Removed open_in_engine argument of create_and_open_tmp_table() and > open_temporary_table(): it became unused after this patch. > > Part of MDEV-17805 - Remove InnoDB cache for temporary tables. Looks ok code wise. But I don't see what are you gaining by this transformation. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] Self Introduction (gsoc)
by shreya pohekar 14 Mar '19

14 Mar '19

Hey! I am Shreya Pohekar. I am currently in my 3rd year of BE in Computer Science and Engineering from University Institute of Technology, RGPV Bhopal. I have a conceptual proficiency on Java and C++. On scrolling through the projects of mariadb, i found most of the projects interesting. But i would like to contribute to the project, "Implement multi-table UPDATE/DELETE returning a result set".@Igor Babaev, If you can guide me on where to get started, that would be great. Looking forward to hearing from you. Thank you.

1 0

[Maria-developers] Self Introduction (gsoc)
by shreya pohekar 14 Mar '19

14 Mar '19

Hey! I am Shreya Pohekar. I am currently in my 3rd year of BE in Computer Science and Engineering from University Institute of Technology, RGPV Bhopal. I have a conceptual proficiency on Java and C++. On scrolling through the projects of mariadb, i found most of the projects interesting. But i would like to contribute to the project, "Implement multi-table UPDATE/DELETE returning a result set". If you can guide me on where to get started, that would be great. Looking forward to hearing from you. Thank you.

1 0

Re: [Maria-developers] 9496394002f: Removed redundant partitioning check
by Sergei Golubchik 14 Mar '19

14 Mar '19

Hi, Sergey! On Mar 13, Sergey Vojtovich wrote: > revision-id: 9496394002f (mariadb-10.4.1-136-g9496394002f) > parent(s): 311beaffa63 > author: Sergey Vojtovich <svoj(a)mariadb.org> > committer: Sergey Vojtovich <svoj(a)mariadb.org> > timestamp: 2019-02-06 13:56:42 +0400 > message: > > Removed redundant partitioning check > > This check was introduced in 602a222 and then became redundant in ad1553e, > where we attempt to open a table even for non-copy algorithms. > > Added missing test case from 602a222. > > Part of MDEV-17805 - Remove InnoDB cache for temporary tables. ok to push Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] 311beaffa63: Simplified dd_recreate_table()
by Sergei Golubchik 14 Mar '19

14 Mar '19

Hi, Sergey! On Mar 13, Sergey Vojtovich wrote: > revision-id: 311beaffa63 (mariadb-10.4.1-135-g311beaffa63) > parent(s): 3323b013c13 > author: Sergey Vojtovich <svoj(a)mariadb.org> > committer: Sergey Vojtovich <svoj(a)mariadb.org> > timestamp: 2019-02-06 13:56:42 +0400 > message: > > Simplified dd_recreate_table() > > It is used only with persistent tables, so remove path argument, which > was introduced in 9594107f and became useless after ce6a63e. > > Part of MDEV-17805 - Remove InnoDB cache for temporary tables. ok to push Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] 3323b013c13: Removed redundant SE lock for tmp tables
by Sergei Golubchik 14 Mar '19

14 Mar '19

Hi, Sergey! On Mar 13, Sergey Vojtovich wrote: > revision-id: 3323b013c13 (mariadb-10.4.1-134-g3323b013c13) > parent(s): 4b3656a44d6 > author: Sergey Vojtovich <svoj(a)mariadb.org> > committer: Sergey Vojtovich <svoj(a)mariadb.org> > timestamp: 2019-02-06 13:56:28 +0400 > message: > > Removed redundant SE lock for tmp tables > > CREATE TEMPORARY TABLE locks SE plugin 6 times. 5 of these locks are > released by the end of the statement. And only 1 acquired by > init_from_binary_frm_image() / plugin_lock() remains. > > The lock removed in this patch was clearly redundant. > > Part of MDEV-17805 - Remove InnoDB cache for temporary tables. > > diff --git a/sql/temporary_tables.cc b/sql/temporary_tables.cc > index 917a85e6c3b..438b670b5f5 100644 > --- a/sql/temporary_tables.cc > +++ b/sql/temporary_tables.cc > @@ -951,8 +947,6 @@ TMP_TABLE_SHARE *THD::create_temporary_table(handlerton *hton, > init_tmp_table_share(this, share, saved_key_cache, key_length, > strend(saved_key_cache) + 1, tmp_path); > > - share->db_plugin= ha_lock_engine(this, hton); > - I suppose, share->db_plugin will be set in the following share->init_from_binary_frm_image ? Ok to push. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] 14c2f90ad08: Idempotent INSERT events for system versioning
by Aleksey Midenkov 13 Mar '19

13 Mar '19

Hi, Sergei, Thank you for observations! This task is in progress. While doing it I found out that RBR doesn't replicate timestamp fractions. That is: it gets seconds from event timestamp marker and adds fractions from system time. This causes bugs in System Versioning. More on it in Bug 6 here: https://github.com/tempesta-tech/mariadb/issues/578#issuecomment-470533050 I propose to fix it by adding timestamp fractions field to Log_event header after FLAGS_OFFSET. To support backwards compatibility it is needed to increment fdle->binlog_version and probably add new flag LOG_EVENT_HAS_SEC_PART_F (for support sending Log_event without fractions). Do you agree? On Thu, Jan 31, 2019 at 1:10 PM Sergei Golubchik <serg(a)mariadb.com> wrote: > Hi, Aleksey! > > On Jan 31, Aleksey Midenkov wrote: > > revision-id: 14c2f90ad08 (versioning-1.0.7-1-g14c2f90ad08) > > parent(s): a8efe7ab1f2 > > author: Aleksey Midenkov <midenok(a)gmail.com> > > committer: Aleksey Midenkov <midenok(a)gmail.com> > > timestamp: 2018-11-17 16:30:10 +0300 > > message: > > > > Idempotent INSERT events for system versioning > > > > Case 1: Rows_log_event::write_row() always overwrite historical row. > > > > Related to MDEV-16370. > > I realized that the whole RBR logging of system versioned tables is > wrong. > > For an update you log update_row event and write_row event separately > and execute them separately. It allows trivially to manipulate the history. > See a test case attached. > > To fix this, the slave should ignore all events that modify the history. > And generate them locally. Say, an update on the master generates > update_row and write_row events. The slave executes update_row event > which updates a current row and this also should generate the historical > row. Then the slave ignores write_row event. > > Of course, as an optimization, the master should not generate historical > write_row events, but as my test case shows, they can be forged, so the > slave should ignore them anyway, if they happen to come. > > I've created a new https://jira.mariadb.org/browse/MDEV-18432 for that, > let's not have it tied to MDEV-16370. > > Regards, > Sergei > Chief Architect MariaDB > and security(a)mariadb.org > -- All the best, Aleksey Midenkov @midenok

3 7

Re: [Maria-developers] 1aa6d5bdbec: MDEV-18888 Server crashes in Item_field::register_field_in_read_map upon...
by Sergei Golubchik 12 Mar '19

12 Mar '19

Hi, Sachin! On Mar 12, Sachin Setiya wrote: > revision-id: 1aa6d5bdbec (mariadb-10.4.3-63-g1aa6d5bdbec) > parent(s): b5d2e8577ef > author: Sachin Setiya <sachin.setiya(a)mariadb.com> > committer: sachin <sachin.setiya(a)mariadb.com> > timestamp: 2019-03-12 19:29:58 +0530 > message: > > MDEV-18888 Server crashes in Item_field::register_field_in_read_map upon... > MODIFY COLUMN > > Do Not create prefix field for long unique key > > diff --git a/sql/table.cc b/sql/table.cc > index 4da873b2e8f..5ca284fd8dd 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -3709,7 +3709,33 @@ enum open_frm_error open_table_from_share(THD *thd, TABLE_SHARE *share, > Field *field= key_part->field= outparam->field[key_part->fieldnr - 1]; > > if (field->key_length() != key_part->length && > - !(field->flags & BLOB_FLAG)) > + !(field->flags & BLOB_FLAG) && > + /* > + MDEV-18888 > + We will not make copy of the field because if we do this this this will > + make this test case to fail > + CREATE TABLE t1 ( > + a CHAR(128), > + b CHAR(128) AS (a), > + c varchar(5000), > + UNIQUE(c,b(64)) > + ) ENGINE=InnoDB; > + And the reason for this is because Item_func_hash will Item_field based on > + this new field. But if this field is virtual field then since it is created > + before calling parse_vcol_defs so it does not have field->vcol_info->expr item > + initiated. And this will make segfault when field->vcol_info->expr is accessed. > + So the question may be why similar unique with just virtual column is not crashing ? > + CREATE TABLE t2 ( > + a CHAR(128), > + b CHAR(128) AS (a), > + c char(128) as (a+b), > + UNIQUE(a, c(64)) > + ) ENGINE=InnoDB; > + In this case also new prefix field will be created whose vcol_info->expr will be NULL. > + But Since this key is not a long unique in innodb_base_col_setup field will be > + table->field[i] which will have not null vcol_info->expr > + */ > + key_info->algorithm != HA_KEY_ALG_LONG_HASH) don't put a comment (particularly a long one) inside the if(), move it to be before the if(), please. and I'd used a simpler comment, like "there's no need to create a prefix Field for HA_KEY_ALG_LONG_HASH indexes, as they implement prefixing via Iten_func_left anyway (see parse_vcol_defs()" > { > /* > We are using only a prefix of the column as a key: Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] b5d2e8577ef: MDEV-18889 Long unique on virtual fields crashes server
by Sergei Golubchik 12 Mar '19

12 Mar '19

Hi, Sachin! On Mar 12, Sachin Setiya wrote: > revision-id: b5d2e8577ef (mariadb-10.4.3-62-gb5d2e8577ef) > parent(s): 202d1b0c897 > author: Sachin Setiya <sachin.setiya(a)mariadb.com> > committer: sachin <sachin.setiya(a)mariadb.com> > timestamp: 2019-03-12 19:29:58 +0530 > message: > > MDEV-18889 Long unique on virtual fields crashes server > > When we receive record from ha_index_read_map it does not have vfield > values. So we need to update it. > > diff --git a/sql/handler.cc b/sql/handler.cc > index 0746f8a81fc..c4f81c6432e 100644 > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -6522,6 +6522,26 @@ static int wsrep_after_row(THD *thd) > } > #endif /* WITH_WSREP */ > > +/* > + Long Unique can have virtual fields. But when we receive record from > + ha_index_read_map it does not have vfield values. So we need to update it. > + IN table Table Object > + IN handler Handler Object > + OUT updated Set to true on successful update Doxygen style for new comments, please. And feel free to rewrite old comments to use Doxygen style whenever you need to edit them anyway. > +*/ > +static void update_table_vcol(TABLE *table, handler *h, bool& updated) > +{ > + uint length= table->s->reclength; > + uchar temp_record[length]; Not a good idea: 1. you'll get compiler warnings for using VLA (-Wvla: Warn if a variable-length array is used") 2. length can be rather large, stack can be rather small. 3. two big memcpy's per row read. Two other options: 1. use field->move(check_unique_buf - record[0]) to move field pointers to check_unique_buf. You can do it once, at the beginning and restore them at the end, no need to do it per read. See TABLE::move_fields() 2. Copy record[0] to check_unique_buf(). And read potential duplicates into record[0]. Then you won't need to do anything special at all, ha_index_read_map() will update vcols automatically. Just don't forget to restore record[0] at the end. See store_record(), restore_record(). I think the second approach might be easier. > + memcpy(temp_record, table->record[0], length); > + restore_record(table, check_unique_buf); > + if (!table->update_virtual_fields(h, VCOL_UPDATE_FOR_READ)) > + updated= true; > + store_record(table, check_unique_buf); > + memcpy(table->record[0], temp_record, length); > + > +} > + Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] 003abb6d9dc: MDEV-18799 Long unique does not work after failed alter table
by Sergei Golubchik 12 Mar '19

12 Mar '19

Hi, Sachin! On Mar 12, Sachin Setiya wrote: > revision-id: 003abb6d9dc (mariadb-10.4.3-58-g003abb6d9dc) > parent(s): e2f8f968785 > author: sachin <sachin.setiya(a)mariadb.com> > committer: sachin <sachin.setiya(a)mariadb.com> > timestamp: 2019-03-12 19:29:57 +0530 > message: > > MDEV-18799 Long unique does not work after failed alter table > > Restore table->key_info after calling setup_keyinfo_hash in > mysql_prepare_alter_table. > > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index e721a868124..2d86035b7fd 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -7923,6 +7923,7 @@ mysql_prepare_alter_table(THD *thd, TABLE *table, > Field **f_ptr,*field; > MY_BITMAP *dropped_fields= NULL; // if it's NULL - no dropped fields > bool drop_period= false; > + bool long_hash_key= false; > DBUG_ENTER("mysql_prepare_alter_table"); > > /* > @@ -8299,10 +8300,14 @@ mysql_prepare_alter_table(THD *thd, TABLE *table, > */ > for (uint i=0 ; i < table->s->keys ; i++,key_info++) > { > + long_hash_key= false; better declare long_hash_key here, you don't use it on a larger scope. > if (key_info->flags & HA_INVISIBLE_KEY) > continue; > if (key_info->algorithm == HA_KEY_ALG_LONG_HASH) > + { > setup_keyinfo_hash(key_info); > + long_hash_key= true; > + } > const char *key_name= key_info->name.str; > Alter_drop *drop; > drop_it.rewind(); > @@ -8476,8 +8481,12 @@ mysql_prepare_alter_table(THD *thd, TABLE *table, > &key_parts, key_info->option_list, DDL_options()); > new_key_list.push_back(key, thd->mem_root); > } > - if (key_info->algorithm == HA_KEY_ALG_LONG_HASH) > + if (long_hash_key) > + { > + key_info->algorithm= HA_KEY_ALG_LONG_HASH; > + key_info->flags&= ~HA_NOSAME; perhaps you could move HA_NOSAME manipulations into [re_]setup_keyinfo_hash ? > re_setup_keyinfo_hash(key_info); > + } > } > { > Key *key; > Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] 202d1b0c897: MDEV-18809 Server crash in fields_in_hash_keyinfo or Assertion...
by Sergei Golubchik 12 Mar '19

12 Mar '19

Hi, Sachin! On Mar 12, Sachin Setiya wrote: > revision-id: 202d1b0c897 (mariadb-10.4.3-61-g202d1b0c897) > parent(s): a4b8e0c30d1 > author: sachin <sachin.setiya(a)mariadb.com> > committer: sachin <sachin.setiya(a)mariadb.com> > timestamp: 2019-03-12 19:29:58 +0530 > message: > > MDEV-18809 Server crash in fields_in_hash_keyinfo or Assertion... > `key_info->key_part->field->flags & (1<< 30)' failed in setup_keyinfo_hash put the subject all on one line please, even when it's (unfortunately) long > > Move calling setup_keyinfo_hash until all continue is exhausted. okay. but there's one `goto err` still in the loop. I suspect it might be a problem, try to do a test case with ALTER TABLE DROP COLUMN from a multi-column long unique constraint. > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index a9c4c71331a..58f175d7a29 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -8304,11 +8304,6 @@ mysql_prepare_alter_table(THD *thd, TABLE *table, > long_hash_key= false; > if (key_info->flags & HA_INVISIBLE_KEY) > continue; > - if (key_info->algorithm == HA_KEY_ALG_LONG_HASH) > - { > - setup_keyinfo_hash(key_info); > - long_hash_key= true; > - } > const char *key_name= key_info->name.str; > Alter_drop *drop; > drop_it.rewind(); > @@ -8339,6 +8334,11 @@ mysql_prepare_alter_table(THD *thd, TABLE *table, > continue; > } > > + if (key_info->algorithm == HA_KEY_ALG_LONG_HASH) > + { > + setup_keyinfo_hash(key_info); > + long_hash_key= true; > + } > const char *dropped_key_part= NULL; > KEY_PART_INFO *key_part= key_info->key_part; > key_parts.empty(); > Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] a4b8e0c30d1: MDEV-18800 Server crash in instant_alter_column_possible or Assertion...
by Sergei Golubchik 12 Mar '19

12 Mar '19

Hi, Sachin! ok to push On Mar 12, Sachin Setiya wrote: > revision-id: a4b8e0c30d1 (mariadb-10.4.3-60-ga4b8e0c30d1) > parent(s): d8e4463d5dc > author: sachin <sachin.setiya(a)mariadb.com> > committer: sachin <sachin.setiya(a)mariadb.com> > timestamp: 2019-03-12 19:29:57 +0530 > message: > > MDEV-18800 Server crash in instant_alter_column_possible or Assertion... > `!pk->has_virtual()' failed in instant_alter_column_possible upon adding key > > Hash key can't be primary key. > > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index 2d86035b7fd..a9c4c71331a 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -4179,6 +4179,7 @@ mysql_prepare_create_table(THD *thd, HA_CREATE_INFO *create_info, > } > if (is_hash_field_needed || > (key_info->algorithm == HA_KEY_ALG_HASH && > + key->type != Key::PRIMARY && > key_info->flags & HA_NOSAME && > !(file->ha_table_flags() & HA_CAN_HASH_KEYS ) && > file->ha_table_flags() & HA_CAN_VIRTUAL_COLUMNS)) > Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] e2f8f968785: MDEV-18790 Server crash in fields_in_hash_keyinfo after unsuccessful...
by Sergei Golubchik 12 Mar '19

12 Mar '19

Hi, Sachin! Ok to push, just one minor detail, see below On Mar 12, Sachin Setiya wrote: > revision-id: e2f8f968785 (mariadb-10.4.3-57-ge2f8f968785) > parent(s): 6d68a3464e9 > author: sachin <sachin.setiya(a)mariadb.com> > committer: sachin <sachin.setiya(a)mariadb.com> > timestamp: 2019-03-12 19:29:57 +0530 > message: > > MDEV-18790 Server crash in fields_in_hash_keyinfo after unsuccessful... > attempt to drop BLOB with long index > > Restore long table->key_info, So that in the case of unsuccessful table > alter table->key_info should have a correct value. In the case of successful > table alter old table is flushed so that is why we don't see this error in > the case of successful alter. > > diff --git a/mysql-test/main/long_unique_bugs.result b/mysql-test/main/long_unique_bugs.result > index 5de7566c8a3..1544565d95e 100644 > --- a/mysql-test/main/long_unique_bugs.result > +++ b/mysql-test/main/long_unique_bugs.result > @@ -57,3 +57,9 @@ create table t (b blob, unique(b)) engine=myisam; > insert into t values ('foo'); > replace into t values ('foo'); > drop table t; > +CREATE TABLE t1 (f INT, x BLOB, UNIQUE (x)); > +INSERT INTO t1 VALUES (1,'foo'); > +ALTER TABLE t1 DROP x, ALGORITHM=INPLACE; > +Got one of the listed errors > +UPDATE t1 SET x = 'bar'; > +DROP TABLE t1; > diff --git a/mysql-test/main/long_unique_bugs.test b/mysql-test/main/long_unique_bugs.test > index 0246ea7962a..14c002f998c 100644 > --- a/mysql-test/main/long_unique_bugs.test > +++ b/mysql-test/main/long_unique_bugs.test > @@ -72,3 +72,13 @@ create table t (b blob, unique(b)) engine=myisam; > insert into t values ('foo'); > replace into t values ('foo'); > drop table t; > + > +# > +# MDEV-18790 Server crash in fields_in_hash_keyinfo after unsuccessful attempt to drop BLOB with long index > +# > +CREATE TABLE t1 (f INT, x BLOB, UNIQUE (x)); > +INSERT INTO t1 VALUES (1,'foo'); > +--error ER_ALTER_OPERATION_NOT_SUPPORTED,ER_ALTER_OPERATION_NOT_SUPPORTED_REASON this makes the test to accept any of the listed error messages and the result file will have "Got one of the listed errors". It's useful when a test can indeed see different errors in a certain place (Elena has probably run the test with different storage engines, and has seen different errors for MyISAM and InnoDB). But this final committed test case is very deterministic, and as far as I can see there can be only one possible error code here. It's generally better to have just this one error code in the `--error` directive and the correct complete error message text in the result file. > +ALTER TABLE t1 DROP x, ALGORITHM=INPLACE; > +UPDATE t1 SET x = 'bar'; > +DROP TABLE t1; Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] 190a29d06: Make Lock Tree's lock waits use externally provided mutexes/conditions
by jocelyn fournier 12 Mar '19

12 Mar '19

Hi Sergei! Just wondering why keeping the toku_ prefix after renaming toku_* to toku_external_* since it’s also used by other engine like RockDB? BR, Jocelyn Fournier > Le 12 mars 2019 à 09:52, Sergei Petrunia <psergey(a)askmonty.org> a écrit : > > revision-id: 190a29d06f4e79d2df4cb513944ac34bd133caa0 (v5.8-1024-g190a29d06) > parent(s): 7a8c777d4880cde8fd96135f6cc216223371a9f4 > author: Sergei Petrunia > committer: Sergei Petrunia > timestamp: 2019-03-12 11:52:17 +0300 > message: > > Make Lock Tree's lock waits use externally provided mutexes/conditions > > In MyRocks, this allows: > - SHOW PROCESSLIST to show "waiting for row lock" for waiting threads > - KILL statement to abort the lock wait in the same way as it does it with > point locking (without use of handlerton::kill_connection() method) > > - Remove previously added TransactionDB::KillLockWait() method as it is > no longer needed. > > --- > include/rocksdb/utilities/transaction_db.h | 1 - > .../transactions/pessimistic_transaction_db.cc | 12 +++- > .../transactions/pessimistic_transaction_db.h | 4 -- > .../range_locking/locktree/lock_request.cc | 35 ++++++---- > .../range_locking/locktree/lock_request.h | 9 ++- > .../range_locking/locktree/locktree.cc | 12 ++-- > .../transactions/range_locking/locktree/locktree.h | 13 ++-- > .../transactions/range_locking/locktree/manager.cc | 17 +++-- > .../portability/toku_external_pthread.h | 81 ++++++++++++++++++++++ > utilities/transactions/transaction_lock_mgr.cc | 15 ++-- > utilities/transactions/transaction_lock_mgr.h | 6 +- > 11 files changed, 155 insertions(+), 50 deletions(-) > > diff --git a/include/rocksdb/utilities/transaction_db.h b/include/rocksdb/utilities/transaction_db.h > index 2a98a8d3d..fb3c6a88c 100644 > --- a/include/rocksdb/utilities/transaction_db.h > +++ b/include/rocksdb/utilities/transaction_db.h > @@ -291,7 +291,6 @@ class TransactionDB : public StackableDB { > bool use_range_locking; > virtual RangeLockMgrControl* get_range_lock_manager() { return nullptr; } > > - virtual void KillLockWait(TransactionID txnid){}; > protected: > // To Create an TransactionDB, call Open() > // The ownership of db is transferred to the base StackableDB > diff --git a/utilities/transactions/pessimistic_transaction_db.cc b/utilities/transactions/pessimistic_transaction_db.cc > index d44f329e5..308187e13 100644 > --- a/utilities/transactions/pessimistic_transaction_db.cc > +++ b/utilities/transactions/pessimistic_transaction_db.cc > @@ -41,7 +41,11 @@ PessimisticTransactionDB::PessimisticTransactionDB( > ? txn_db_options_.custom_mutex_factory > : std::shared_ptr<TransactionDBMutexFactory>( > new TransactionDBMutexFactoryImpl())), > - range_lock_mgr_(this) { > + range_lock_mgr_(this, > + txn_db_options_.custom_mutex_factory? > + txn_db_options_.custom_mutex_factory : > + std::shared_ptr<TransactionDBMutexFactory>( > + new TransactionDBMutexFactoryImpl())) { > assert(db_impl_ != nullptr); > info_log_ = db_impl_->GetDBOptions().info_log; > } > @@ -73,7 +77,11 @@ PessimisticTransactionDB::PessimisticTransactionDB( > ? txn_db_options_.custom_mutex_factory > : std::shared_ptr<TransactionDBMutexFactory>( > new TransactionDBMutexFactoryImpl())), > - range_lock_mgr_(this) { > + range_lock_mgr_(this, > + txn_db_options_.custom_mutex_factory > + ? txn_db_options_.custom_mutex_factory > + : std::shared_ptr<TransactionDBMutexFactory>( > + new TransactionDBMutexFactoryImpl())) { > assert(db_impl_ != nullptr); > } > > diff --git a/utilities/transactions/pessimistic_transaction_db.h b/utilities/transactions/pessimistic_transaction_db.h > index 07ed3fed2..388ed4099 100644 > --- a/utilities/transactions/pessimistic_transaction_db.h > +++ b/utilities/transactions/pessimistic_transaction_db.h > @@ -127,10 +127,6 @@ class PessimisticTransactionDB : public TransactionDB { > virtual void UpdateCFComparatorMap(const std::vector<ColumnFamilyHandle*>&) {} > virtual void UpdateCFComparatorMap(ColumnFamilyHandle*) {} > > - void KillLockWait(TransactionID txnid) override { > - if (use_range_locking) > - range_lock_mgr_.KillLockWait(txnid); > - } > protected: > DBImpl* db_impl_; > std::shared_ptr<Logger> info_log_; > diff --git a/utilities/transactions/range_locking/locktree/lock_request.cc b/utilities/transactions/range_locking/locktree/lock_request.cc > index 0b96b2051..84cfdbf62 100644 > --- a/utilities/transactions/range_locking/locktree/lock_request.cc > +++ b/utilities/transactions/range_locking/locktree/lock_request.cc > @@ -59,7 +59,7 @@ Copyright (c) 2006, 2015, Percona and/or its affiliates. All rights reserved. > namespace toku { > > // initialize a lock request's internals > -void lock_request::create(void) { > +void lock_request::create(toku_external_mutex_factory_t mutex_factory) { > m_txnid = TXNID_NONE; > m_conflicting_txnid = TXNID_NONE; > m_start_time = 0; > @@ -75,7 +75,9 @@ void lock_request::create(void) { > m_state = state::UNINITIALIZED; > m_info = nullptr; > > - toku_cond_init(*lock_request_m_wait_cond_key, &m_wait_cond, nullptr); > + //psergey-todo: this condition is for interruptible wait > + // note: moved to here from lock_request::create: > + toku_external_cond_init(mutex_factory, &m_wait_cond); > > m_start_test_callback = nullptr; > m_start_before_pending_test_callback = nullptr; > @@ -89,13 +91,14 @@ void lock_request::destroy(void) { > m_state = state::DESTROYED; > toku_destroy_dbt(&m_left_key_copy); > toku_destroy_dbt(&m_right_key_copy); > - toku_cond_destroy(&m_wait_cond); > + toku_external_cond_destroy(&m_wait_cond); > } > > // set the lock request parameters. this API allows a lock request to be reused. > void lock_request::set(locktree *lt, TXNID txnid, const DBT *left_key, const DBT *right_key, lock_request::type lock_type, bool big_txn, void *extra) { > invariant(m_state != state::PENDING); > m_lt = lt; > + > m_txnid = txnid; > m_left_key = left_key; > m_right_key = right_key; > @@ -190,13 +193,13 @@ int lock_request::start(void) { > m_conflicting_txnid = conflicts.get(0); > if (m_start_before_pending_test_callback) > m_start_before_pending_test_callback(); > - toku_mutex_lock(&m_info->mutex); > + toku_external_mutex_lock(&m_info->mutex); > insert_into_lock_requests(); > if (deadlock_exists(conflicts)) { > remove_from_lock_requests(); > r = DB_LOCK_DEADLOCK; > } > - toku_mutex_unlock(&m_info->mutex); > + toku_external_mutex_unlock(&m_info->mutex); > if (m_start_test_callback) > m_start_test_callback(); // test callback > } > @@ -221,7 +224,7 @@ int lock_request::wait(uint64_t wait_time_ms, uint64_t killed_time_ms, int (*kil > uint64_t t_start = t_now; > uint64_t t_end = t_start + wait_time_ms * 1000; > > - toku_mutex_lock(&m_info->mutex); > + toku_external_mutex_lock(&m_info->mutex); > > // check again, this time locking out other retry calls > if (m_state == state::PENDING) { > @@ -252,10 +255,14 @@ int lock_request::wait(uint64_t wait_time_ms, uint64_t killed_time_ms, int (*kil > if (t_wait > t_end) > t_wait = t_end; > } > + /* > + PORT: don't need to compute "time when the wait should end" anymore. > struct timespec ts = {0, 0}; > ts.tv_sec = t_wait / 1000000; > ts.tv_nsec = (t_wait % 1000000) * 1000; > - int r = toku_cond_timedwait(&m_wait_cond, &m_info->mutex, &ts); > + */ > + int r = toku_external_cond_timedwait(&m_wait_cond, &m_info->mutex, > + int32_t(wait_time_ms)*1000); > invariant(r == 0 || r == ETIMEDOUT); > > t_now = toku_current_time_microsec(); > @@ -279,7 +286,7 @@ int lock_request::wait(uint64_t wait_time_ms, uint64_t killed_time_ms, int (*kil > m_info->counters.long_wait_count += 1; > m_info->counters.long_wait_time += duration; > } > - toku_mutex_unlock(&m_info->mutex); > + toku_external_mutex_unlock(&m_info->mutex); > > invariant(m_state == state::COMPLETE); > return m_complete_r; > @@ -332,7 +339,7 @@ int lock_request::retry(GrowableArray<TXNID> *conflicts_collector) { > complete(r); > if (m_retry_test_callback) > m_retry_test_callback(); // test callback > - toku_cond_broadcast(&m_wait_cond); > + toku_external_cond_broadcast(&m_wait_cond); > } else { > m_conflicting_txnid = conflicts.get(0); > add_conflicts_to_waits(&conflicts, conflicts_collector); > @@ -393,7 +400,7 @@ void lock_request::retry_all_lock_requests( > } > > void lock_request::retry_all_lock_requests_info(lt_lock_request_info *info, GrowableArray<TXNID> *collector) { > - toku_mutex_lock(&info->mutex); > + toku_external_mutex_lock(&info->mutex); > // retry all of the pending lock requests. > for (size_t i = 0; i < info->pending_lock_requests.size();) { > lock_request *request; > @@ -412,7 +419,7 @@ void lock_request::retry_all_lock_requests_info(lt_lock_request_info *info, Grow > > // future threads should only retry lock requests if some still exist > info->should_retry_lock_requests = info->pending_lock_requests.size() > 0; > - toku_mutex_unlock(&info->mutex); > + toku_external_mutex_unlock(&info->mutex); > } > > void lock_request::add_conflicts_to_waits(txnid_set *conflicts, > @@ -444,12 +451,12 @@ void *lock_request::get_extra(void) const { > void lock_request::kill_waiter(void) { > remove_from_lock_requests(); > complete(DB_LOCK_NOTGRANTED); > - toku_cond_broadcast(&m_wait_cond); > + toku_external_cond_broadcast(&m_wait_cond); > } > > void lock_request::kill_waiter(locktree *lt, void *extra) { > lt_lock_request_info *info = lt->get_lock_request_info(); > - toku_mutex_lock(&info->mutex); > + toku_external_mutex_lock(&info->mutex); > for (size_t i = 0; i < info->pending_lock_requests.size(); i++) { > lock_request *request; > int r = info->pending_lock_requests.fetch(i, &request); > @@ -458,7 +465,7 @@ void lock_request::kill_waiter(locktree *lt, void *extra) { > break; > } > } > - toku_mutex_unlock(&info->mutex); > + toku_external_mutex_unlock(&info->mutex); > } > > // find another lock request by txnid. must hold the mutex. > diff --git a/utilities/transactions/range_locking/locktree/lock_request.h b/utilities/transactions/range_locking/locktree/lock_request.h > index 8b9529344..b17b639c8 100644 > --- a/utilities/transactions/range_locking/locktree/lock_request.h > +++ b/utilities/transactions/range_locking/locktree/lock_request.h > @@ -84,7 +84,7 @@ public: > }; > > // effect: Initializes a lock request. > - void create(void); > + void create(toku_external_mutex_factory_t mutex_factory); > > // effect: Destroys a lock request. > void destroy(void); > @@ -171,7 +171,7 @@ public: > int m_complete_r; > state m_state; > > - toku_cond_t m_wait_cond; > + toku_external_cond_t m_wait_cond; > > bool m_big_txn; > > @@ -227,6 +227,9 @@ public: > > friend class lock_request_unit_test; > }; > -ENSURE_POD(lock_request); > +// PORT: lock_request is not a POD anymore due to use of toku_external_cond_t > +// This is ok as the PODness is not really required: lock_request objects are > +// not moved in memory or anything. > +//ENSURE_POD(lock_request); > > } /* namespace toku */ > diff --git a/utilities/transactions/range_locking/locktree/locktree.cc b/utilities/transactions/range_locking/locktree/locktree.cc > index 4ecfb2d26..9b530c7b0 100644 > --- a/utilities/transactions/range_locking/locktree/locktree.cc > +++ b/utilities/transactions/range_locking/locktree/locktree.cc > @@ -74,7 +74,9 @@ namespace toku { > // but does nothing based on the value of the reference count - it is > // up to the user of the locktree to destroy it when it sees fit. > > -void locktree::create(locktree_manager *mgr, DICTIONARY_ID dict_id, const comparator &cmp) { > +void locktree::create(locktree_manager *mgr, DICTIONARY_ID dict_id, > + const comparator &cmp, > + toku_external_mutex_factory_t mutex_factory) { > m_mgr = mgr; > m_dict_id = dict_id; > > @@ -91,14 +93,14 @@ void locktree::create(locktree_manager *mgr, DICTIONARY_ID dict_id, const compar > m_sto_end_early_count = 0; > m_sto_end_early_time = 0; > > - m_lock_request_info.init(); > + m_lock_request_info.init(mutex_factory); > } > > -void lt_lock_request_info::init(void) { > +void lt_lock_request_info::init(toku_external_mutex_factory_t mutex_factory) { > pending_lock_requests.create(); > pending_is_empty = true; > ZERO_STRUCT(mutex); > - toku_mutex_init(*locktree_request_info_mutex_key, &mutex, nullptr); > + toku_external_mutex_init(mutex_factory, &mutex); > retry_want = retry_done = 0; > ZERO_STRUCT(counters); > ZERO_STRUCT(retry_mutex); > @@ -124,7 +126,7 @@ void locktree::destroy(void) { > > void lt_lock_request_info::destroy(void) { > pending_lock_requests.destroy(); > - toku_mutex_destroy(&mutex); > + toku_external_mutex_destroy(&mutex); > toku_mutex_destroy(&retry_mutex); > toku_cond_destroy(&retry_cv); > } > diff --git a/utilities/transactions/range_locking/locktree/locktree.h b/utilities/transactions/range_locking/locktree/locktree.h > index 4f0530855..5ff4f7449 100644 > --- a/utilities/transactions/range_locking/locktree/locktree.h > +++ b/utilities/transactions/range_locking/locktree/locktree.h > @@ -55,6 +55,7 @@ Copyright (c) 2006, 2015, Percona and/or its affiliates. All rights reserved. > > #include <db.h> > #include <toku_pthread.h> > +#include <toku_external_pthread.h> > #include <toku_time.h> > > #include <ft/comparator.h> > @@ -102,7 +103,7 @@ namespace toku { > struct lt_lock_request_info { > omt<lock_request *> pending_lock_requests; > std::atomic_bool pending_is_empty; > - toku_mutex_t mutex; > + toku_external_mutex_t mutex; > bool should_retry_lock_requests; > lt_counters counters; > std::atomic_ullong retry_want; > @@ -111,7 +112,7 @@ namespace toku { > toku_cond_t retry_cv; > bool running_retry; > > - void init(void); > + void init(toku_external_mutex_factory_t mutex_factory); > void destroy(void); > }; > > @@ -127,7 +128,8 @@ namespace toku { > void create(lt_create_cb create_cb, > lt_destroy_cb destroy_cb, > lt_escalate_cb escalate_cb, > - void *extra); > + void *extra, > + toku_external_mutex_factory_t mutex_factory_arg); > > void destroy(void); > > @@ -212,6 +214,8 @@ namespace toku { > > omt<locktree *> m_locktree_map; > > + toku_external_mutex_factory_t mutex_factory; > + > // the manager's mutex protects the locktree map > toku_mutex_t m_mutex; > > @@ -275,7 +279,8 @@ namespace toku { > class locktree { > public: > // effect: Creates a locktree > - void create(locktree_manager *mgr, DICTIONARY_ID dict_id, const comparator &cmp); > + void create(locktree_manager *mgr, DICTIONARY_ID dict_id, const comparator &cmp, > + toku_external_mutex_factory_t mutex_factory); > > void destroy(void); > > diff --git a/utilities/transactions/range_locking/locktree/manager.cc b/utilities/transactions/range_locking/locktree/manager.cc > index c31158bb8..47a782565 100644 > --- a/utilities/transactions/range_locking/locktree/manager.cc > +++ b/utilities/transactions/range_locking/locktree/manager.cc > @@ -60,7 +60,12 @@ Copyright (c) 2006, 2015, Percona and/or its affiliates. All rights reserved. > > namespace toku { > > -void locktree_manager::create(lt_create_cb create_cb, lt_destroy_cb destroy_cb, lt_escalate_cb escalate_cb, void *escalate_extra) { > +void locktree_manager::create(lt_create_cb create_cb, > + lt_destroy_cb destroy_cb, > + lt_escalate_cb escalate_cb, > + void *escalate_extra, > + toku_external_mutex_factory_t mutex_factory_arg) { > + mutex_factory= mutex_factory_arg; > m_max_lock_memory = DEFAULT_MAX_LOCK_MEMORY; > m_current_lock_memory = 0; > > @@ -151,7 +156,7 @@ locktree *locktree_manager::get_lt(DICTIONARY_ID dict_id, > locktree *lt = locktree_map_find(dict_id); > if (lt == nullptr) { > XCALLOC(lt); > - lt->create(this, dict_id, cmp); > + lt->create(this, dict_id, cmp, mutex_factory); > > // new locktree created - call the on_create callback > // and put it in the locktree map > @@ -320,7 +325,7 @@ int locktree_manager::iterate_pending_lock_requests(lock_request_iterate_callbac > invariant_zero(r); > > struct lt_lock_request_info *info = lt->get_lock_request_info(); > - toku_mutex_lock(&info->mutex); > + toku_external_mutex_lock(&info->mutex); > > size_t num_requests = info->pending_lock_requests.size(); > for (size_t k = 0; k < num_requests && r == 0; k++) { > @@ -332,7 +337,7 @@ int locktree_manager::iterate_pending_lock_requests(lock_request_iterate_callbac > req->get_conflicting_txnid(), req->get_start_time(), extra); > } > > - toku_mutex_unlock(&info->mutex); > + toku_external_mutex_unlock(&info->mutex); > } > mutex_unlock(); > return r; > @@ -472,10 +477,10 @@ void locktree_manager::get_status(LTM_STATUS statp) { > locktree *lt; > int r = m_locktree_map.fetch(i, &lt); > invariant_zero(r); > - if (toku_mutex_trylock(&lt->m_lock_request_info.mutex) == 0) { > + if (toku_external_mutex_trylock(&lt->m_lock_request_info.mutex) == 0) { > lock_requests_pending += lt->m_lock_request_info.pending_lock_requests.size(); > lt_counters.add(lt->get_lock_request_info()->counters); > - toku_mutex_unlock(&lt->m_lock_request_info.mutex); > + toku_external_mutex_unlock(&lt->m_lock_request_info.mutex); > } > sto_num_eligible += lt->sto_txnid_is_valid_unsafe() ? 1 : 0; > sto_end_early_count += lt->m_sto_end_early_count; > diff --git a/utilities/transactions/range_locking/portability/toku_external_pthread.h b/utilities/transactions/range_locking/portability/toku_external_pthread.h > new file mode 100644 > index 000000000..2482ec507 > --- /dev/null > +++ b/utilities/transactions/range_locking/portability/toku_external_pthread.h > @@ -0,0 +1,81 @@ > +/* > + A wrapper around rocksdb::TransactionDBMutexFactory-provided condition and mutex > + that provides toku_pthread_*-like interface. The functions are named > + > + toku_external_{mutex|cond}_XXX > + > + Lock Tree uses this mutex and condition for interruptible (long) lock waits. > + > + (It also still uses toku_pthread_XXX calls for mutexes/conditions for > + shorter waits on internal objects) > +*/ > + > +#pragma once > + > +#include <pthread.h> > +#include <time.h> > +#include <stdint.h> > + > +#include "toku_portability.h" > + > +#include "rocksdb/utilities/transaction_db.h" > +#include "rocksdb/utilities/transaction_db_mutex.h" > + > +using rocksdb::TransactionDBMutex; > +using rocksdb::TransactionDBCondVar; > + > +typedef std::shared_ptr<rocksdb::TransactionDBMutexFactory> toku_external_mutex_factory_t; > + > +typedef std::shared_ptr<TransactionDBMutex> toku_external_mutex_t; > +typedef std::shared_ptr<TransactionDBCondVar> toku_external_cond_t; > + > +static inline > +void toku_external_cond_init(toku_external_mutex_factory_t mutex_factory, > + toku_external_cond_t *cond) { > + *cond= mutex_factory->AllocateCondVar(); > +} > + > +inline void toku_external_cond_destroy(toku_external_cond_t *cond) { > + cond->reset(); // this will destroy the managed object > +} > + > +inline void toku_external_cond_signal(toku_external_cond_t *cond) { > + (*cond)->Notify(); > +} > + > +inline void toku_external_cond_broadcast(toku_external_cond_t *cond) { > + (*cond)->NotifyAll(); > +} > + > +inline int toku_external_cond_timedwait(toku_external_cond_t *cond, > + toku_external_mutex_t *mutex, > + int32_t timeout_microsec) { > + auto res= (*cond)->WaitFor(*mutex, timeout_microsec); > + if (res.ok()) > + return 0; > + else > + return ETIMEDOUT; > +} > + > +inline void toku_external_mutex_init(toku_external_mutex_factory_t factory, > + toku_external_mutex_t *mutex) { > + *mutex = factory->AllocateMutex(); > +} > + > +inline void toku_external_mutex_lock(toku_external_mutex_t *mutex) { > + (*mutex)->Lock(); > +} > + > +inline int toku_external_mutex_trylock(toku_external_mutex_t *mutex) { > + (*mutex)->Lock(); > + return 0; > +} > + > +inline void toku_external_mutex_unlock(toku_external_mutex_t *mutex) { > + (*mutex)->UnLock(); > +} > + > +inline void toku_external_mutex_destroy(toku_external_mutex_t *mutex) { > + mutex->reset(); // this will destroy the managed object > +} > + > diff --git a/utilities/transactions/transaction_lock_mgr.cc b/utilities/transactions/transaction_lock_mgr.cc > index 56f3295fd..79164a962 100644 > --- a/utilities/transactions/transaction_lock_mgr.cc > +++ b/utilities/transactions/transaction_lock_mgr.cc > @@ -790,11 +790,6 @@ public: > bool releasing_locks_; > }; > > - > -void RangeLockMgr::KillLockWait(TransactionID txnid) { > - ltm.kill_waiter((void*)txnid); > -} > - > // Get a range lock on [start_key; end_key] range > Status RangeLockMgr::TryRangeLock(PessimisticTransaction* txn, > uint32_t column_family_id, > @@ -802,7 +797,7 @@ Status RangeLockMgr::TryRangeLock(PessimisticTransaction* txn, > const rocksdb::Slice &end_key, > bool exclusive) { > toku::lock_request request; > - request.create(); > + request.create(mutex_factory_); > DBT start_key_dbt, end_key_dbt; > > toku_fill_dbt(&start_key_dbt, start_key.data(), start_key.size()); > @@ -810,6 +805,8 @@ Status RangeLockMgr::TryRangeLock(PessimisticTransaction* txn, > > // Use the txnid as "extra" in the lock_request. Then, KillLockWait() > // will be able to use kill_waiter(txn_id) to kill the wait if needed > + // TODO: KillLockWait is gone and we are no longer using > + // locktree::kill_waiter call. Do we need this anymore? > TransactionID wait_txn_id = txn->GetID(); > > request.set(lt, (TXNID)txn, &start_key_dbt, &end_key_dbt, > @@ -1004,8 +1001,10 @@ int RangeLockMgr::compare_dbt_endpoints(__toku_db*, void *arg, > } > > > -RangeLockMgr::RangeLockMgr(TransactionDB* txn_db) : my_txn_db(txn_db) { > - ltm.create(on_create, on_destroy, on_escalate, NULL); > +RangeLockMgr::RangeLockMgr(TransactionDB* txn_db, > + std::shared_ptr<TransactionDBMutexFactory> mutex_factory) : > + my_txn_db(txn_db), mutex_factory_(mutex_factory) { > + ltm.create(on_create, on_destroy, on_escalate, NULL, mutex_factory_); > lt= nullptr; > cmp_initialized_= false; > } > diff --git a/utilities/transactions/transaction_lock_mgr.h b/utilities/transactions/transaction_lock_mgr.h > index 0c3284519..9f73e5a07 100644 > --- a/utilities/transactions/transaction_lock_mgr.h > +++ b/utilities/transactions/transaction_lock_mgr.h > @@ -215,11 +215,10 @@ class RangeLockMgr : > void UnLock(PessimisticTransaction* txn, uint32_t column_family_id, > const std::string& key, Env* env) override ; > > - RangeLockMgr(TransactionDB* txn_db); > + RangeLockMgr(TransactionDB* txn_db, > + std::shared_ptr<TransactionDBMutexFactory> mutex_factory); > ~RangeLockMgr(); > > - void KillLockWait(TransactionID txnid); > - > int set_max_lock_memory(size_t max_lock_memory) override > { > return ltm.set_max_lock_memory(max_lock_memory); > @@ -251,6 +250,7 @@ class RangeLockMgr : > compare_endpoints_func compare_endpoints; > > TransactionDB* my_txn_db; > + std::shared_ptr<TransactionDBMutexFactory> mutex_factory_; > > static int compare_dbt_endpoints(__toku_db*, void *arg, const DBT *a_key, const DBT *b_key); > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

2 1

Re: [Maria-developers] [Commits] test
by Roberto Spadim 11 Mar '19

11 Mar '19

noise ignored =) Em seg, 11 de mar de 2019 às 15:11, Faustin Lammler <faustin(a)mariadb.org> escreveu: > Hi, > this is a test, please do not respond. > > And sorry for the noise. > > -- > Faustin Lammler > MariaDB Foundation > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- Roberto Spadim SPAEmpresarial - Software ERP Eng. Automação e Controle

1 0

[Maria-developers] MariaDB-10.4 / Galera and GTIDs - aka we now have two different GTID implementations
by Daniel Black 04 Mar '19

04 Mar '19

Hi, I was excited to come across the Galera-4 release notes and blog. http://galeracluster.com/2019/02/galera-cluster-4-available-for-use-in-the-… It was good to see causal read functions that I'd hoped for in https://jira.mariadb.org/browse/MDEV-10715. WSREP_LAST_WRITTEN_GTID/WSREP_LAST_SEEN_GTID/ WSREP_SYNC_WAIT_UPTO_GTID functions which seemed to mirror existing functions like MASTER_GTID_WAIT and GTID variables. The reason these are different names is because a MariaDB GTID is different from a Galera one. *Having two GTID implementations is insane*. To state the obvious from a user point of view, we've now got two Global Transaction Identification implementations. Application and framework writers like Django/Drupal now have to handle now try to handle consistently with 00000000-0000-0000-0000-000000000000:25 for Galera and 0-1-33 for MariaDB async gtid replication. This is not just a format string difference. The an implementation requires a disparate set of functions to implement when they could of been consolidated. A mixed topology setup could be hidden from the application implementation if the same functions where used. There is/was also a option/proposal that had MariaDB gtid in the protocol save a round trip retrieval of a GTID - quite useful. There is also a bunch of documentation that refers to GTID and its going to absolutely horrible to try to keep the concepts separated as you try to configure a system. An issue like MDEV-10715, the most voted for and watched open issue in JIRA, (and related MDEV-14153), should of had a very priority than this when the implementation from Codership was been merged. This shoehorned in implementation from Codership is a very poor fit. Giving MDEV-10715 a critical priority and a 10.5 target is fiction when there isn't going to be a new galera wsrep api implementation for a number of years. An extra field in the protocol is all it would of taken (comment in MDEV-10715 from 8 months ago). 10.4.3 is out (really an RC?), the galera code got merge in 10.4.2, and the source code of galera-4 isn't even released. Hopefully the tight relationship between MariaDB Corporation and Codership is enough to get this fixed because from an architectural point of view, because the current implementation shouldn't of been merged (contractual arrangement or not). Developers, MariaDB and Codership you've lost track of your user base because the feedback, that is readily available, is ignored. Sadly, Daniel

2 1

Re: [Maria-developers] [Commits] ce29dfac72e: MDEV-18601: Can't create table with ENCRYPTED=DEFAULT when innodb_default_encryption_key_id!=1
by Marko Mäkelä 28 Feb '19

28 Feb '19

Greetings from MariaDB Openworks in New York City. I just discussed this with Monty. His idea is that "don’t care" attributes should be ignored, or at most a warning could be issued for them. This allows ALTER TABLE from one storage engine to another while preserving attributes that only matter for some storage engines. Based on that basic principle, we should ignore the encryption_key_id attribute (whether or not it was explicitly specified by the user, or inherited from the session variable innodb_default_encryption_key_id). We could issue a warning if innodb_default_encryption_key_id is assigned to an invalid value, or an invalid encryption_key_id is being implicitly or explicitly passed to CREATE TABLE or ALTER TABLE. On Thu, Feb 21, 2019 at 2:53 PM jan <jan.lindstrom(a)mariadb.com> wrote: [snip] > diff --git a/mysql-test/suite/encryption/r/innodb-encryption-alter.result b/mysql-test/suite/encryption/r/innodb-encryption-alter.result > index 5245d1da7d0..038e2a2fd47 100644 > --- a/mysql-test/suite/encryption/r/innodb-encryption-alter.result > +++ b/mysql-test/suite/encryption/r/innodb-encryption-alter.result > @@ -3,24 +3,18 @@ SET GLOBAL innodb_file_per_table = ON; > SET GLOBAL innodb_encrypt_tables = ON; > SET GLOBAL innodb_encryption_threads = 4; > CREATE TABLE t1 (pk INT PRIMARY KEY AUTO_INCREMENT, c VARCHAR(256)) ENGINE=INNODB ENCRYPTED=NO ENCRYPTION_KEY_ID=4; > -Warnings: > -Warning 140 InnoDB: Ignored ENCRYPTION_KEY_ID 4 when encryption is disabled > DROP TABLE t1; > set innodb_default_encryption_key_id = 99; > +Warnings: > +Warning 1210 InnoDB: Ignored innodb_default_encryption_key_id=99 as it is not available in the key file. Using default=1. I think that the warning during CREATE TABLE was appropriate and should not be removed. A warning for the SET is appropriate, but I do not like the "Using default=1" part. I think that we should keep the invalid attribute, and throw an error or warning during CREATE TABLE as appropriate. > CREATE TABLE t1 (pk INT PRIMARY KEY AUTO_INCREMENT, c VARCHAR(256)) ENGINE=INNODB; > -ERROR HY000: Can't create table `test`.`t1` (errno: 140 "Wrong create options") > SHOW WARNINGS; > Level Code Message > -Warning 140 InnoDB: ENCRYPTION_KEY_ID 99 not available > -Error 1005 Can't create table `test`.`t1` (errno: 140 "Wrong create options") > -Warning 1030 Got error 140 "Wrong create options" from storage engine InnoDB I believe that this CREATE TABLE should continue to be disallowed, because innodb_encrypt_tables=ON and innodb_default_encryption_key_id=9 should imply ENCRYPTED=YES and ENCRYPTION_KEY_ID=99, and the key 99 is not available. Even if you think that it should be allowed (I find it hard to agree with that), I believe that we should at least be issuing warnings. [snip] > SET GLOBAL innodb_encrypt_tables=OFF; > CREATE TABLE t1 (a int not null primary key) engine=innodb; > ALTER TABLE t1 ENCRYPTION_KEY_ID=4; > -ERROR HY000: Table storage engine 'InnoDB' does not support the create option 'ENCRYPTION_KEY_ID' > -SHOW WARNINGS; > -Level Code Message > -Warning 140 InnoDB: innodb_encrypt_tables=OFF only allows ENCRYPTION_KEY_ID=1 > -Error 1478 Table storage engine 'InnoDB' does not support the create option 'ENCRYPTION_KEY_ID' Here, it is correct that we should not throw a warning, because innodb_encrypt_tables=OFF causes the encryption_key_id to not matter. But we could issue a warning if ENCRYPTION_KEY_ID is not available. I am going to push a revised patch to the bb-10.1-marko branch and ask for your review. With best regards, Marko -- Marko Mäkelä, Lead Developer InnoDB MariaDB Corporation

2 2

Re: [Maria-developers] 0608c273598: MDEV-15966: Behavior for TRUNCATE versioned table is not documented and not covered by tests
by Sergei Golubchik 28 Feb '19

28 Feb '19

Hi, Nikita! On Dec 26, Nikita Malyavin wrote: > revision-id: 0608c273598 (versioning-1.0.5-112-g0608c273598) > parent(s): 49d506cd1d7 > author: Nikita Malyavin <nikitamalyavin(a)gmail.com> > committer: Nikita Malyavin <nikitamalyavin(a)gmail.com> > timestamp: 2018-09-05 04:42:17 +1000 > message: > > MDEV-15966: Behavior for TRUNCATE versioned table is not documented and not covered by tests > > * MDEV-15966: Behavior for TRUNCATE versioned table is not documented and not covered by tests > > * add error for truncation of versioned tables: `ER_TRUNCATE_ILLEGAL_VERS` > * extract reading `extra2` section in function `dd_read_extra2` > * update `dd_frm_type` and `ha_table_exists` signatures: add new parameter `is_versioned` > > test suites run: main, parts, versioning > > [fixes tempesta-tech/mariadb#261] I don't quite like that a small and simple function dd_frm_type(), which used to just read few first bytes of the frm file, gets more and more complex, growing into a complete frm-open method. So, perhaps, I'd consider changing TRUNCATE to open the table properly without all that just-read-few-bytes shortcuts. Or, may be, simply document that TRUNCATE works on versioning tables just as DROP+CREATE does. There is no logical reason why it shouldn't - if one can do SHOW CREATE TABLE and CREATE OR REPLACE, then TRUNCATE won't add any new functionality on top of that. That's the easiest and most logical "fix" to this bug. Agree? Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 5

Re: [Maria-developers] df203d1daa1: MDEV-15990: REPLACE on a precise-versioned table returns duplicate key error (ER_DUP_ENTRY)
by Sergei Golubchik 28 Feb '19

28 Feb '19

Hi, Nikita! Sorry, I don't understand what are you fixing and why :( Could you please elaborate? See few minor comments below and the main question at the very end. On Dec 26, Nikita Malyavin wrote: > revision-id: df203d1daa1 (versioning-1.0.5-42-gdf203d1daa1) > parent(s): 5e89a0dc25a > author: Nikita Malyavin <nikitamalyavin(a)gmail.com> > committer: Nikita Malyavin <nikitamalyavin(a)gmail.com> > timestamp: 2018-07-21 00:14:57 +1000 > message: > > MDEV-15990: REPLACE on a precise-versioned table returns duplicate key error (ER_DUP_ENTRY) > > * handle zero value in `row_start` `trx_id` > * refuse to do more than one versioned insert on the same transaction and same row > * generalize versioned cases with and without triggers > > [closes tempesta-tech/mariadb#517] > [closes tempesta-tech/mariadb#520] > > diff --git a/mysql-test/suite/versioning/r/replace.result b/mysql-test/suite/versioning/r/replace.result > --- a/mysql-test/suite/versioning/r/replace.result > +++ b/mysql-test/suite/versioning/r/replace.result > @@ -7,10 +7,31 @@ period for system_time (row_start, row_end) > ) with system versioning; > insert t values (1, 2); > replace t values (1, 3); > -select *, current_row(row_end) as current from t for system_time all order by x; > -id x current > -1 2 0 > -1 3 1 > +replace t values (1, 3); > +# MDEV-15990: REPLACE on a precise-versioned table > +# returns duplicate key error (ER_DUP_ENTRY) > +replace t values (1, 3), (1, 30), (1, 40); > +create table log(s varchar(3)); > +create trigger tr1del before delete on t > +for each row insert into log values('DEL'); > +replace t values (1, 4), (1, 40), (1, 400); > +select *, check_row(row_start, row_end), row_start != 0 as start_nonzero > +from t for system_time all order by x, row_end; > +id x check_row(row_start, row_end) start_nonzero > +1 2 HISTORICAL ROW 1 > +1 3 HISTORICAL ROW 1 > +1 3 HISTORICAL ROW 1 > +1 40 HISTORICAL ROW 1 > +1 400 CURRENT ROW 1 > +select * from log; > +s > +DEL > +DEL > +DEL better to insert OLD.x too, to be able to see where the DEL comes from > +# ensure that row_start is not duplicated > +select count(row_start) as empty from t for system_time all > +group by row_start having count(row_start) > 1; > +empty > drop table t; > create table t ( > id int unique, > diff --git a/sql/sql_insert.cc b/sql/sql_insert.cc > --- a/sql/sql_insert.cc > +++ b/sql/sql_insert.cc > @@ -1969,23 +1952,22 @@ int write_record(THD *thd, TABLE *table,COPY_INFO *info) > TRG_ACTION_BEFORE, TRUE)) > goto before_trg_err; > > - if (!table->versioned(VERS_TIMESTAMP)) > + if (table->versioned(VERS_TRX_ID) && table->vers_write) > + { can table->vers_write be false here? I suspect it can never be false for REPLACE, so I'd write it as if (table->versioned(VERS_TRX_ID)) { DBUG_ASSERT(table->vers_write); > + bitmap_set_bit(table->write_set, table->vers_start_field()->field_index); > + table->vers_start_field()->store(0, false); > + } > + > + if (!table->versioned()) > error= table->file->ha_delete_row(table->record[1]); > else > - { > - store_record(table, record[2]); > - restore_record(table, record[1]); > - table->vers_update_end(); > error= table->file->ha_update_row(table->record[1], > table->record[0]); > - restore_record(table, record[2]); > - } > - if (unlikely(error)) > + if (unlikely(error && error != HA_ERR_RECORD_IS_THE_SAME)) > goto err; > - if (!table->versioned(VERS_TIMESTAMP)) > + if (likely(!error)) > info->deleted++; > - else > - info->updated++; > + error= 0; > if (!table->file->has_transactions()) > thd->transaction.stmt.modified_non_trans_table= TRUE; > if (table->triggers && > @@ -1995,6 +1977,17 @@ int write_record(THD *thd, TABLE *table,COPY_INFO *info) > trg_error= 1; > goto ok_or_after_trg_err; > } > + if (table->versioned(VERS_TIMESTAMP) && table->vers_write) same > + { > + store_record(table, record[2]); > + error = vers_insert_history_row(table); why are you doing that (insert + goto after_trg_n_copied_inc) instead of going the normal REPLACE code path? > + restore_record(table, record[2]); > + if (unlikely(error)) > + goto err; > + } > + if (table->versioned()) > + goto after_trg_n_copied_inc; > + > /* Let us attempt do write_row() once more */ > } > } > diff --git a/storage/innobase/handler/ha_innodb.cc b/storage/innobase/handler/ha_innodb.cc > index db755077e0b..139c43ced38 100644 > --- a/storage/innobase/handler/ha_innodb.cc > +++ b/storage/innobase/handler/ha_innodb.cc > @@ -8916,6 +8916,19 @@ ha_innobase::update_row( > upd_t* uvect = row_get_prebuilt_update_vector(m_prebuilt); > ib_uint64_t autoinc; > > + bool force_insert= table->versioned_write(VERS_TRX_ID) > + && table->vers_start_id() == 0; hmm, so you're using vers_start_id() == 0 as a special hack to signal REPLACE to the storage engine... > + > + if (force_insert) { > + table->vers_start_field()->store(trx->id); > + > + Field *start= table->vers_start_field(); > + trx_id_t old_trx_id= uint8korr(start->ptr_in_record(old_row)); > + > + /* avoid double versioned insert on the same transaction and same row */ > + force_insert= (old_trx_id != trx->id); okay, I'm confused. What was wrong in the old code? Normally REPLACE is DELETE + INSERT, why was it not enough for InnoDB and you needed some special logic here to handle the REPLACE case? > + } > + > /* Build an update vector from the modified fields in the rows > (uses m_upd_buf of the handle) */ > Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 1

Re: [Maria-developers] 773d24d457a: MDEV-15951 system versioning by trx id doesn't work with partitioning
by Sergei Golubchik 28 Feb '19

28 Feb '19

Hi, Nikita! On Dec 25, Nikita Malyavin wrote: > revision-id: 773d24d457a (versioning-1.0.6-100-g773d24d457a) > parent(s): b26736cdb11 > author: Nikita Malyavin <nikitamalyavin(a)gmail.com> > committer: Nikita Malyavin <nikitamalyavin(a)gmail.com> > timestamp: 2018-12-21 03:13:47 +1000 > message: > > MDEV-15951 system versioning by trx id doesn't work with partitioning > > Fix partitioning for trx_id-versioned tables. > `partition by hash`, `range` and others now work. > `partition by system_time` is forbidden. > Currently we cannot use row_start and row_end in `partition by`, because > insertion of versioned field is done by engine's handler, as well as > row_start/row_end's value set up, which is a transaction id -- so it's > also forbidden. > > The drawback is that it's now impossible to use `partition by key()` > without parameters for such tables, because it references row_start and > row_end implicitly. > > * add handler::vers_can_native() > * drop Table_scope_and_contents_source_st::vers_native() > * drop partition_element::find_engine_flag as unused > * forbid versioning partitioning for trx_id as not supported > * adopt vers tests for trx_id partitioning > * forbid any row_end referencing in `partition by` clauses, > including implicit `by key()` > > diff --git a/sql/sql_partition.cc b/sql/sql_partition.cc > index 9e6c333d3c9..48bcb828d56 100644 > --- a/sql/sql_partition.cc > +++ b/sql/sql_partition.cc > @@ -341,7 +341,16 @@ static bool set_up_field_array(THD *thd, TABLE *table, > while ((field= *(ptr++))) > { > if (field->flags & GET_FIXED_FIELDS_FLAG) > + { > + if (table->versioned(VERS_TRX_ID) > + && unlikely(field->flags & VERS_SYSTEM_FIELD)) > + { > + my_error(ER_VERS_NOT_SUPPORTED, MYF(0), > + "Using ROW START/END for patitioning", "transactional"); This is incorrect usage of error message parameters. The error message can be translated, parameters can be reordered - you should not use English as parameter values. Consider: transactional cистемно-версионированные таблицы не поддерживают Using ROW START/END for patitioning But looks like ER_VERS_NOT_SUPPORTED is not used anywhere at the moment. So you can change it to be anything you want, even remove all parameters: Transactional system versioned tables do not support partitioning by ROW START or ROW END > + DBUG_RETURN(TRUE); > + } > num_fields++; > + } > } > if (unlikely(num_fields > MAX_REF_PARTS)) > { > diff --git a/sql/table.cc b/sql/table.cc > index ce7a34a8fe2..0c2ff1eea5a 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -1776,7 +1776,7 @@ int TABLE_SHARE::init_from_binary_frm_image(THD *thd, bool write, > goto err; > DBUG_PRINT("info", ("Columns with system versioning: [%d, %d]", row_start, row_end)); > versioned= VERS_TIMESTAMP; > - vers_can_native= plugin_hton(se_plugin)->flags & HTON_NATIVE_SYS_VERSIONING; > + vers_can_native= handler_file->vers_can_native(thd); Interesting. How does that work? Your ha_partition::vers_can_native() is bool can= !thd->lex->part_info || thd->lex->part_info->part_type != VERSIONING_PARTITION; for (uint i= 0; i < m_tot_parts && can; i++) can= can && m_file[i]->vers_can_native(thd); return can; But it doesn't look like thd->lex->part_info->part_type is initialized at this point in time. Try to add to your test case flush tables; select * from t1; Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 2

Re: [Maria-developers] bc7d600a27f: MDEV-15951 system versioning by trx id doesn't work with partitioning
by Sergei Golubchik 28 Feb '19

28 Feb '19

Hi, Nikita! On Dec 21, Nikita Malyavin wrote: > > > @@ -4938,6 +4938,7 @@ int ha_create_table(THD *thd, const char *path, > > > !create_info->tmp_table(); > > > > > > share.frm_image= frm; > > > + share.orig_table_name= table_name; > > > > why? if orig_table_name is NULL, error_table_name() will use table_name, > > so there's never need to make orig_table_name==table_name. > > > > and I don't understand all your other manipulations with > > orig_table_name. Could you please explain that? > > It affects ALTER TABLE error messages. table_name will contain temporary name, > and error_table_name() will output "(temporary)", which is rather inconvenient. > AFAICS, orig_table_name purpose is exactly for fixing that, but some > corner cases were not covered to this moment. I don't understand that. You've made a lot of changes in many places passing orig table name around, it's difficult to see what corner cases you were trying to fix. Could you please fix it in a separate commit? With tests and an explanation in a commit comment, describing what exactly corner cases were not covered. > > > // open an frm image > > > if (share.init_from_binary_frm_image(thd, write_frm_now, Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 1

[Maria-developers] Exposing Item::store_native and val_native to the SQL layer
by Eric Herman 25 Feb '19

25 Feb '19

Hello Bar, As you have done the work on store_native and val_native, I would especially like to know your thoughts with regards to how to expose these to the SQL layer. In my particular case, I'd like to re-work the "Add FLOAT and DOUBLE data interchange functions" patch ( https://github.com/ericherman/mariadb-server/commit/dcc7ccf65689bd7d84bef37… ) to use these new API functions. I've only just started on this task ( https://github.com/ericherman/mariadb-server/tree/10.4-eherman-float-native ) and first thing that I need to figure out is how to write the tests. In commit 34eb98387f8f46a80fb053081dbe20d415f23b39 ( MDEV-13995 MAX(timestamp) returns a wrong result near DST change ), I see that in tests the SQL calls various time functions which ultimately call into these functions, yet I do not see any way to access these two new functions directly. Are you thinking of adding new SQL functions to expose the underlying bytes to the user? If not, I have a sense that casting to and from BINARY could expose the native byte arrays, but I'm not sure if that's the direction you imagine. If you tell me which direction to go, I will try to make that happen. Cheers, -Eric Other foo_native related commits: * commit a8a757c6bb32bbf291afdf33df861127489889ab wl#173 - temporal types with sub-second resolution * commit c353b2a8fc10e16107ee6c7e26877f4243d4eaef MDEV-17979 Assertion `0' failed in Item::val_native upon SELECT with timestamp, NULLIF, GROUP BY --

2 1

Re: [Maria-developers] [Commits] dadf5f78411: Fix few compiler warnings.
by Marko Mäkelä 20 Feb '19

20 Feb '19

On February 20, 2019, Jan Lindström wrote: > revision-id: dadf5f78411b8f46ac53b8ea12c08ec317d9aab4 (mariadb-10.1.38-7-gdadf5f78411) > parent(s): 346e46089621e6951e076c82ed5690aa23dcb5fe > author: Jan Lindström > committer: Jan Lindström > timestamp: 2019-02-20 08:10:26 +0200 > message: > > Fix few compiler warnings. The commit message should include the ticket number. I would also suggest fixing the grammar: MDEV-18659: Fix a few compiler warnings Could you make the title more specific? > dict_table_rename_in_cache > dict_create_add_foreign_id > Use of strncpy caused string truncation warnings and is > replaced with memcpy. Specifically, which warnings were issued? I am concerned that this could be replacing dubious code with worse. With memcpy(), you should be sure that the source strings is really safe to access up to the specified length. With strncpy(), the copying will stop at the first NUL character in the output, or at the specified length, whichever comes first. > fts_fetch_index_words > dfield_get_len() can return UNIV_SQL_NULL (=ULINT32_UNDEFINED) > that is too big for used short datatype. Replaced with correct > ulint. Is NULL really an allowed value there? I do not think that the "words" or "tokens" (the keys of the inverted index) should ever be NULL. Likewise, if the entire fulltext-indexed column is NULL, we should not be inserting anything into the inverted index. Please ensure that these cases are covered by test cases in mysql-test-run, and share the detailed results: especially, which pieces of code are handling the NULL values. The data that is being fed to fts_fetch_index_words() comes from this query to the InnoDB internal SQL parser: " SELECT word\n" " FROM $table_name\n" " WHERE word > :word\n" " ORDER BY word;\n" These tables appear to be created in fts_create_one_index_table(), and it seems to me that it is PRIMARY KEY(word,first_doc_id). Strangely, I do not see DATA_NOT_NULL in the column creation, so it is possible that we are reserving NULL flags for the "word" column. But, InnoDB is not prepared to deal with NULL values in the primary key. Lots of things would break if "word" were NULL. Please add assertions to the places where "word" is being written or read in these tables. Here is the call that forgets to declare "word" as NOT NULL: dict_mem_table_add_col(new_table, heap, "word", charset == &my_charset_latin1 ? DATA_VARCHAR : DATA_VARMYSQL, field->col->prtype, FTS_MAX_WORD_LEN_IN_CHAR * unsigned(field->col->mbmaxlen)); > row_rename_table_for_mysql > Target string was exactly same length as maximum length > of copied string. Fixing this does make sense. > diff --git a/storage/innobase/dict/dict0dict.cc b/storage/innobase/dict/dict0dict.cc > index 06c6c3effab..04b14e3ae06 100644 > --- a/storage/innobase/dict/dict0dict.cc > +++ b/storage/innobase/dict/dict0dict.cc > @@ -1845,7 +1845,7 @@ dict_table_rename_in_cache( > > ulint db_len; > char* old_id; > - char old_name_cs_filename[MAX_TABLE_NAME_LEN+20]; > + char old_name_cs_filename[MAX_TABLE_NAME_LEN+20]=""; > uint errors = 0; > > /* All table names are internally stored in charset What is the initialization needed for? It was not mentioned in the commit comment. Note that this would likely not only initialize the first byte of the string, but all bytes. What kind of machine code is this generating? memcpy()? Or is the compiler smart enough to recognize that the stack variable is going to be zero-filled, and will emit something like memset() instead? Either way, it is not good to write MAX_TABLE_NAME_LEN+20 bytes if only 1 byte (a terminating NUL character at the start of the array) would suffice. And you would have to convince me that this initialization is necessary in the first place. Best regards, Marko Mäkelä Lead Developer InnoDB MariaDB Corporation

1 0

Re: [Maria-developers] 46734eb1632: MDEV-371 long unique
by Sergei Golubchik 19 Feb '19

19 Feb '19

Hi, Sachin! On Feb 14, Sachin Setiya wrote: > revision-id: 46734eb1632 (mariadb-10.4.1-97-g46734eb1632) > parent(s): acf4269a5b1 > author: Sachin <sachin.setiya(a)mariadb.com> > committer: Sachin <sachin.setiya(a)mariadb.com> > timestamp: 2019-02-12 19:01:17 +0530 > message: > diff --git a/mysql-test/main/long_unique.result b/mysql-test/main/long_unique.result > --- /dev/null > +++ b/mysql-test/main/long_unique.result > @@ -0,0 +1,1391 @@ please add a test with 64 indexes. Say, one long unique and 63 normal indexes on ints or whatever. The point is to show that long unique creates internally only _one_ index, not two, so it does not limit user's ability to create all MAX_INDEXES indexes > +#Structure of tests > +#First we will check all option for > +#table containing single unique column > +#table containing keys like unique(a,b,c,d) etc > +#then table containing 2 blob unique etc > +set @allowed_packet= @@max_allowed_packet; > +#table with single long blob column; > +create table t1(a blob unique ); > +insert into t1 values(1),(2),(3),(56),('sachin'),('maria'),(123456789034567891),(null),(null),(123456789034567890); > +#blob with primary key not allowed > +create table t2(a blob,primary key(a(10000))); > +ERROR 42000: Specified key was too long; max key length is 1000 bytes > +create table t3(a varchar(10000) primary key); > +ERROR 42000: Specified key was too long; max key length is 1000 bytes > +insert into t1 values(2); > +ERROR 23000: Duplicate entry '2' for key 'a' > +#table structure; > +desc t1; > +Field Type Null Key Default Extra > +a blob YES UNI NULL > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + UNIQUE KEY `a` (`a`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table t1 > +Non_unique 0 > +Key_name a > +Seq_in_index 1 > +Column_name a > +Collation A > +Cardinality NULL > +Sub_part NULL > +Packed NULL > +Null YES > +Index_type HASH > +Comment > +Index_comment > + > +MyISAM file: DATADIR/test/t1 > +Record format: Packed > +Character set: latin1_swedish_ci (8) > +Data records: 10 Deleted blocks: 0 > +Recordlength: 12 > + > +table description: > +Key Start Len Index Type > +1 12 8 multip. ulonglong NULL > +select * from information_schema.columns where table_schema = 'test' and table_name = 't1'; > +TABLE_CATALOG TABLE_SCHEMA TABLE_NAME COLUMN_NAME ORDINAL_POSITION COLUMN_DEFAULT IS_NULLABLE DATA_TYPE CHARACTER_MAXIMUM_LENGTH CHARACTER_OCTET_LENGTH NUMERIC_PRECISION NUMERIC_SCALE DATETIME_PRECISION CHARACTER_SET_NAME COLLATION_NAME COLUMN_TYPE COLUMN_KEY EXTRA PRIVILEGES COLUMN_COMMENT IS_GENERATED GENERATION_EXPRESSION > +def test t1 a 1 NULL YES blob 65535 65535 NULL NULL NULL NULL NULL blob UNI select,insert,update,references NEVER NULL > +select * from information_schema.statistics where table_schema = 'test' and table_name = 't1'; > +TABLE_CATALOG TABLE_SCHEMA TABLE_NAME NON_UNIQUE INDEX_SCHEMA INDEX_NAME SEQ_IN_INDEX COLUMN_NAME COLLATION CARDINALITY SUB_PART PACKED NULLABLE INDEX_TYPE COMMENT INDEX_COMMENT > +def test t1 0 test a 1 a A NULL NULL NULL YES HASH > +select * from information_schema.key_column_usage where table_schema= 'test' and table_name= 't1'; > +CONSTRAINT_CATALOG def > +CONSTRAINT_SCHEMA test > +CONSTRAINT_NAME a > +TABLE_CATALOG def > +TABLE_SCHEMA test > +TABLE_NAME t1 > +COLUMN_NAME a > +ORDINAL_POSITION 1 > +POSITION_IN_UNIQUE_CONSTRAINT NULL > +REFERENCED_TABLE_SCHEMA NULL > +REFERENCED_TABLE_NAME NULL > +REFERENCED_COLUMN_NAME NULL > +# table select we should not be able to see db_row_hash_column; > +select * from t1 order by a; > +a > +NULL > +NULL > +1 > +123456789034567890 > +123456789034567891 > +2 > +3 > +56 > +maria > +sachin > +select db_row_hash_1 from t1; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 'field list' > +#duplicate entry test; > +insert into t1 values(2); > +ERROR 23000: Duplicate entry '2' for key 'a' > +insert into t1 values('sachin'); > +ERROR 23000: Duplicate entry 'sachin' for key 'a' > +insert into t1 values(123456789034567891); > +ERROR 23000: Duplicate entry '123456789034567891' for key 'a' > +select * from t1 order by a; > +a > +NULL > +NULL > +1 > +123456789034567890 > +123456789034567891 > +2 > +3 > +56 > +maria > +sachin > +insert into t1 values(11),(22),(33); > +insert into t1 values(12),(22); > +ERROR 23000: Duplicate entry '22' for key 'a' > +select * from t1 order by a; > +a > +NULL > +NULL > +1 > +11 > +12 > +123456789034567890 > +123456789034567891 > +2 > +22 > +3 > +33 > +56 > +maria > +sachin > +insert into t1 values(repeat('s',4000*10)),(repeat('s',4001*10)); > +insert into t1 values(repeat('m',4000*10)),(repeat('m',4000*10)); > +ERROR 23000: Duplicate entry 'mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm' for key 'a' > +insert into t1 values(repeat('m',4001)),(repeat('m',4002)); > +truncate table t1; > +insert into t1 values(1),(2),(3),(4),(5),(8),(7); > + > +MyISAM file: DATADIR/test/t1 > +Record format: Packed > +Character set: latin1_swedish_ci (8) > +Data records: 7 Deleted blocks: 0 > +Recordlength: 12 > + > +table description: > +Key Start Len Index Type > +1 12 8 multip. ulonglong NULL > +#now some alter commands; > +alter table t1 add column b int; > +desc t1; > +Field Type Null Key Default Extra > +a blob YES UNI NULL > +b int(11) YES NULL > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + `b` int(11) DEFAULT NULL, > + UNIQUE KEY `a` (`a`(65535)) hmm, again, (65535) appeared from nowhere > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +insert into t1 values(1,2); > +ERROR 23000: Duplicate entry '1' for key 'a' > +insert into t1 values(2,2); > +ERROR 23000: Duplicate entry '2' for key 'a' > +select db_row_hash_1 from t1; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 'field list' > +#now try to change db_row_hash_1 column; > +alter table t1 drop column db_row_hash_1; > +ERROR 42000: Can't DROP COLUMN `db_row_hash_1`; check that it exists > +alter table t1 add column d int , add column e int , drop column db_row_hash_1; ... > +insert into t2 values(1,1); > +insert into t2 values(2,1); > +drop table t2; > +#not null test //todo solve warnings is that solved? I don't see any warnings > +create table t1(a blob unique not null); > +desc t1; > +Field Type Null Key Default Extra > +a blob NO UNI NULL > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob NOT NULL, > + UNIQUE KEY `a` (`a`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +insert into t1 values(1); > +insert into t1 values(3); > +insert into t1 values(1); > +ERROR 23000: Duplicate entry '1' for key 'a' > +drop table t1; > +create table t1(a int primary key, b blob unique , c blob unique not null); > +insert into t1 values(1,1,1); > +insert into t1 values(2,1,2); > +ERROR 23000: Duplicate entry '1' for key 'b' > +insert into t1 values(3,3,1); > +ERROR 23000: Duplicate entry '1' for key 'c' > +drop table t1; > +create table t1 (a blob unique not null, b blob not null, c blob not null, unique(b,c)); > +desc t1; > +Field Type Null Key Default Extra > +a blob NO UNI NULL > +b blob NO MUL NULL > +c blob NO NULL > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob NOT NULL, > + `b` blob NOT NULL, > + `c` blob NOT NULL, > + UNIQUE KEY `a` (`a`), > + UNIQUE KEY `b` (`b`,`c`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +insert into t1 values (1, 2, 3); > +insert into t1 values (2, 1, 3); > +insert into t1 values (2, 1, 3); > +ERROR 23000: Duplicate entry '2' for key 'a' > +drop table t1; > +#partition > +create table t1(a blob unique) partition by hash(a); > +ERROR HY000: A BLOB field is not allowed in partition function > +#key length > 2^15 -1 > +create table t1(a blob, unique(a(100001))); > +ERROR 42000: Specified key was too long; max key length is 1000 bytes better to say "max length is 32767 bytes" or, may be, a new error message "key segment length is 32767 bytes"? > +set @@GLOBAL.max_allowed_packet= @allowed_packet; > diff --git a/mysql-test/main/long_unique_debug.result b/mysql-test/main/long_unique_debug.result > --- /dev/null > +++ b/mysql-test/main/long_unique_debug.result > @@ -0,0 +1,574 @@ > +#In this test case we will check what will happen in the case of hash collision > +SET debug_dbug="d,same_long_unique_hash"; > +create table t1(a blob unique); > +insert into t1 values('xyz'); > +insert into t1 values('abc'); > +insert into t1 values('sachin'); > +insert into t1 values('sachin'); > +ERROR 23000: Duplicate entry 'sachin' for key 'a' > +insert into t1 values('maria'); > +insert into t1 values('maria'); > +ERROR 23000: Duplicate entry 'maria' for key 'a' I think you can use SHOW STATUS LIKE 'Handler_read_key' (or some other Handler_% variable) to verify here that there was indeed a hash collision - that is, this variable counts row reads and you have an extra row read here, so it should be visible. Try: add FLUSH STATUS at the beginning of the test, SHOW STATUS LIKE 'Handler_read_key' after errors and run the test with and without d,same_long_unique_hash if there'll be a difference, let's keep SHOW STATUS in the test. > +drop table t1; > +create table t1(a blob unique, b blob unique); > +insert into t1 values('xyz', 11); > +insert into t1 values('abc', 22); > +insert into t1 values('sachin', 1); > +insert into t1 values('sachin', 4); > +ERROR 23000: Duplicate entry 'sachin' for key 'a' > +insert into t1 values('maria', 2); > +insert into t1 values('maria', 3); > +ERROR 23000: Duplicate entry 'maria' for key 'a' > +drop table t1; > +create table t1(a blob , b blob , unique(a,b)); > +insert into t1 values('xyz', 11); > +insert into t1 values('abc', 22); > +insert into t1 values('sachin', 1); > +insert into t1 values('sachin', 1); > +ERROR 23000: Duplicate entry 'sachin-1' for key 'a' > +insert into t1 values('maria', 2); > +insert into t1 values('maria', 2); > +ERROR 23000: Duplicate entry 'maria-2' for key 'a' > +drop table t1; > +##Internal State of long unique tables > +SET debug_dbug="d,print_long_unique_internal_state"; > +create table t1 ( a blob unique); > +Warnings: > +Note 1 Printing Table state, It will print table fields, fields->offset,field->null_bit, field->null_pos and key_info ... > + > +Printing Table keyinfo > + > +table->s->reclength 19 > +table->s->fields 2 Rather unorthodox, but fine, as you like :) > diff --git a/mysql-test/main/long_unique_innodb.result b/mysql-test/main/long_unique_innodb.result > --- /dev/null > +++ b/mysql-test/main/long_unique_innodb.result > @@ -0,0 +1,121 @@ > +create table t1(a blob unique) engine= InnoDB; > +insert into t1 values('RUC'); > +insert into t1 values ('RUC'); > +ERROR 23000: Duplicate entry 'RUC' for key 'a' > +drop table t1; > +#test for concurrent insert of long unique in innodb > +create table t1(a blob unique) engine= InnoDB; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + UNIQUE KEY `a` (`a`) > +) ENGINE=InnoDB DEFAULT CHARSET=latin1 > +connect 'con1', localhost, root,,; > +connect 'con2', localhost, root,,; > +connection con1; > +set transaction isolation level READ UNCOMMITTED; > +start transaction; > +insert into t1 values('RUC'); > +connection con2; > +set transaction isolation level READ UNCOMMITTED; > +start transaction; > +insert into t1 values ('RUC'); > +ERROR 23000: Duplicate entry 'RUC' for key 'a' > +connection con1; > +commit; > +set transaction isolation level READ COMMITTED; > +start transaction; > +insert into t1 values('RC'); > +connection con2; > +commit; > +set transaction isolation level READ COMMITTED; > +start transaction; > +insert into t1 values ('RC'); > +ERROR 23000: Duplicate entry 'RC' for key 'a' Is this your HA_ERR_LOCK_WAIT_TIMEOUT that you replace with HA_ERR_FOUND_DUPP_KEY? I think HA_ERR_LOCK_WAIT_TIMEOUT is correct here. Compare with normal indexes, there should be HA_ERR_LOCK_WAIT_TIMEOUT with normal indexes too. Because the second INSERT should wait for the first to commit or rollback. And if the first doesn't neither, the second times out. Looks reasonable. > +commit; > +connection con1; > +commit; > +set transaction isolation level REPEATABLE READ; > +start transaction; > +insert into t1 values('RR'); > +connection con2; > +commit; > +set transaction isolation level REPEATABLE READ; > +start transaction; > +insert into t1 values ('RR'); > +ERROR 23000: Duplicate entry 'RR' for key 'a' Same here, I guess > +connection con1; > +commit; > +set transaction isolation level SERIALIZABLE; > +start transaction; > +insert into t1 values('S'); > +connection con2; > +commit; > +set transaction isolation level SERIALIZABLE; > +start transaction; > +insert into t1 values ('S'); > +ERROR 23000: Duplicate entry 'S' for key 'a' and here > +commit; > +connection con1; > +commit; > +select * from t1; > +a > +RUC > +RC > +RR > +S > +drop table t1; > +create table t1(a blob unique) engine=Innodb; > +connection con1; > +set transaction isolation level READ UNCOMMITTED; > +start transaction; > +insert into t1 values('RUC'); > +connection con2; > +set transaction isolation level READ UNCOMMITTED; > +start transaction; > +insert into t1 values ('RUC');; > +connection con1; > +rollback; > +connection con2; > +commit; > +connection con1; > +set transaction isolation level READ COMMITTED; > +start transaction; > +insert into t1 values('RC'); > +connection con2; > +set transaction isolation level READ COMMITTED; > +start transaction; > +insert into t1 values ('RC');; > +connection con1; > +rollback; > +connection con2; > +commit; > +connection con1; > +set transaction isolation level REPEATABLE READ; > +start transaction; > +insert into t1 values('RR'); > +connection con2; > +set transaction isolation level REPEATABLE READ; > +start transaction; > +insert into t1 values ('RR');; > +connection con1; > +rollback; > +connection con2; > +commit; > +connection con1; > +set transaction isolation level SERIALIZABLE; > +start transaction; > +insert into t1 values('S'); > +connection con2; > +set transaction isolation level SERIALIZABLE; > +start transaction; > +insert into t1 values ('S');; > +connection con1; > +rollback; > +connection con2; > +commit; > +connection default; > +drop table t1; > +disconnect con1; > +disconnect con2; > diff --git a/mysql-test/main/long_unique_innodb.test b/mysql-test/main/long_unique_innodb.test > --- /dev/null > +++ b/mysql-test/main/long_unique_innodb.test > @@ -0,0 +1,131 @@ > +--source include/have_innodb.inc > +--source include/big_test.inc instead of big_test, better set the innodb lock timeout to something small for the first half of your test. and please make sure all tests pass. This one doesn't even start for me now in your branch. > + > +create table t1(a blob unique) engine= InnoDB; > +insert into t1 values('RUC'); > +--error ER_DUP_ENTRY > +insert into t1 values ('RUC'); > +drop table t1; > + > +--echo #test for concurrent insert of long unique in innodb > +create table t1(a blob unique) engine= InnoDB; > +show create table t1; > +connect ('con1', localhost, root,,); > diff --git a/mysql-test/main/mdev-504.test b/mysql-test/main/mdev-504.test > --- a/mysql-test/main/mdev-504.test > +++ b/mysql-test/main/mdev-504.test > @@ -1,4 +1,3 @@ > ---source include/not_valgrind.inc and restore this line too, please. Or, if it make sense to remove it, make sure you'll push it in a separate commit, not in MDEV-371 > --source include/no_protocol.inc > > set @save_use_stat_tables=@@global.use_stat_tables; > diff --git a/sql/field.h b/sql/field.h > --- a/sql/field.h > +++ b/sql/field.h > @@ -553,7 +553,7 @@ class Virtual_column_info: public Sql_alloc > name.str= NULL; > name.length= 0; > }; > - ~Virtual_column_info() {} > + ~Virtual_column_info(); Hmm, I didn't find ~Virtual_column_info() destructor anywhere. Where is it now? > enum_vcol_info_type get_vcol_type() const > { > return vcol_type; > diff --git a/sql/item_func.h b/sql/item_func.h > --- a/sql/item_func.h > +++ b/sql/item_func.h > @@ -1019,6 +1019,19 @@ class Item_long_func: public Item_int_func > }; > > > +class Item_func_hash: public Item_int_func > +{ > +public: > + Item_func_hash(THD *thd, List<Item> &item): Item_int_func(thd, item) > + {} > + longlong val_int(); > + bool fix_length_and_dec(); > + const Type_handler *type_handler() const { return &type_handler_long; } > + Item *get_copy(THD *thd) > + { return get_item_copy<Item_func_hash>(thd, this); } > + const char *func_name() const { return "HASH"; } perhaps "<hash>" ? Like other internal items, e.g. <in_optimizer> or <rowid> > +}; > + > class Item_longlong_func: public Item_int_func > { > public: > diff --git a/sql/sql_show.cc b/sql/sql_show.cc > --- a/sql/sql_show.cc > +++ b/sql/sql_show.cc > @@ -2341,7 +2341,7 @@ int show_create_table(THD *thd, TABLE_LIST *table_list, String *packet, > } > } > packet->append(')'); > - store_key_options(thd, packet, table, key_info); > + store_key_options(thd, packet, table, &table->key_info[i]); why? table->s->key_info[i] doesn't show correct options? I've tried to revert this change and long_unique test passed. > if (key_info->parser) > { > LEX_CSTRING *parser_name= plugin_name(key_info->parser); > @@ -6586,15 +6586,20 @@ static int get_schema_stat_record(THD *thd, TABLE_LIST *tables, > table->field[8]->set_notnull(); > } > KEY *key=show_table->key_info+i; > - if (key->rec_per_key[j]) > + if (key->rec_per_key[j] && key->algorithm != HA_KEY_ALG_LONG_HASH) > { > ha_rows records= (ha_rows) ((double) show_table->stat_records() / > key->actual_rec_per_key(j)); > table->field[9]->store((longlong) records, TRUE); > table->field[9]->set_notnull(); > } When you'll do the optimizer support, I think you can print rec_per_key[] here for your virtual HASH key. But it's not important now in this patch, so let's not delay the push because of it. > - const char *tmp= show_table->file->index_type(i); > - table->field[13]->store(tmp, strlen(tmp), cs); > + if (key->algorithm == HA_KEY_ALG_LONG_HASH) > + table->field[13]->store(STRING_WITH_LEN("HASH"), cs); > + else > + { > + const char *tmp= show_table->file->index_type(i); > + table->field[13]->store(tmp, strlen(tmp), cs); > + } > } > if (!(key_info->flags & HA_FULLTEXT) && > (key_part->field && > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -4106,12 +4127,27 @@ mysql_prepare_create_table(THD *thd, HA_CREATE_INFO *create_info, > if (key->type == Key::UNIQUE && !(key_info->flags & HA_NULL_PART_KEY)) > unique_key=1; > key_info->key_length=(uint16) key_length; > - if (key_length > max_key_length && key->type != Key::FULLTEXT) > + if (key_length > max_key_length && key->type != Key::FULLTEXT && > + !is_hash_field_needed) > { > my_error(ER_TOO_LONG_KEY,MYF(0),max_key_length); > DBUG_RETURN(TRUE); > } > > + if (is_hash_field_needed) > + { > + if (key_info->algorithm != HA_KEY_ALG_UNDEF && > + key_info->algorithm != HA_KEY_ALG_HASH ) > + { > + my_error(ER_TOO_LONG_KEY, MYF(0), file->max_key_length()); > + DBUG_RETURN(TRUE); > + } > + key_info->algorithm= HA_KEY_ALG_LONG_HASH; > + } > + // If user forces hash index for storage engine other then memory > + else if (key_info->algorithm == HA_KEY_ALG_HASH && > + create_info->db_type->db_type != DB_TYPE_HEAP) Please, no. No engine type checks. Better add a new engine capability flag HA_CAN_HASH_KEYS (see HA_CAN_RTREEKEYS for example). And only MEMORY engine will have it for now. > + key_info->algorithm= HA_KEY_ALG_LONG_HASH; > if (validate_comment_length(thd, &key->key_create_info.comment, > INDEX_COMMENT_MAXLEN, > ER_TOO_LONG_INDEX_COMMENT, > @@ -8319,6 +8355,11 @@ mysql_prepare_alter_table(THD *thd, TABLE *table, > enum Key::Keytype key_type; > LEX_CSTRING tmp_name; > bzero((char*) &key_create_info, sizeof(key_create_info)); > + if (key_info->algorithm == HA_KEY_ALG_LONG_HASH) > + { > + key_info->flags|= HA_NOSAME; > + key_info->algorithm= HA_KEY_ALG_UNDEF; Why HA_KEY_ALG_UNDEF? > + } > key_create_info.algorithm= key_info->algorithm; > /* > We copy block size directly as some engines, like Area, sets this > @@ -10262,6 +10304,8 @@ copy_data_between_tables(THD *thd, TABLE *from, TABLE *to, > to->s->default_fields= 0; > for (Field **ptr=to->field ; *ptr ; ptr++) > { > + if ((*ptr)->flags & LONG_UNIQUE_HASH_FIELD) > + continue; Really? I'd think it would skip all virtual fields anyway, so you should not need to have any special checks for your HASH field. > def=it++; > if (def->field) > { > @@ -10485,14 +10529,23 @@ copy_data_between_tables(THD *thd, TABLE *from, TABLE *to, > if ((int) key_nr >= 0) > { > const char *err_msg= ER_THD(thd, ER_DUP_ENTRY_WITH_KEY_NAME); > + KEY *long_key= NULL; > if (key_nr == 0 && to->s->keys > 0 && > (to->key_info[0].key_part[0].field->flags & > AUTO_INCREMENT_FLAG)) > err_msg= ER_THD(thd, ER_DUP_ENTRY_AUTOINCREMENT_CASE); > + if (key_nr <= to->s->keys && to->key_info[key_nr].algorithm > + == HA_KEY_ALG_LONG_HASH) > + { > + long_key= to->key_info + key_nr; > + setup_keyinfo_hash(long_key); > + } > print_keydup_error(to, > key_nr >= to->s->keys ? NULL : > &to->key_info[key_nr], > err_msg, MYF(0)); > + if (long_key) > + re_setup_keyinfo_hash(long_key); You've already had this exactly code in handler::print_error. May be it'd be better to move it inside print_keydup_error() to have it only in one place? > } > else > to->file->print_error(error, MYF(0)); > diff --git a/sql/table.cc b/sql/table.cc > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -1143,10 +1150,59 @@ bool parse_vcol_defs(THD *thd, MEM_ROOT *mem_root, TABLE *table, > pos+= expr_length; > } > > - /* Now, initialize CURRENT_TIMESTAMP fields */ > + /* Now, initialize CURRENT_TIMESTAMP and UNIQUE_INDEX_HASH_FIELD fields */ > for (field_ptr= table->field; *field_ptr; field_ptr++) > { > Field *field= *field_ptr; > + if (field->flags & LONG_UNIQUE_HASH_FIELD) > + { > + List<Item > *field_list= new (mem_root) List<Item >(); Why do you add a space before a closing angle bracket? I don't think I've see this style anywhere in our codebase. > + Item *list_item; > + KEY *key; > + uint key_index; > + for (key_index= 0; key_index < table->s->keys; key_index++) > + { > + key=table->key_info + key_index; > + if (key->algorithm == HA_KEY_ALG_LONG_HASH && > + key->key_part[key->user_defined_key_parts].fieldnr == field->field_index+ 1) > + break; > + } > + KEY_PART_INFO *keypart; > + for (uint i=0; i < key->user_defined_key_parts; i++) > + { > + keypart= key->key_part + i; > + if (!keypart->length) > + { > + list_item= new(mem_root)Item_field(thd, keypart->field); > + } > + else > + { > + list_item= new(mem_root)Item_func_left(thd, > + new (mem_root)Item_field(thd, keypart->field), > + new (mem_root) Item_int(thd, keypart->length)); > + list_item->fix_fields(thd, NULL); > + } > + field_list->push_back(list_item, mem_root); > + } > + Item_func_hash *hash_item= new(mem_root)Item_func_hash(thd, *field_list); > + Virtual_column_info *v= new (mem_root) Virtual_column_info(); > + field->vcol_info= v; > + field->vcol_info->expr= hash_item; > + uint parts= key->user_defined_key_parts; > + key->user_defined_key_parts= key->ext_key_parts= key->usable_key_parts= 1; > + key->key_part+= parts; > + > + if (key->flags & HA_NULL_PART_KEY) > + key->key_length= HA_HASH_KEY_LENGTH_WITH_NULL; > + else > + key->key_length= HA_HASH_KEY_LENGTH_WITHOUT_NULL; > + > + if (key->flags & HA_NULL_PART_KEY) > + key->key_length= HA_HASH_KEY_LENGTH_WITH_NULL; > + else > + key->key_length= HA_HASH_KEY_LENGTH_WITHOUT_NULL; copy-paste error? same if() repeated twice > + *(vfield_ptr++)= *field_ptr; > + } > if (field->has_default_now_unireg_check()) > { > expr_str.length(parse_vcol_keyword.length); > @@ -1287,6 +1366,7 @@ int TABLE_SHARE::init_from_binary_frm_image(THD *thd, bool write, > > keyinfo= &first_keyinfo; > share->ext_key_parts= 0; > + share->long_unique_table= 0; I don't think that's needed, TABLE_SHARE is probably bzero-ed by this time (otherwise there would've been a lot more initializations here) > thd->mem_root= &share->mem_root; > > if (write && write_frm_image(frm_image, frm_length)) > @@ -2162,6 +2255,51 @@ int TABLE_SHARE::init_from_binary_frm_image(THD *thd, bool write, > share->default_fields++; > } > } > + > + keyinfo= share->key_info; > + for (uint i=0; i < hash_fields; i++, field_ptr++) > + { > + LEX_CSTRING comment, name; > + uint32 flags= 0; > + Column_definition_attributes attr; > + comment.str= (char*) ""; > + comment.length=0; > + //There are n no of hash fields for n long unique key > + while(keyinfo->algorithm != HA_KEY_ALG_LONG_HASH) > + keyinfo++; > + attr.length= HA_HASH_FIELD_LENGTH; > + if (keyinfo->flags & HA_NULL_PART_KEY) > + attr.pack_flag= FIELDFLAG_MAYBE_NULL; > + attr.unireg_check= Field::NONE; > + attr.charset= share->table_charset;//STODO > + name.str= (const char *)alloc_root(&share->mem_root, LONG_HASH_FIELD_NAME_LENGTH); > + make_long_hash_field_name(&name, &fieldnames); > + fieldnames.count++; > + fieldnames.type_names[share->fields + i] = name.str; > + Record_addr addr(share->default_values + share->reclength, null_pos, null_bit_pos); > + *field_ptr= reg_field= > + attr.make_field(share, &share->mem_root, &addr, &type_handler_longlong, &name, flags); > + if (!reg_field) // Not supported field type > + goto err; > + reg_field->field_index= i + share->fields; > + reg_field->comment=comment; > + reg_field->flags|= LONG_UNIQUE_HASH_FIELD; > + reg_field->invisible= INVISIBLE_FULL; > + if (!(reg_field->flags & NOT_NULL_FLAG)) > + { > + if (!(null_bit_pos= (null_bit_pos + 1) & 7)) > + null_pos++; > + } > + share->reclength+= HA_HASH_FIELD_LENGTH; > + keyinfo++; > + } I'm sorry :( After I saw your chat with Marko I've realized a big problem with this approach. If we do it this way, it'll be very diffucult to change anything in the future. Say, you'd want to do persistent generated columns (I'm not saying you should, it's just an example) - if LONG_UNIQUE_HASH_FIELD fields are stored in the frm, you simply make generated columns persistent, and everything works. If LONG_UNIQUE_HASH_FIELD fields are generated on the fly, you'd need here to generate persistent or virtual fields to match what the actual table in the storage engine has, so here you'd still need to be able to write in the frm that some LONG_UNIQUE_HASH_FIELD are persistent. It looks like it'd be more future-proof just to write LONG_UNIQUE_HASH_FIELD fields into frm as you did in a previous patch. Sorry, for this. > + keyinfo= share->key_info; > + for (uint i =0; hash_fields && i < share->keys; i++) > + { > + if(keyinfo[i].algorithm != HA_KEY_ALG_LONG_HASH) > + for (uint j= 0; j < keyinfo->user_defined_key_parts; j++) > + keyinfo[i].key_part[j].offset+= extra_null_field_bytes; > + } > *field_ptr=0; // End marker > /* Sanity checks: */ > DBUG_ASSERT(share->fields>=share->stored_fields); > diff --git a/sql/table_cache.cc b/sql/table_cache.cc > --- a/sql/table_cache.cc > +++ b/sql/table_cache.cc > @@ -824,11 +824,9 @@ TABLE_SHARE *tdc_acquire_share(THD *thd, TABLE_LIST *tl, uint flags, > DBUG_RETURN(0); > else if (res == 1) > continue; > - > element= (TDC_element*) lf_hash_search_using_hash_value(&tdc_hash, > thd->tdc_hash_pins, hash_value, (uchar*) key, key_length); > lf_hash_search_unpin(thd->tdc_hash_pins); > - DBUG_ASSERT(element); Why? > > if (!(share= alloc_table_share(tl->db.str, tl->table_name.str, key, key_length))) > { Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 1

Re: [Maria-developers] 5e265de00c5: triggers fix: delete
by Sergei Golubchik 19 Feb '19

19 Feb '19

Hi, Nikita! On Feb 18, Nikita Malyavin wrote: > revision-id: 5e265de00c5 (mariadb-10.4.1-185-g5e265de00c5) > parent(s): 8ec8242e962 > author: Nikita Malyavin <nikitamalyavin(a)gmail.com> > committer: Nikita Malyavin <nikitamalyavin(a)gmail.com> > timestamp: 2019-02-18 23:48:33 +1000 > message: > > triggers fix: delete Commit comment, please. Like MDEV-16973 Application-time periods: DELETE fire triggers in the order: BEFORE DELETE, BEFORE INSERT, AFTER INSERT, AFTER DELETE. > --- > mysql-test/suite/period/r/delete.result | 74 ++++++++++++++++----------------- > sql/sql_delete.cc | 45 ++++++++++---------- > 2 files changed, 58 insertions(+), 61 deletions(-) > > diff --git a/mysql-test/suite/period/r/delete.result b/mysql-test/suite/period/r/delete.result > index f0949f5b0d9..0ac11f518f8 100644 > --- a/mysql-test/suite/period/r/delete.result > +++ b/mysql-test/suite/period/r/delete.result > @@ -35,33 +35,33 @@ id s e > select * from log_tbl order by id; > id log > 1 >DEL: 1, 1999-01-01, 2018-12-12 > -2 <DEL: 1, 1999-01-01, 2018-12-12 > -3 >INS: 1, 1999-01-01, 2000-01-01 > -4 <INS: 1, 1999-01-01, 2000-01-01 > -5 >INS: 1, 2018-01-01, 2018-12-12 > -6 <INS: 1, 2018-01-01, 2018-12-12 > +2 >INS: 1, 1999-01-01, 2000-01-01 > +3 <INS: 1, 1999-01-01, 2000-01-01 > +4 >INS: 1, 2018-01-01, 2018-12-12 > +5 <INS: 1, 2018-01-01, 2018-12-12 > +6 <DEL: 1, 1999-01-01, 2018-12-12 > 7 >DEL: 1, 1999-01-01, 2017-01-01 > -8 <DEL: 1, 1999-01-01, 2017-01-01 > -9 >INS: 1, 1999-01-01, 2000-01-01 > -10 <INS: 1, 1999-01-01, 2000-01-01 > +8 >INS: 1, 1999-01-01, 2000-01-01 > +9 <INS: 1, 1999-01-01, 2000-01-01 > +10 <DEL: 1, 1999-01-01, 2017-01-01 > 11 >DEL: 1, 2017-01-01, 2019-01-01 > -12 <DEL: 1, 2017-01-01, 2019-01-01 > -13 >INS: 1, 2018-01-01, 2019-01-01 > -14 <INS: 1, 2018-01-01, 2019-01-01 > +12 >INS: 1, 2018-01-01, 2019-01-01 > +13 <INS: 1, 2018-01-01, 2019-01-01 > +14 <DEL: 1, 2017-01-01, 2019-01-01 > 15 >DEL: 2, 1998-01-01, 2018-12-12 > -16 <DEL: 2, 1998-01-01, 2018-12-12 > -17 >INS: 2, 1998-01-01, 2000-01-01 > -18 <INS: 2, 1998-01-01, 2000-01-01 > -19 >INS: 2, 2018-01-01, 2018-12-12 > -20 <INS: 2, 2018-01-01, 2018-12-12 > +16 >INS: 2, 1998-01-01, 2000-01-01 > +17 <INS: 2, 1998-01-01, 2000-01-01 > +18 >INS: 2, 2018-01-01, 2018-12-12 > +19 <INS: 2, 2018-01-01, 2018-12-12 > +20 <DEL: 2, 1998-01-01, 2018-12-12 > 21 >DEL: 3, 1997-01-01, 2015-01-01 > -22 <DEL: 3, 1997-01-01, 2015-01-01 > -23 >INS: 3, 1997-01-01, 2000-01-01 > -24 <INS: 3, 1997-01-01, 2000-01-01 > +22 >INS: 3, 1997-01-01, 2000-01-01 > +23 <INS: 3, 1997-01-01, 2000-01-01 > +24 <DEL: 3, 1997-01-01, 2015-01-01 > 25 >DEL: 4, 2016-01-01, 2020-01-01 > -26 <DEL: 4, 2016-01-01, 2020-01-01 > -27 >INS: 4, 2018-01-01, 2020-01-01 > -28 <INS: 4, 2018-01-01, 2020-01-01 > +26 >INS: 4, 2018-01-01, 2020-01-01 > +27 <INS: 4, 2018-01-01, 2020-01-01 > +28 <DEL: 4, 2016-01-01, 2020-01-01 > 29 >DEL: 5, 2010-01-01, 2015-01-01 > 30 <DEL: 5, 2010-01-01, 2015-01-01 add a test case with no BEFORE INSERT trigger. For example, here drop the BEFORE INSERT trigger (but keep the other three) and delete something again. Please, make sure that "delete something" covers all cases, like above (left end, right end, middle, etc). > # INSERT trigger only also works > diff --git a/sql/sql_delete.cc b/sql/sql_delete.cc > index 9607718d755..d79a9deaa4d 100644 > --- a/sql/sql_delete.cc > +++ b/sql/sql_delete.cc > @@ -802,9 +796,11 @@ bool mysql_delete(THD *thd, TABLE_LIST *table_list, COND *conds, > only if there are no triggers set. > It is also meaningless for system-versioned table Fix the comment above to explain why you only look at BEFORE INSERT trigger here. > */ > - portion_of_time_through_update= !has_triggers > - && !table->versioned() > - && table->file->has_transactions(); > + portion_of_time_through_update= > + !(table->triggers && table->triggers->has_triggers(TRG_EVENT_INSERT, > + TRG_ACTION_BEFORE)) > + && !table->versioned() > + && table->file->has_transactions(); > > THD_STAGE_INFO(thd, stage_updating); > while (likely(!(error=info.read_record())) && likely(!thd->killed) && > @@ -840,6 +836,12 @@ bool mysql_delete(THD *thd, TABLE_LIST *table_list, COND *conds, > else > { > error= table->delete_row(); > + > + ha_rows rows_inserted; > + if (likely(!error) && table_list->has_period() > + && !portion_of_time_through_update) > + error= table->insert_portion_of_time(thd, table_list->period_conditions, > + &rows_inserted); > } Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 2

[Maria-developers] Input on Optimizer trace code, part #2
by Sergey Petrunia 14 Feb '19

14 Feb '19

Hi Varun, Please find more input below. > --- 10.4-optimizer-trace-orig/sql/sql_derived.cc 2019-02-10 14:25:59.217015365 +0300 > +++ 10.4-optimizer-trace-cl/sql/sql_derived.cc 2019-02-12 14:04:05.446571305 +0300 > @@ -486,6 +486,24 @@ exit_merge: > DBUG_RETURN(res); > > unconditional_materialization: > + > + if (unlikely(thd->trace_started())) > + { > + /* > + Add to the optimizer trace the change in choice for merged > + derived tables/views to materialised ones. > + */ > + Json_writer_object trace_wrapper(thd); > + Json_writer_object trace_derived(thd, derived->is_derived() ? > + "derived" : "view"); > + trace_derived.add("table", derived->alias.str ? derived->alias.str : "<NULL>") > + .add_select_number(derived->get_unit()-> > + first_select()->select_number) > + .add("initial_choice", "merged") > + .add("final_choice", "materialized") > + .add("cause", cause); Please add test coverage for this (I cannot find it in the testsuite). > +select * from v1 { > + "steps": [ > + { > + "join_preparation": { > + "select_id": 1, > + "steps": [ > + { > + "view": { > + "table": "v1", > + "select_id": 2, > + "merged": true The above looks a bit odd, "view", inside it "table", and then either "materilized" or "merged". I would prefer "algorithm": "materialize" or "algorithm": "merge" > diff -urpN '--exclude=.*' 10.4-optimizer-trace-orig/sql/sql_select.cc 10.4-optimizer-trace-cl/sql/sql_select.cc > --- 10.4-optimizer-trace-orig/sql/sql_select.cc 2019-02-10 14:25:59.225015425 +0300 > +++ 10.4-optimizer-trace-cl/sql/sql_select.cc 2019-02-12 14:04:05.450571322 +0300 > @@ -345,6 +349,40 @@ bool dbug_user_var_equals_int(THD *thd, > } > #endif > > +static void trace_table_dependencies(THD *thd, > + JOIN_TAB *join_tabs, uint table_count) > +{ > + Json_writer_object trace_wrapper(thd); > + Json_writer_array trace_dep(thd, "table_dependencies"); > + for (uint i = 0; i < table_count; i++) > + { > + TABLE_LIST *table_ref = join_tabs[i].tab_list; > + Json_writer_object trace_one_table(thd); > + trace_one_table.add_table_name(&join_tabs[i]); > + trace_one_table.add("row_may_be_null", > + (bool)table_ref->table->maybe_null); > + const table_map map = table_ref->get_map(); > + DBUG_ASSERT(map < (1ULL << table_count)); > + for (uint j = 0; j < table_count; j++) > + { > + if (map & (1ULL << j)) > + { > + trace_one_table.add("map_bit", static_cast<longlong>(j)); > + break; > + } > + } > + Json_writer_array depends_on(thd, "depends_on_map_bits"); > + static_assert(sizeof(table_ref->get_map()) <= 64, > + "RAND_TABLE_BIT may be in join_tabs[i].dependent, so we test " > + "all 64 bits."); sizeof() is in bytes, not bits, so the above assert actually checks for whether table_map is 64 bytes large (which is incorrect). I would suggest to switch to using Table_map_iterator and forget about the 64-byte limitation (and the value of RAND_TABLE_BIT, too) > + for (uint j = 0; j < 64; j++) > + { > + if (join_tabs[i].dependent & (1ULL << j)) > + depends_on.add(static_cast<longlong>(j)); > + } > --- 10.4-optimizer-trace-orig/sql/opt_table_elimination.cc 2019-02-10 14:25:59.209015304 +0300 > +++ 10.4-optimizer-trace-cl/sql/opt_table_elimination.cc 2019-02-12 14:04:05.442571289 +0300 > @@ -522,7 +524,8 @@ eliminate_tables_for_list(JOIN *join, > List<TABLE_LIST> *join_list, > table_map tables_in_list, > Item *on_expr, > - table_map tables_used_elsewhere); > + table_map tables_used_elsewhere, > + Json_writer_array* eliminate_tables); I think I wrote this already - I think 'eliminate_tables' is a very poor name for the argument- it doesn't show that this is just a trace output object. If something is named 'eliminate_tables' I would expect it to play some active role (e.g. being a list of tables we are trying to eliminate or something like that). Please change to something like trace, trace_eliminated, trace_for_eliminated, ... > @@ -2340,8 +2377,13 @@ int pull_out_semijoin_tables(JOIN *join) > bool optimize_semijoin_nests(JOIN *join, table_map all_table_map) > { > DBUG_ENTER("optimize_semijoin_nests"); > + THD *thd= join->thd; > List_iterator<TABLE_LIST> sj_list_it(join->select_lex->sj_nests); > TABLE_LIST *sj_nest; > + Json_writer_object wrapper(thd); > + Json_writer_object trace_semijoin_nest(thd, > + "execution_plan_for_potential_materialization"); > + Json_writer_array trace_steps_array(thd, "steps"); > while ((sj_nest= sj_list_it++)) > { > /* semi-join nests with only constant tables are not valid */ The most common output this produces is a useless "execution_plan_for_potential_materialization": { "steps": [] } Please add "if (!join->select_lex->sj_nests.elements) \n DBUG_RETURN(FALSE);" to avoid it. Please also add a testcase where a non-empty output is produced (it's nice to have a semi-representative set of outputs in main/opt_trace*result). > --- 10.4-optimizer-trace-orig/mysql-test/main/opt_trace.result 1970-01-01 03:00:00.000000000 +0300 > +++ 10.4-optimizer-trace-cl/mysql-test/main/opt_trace.result 2019-02-12 14:04:05.438571272 +0300 > +QUERY TRACE MISSING_BYTES_BEYOND_MAX_MEM_SIZE INSUFFICIENT_PRIVILEGES > +EXPLAIN SELECT DISTINCT a FROM t1 { ... > + "best_group_range_summary": { > + "type": "index_group", > + "index": "a", > + "group_attribute": null, > + "min_aggregate": false, > + "max_aggregate": false, > + "distinct_aggregate": false, > + "rows": 5, > + "cost": 7.5, > + "key_parts_used_for_access": ["a"], > + "ranges": [], > + "chosen": true > + }, How come group_attribute is NULL? Loose Index Scan requires groping on a column. Are you sure you are printing the right value? --> - 10.4-optimizer-trace-orig/sql/opt_trace.cc 1970-01-01 03:00:00.000000000 +0300 > +++ 10.4-optimizer-trace-cl/sql/opt_trace.cc 2019-02-12 14:04:05.442571289 +0300 > ... > +void opt_trace_print_expanded_query(THD *thd, SELECT_LEX *select_lex, > + Json_writer_object *writer) > + > +{ > + if (!thd->trace_started()) > + return; > + char buff[1024]; > + String str(buff, sizeof(buff), system_charset_info); > + str.length(0); Please use StringBuffer<1024> instead. > + select_lex->print(thd, &str, > + enum_query_type(QT_TO_SYSTEM_CHARSET | > + QT_SHOW_SELECT_NUMBER | > + QT_ITEM_IDENT_SKIP_DB_NAMES | > + QT_VIEW_INTERNAL > + )); > + /* > + The output is not very pretty lots of back-ticks, the output > + is as the one in explain extended , lets try to improved it here. > + */ > + writer->add("expanded_query", str.c_ptr_safe(), str.length()); > +} Did you try using QT_EXPLAIN? (if it doesn't work, I'd like to know why) ... > + trace->missing_privilege(); > + return; > +} A 'return;' at the end of void function looks very confusing. Please remove it > +void opt_trace_disable_if_no_stored_proc_func_access(THD *thd, sp_head *sp) > +{ > + if (likely(!(thd->variables.optimizer_trace & > + Opt_trace_context::FLAG_ENABLED)) || > + thd->system_thread) > + return; > + > + Opt_trace_context *const trace = &thd->opt_trace; > + if (!thd->trace_started()) > + return; > + bool full_access; > + Security_context *const backup_thd_sctx = thd->security_context(); > + thd->set_security_context(&thd->main_security_ctx); > + const bool rc = check_show_routine_access(thd, sp, &full_access) || !full_access; > + thd->set_security_context(backup_thd_sctx); > + if (rc) > + trace->missing_privilege(); > + return; > +} A 'return;' at the end of void function looks very confusing. Please remove it. Also, check the assignments in the function: they should follow coding style and be "x= y" and not "x = y". > +int fill_optimizer_trace_info(THD *thd, TABLE_LIST *tables, Item *) > +{ > + TABLE *table = tables->table; > + Opt_trace_info info; > + > + /* get_values of trace, query , missing bytes and missing_priv > + > + @todo: Need an iterator here to walk over all the traces > + */ > + Opt_trace_context* ctx= &thd->opt_trace; > + > + if (thd->opt_trace.empty()) > + { So thd->opt_trace.empty() returns FALSE when there are NO traces. And it returns TRUE when there are traces. THIS IS EXTREMELY CONFUSING! Please fix it. > + Opt_trace_stmt *stmt= ctx->get_top_trace(); > + stmt->fill_info(&info); > + > + table->field[0]->store(info.query_ptr, static_cast<uint>(info.query_length), > + info.query_charset); ... > +void Json_writer::add_str(Item *item) > +{ > + if (item) > + { > + THD *thd= current_thd; > + char buff[256]; > + String str(buff, sizeof(buff), system_charset_info); > + str.length(0); Please use StringBuffer. > + ~Opt_trace_stmt() > + { > + delete current_json; > + missing_priv= false; > + ctx= NULL; > + I_S_disabled= 0; > + } What is this for? Why set member variables in destructor? This either does nothing, or is a unsuccessful attempt to prevent "reading freed memory"-type crashes. Please remove this. > +private: > + Opt_trace_context *ctx; > + String query; // store the query sent by the user > + Json_writer *current_json; // stores the trace > + bool missing_priv; ///< whether user lacks privilege to see this trace > + uint I_S_disabled; > +}; What is I_S_disabled and why is it called like that? Please document as it is totally confusing. > +void opt_trace_disable_if_no_security_context_access(THD *thd) > ... > + if (!thd->trace_started()) > + { > + /* > + @@optimizer_trace has "enabled=on" but trace is not started. > + Either Opt_trace_start ctor was not called for our statement (3), or it > + was called but at that time, the variable had "enabled=off" (4). > + > + There are no known cases of (3). > + > + (4) suggests that the user managed to change the variable during > + execution of the statement, and this statement is using > + view/routine (note that we have not been able to provoke this, maybe > + this is impossible). If it happens it is suspicious. > + > + We disable I_S output. And we cannot do otherwise: we have no place to > + store a possible "missing privilege" information (no Opt_trace_stmt, as > + is_started() is false), so cannot do security checks, so cannot safely > + do tracing, so have to disable I_S output. And even then, we don't know > + when to re-enable I_S output, as we have no place to store the > + information "re-enable tracing at the end of this statement", and we > + don't even have a notion of statement here (statements in the optimizer > + trace world mean an Opt_trace_stmt object, and there is none here). So > + we must disable for the session's life. > + > + COM_FIELD_LIST opens views, thus used to be a case of (3). To avoid > + disabling I_S output for the session's life when this command is issued > + (like in: "SET OPTIMIZER_TRACE='ENABLED=ON';USE somedb;" in the 'mysql' > + command-line client), we have decided to create a Opt_trace_start for > + this command. The command itself is not traced though > + (SQLCOM_SHOW_FIELDS does not have CF_OPTIMIZER_TRACE). > + */ > + return; > + } As far as I understand, this was copied from MySQL? Is this still true in MariaDB codebase? It sounds scary, let's discuss it. > diff -urpN '--exclude=.*' 10.4-optimizer-trace-orig/sql/my_json_writer.cc 10.4-optimizer-trace-cl/sql/my_json_writer.cc > --- 10.4-optimizer-trace-orig/sql/my_json_writer.cc 2019-02-10 14:25:59.201015244 +0300 > +++ 10.4-optimizer-trace-cl/sql/my_json_writer.cc 2019-02-12 14:04:05.438571272 +0300 > ... > +Json_writer_array::Json_writer_array(THD *thd) : > + Json_writer_struct(thd) Json_writer_object and Json_writer_array are widely used. I guess performance-wise, it is be beneficial if their calls are inlined. A while ago, compiler would only inline a function if its body is in the .h file (and so is visible in the compilation unit where inlining should occur). I'm wondering if this is still true? (if yes, we better move these methods to the .h file). Can you look at release-build disassembly and check if there are calls to Json_writer_array member functions there or they were inlined? (the same question for THD::trace_started) > +{ > + if (my_writer) > + my_writer->start_array(); > +} > + > +Json_writer_array::Json_writer_array(THD *thd, const char *str) > + :Json_writer_struct(thd) Please change the indentation to be like in the ctor above. > +{ > + if (my_writer) ... > @@ -123,9 +209,18 @@ public: > + /* > + One can set a limit of how large a JSON document should be. > + Writes beyond that size will be counted, but will not be collected. > + */ > + void set_size_limit(size_t mem_size) { output.set_size_limit(mem_size); } > + > + // psergey: return how many bytes would be required to store everything Remove the comment. > + size_t get_truncated_bytes() { return output.get_truncated_bytes(); } > + > Json_writer() : > indent_level(0), document_start(true), element_started(false), > - first_child(true) > + first_child(true), allowed_mem_size(0) > { > fmt_helper.init(this); > } > @@ -140,15 +235,335 @@ private: > bool element_started; > bool first_child; > > + /* > + True when we are using the optimizer trace > + FALSE otherwise > + */ > + size_t allowed_mem_size; The comment above is wrong, remove it. Also remove the 'allowed_mem_size' as it is not used anymore. > +void Opt_trace_context::flush_optimizer_trace() > +{ > + inited= false; > + if (traces) > + { > + while (traces->elements()) > + { > + Opt_trace_stmt *prev= traces->at(0); > + delete prev; > + traces->del(0); > + } > + delete traces; > + traces= NULL; > + } > +} The name of this function is confusing. It is not clear what is flushed and where. How about renaming it to "delete_traces()" ? > +Opt_trace_context::~Opt_trace_context() > +{ > + inited= FALSE; > + /* > + would be nice to move this to a function > + */ And this function is already there - it's currently called flush_optimizer_trace(). I am also confused with setting member variables (inited, max_mem_size) in the destructor. This object is going away, why do it? Please remove that. > + if (traces) > + { > + while (traces->elements()) > + { > + Opt_trace_stmt *prev= traces->at(0); > + delete prev; > + traces->del(0); > + } > + delete traces; > + traces= NULL; > + } > + max_mem_size= 0; > +} > + > --- 10.4-optimizer-trace-orig/sql/opt_range.cc 2019-02-10 14:25:59.209015304 +0300 > +++ 10.4-optimizer-trace-cl/sql/opt_range.cc 2019-02-12 14:04:05.438571272 +0300 > @@ -3214,17 +3431,25 @@ bool calculate_cond_selectivity_for_tabl > SEL_ARG *key= tree->keys[idx]; > if (key) > { ... > rows= records_in_column_ranges(&param, idx, key); > if (rows != DBL_MAX) > + { > key->field->cond_selectivity= rows/table_records; > + selectivity_for_column.add("selectivity_from_histograms", > + key->field->cond_selectivity); > + } That's not from multiple histogramS, that's from one histogram. Please change the name. And I think I have requested this already: at the end of the function, please print the table->cond_selectivity as the total selectivity of condition for this table. ... Coding style: can you search through your patch for instances of if (x) { } and change them to if (x) { } ? The same for "x = y;" -> "x= y;" > --- 10.4-optimizer-trace-orig/sql/opt_trace_context.h 1970-01-01 03:00:00.000000000 +0300 > +++ 10.4-optimizer-trace-cl/sql/opt_trace_context.h 2019-02-10 14:25:38.576860203 +0300 > + Opt_trace_stmt *current_trace; > + /* > + TRUE: if we allocate memory for list of traces > + FALSE: otherwise > + */ > + bool inited; 'inited' is not necessary, it would be sufficient to check if 'traces' is NULL. (On the other hand, why traces is a pointer?) BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] [Commits] b428e82: MDEV-7974 backport fix for mysql bug#12161 (XA and binlog).
by andrei.elkin＠pp.inet.fi 14 Feb '19

14 Feb '19

Привет, Алексей! Я просматриваю патч. Где-нить через 2-3 часа должны быть комментарии от меня. Можно будет созвониться. И сразу - полагаю, ты сейчас прямо усердно трудишься над тестами (в патче не присутствующими). Andrei. holyfoot(a)askmonty.org (Alexey Botchkov) writes: > revision-id: b428e822da09b2bc82a2447332bb980f93c80262 (mariadb-10.4.1-103-gb428e82) > parent(s): 8aae31cf494678b6253031c627566e50bc666920 > committer: Alexey Botchkov > timestamp: 2019-02-14 02:46:57 +0400 > message: > > MDEV-7974 backport fix for mysql bug#12161 (XA and binlog). > > XA transactions now are kept persistent after prepare. > XA_prepare_log_event implamented, and XA tranasctions are logged > as XA transactions. > > --- > sql/handler.cc | 9 ++ > sql/handler.h | 10 ++ > sql/log.cc | 115 +++++++++++++--- > sql/log.h | 10 ++ > sql/log_event.cc | 397 +++++++++++++++++++++++++++++++++++++++++++++++++++-- > sql/log_event.h | 81 +++++++++++ > sql/sql_class.cc | 18 +-- > sql/sql_class.h | 20 ++- > sql/sql_connect.cc | 1 + > sql/transaction.cc | 69 +++++++++- > sql/transaction.h | 1 + > 11 files changed, 691 insertions(+), 40 deletions(-) > > diff --git a/sql/handler.cc b/sql/handler.cc > index 001055c..3b2a3e0 100644 > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -1214,6 +1214,9 @@ static int prepare_or_error(handlerton *ht, THD *thd, bool all) > } > > > +/*static inline */int > +binlog_commit_flush_xid_caches(THD *thd, binlog_cache_mngr *cache_mngr, > + bool all, my_xid xid); > /** > @retval > 0 ok > @@ -1225,6 +1228,7 @@ int ha_prepare(THD *thd) > int error=0, all=1; > THD_TRANS *trans=all ? &thd->transaction.all : &thd->transaction.stmt; > Ha_trx_info *ha_info= trans->ha_list; > + > DBUG_ENTER("ha_prepare"); > > if (ha_info) > @@ -1250,6 +1254,11 @@ int ha_prepare(THD *thd) > > } > } > + if (unlikely(tc_log->log_prepare(thd, all))) > + { > + ha_rollback_trans(thd, all); > + error=1; > + } > } > > DBUG_RETURN(error); > diff --git a/sql/handler.h b/sql/handler.h > index fc6246c..613c1c3 100644 > --- a/sql/handler.h > +++ b/sql/handler.h > @@ -810,6 +810,16 @@ struct xid_t { > long gtrid_length; > long bqual_length; > char data[XIDDATASIZE]; // not \0-terminated ! > + /* > + The size of the string containing serialized Xid representation > + is computed as a sum of > + eight as the number of formatting symbols (X'',X'',) > + plus 2 x XIDDATASIZE (2 due to hex format), > + plus space for decimal digits of XID::formatID, > + plus one for 0x0. > + */ > + static const uint ser_buf_size= > + 8 + 2 * XIDDATASIZE + 4 * sizeof(long) + 1; > > xid_t() {} /* Remove gcc warning */ > bool eq(struct xid_t *xid) > diff --git a/sql/log.cc b/sql/log.cc > index a56117a..316b871 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -87,6 +87,9 @@ static bool binlog_savepoint_rollback_can_release_mdl(handlerton *hton, > static int binlog_commit(handlerton *hton, THD *thd, bool all); > static int binlog_rollback(handlerton *hton, THD *thd, bool all); > static int binlog_prepare(handlerton *hton, THD *thd, bool all); > +static int binlog_xa_recover(handlerton *hton, XID *xid_list, uint len); > +static int binlog_commit_by_xid(handlerton *hton, XID *xid); > +static int binlog_rollback_by_xid(handlerton *hton, XID *xid); > static int binlog_start_consistent_snapshot(handlerton *hton, THD *thd); > > static const LEX_CSTRING write_error_msg= > @@ -1688,6 +1691,9 @@ int binlog_init(void *p) > binlog_hton->commit= binlog_commit; > binlog_hton->rollback= binlog_rollback; > binlog_hton->prepare= binlog_prepare; > + binlog_hton->recover= binlog_xa_recover; > + binlog_hton->commit_by_xid = binlog_commit_by_xid; > + binlog_hton->rollback_by_xid = binlog_rollback_by_xid; > binlog_hton->start_consistent_snapshot= binlog_start_consistent_snapshot; > binlog_hton->flags= HTON_NOT_USER_SELECTABLE | HTON_HIDDEN; > return 0; > @@ -1883,23 +1889,16 @@ static inline int > binlog_commit_flush_xid_caches(THD *thd, binlog_cache_mngr *cache_mngr, > bool all, my_xid xid) > { > - if (xid) > - { > - Xid_log_event end_evt(thd, xid, TRUE); > - return (binlog_flush_cache(thd, cache_mngr, &end_evt, all, TRUE, TRUE)); > - } > - else > + /* Mask XA COMMIT ... ONE PHASE as plain BEGIN ... COMMIT */ > + if (!xid) > { > - /* > - Empty xid occurs in XA COMMIT ... ONE PHASE. > - In this case, we do not have a MySQL xid for the transaction, and the > - external XA transaction coordinator will have to handle recovery if > - needed. So we end the transaction with a plain COMMIT query event. > - */ > - Query_log_event end_evt(thd, STRING_WITH_LEN("COMMIT"), > - TRUE, TRUE, TRUE, 0); > - return (binlog_flush_cache(thd, cache_mngr, &end_evt, all, TRUE, TRUE)); > + DBUG_ASSERT(thd->transaction.xid_state.xa_state == XA_IDLE && > + thd->lex->xa_opt == XA_ONE_PHASE); > + xid= thd->query_id; > } > + > + Xid_log_event end_evt(thd, xid, TRUE); > + return (binlog_flush_cache(thd, cache_mngr, &end_evt, all, TRUE, TRUE)); > } > > /** > @@ -1961,11 +1960,77 @@ static int binlog_prepare(handlerton *hton, THD *thd, bool all) > do nothing. > just pretend we can do 2pc, so that MySQL won't > switch to 1pc. > - real work will be done in MYSQL_BIN_LOG::log_and_order() > + real work is done in MYSQL_BIN_LOG::log_and_order() > */ > return 0; > } > > + > +static int serialize_xid(XID *xid, char *buf) > +{ > + size_t size; > + buf[0]= '\''; > + memcpy(buf+1, xid->data, xid->gtrid_length); > + size= xid->gtrid_length + 2; > + buf[size-1]= '\''; > + if (xid->bqual_length == 0 && xid->formatID == 1) > + return size; > + > + memcpy(buf+size, ", '", 3); > + memcpy(buf+size+3, xid->data+xid->gtrid_length, xid->bqual_length); > + size+= 3 + xid->bqual_length; > + buf[size]= '\''; > + size++; > + if (xid->formatID != 1) > + size+= sprintf(buf+size, ", %ld", xid->formatID); > + return size; > +} > + > + > +static int binlog_xa_recover(handlerton *hton __attribute__((unused)), > + XID *xid_list __attribute__((unused)), > + uint len __attribute__((unused))) > +{ > + /* Does nothing. */ > + return 0; > +} > + > + > +static int binlog_commit_by_xid(handlerton *hton, XID *xid) > +{ > + THD *thd= current_thd; > + const size_t xc_len= sizeof("XA COMMIT ") - 1; // do not count trailing 0 > + char buf[xc_len + xid_t::ser_buf_size]; > + size_t buflen; > + binlog_cache_mngr *const cache_mngr= thd->binlog_setup_trx_data(); > + > + DBUG_ASSERT(thd->lex->sql_command == SQLCOM_XA_COMMIT); > + > + if (!cache_mngr) > + return 1; > + > + memcpy(buf, "XA COMMIT ", xc_len); > + buflen= xc_len + serialize_xid(xid, buf+xc_len); > + Query_log_event qe(thd, buf, buflen, TRUE, FALSE, FALSE, 0); > + return binlog_flush_cache(thd, cache_mngr, &qe, TRUE, TRUE, TRUE); > +} > + > + > +static int binlog_rollback_by_xid(handlerton *hton, XID *xid) > +{ > + THD *thd= current_thd; > + const size_t xr_len= sizeof("XA ROLLBACK ") - 1; // do not count trailing 0 > + char buf[xr_len + xid_t::ser_buf_size]; > + size_t buflen; > + > + DBUG_ASSERT(thd->lex->sql_command == SQLCOM_XA_ROLLBACK); > + memcpy(buf, "XA ROLLBACK ", xr_len); > + buflen= xr_len + serialize_xid(xid, buf+xr_len); > + Query_log_event qe(thd, buf, buflen, FALSE, TRUE, TRUE, 0); > + return mysql_bin_log.write_event(&qe); > +} > + > + > /* > We flush the cache wrapped in a beging/rollback if: > . aborting a single or multi-statement transaction and; > @@ -9809,6 +9874,24 @@ int TC_LOG_BINLOG::unlog(ulong cookie, my_xid xid) > DBUG_RETURN(BINLOG_COOKIE_GET_ERROR_FLAG(cookie)); > } > > + > +int TC_LOG_BINLOG::log_prepare(THD *thd, bool all) > +{ > + XID *xid= &thd->transaction.xid_state.xid; > + XA_prepare_log_event end_evt(thd, xid, FALSE); > + binlog_cache_mngr *cache_mngr= thd->binlog_setup_trx_data(); > + > + if (!cache_mngr) > + { > + WSREP_DEBUG("Skipping empty log_xid: %s", thd->query()); > + return 0; > + } > + > + cache_mngr->using_xa= FALSE; > + return (binlog_flush_cache(thd, cache_mngr, &end_evt, all, TRUE, TRUE)); > +} > + > + > void > TC_LOG_BINLOG::commit_checkpoint_notify(void *cookie) > { > diff --git a/sql/log.h b/sql/log.h > index 7dfdb36..92fdf95 100644 > --- a/sql/log.h > +++ b/sql/log.h > @@ -61,6 +61,7 @@ class TC_LOG > bool need_prepare_ordered, > bool need_commit_ordered) = 0; > virtual int unlog(ulong cookie, my_xid xid)=0; > + virtual int log_prepare(THD *thd, bool all)= 0; > virtual void commit_checkpoint_notify(void *cookie)= 0; > > protected: > @@ -115,6 +116,10 @@ class TC_LOG_DUMMY: public TC_LOG // use it to disable the logging > return 1; > } > int unlog(ulong cookie, my_xid xid) { return 0; } > + int log_prepare(THD *thd, bool all) > + { > + return 0; > + } > void commit_checkpoint_notify(void *cookie) { DBUG_ASSERT(0); }; > }; > > @@ -198,6 +203,10 @@ class TC_LOG_MMAP: public TC_LOG > int log_and_order(THD *thd, my_xid xid, bool all, > bool need_prepare_ordered, bool need_commit_ordered); > int unlog(ulong cookie, my_xid xid); > + int log_prepare(THD *thd, bool all) > + { > + return 0; > + } > void commit_checkpoint_notify(void *cookie); > int recover(); > > @@ -698,6 +707,7 @@ class MYSQL_BIN_LOG: public TC_LOG, private MYSQL_LOG > int log_and_order(THD *thd, my_xid xid, bool all, > bool need_prepare_ordered, bool need_commit_ordered); > int unlog(ulong cookie, my_xid xid); > + int log_prepare(THD *thd, bool all); > void commit_checkpoint_notify(void *cookie); > int recover(LOG_INFO *linfo, const char *last_log_name, IO_CACHE *first_log, > Format_description_log_event *fdle, bool do_xa); > diff --git a/sql/log_event.cc b/sql/log_event.cc > index 7a0d0be..354c5f3 100644 > --- a/sql/log_event.cc > +++ b/sql/log_event.cc > @@ -2139,6 +2139,9 @@ Log_event* Log_event::read_log_event(const char* buf, uint event_len, > case XID_EVENT: > ev = new Xid_log_event(buf, fdle); > break; > + case XA_PREPARE_LOG_EVENT: > + ev = new XA_prepare_log_event(buf, fdle); > + break; > case RAND_EVENT: > ev = new Rand_log_event(buf, fdle); > break; > @@ -2190,7 +2193,6 @@ Log_event* Log_event::read_log_event(const char* buf, uint event_len, > case PREVIOUS_GTIDS_LOG_EVENT: > case TRANSACTION_CONTEXT_EVENT: > case VIEW_CHANGE_EVENT: > - case XA_PREPARE_LOG_EVENT: > ev= new Ignorable_log_event(buf, fdle, > get_type_str((Log_event_type) event_type)); > break; > @@ -6222,6 +6224,7 @@ Format_description_log_event(uint8 binlog_ver, const char* server_ver) > post_header_len[USER_VAR_EVENT-1]= USER_VAR_HEADER_LEN; > post_header_len[FORMAT_DESCRIPTION_EVENT-1]= FORMAT_DESCRIPTION_HEADER_LEN; > post_header_len[XID_EVENT-1]= XID_HEADER_LEN; > + post_header_len[XA_PREPARE_LOG_EVENT-1]= XA_PREPARE_HEADER_LEN; > post_header_len[BEGIN_LOAD_QUERY_EVENT-1]= BEGIN_LOAD_QUERY_HEADER_LEN; > post_header_len[EXECUTE_LOAD_QUERY_EVENT-1]= EXECUTE_LOAD_QUERY_HEADER_LEN; > /* > @@ -7874,7 +7877,7 @@ Gtid_log_event::Gtid_log_event(const char *buf, uint event_len, > buf+= 8; > domain_id= uint4korr(buf); > buf+= 4; > - flags2= *buf; > + flags2= *(buf++); > if (flags2 & FL_GROUP_COMMIT_ID) > { > if (event_len < (uint)header_size + GTID_HEADER_LEN + 2) > @@ -7882,8 +7885,22 @@ Gtid_log_event::Gtid_log_event(const char *buf, uint event_len, > seq_no= 0; // So is_valid() returns false > return; > } > - ++buf; > commit_id= uint8korr(buf); > + buf+= 8; > + } > + if (flags2 & FL_XA_TRANSACTION) > + { > + xid.formatID= (long) buf[0]; > + xid.gtrid_length= (long) buf[1]; > + xid.bqual_length= (long) buf[2]; > + > + buf+= 3; > + if (xid.formatID >= 0) > + { > + long data_length= xid.bqual_length + xid.gtrid_length; > + memcpy(xid.data, buf, data_length); > + buf+= data_length; > + } > } > } > > @@ -7914,6 +7931,12 @@ Gtid_log_event::Gtid_log_event(THD *thd_arg, uint64 seq_no_arg, > /* Preserve any DDL or WAITED flag in the slave's binlog. */ > if (thd_arg->rgi_slave) > flags2|= (thd_arg->rgi_slave->gtid_ev_flags2 & (FL_DDL|FL_WAITED)); > + if (thd->transaction.xid_state.xa_state == XA_IDLE && > + thd->lex->xa_opt != XA_ONE_PHASE) > + { > + flags2|= FL_XA_TRANSACTION; > + xid= thd->transaction.xid_state.xid; > + } > } > > > @@ -7956,7 +7979,7 @@ Gtid_log_event::peek(const char *event_start, size_t event_len, > bool > Gtid_log_event::write() > { > - uchar buf[GTID_HEADER_LEN+2]; > + uchar buf[GTID_HEADER_LEN+2+sizeof(XID)]; > size_t write_len; > > int8store(buf, seq_no); > @@ -7968,8 +7991,25 @@ Gtid_log_event::write() > write_len= GTID_HEADER_LEN + 2; > } > else > + write_len= 13; > + > + if (flags2 & FL_XA_TRANSACTION) > + { > + buf[write_len]= (uchar) ((char) xid.formatID); > + buf[write_len+1]= (uchar) xid.gtrid_length; > + buf[write_len+2]= (uchar) xid.bqual_length; > + write_len+= 3; > + if (xid.formatID >= 0) > + { > + long data_length= xid.bqual_length + xid.gtrid_length; > + memcpy(buf+write_len, xid.data, data_length); > + write_len+= data_length; > + } > + } > + > + if (write_len < GTID_HEADER_LEN) > { > - bzero(buf+13, GTID_HEADER_LEN-13); > + bzero(buf+write_len, GTID_HEADER_LEN-write_len); > write_len= GTID_HEADER_LEN; > } > return write_header(write_len) || > @@ -8012,7 +8052,7 @@ Gtid_log_event::make_compatible_event(String *packet, bool *need_dummy_event, > void > Gtid_log_event::pack_info(Protocol *protocol) > { > - char buf[6+5+10+1+10+1+20+1+4+20+1]; > + char buf[6+5+10+1+10+1+20+1+4+20+1+5+128]; > char *p; > p = strmov(buf, (flags2 & FL_STANDALONE ? "GTID " : "BEGIN GTID ")); > p= longlong10_to_str(domain_id, p, 10); > @@ -8026,6 +8066,12 @@ Gtid_log_event::pack_info(Protocol *protocol) > p= longlong10_to_str(commit_id, p, 10); > } > > + if (flags2 & FL_XA_TRANSACTION) > + { > + p= strmov(p, " XID :"); > + p= strnmov(p, xid.data, xid.bqual_length + xid.gtrid_length); > + } > + > protocol->store(buf, p-buf, &my_charset_bin); > } > > @@ -8079,11 +8125,25 @@ Gtid_log_event::do_apply_event(rpl_group_info *rgi) > thd->lex->sql_command= SQLCOM_BEGIN; > thd->is_slave_error= 0; > status_var_increment(thd->status_var.com_stat[thd->lex->sql_command]); > - if (trans_begin(thd, 0)) > + if (flags2 & FL_XA_TRANSACTION) > { > - DBUG_PRINT("error", ("trans_begin() failed")); > - thd->is_slave_error= 1; > + thd->lex->xid= &xid; > + thd->lex->xa_opt= XA_NONE; > + if (trans_xa_start(thd)) > + { > + DBUG_PRINT("error", ("trans_xa_start() failed")); > + thd->is_slave_error= 1; > + } > + } > + else > + { > + if (trans_begin(thd, 0)) > + { > + DBUG_PRINT("error", ("trans_begin() failed")); > + thd->is_slave_error= 1; > + } > } > + > thd->update_stats(); > > if (likely(!thd->is_slave_error)) > @@ -8202,9 +8262,29 @@ Gtid_log_event::print(FILE *file, PRINT_EVENT_INFO *print_event_info) > buf, print_event_info->delimiter)) > goto err; > } > - if (!(flags2 & FL_STANDALONE)) > - if (my_b_printf(&cache, is_flashback ? "COMMIT\n%s\n" : "BEGIN\n%s\n", print_event_info->delimiter)) > + if ((flags2 & FL_XA_TRANSACTION) && !is_flashback) > + { > + my_b_write_string(&cache, "XA START '"); > + my_b_write(&cache, (uchar *) xid.data, xid.gtrid_length); > + my_b_write_string(&cache, "'"); > + if (xid.bqual_length > 0 || xid.formatID != 1) > + { > + my_b_write_string(&cache, ", '"); > + my_b_write(&cache, (uchar *) xid.data+xid.gtrid_length, xid.bqual_length); > + my_b_write_string(&cache, "'"); > + if (xid.formatID != 1) > + if (my_b_printf(&cache, ", %d", xid.formatID)) > + goto err; > + } > + if (my_b_printf(&cache, "%s\n", print_event_info->delimiter)) > + goto err; > + } > + else if (!(flags2 & FL_STANDALONE)) > + { > + if (my_b_printf(&cache, is_flashback ? "COMMIT\n%s\n" : "BEGIN\n%s\n", > + print_event_info->delimiter)) > goto err; > + } > > return cache.flush_data(); > err: > @@ -9003,6 +9083,300 @@ Xid_log_event::do_shall_skip(rpl_group_info *rgi) > #endif /* !MYSQL_CLIENT */ > > > +/** > + Function serializes XID which is characterized by by four last arguments > + of the function. > + Serialized XID is presented in valid hex format and is returned to > + the caller in a buffer pointed by the first argument. > + The buffer size provived by the caller must be not less than > + 8 + 2 * XIDDATASIZE + 4 * sizeof(XID::formatID) + 1, see > + XID::serialize_xid() that is a caller and plugin.h for XID declaration. > + > + @param buf pointer to a buffer allocated for storing serialized data > + > + @return the value of the buffer pointer > +*/ > + > +char *XA_prepare_log_event::event_xid_t::serialize(char *buf) const > +{ > + int i; > + char *c= buf; > + /* > + Build a string like following pattern: > + X'hex11hex12...hex1m',X'hex21hex22...hex2n',11 > + and store it into buf. > + Here hex1i and hex2k are hexadecimals representing XID's internal > + raw bytes (1 <= i <= m, 1 <= k <= n), and `m' and `n' even numbers > + half of which corresponding to the lengths of XID's components. > + */ > + c[0]= 'X'; > + c[1]= '\''; > + c+= 2; > + for (i= 0; i < gtrid_length; i++) > + { > + c[0]=_dig_vec_lower[((uchar*) data)[i] >> 4]; > + c[1]=_dig_vec_lower[((uchar*) data)[i] & 0x0f]; > + c+= 2; > + } > + c[0]= '\''; > + c[1]= ','; > + c[2]= 'X'; > + c[3]= '\''; > + c+= 4; > + > + for (; i < gtrid_length + bqual_length; i++) > + { > + c[0]=_dig_vec_lower[((uchar*) data)[i] >> 4]; > + c[1]=_dig_vec_lower[((uchar*) data)[i] & 0x0f]; > + c+= 2; > + } > + c[0]= '\''; > + sprintf(c+1, ",%lu", formatID); > + > + return buf; > +} > + > + > +/************************************************************************** > + XA_prepare_log_event methods > +**************************************************************************/ > +/** > + @note > + It's ok not to use int8store here, > + as long as xid_t::set(ulonglong) and > + xid_t::get_n_xid doesn't do it either. > + We don't care about actual values of xids as long as > + identical numbers compare identically > +*/ > + > +XA_prepare_log_event:: > +XA_prepare_log_event(const char* buf, > + const Format_description_log_event *description_event) > + :Log_event(buf, description_event) > +{ > + uint32 temp= 0; > + uint8 temp_byte; > + > + buf+= description_event->common_header_len + > + description_event->post_header_len[XA_PREPARE_LOG_EVENT-1]; > + memcpy(&temp_byte, buf, 1); > + one_phase= (bool) temp_byte; > + buf += sizeof(temp_byte); > + memcpy(&temp, buf, sizeof(temp)); > + m_xid.formatID= le32toh(temp); > + buf += sizeof(temp); > + memcpy(&temp, buf, sizeof(temp)); > + m_xid.gtrid_length= le32toh(temp); > + buf += sizeof(temp); > + memcpy(&temp, buf, sizeof(temp)); > + m_xid.bqual_length= le32toh(temp); > + buf += sizeof(temp); > + memcpy(m_xid.data, buf, m_xid.gtrid_length + m_xid.bqual_length); > + > + xid= NULL; > +} > + > + > +#if defined(HAVE_REPLICATION) && !defined(MYSQL_CLIENT) > +void XA_prepare_log_event::pack_info(Protocol *protocol) > +{ > + char buf[ser_buf_size]; > + char query[sizeof("XA COMMIT ONE PHASE") + 1 + sizeof(buf)]; > + > + /* RHS of the following assert is unknown to client sources */ > + compile_time_assert(ser_buf_size == XID::ser_buf_size); > + m_xid.serialize(buf); > + sprintf(query, > + (one_phase ? "XA COMMIT %s ONE PHASE" : "XA PREPARE %s"), > + buf); > + > + protocol->store(query, strlen(query), &my_charset_bin); > +} > +#endif > + > + > +#ifndef MYSQL_CLIENT > +bool XA_prepare_log_event::write() > +{ > + uchar data[1 + 4 + 4 + 4]; > + uint8 one_phase_byte= one_phase; > + > + data[0]= one_phase; > + int4store(data+1, static_cast<XID*>(xid)->formatID); > + int4store(data+(1+4), static_cast<XID*>(xid)->gtrid_length); > + int4store(data+(1+4+4), static_cast<XID*>(xid)->bqual_length); > + > + DBUG_ASSERT(xid_bufs_size == sizeof(data) - 1); > + > + return write_header(sizeof(one_phase_byte) + xid_bufs_size + > + static_cast<XID*>(xid)->gtrid_length + > + static_cast<XID*>(xid)->bqual_length) || > + write_data(data, sizeof(data)) || > + write_data((uchar*) static_cast<XID*>(xid)->data, > + static_cast<XID*>(xid)->gtrid_length + > + static_cast<XID*>(xid)->bqual_length) || > + write_footer(); > +} > +#endif > + > + > +#ifdef MYSQL_CLIENT > +bool XA_prepare_log_event::print(FILE* file, PRINT_EVENT_INFO* print_event_info) > +{ > + Write_on_release_cache cache(&print_event_info->head_cache, file, > + Write_on_release_cache::FLUSH_F, this); > + char buf[ser_buf_size]; > + > + m_xid.serialize(buf); > + > + if (!print_event_info->short_form) > + { > + print_header(&cache, print_event_info, FALSE); > + if (my_b_printf(&cache, "\tXID = %s\n", buf)) > + goto error; > + } > + > + if (my_b_printf(&cache, "XA END %s\n%s\n", > + buf, print_event_info->delimiter) || > + my_b_printf(&cache, "XA PREPARE %s\n%s\n", > + buf, print_event_info->delimiter)) > + goto error; > + > + return cache.flush_data(); > +error: > + return TRUE; > +} > +#endif /* MYSQL_CLIENT */ > + > + > +#if defined(HAVE_REPLICATION) && !defined(MYSQL_CLIENT) > +int XA_prepare_log_event::do_apply_event(rpl_group_info *rgi) > +{ > + bool res; > + int err; > + rpl_gtid gtid; > + uint64 sub_id= 0; > + Relay_log_info const *rli= rgi->rli; > + xid_t xid; > + void *hton= NULL; > + > + /* > + XID_EVENT works like a COMMIT statement. And it also updates the > + mysql.gtid_slave_pos table with the GTID of the current transaction. > + > + Therefore, it acts much like a normal SQL statement, so we need to do > + THD::reset_for_next_command() as if starting a new statement. > + */ > + thd->reset_for_next_command(); > + /* > + Record any GTID in the same transaction, so slave state is transactionally > + consistent. > + */ > +#ifdef WITH_WSREP > + thd->wsrep_affected_rows= 0; > +#endif > + > + if (rgi->gtid_pending) > + { > + xa_states c_state= thd->transaction.xid_state.xa_state; > + sub_id= rgi->gtid_sub_id; > + rgi->gtid_pending= false; > + > + gtid= rgi->current_gtid; > + > + thd->transaction.xid_state.xa_state= XA_ACTIVE; > + err= rpl_global_gtid_slave_state->record_gtid(thd, &gtid, sub_id, true, > + false, &hton); > + thd->transaction.xid_state.xa_state= c_state; > + if (err) > + { > + int ec= thd->get_stmt_da()->sql_errno(); > + /* > + Do not report an error if this is really a kill due to a deadlock. > + In this case, the transaction will be re-tried instead. > + */ > + if (!is_parallel_retry_error(rgi, ec)) > + rli->report(ERROR_LEVEL, ER_CANNOT_UPDATE_GTID_STATE, rgi->gtid_info(), > + "Error during XID COMMIT: failed to update GTID state in " > + "%s.%s: %d: %s", > + "mysql", rpl_gtid_slave_state_table_name.str, ec, > + thd->get_stmt_da()->message()); > + thd->is_slave_error= 1; > + return err; > + } > + > + DBUG_EXECUTE_IF("gtid_fail_after_record_gtid", > + { my_error(ER_ERROR_DURING_COMMIT, MYF(0), HA_ERR_WRONG_COMMAND); > + thd->is_slave_error= 1; > + return 1; > + }); > + } > + /* For a slave XA_prepare_log_event is COMMIT */ > + general_log_print(thd, COM_QUERY, > + "COMMIT /* implicit, from Xid_log_event */"); > + thd->variables.option_bits&= ~OPTION_GTID_BEGIN; > + > + xid.set(m_xid.formatID, > + m_xid.data, m_xid.gtrid_length, > + m_xid.data + m_xid.gtrid_length, m_xid.bqual_length); > + > + thd->lex->xid= &xid; > + if (trans_xa_end(thd)) > + return 1; > + > + if (!one_phase) > + { > + res= trans_xa_prepare(thd); > + } > + else > + { > + res= trans_xa_commit(thd); > + thd->mdl_context.release_transactional_locks(); > + } > + > + > + if (!res && sub_id) > + rpl_global_gtid_slave_state->update_state_hash(sub_id, &gtid, hton, rgi); > + > + /* > + Increment the global status commit count variable > + */ > + status_var_increment(thd->status_var.com_stat[SQLCOM_COMMIT]); > + > + return res; > +} > + > + > +Log_event::enum_skip_reason > +XA_prepare_log_event::do_shall_skip(rpl_group_info *rgi) > +{ > + DBUG_ENTER("Xid_log_event::do_shall_skip"); > + if (rgi->rli->slave_skip_counter > 0) > + { > + DBUG_ASSERT(!rgi->rli->get_flag(Relay_log_info::IN_TRANSACTION)); > + thd->variables.option_bits&= ~(OPTION_BEGIN | OPTION_GTID_BEGIN); > + DBUG_RETURN(Log_event::EVENT_SKIP_COUNT); > + } > +#ifdef WITH_WSREP > + else if (wsrep_mysql_replication_bundle && WSREP_ON && > + opt_slave_domain_parallel_threads == 0) > + { > + if (++thd->wsrep_mysql_replicated < (int)wsrep_mysql_replication_bundle) > + { > + WSREP_DEBUG("skipping wsrep commit %d", thd->wsrep_mysql_replicated); > + DBUG_RETURN(Log_event::EVENT_SKIP_IGNORE); > + } > + else > + { > + thd->wsrep_mysql_replicated = 0; > + } > + } > +#endif > + DBUG_RETURN(Log_event::do_shall_skip(rgi)); > +} > +#endif /* !MYSQL_CLIENT */ > + > + > /************************************************************************** > User_var_log_event methods > **************************************************************************/ > @@ -14789,7 +15163,6 @@ bool event_that_should_be_ignored(const char *buf) > event_type == PREVIOUS_GTIDS_LOG_EVENT || > event_type == TRANSACTION_CONTEXT_EVENT || > event_type == VIEW_CHANGE_EVENT || > - event_type == XA_PREPARE_LOG_EVENT || > (uint2korr(buf + FLAGS_OFFSET) & LOG_EVENT_IGNORABLE_F)) > return 1; > return 0; > diff --git a/sql/log_event.h b/sql/log_event.h > index 38a40c9..b5c48c9 100644 > --- a/sql/log_event.h > +++ b/sql/log_event.h > @@ -217,6 +217,7 @@ class String; > #define GTID_HEADER_LEN 19 > #define GTID_LIST_HEADER_LEN 4 > #define START_ENCRYPTION_HEADER_LEN 0 > +#define XA_PREPARE_HEADER_LEN 0 > > /* > Max number of possible extra bytes in a replication event compared to a > @@ -3064,6 +3065,79 @@ class Xid_log_event: public Log_event > #endif > }; > > + > +/** > + @class XA_prepare_log_event > + > + Similar to Xid_log_event except that > + - it is specific to XA transaction > + - it carries out the prepare logics rather than the final committing > + when @c one_phase member is off. > + From the groupping perspective the event finalizes the current "prepare" group > + started with XA START Query-log-event. > + When @c one_phase is false Commit of Rollback for XA transaction are > + logged separately to the prepare-group events so being a groups of > + their own. > +*/ > + > +class XA_prepare_log_event: public Log_event > +{ > +protected: > + /* The event_xid_t members were copied from handler.h */ > + struct event_xid_t > + { > + long formatID; > + long gtrid_length; > + long bqual_length; > + char data[MYSQL_XIDDATASIZE]; // not \0-terminated ! > + char *serialize(char *buf) const; > + }; > + > + /* size of serialization buffer is explained in $MYSQL/sql/xa.h. */ > + static const uint ser_buf_size= > + 8 + 2 * MYSQL_XIDDATASIZE + 4 * sizeof(long) + 1; > + > + /* Total size of buffers to hold serialized members of XID struct */ > + static const int xid_bufs_size= 12; > + event_xid_t m_xid; > + void *xid; > + bool one_phase; > + > +public: > +#ifdef MYSQL_SERVER > + XA_prepare_log_event(THD* thd_arg, XID *xid_arg, bool one_phase_arg): > + Log_event(thd_arg, 0, TRUE), xid(xid_arg), one_phase(one_phase_arg) > + { > + cache_type= Log_event::EVENT_NO_CACHE; > + } > +#ifdef HAVE_REPLICATION > + void pack_info(Protocol* protocol); > +#endif /* HAVE_REPLICATION */ > +#else > + bool print(FILE* file, PRINT_EVENT_INFO* print_event_info); > +#endif > + XA_prepare_log_event(const char* buf, > + const Format_description_log_event *description_event); > + ~XA_prepare_log_event() {} > + Log_event_type get_type_code() { return XA_PREPARE_LOG_EVENT; } > + int get_data_size() > + { > + return xid_bufs_size + m_xid.gtrid_length + m_xid.bqual_length; > + } > + > +#ifdef MYSQL_SERVER > + bool write(); > +#endif > + bool is_valid() const { return 1; } > + > +private: > +#if defined(MYSQL_SERVER) && defined(HAVE_REPLICATION) > + virtual int do_apply_event(rpl_group_info *rgi); > + enum_skip_reason do_shall_skip(rpl_group_info *rgi); > +#endif > +}; > + > + > /** > @class User_var_log_event > > @@ -3376,6 +3450,11 @@ class Gtid_log_event: public Log_event > uint64 seq_no; > uint64 commit_id; > uint32 domain_id; > +#ifdef MYSQL_SERVER > + XID xid; > +#else > + struct st_mysql_xid xid; > +#endif > uchar flags2; > > /* Flags2. */ > @@ -3404,6 +3483,8 @@ class Gtid_log_event: public Log_event > static const uchar FL_WAITED= 16; > /* FL_DDL is set for event group containing DDL. */ > static const uchar FL_DDL= 32; > + /* FL_XA_TRANSACTION is set for XA transaction. */ > + static const uchar FL_XA_TRANSACTION= 64; > > #ifdef MYSQL_SERVER > Gtid_log_event(THD *thd_arg, uint64 seq_no, uint32 domain_id, bool standalone, > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index fa2f866..cc75da9 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -1461,12 +1461,19 @@ void THD::cleanup(void) > DBUG_ASSERT(cleanup_done == 0); > > set_killed(KILL_CONNECTION); > -#ifdef ENABLE_WHEN_BINLOG_WILL_BE_ABLE_TO_PREPARE > if (transaction.xid_state.xa_state == XA_PREPARED) > { > -#error xid_state in the cache should be replaced by the allocated value > + trans_detach(this); > + transaction.xid_state.xa_state= XA_NOTR; > + transaction.xid_state.rm_error= 0; > + } > + else > + { > + transaction.xid_state.xa_state= XA_NOTR; > + transaction.xid_state.rm_error= 0; > + trans_rollback(this); > + xid_cache_delete(this, &transaction.xid_state); > } > -#endif > > mysql_ha_cleanup(this); > locked_tables_list.unlock_locked_tables(this); > @@ -1474,11 +1481,6 @@ void THD::cleanup(void) > delete_dynamic(&user_var_events); > close_temporary_tables(); > > - transaction.xid_state.xa_state= XA_NOTR; > - transaction.xid_state.rm_error= 0; > - trans_rollback(this); > - xid_cache_delete(this, &transaction.xid_state); > - > DBUG_ASSERT(open_tables == NULL); > /* > If the thread was in the middle of an ongoing transaction (rolled > diff --git a/sql/sql_class.h b/sql/sql_class.h > index 69fabee..76befcb 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -1255,6 +1255,18 @@ typedef struct st_xid_state { > /* Error reported by the Resource Manager (RM) to the Transaction Manager. */ > uint rm_error; > XID_cache_element *xid_cache_element; > + /* > + Binary logging status. > + It is set to TRUE at XA PREPARE if the transaction was written > + to the binlog. > + Naturally FALSE means the transaction was not written to > + the binlog. Happens if the trnasaction did not modify anything > + or binlogging was turned off. In that case we shouldn't binlog > + the consequent XA COMMIT/ROLLBACK. > + The recovered transaction after server restart sets it to TRUE always. > + That can cause inconsistencies (shoud be fixed?). > + */ > + bool is_binlogged; > > /** > Check that XA transaction has an uncommitted work. Report an error > @@ -1278,6 +1290,12 @@ typedef struct st_xid_state { > } > return false; > } > + > + void reset() > + { > + xid.null(); > + is_binlogged= FALSE; > + } > } XID_STATE; > > void xid_cache_init(void); > @@ -2603,7 +2621,7 @@ class THD :public Statement, > then. > */ > if (!xid_state.rm_error) > - xid_state.xid.null(); > + xid_state.reset(); > free_root(&mem_root,MYF(MY_KEEP_PREALLOC)); > DBUG_VOID_RETURN; > } > diff --git a/sql/sql_connect.cc b/sql/sql_connect.cc > index b48070b..3e4a067 100644 > --- a/sql/sql_connect.cc > +++ b/sql/sql_connect.cc > @@ -1414,6 +1414,7 @@ void do_handle_one_connection(CONNECT *connect) > #endif > end_thread: > close_connection(thd); > + thd->get_stmt_da()->reset_diagnostics_area(); > > if (thd->userstat_running) > update_global_user_stats(thd, create_user, time(NULL)); > diff --git a/sql/transaction.cc b/sql/transaction.cc > index 13614d3..64533d7 100644 > --- a/sql/transaction.cc > +++ b/sql/transaction.cc > @@ -790,6 +790,44 @@ bool trans_release_savepoint(THD *thd, LEX_CSTRING name) > > > /** > + Detach the current XA transaction; > + > + @param thd Current thread > + > + @retval FALSE Success > + @retval TRUE Failure > +*/ > + > +bool trans_detach(THD *thd) > +{ > + XID_STATE *xid_s= &thd->transaction.xid_state; > + Ha_trx_info *ha_info, *ha_info_next; > + > + DBUG_ENTER("trans_detach"); > + > +// DBUG_ASSERT(xid_s->xa_state == XA_PREPARED && > +// xid_cache_search(thd, &xid_s->xid)); > + > + xid_cache_delete(thd, xid_s); > + if (xid_cache_insert(&xid_s->xid, XA_PREPARED)) > + DBUG_RETURN(TRUE); > + > + for (ha_info= thd->transaction.all.ha_list; > + ha_info; > + ha_info= ha_info_next) > + { > + ha_info_next= ha_info->next(); > + ha_info->reset(); /* keep it conveniently zero-filled */ > + } > + > + thd->transaction.all.ha_list= 0; > + thd->transaction.all.no_2pc= 0; > + > + DBUG_RETURN(FALSE); > +} > + > + > +/** > Starts an XA transaction with the given xid value. > > @param thd Current thread > @@ -928,6 +966,12 @@ bool trans_xa_commit(THD *thd) > res= !xs; > if (res) > my_error(ER_XAER_NOTA, MYF(0)); > + else if (thd->in_multi_stmt_transaction_mode()) > + { > + my_error(ER_XAER_RMFAIL, MYF(0), > + xa_state_names[thd->transaction.xid_state.xa_state]); > + res= TRUE; > + } > else > { > res= xa_trans_rolled_back(xs); > @@ -978,8 +1022,16 @@ bool trans_xa_commit(THD *thd) > { > DEBUG_SYNC(thd, "trans_xa_commit_after_acquire_commit_lock"); > > - res= MY_TEST(ha_commit_one_phase(thd, 1)); > - if (res) > + if(WSREP_EMULATE_BINLOG(thd) || mysql_bin_log.is_open()) > + { > + res= thd->binlog_query(THD::THD::STMT_QUERY_TYPE, > + thd->query(), thd->query_length(), > + FALSE, FALSE, FALSE, 0); > + } > + else > + res= 0; > + > + if (res || (res= MY_TEST(ha_commit_one_phase(thd, 1)))) > my_error(ER_XAER_RMERR, MYF(0)); > } > } > @@ -1044,7 +1096,18 @@ bool trans_xa_rollback(THD *thd) > DBUG_RETURN(TRUE); > } > > - res= xa_trans_force_rollback(thd); > + if(WSREP_EMULATE_BINLOG(thd) || mysql_bin_log.is_open()) > + { > + res= thd->binlog_query(THD::THD::STMT_QUERY_TYPE, > + thd->query(), thd->query_length(), > + FALSE, FALSE, FALSE, 0); > + } > + else > + res= 0; > + > + res= res || xa_trans_force_rollback(thd); > + if (res || (res= MY_TEST(xa_trans_force_rollback(thd)))) > + my_error(ER_XAER_RMERR, MYF(0)); > > thd->variables.option_bits&= ~(OPTION_BEGIN | OPTION_KEEP_LOG); > thd->transaction.all.reset(); > diff --git a/sql/transaction.h b/sql/transaction.h > index 7e34693..f228cc6 100644 > --- a/sql/transaction.h > +++ b/sql/transaction.h > @@ -29,6 +29,7 @@ bool trans_commit(THD *thd); > bool trans_commit_implicit(THD *thd); > bool trans_rollback(THD *thd); > bool trans_rollback_implicit(THD *thd); > +bool trans_detach(THD *thd); > > bool trans_commit_stmt(THD *thd); > bool trans_rollback_stmt(THD *thd); > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

Re: [Maria-developers] 297ac6aa5b0: MDEV-7597 Expiration of user passwords
by Sergei Golubchik 13 Feb '19

13 Feb '19

Hi, Robert! On Feb 13, Robert Bindar wrote: > revision-id: 297ac6aa5b0 (mariadb-10.4.1-101-g297ac6aa5b0) > parent(s): ce02e7c9acc > author: Robert Bindar <robert(a)mariadb.org> > committer: Robert Bindar <robert(a)mariadb.org> > timestamp: 2019-02-07 17:47:13 +0200 > message: > > MDEV-7597 Expiration of user passwords > > This patch adds support for expiring user passwords. > The following statements are extended: > CREATE USER user@localhost PASSWORD EXPIRE [option] > ALTER USER user@localhost PASSWORD EXPIRE [option] > If no option is specified, the password is expired with immediate > effect. If option is DEFAULT, global policy applies according to > the default_password_lifetime system var (if 0, password never > expires, if N, password expires every N days). If option is NEVER, > the password never expires and if option is INTERVAL N DAY, the > password expires every N days. > The feature also supports the disconnect_on_expired_password system > var and the --connect-expired-password client option. > > diff --git a/libmariadb b/libmariadb > index 34f8887af03..70f2964dc4d 160000 > --- a/libmariadb > +++ b/libmariadb > @@ -1 +1 @@ > -Subproject commit 34f8887af03d022416dd6593de91d0706e57f46b > +Subproject commit 70f2964dc4de116f4b50732cfec7cb566e082b4c This shouldn't be in your commit > diff --git a/mysql-test/main/grant5.result b/mysql-test/main/grant5.result > index 3d0f331e58a..3bb81266b4d 100644 > --- a/mysql-test/main/grant5.result > +++ b/mysql-test/main/grant5.result > @@ -67,21 +67,21 @@ show create user u1@h; > ERROR 28000: Can't find any matching row in the user table > show create user u2@h; > CREATE USER for u2@h > -CREATE USER 'u2'@'h' IDENTIFIED BY PASSWORD '*975B2CD4FF9AE554FE8AD33168FBFC326D2021DD' ACCOUNT UNLOCK > +CREATE USER 'u2'@'h' IDENTIFIED BY PASSWORD '*975B2CD4FF9AE554FE8AD33168FBFC326D2021DD' PASSWORD EXPIRE DEFAULT ACCOUNT UNLOCK Just like with account locking, I wouldn't print default behavior clauses here For example, we don't print REQUIRE NONE WITH MAX_QUERIES_PER_HOUR 0 MAX_UPDATES_PER_HOUR 0 MAX_CONNECTIONS_PER_HOUR 0 MAX_USER_CONNECTIONS 0 > show create user u3@h; > CREATE USER for u3@h > -CREATE USER 'u3'@'h' IDENTIFIED BY PASSWORD '*975B2CD4FF9AE554FE8AD33168FBFC326D2021DD' ACCOUNT UNLOCK > +CREATE USER 'u3'@'h' IDENTIFIED BY PASSWORD '*975B2CD4FF9AE554FE8AD33168FBFC326D2021DD' PASSWORD EXPIRE DEFAULT ACCOUNT UNLOCK > show create user u4@h; > CREATE USER for u4@h > -CREATE USER 'u4'@'h' IDENTIFIED BY PASSWORD '*975B2CD4FF9AE554FE8AD33168FBFC326D2021DD' ACCOUNT UNLOCK > +CREATE USER 'u4'@'h' IDENTIFIED BY PASSWORD '*975B2CD4FF9AE554FE8AD33168FBFC326D2021DD' PASSWORD EXPIRE DEFAULT ACCOUNT UNLOCK > show create user u5@h; > ERROR 28000: Can't find any matching row in the user table > show create user u6@h; > CREATE USER for u6@h > -CREATE USER 'u6'@'h' IDENTIFIED BY PASSWORD '78a302dd267f6044' ACCOUNT UNLOCK > +CREATE USER 'u6'@'h' IDENTIFIED BY PASSWORD '78a302dd267f6044' PASSWORD EXPIRE DEFAULT ACCOUNT UNLOCK > show create user u7@h; > CREATE USER for u7@h > -CREATE USER 'u7'@'h' IDENTIFIED BY PASSWORD '78a302dd267f6044' ACCOUNT UNLOCK > +CREATE USER 'u7'@'h' IDENTIFIED BY PASSWORD '78a302dd267f6044' PASSWORD EXPIRE DEFAULT ACCOUNT UNLOCK > show create user u8@h; > ERROR 28000: Can't find any matching row in the user table > grant select on *.* to u1@h; > diff --git a/mysql-test/main/password_expiration_dbug.test b/mysql-test/main/password_expiration_dbug.test > new file mode 100644 > index 00000000000..99cbf5a6efb > --- /dev/null > +++ b/mysql-test/main/password_expiration_dbug.test > @@ -0,0 +1,57 @@ > +# > +# Test password expiration INTERVAL and default_password_lifetime options > +# > + > +--source include/have_debug.inc > +--source include/not_embedded.inc > + > +set @old_dbug=@@global.debug_dbug; > +set global debug_dbug="+d,password_expiration_interval_sec"; > + > +--echo # > +--echo # PASSWORD EXPIRE DEFAULT should use the default_password_lifetime > +--echo # system var to set the number of days till expiration > +--echo # > +set global disconnect_on_expired_password=ON; > +set global default_password_lifetime=1; > +create user user1@localhost password expire default; > +--sleep 2 I don't like that. all sleep-based tests have proven to be very unreliable in buildbot :( I don't have any good ideas, you cannot change the global server time from the test. May be, mess with a timezone? Like, set the expiration for a few minutes (like you do) and change server's timezone to move the time forward? Might work. Depends on what time you use for expiration - local or UTC. > +--replace_result $MASTER_MYPORT MYSQL_PORT $MASTER_MYSOCK MYSQL_SOCK > +--error ER_MUST_CHANGE_PASSWORD_LOGIN > +connect(con1,localhost,user1); > +drop user user1@localhost; > + > +--echo # > +--echo # PASSWORD EXPIRE INTERVAL should expire a client's password after > +--echo # X seconds and not before > +--echo # > +set global disconnect_on_expired_password=ON; > +create user user1@localhost password expire interval 2 day; > +--replace_result $MASTER_MYPORT MYSQL_PORT $MASTER_MYSOCK MYSQL_SOCK > +connect(con1,localhost,user1); > +disconnect con1; > +connection default; > +--sleep 3 > +--replace_result $MASTER_MYPORT MYSQL_PORT $MASTER_MYSOCK MYSQL_SOCK > +--error ER_MUST_CHANGE_PASSWORD_LOGIN > +connect(con1,localhost,user1); > +drop user user1@localhost; > + > +--echo # > +--echo # PASSWORD EXPIRE NEVER should override the other policies and never > +--echo # expire a client's password > +--echo # > +set global disconnect_on_expired_password=ON; > +create user user1@localhost password expire interval 2 day; > +alter user user1@localhost password expire never; > +--sleep 3 > +--replace_result $MASTER_MYPORT MYSQL_PORT $MASTER_MYSOCK MYSQL_SOCK > +connect(con1,localhost,user1); > +disconnect con1; > +connection default; > +drop user user1@localhost; > + > +set global debug_dbug=@old_dbug; > +set global disconnect_on_expired_password=default; > +set global default_password_lifetime=default; > + > diff --git a/mysql-test/main/system_mysql_db_fix40123.result b/mysql-test/main/system_mysql_db_fix40123.result > index 3903dbdad94..a96a33ce54f 100644 > --- a/mysql-test/main/system_mysql_db_fix40123.result > +++ b/mysql-test/main/system_mysql_db_fix40123.result > @@ -108,6 +108,8 @@ user CREATE TABLE `user` ( > `plugin` char(64) CHARACTER SET latin1 NOT NULL DEFAULT '', > `authentication_string` text COLLATE utf8_bin NOT NULL, > `password_expired` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N', > + `password_last_changed` timestamp NOT NULL DEFAULT current_timestamp(), > + `password_lifetime` smallint(5) unsigned NOT NULL DEFAULT 0, I've fixed it, so when you rebase they should go away, make sure to rerun these tests. > `account_locked` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N', > `is_role` enum('N','Y') CHARACTER SET utf8 NOT NULL DEFAULT 'N', > `default_role` char(80) COLLATE utf8_bin NOT NULL DEFAULT '', > diff --git a/scripts/mysql_system_tables_fix.sql b/scripts/mysql_system_tables_fix.sql > index 04a9bd9785a..e8af55993f5 100644 > --- a/scripts/mysql_system_tables_fix.sql > +++ b/scripts/mysql_system_tables_fix.sql > @@ -805,6 +807,9 @@ IF 'BASE TABLE' = (select table_type from information_schema.tables where table_ > 'max_statement_time', max_statement_time, > 'plugin', if(plugin>'',plugin,if(length(password)=16,'mysql_old_password','mysql_native_password')), > 'authentication_string', if(plugin>'',authentication_string,password), > + 'password_expired', 'Y'=password_expired, Why do you need an explicit password_expired value? I'd just use password_last_changed=0 for that. It'll work automatically (changed too back in the past so it makes it expired) without an explicit check for password_expired==1 > + 'password_last_changed', UNIX_TIMESTAMP(password_last_changed), > + 'password_lifetime', ifnull(password_lifetime, -1), > 'account_locked', 'Y'=account_locked, > 'default_role', default_role, > 'is_role', 'Y'=is_role)) as Priv > diff --git a/sql/share/errmsg-utf8.txt b/sql/share/errmsg-utf8.txt > index 4536abd2907..e0db27165c4 100644 > --- a/sql/share/errmsg-utf8.txt > +++ b/sql/share/errmsg-utf8.txt > @@ -6940,8 +6940,9 @@ ER_NOT_VALID_PASSWORD > eng "Your password does not satisfy the current policy requirements" > > ER_MUST_CHANGE_PASSWORD > - eng "You must SET PASSWORD before executing this statement" > + eng "You must change your password using either ALTER USER or SET PASSWORD before executing this statement" > bgn "Трябва първо да си смените паролата със SET PASSWORD за да можете да изпълните тази команда" > + rum "Trebuie sa iti schimbi parola folosind ALTER USER sau SET PASSWORD inainte de a executa aceasta comanda" Let's keep the old SET PASSWORD message. It seems that ALTER USER is more appropriate for DBA, for account administration, not for mere users. > > ER_FK_NO_INDEX_CHILD > eng "Failed to add the foreign key constaint. Missing index for constraint '%s' in the foreign table '%s'" > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index 983686d51c7..f0a8ffb1a60 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -634,7 +638,7 @@ static ACL_ROLE *find_acl_role(const char *user); > static ROLE_GRANT_PAIR *find_role_grant_pair(const LEX_CSTRING *u, const LEX_CSTRING *h, const LEX_CSTRING *r); > static ACL_USER_BASE *find_acl_user_base(const char *user, const char *host); > static bool update_user_table_password(THD *, const User_table&, > - const ACL_USER &); > + ACL_USER &); use only const references or pointers. So, this one should become a pointer. > static bool acl_load(THD *thd, const Grant_tables& grant_tables); > static inline void get_grantor(THD *thd, char* grantor); > static bool add_role_user_mapping(const char *uname, const char *hname, const char *rname); > @@ -2705,6 +2806,7 @@ bool acl_getroot(Security_context *sctx, const char *user, const char *host, > sctx->master_access= 0; > sctx->db_access= 0; > *sctx->priv_user= *sctx->priv_host= *sctx->priv_role= 0; > + sctx->set_password_expired(false); I'd rather call init() here. Future-proof. > > if (host[0]) // User, not Role > { > @@ -3457,6 +3565,17 @@ bool change_password(THD *thd, LEX_USER *user) > goto end; > } > > + if (sctx->password_expired() && > + (strcmp(acl_user->user.str, sctx->priv_user) || > + my_strcasecmp(system_charset_info, > + acl_user->host.hostname, > + sctx->priv_host))) > + { > + mysql_mutex_unlock(&acl_cache->lock); > + my_error(ER_MUST_CHANGE_PASSWORD, MYF(0)); > + goto end; > + } > + technically, check_alter_user() should've done that already, I'm not sure you need to repeat the check here. > if (update_user_table_password(thd, tables.user_table(), *acl_user)) > { > mysql_mutex_unlock(&acl_cache->lock); /* purecov: deadcode */ > @@ -3930,6 +4049,12 @@ static bool update_user_table_password(THD *thd, const User_table& user_table, > user_table.set_auth(user.plugin.str, user.plugin.length, > user.auth_string.str, user.auth_string.length); > > + /* Update the persistent password expired state of user */ > + user_table.set_password_expired(false); > + thd->set_time(); Hmm, why? This is done at the start of every statement, why do you repeat it here? > + Timeval now= thd->query_start_timeval(); Just simply m_time_t now= thd->query_start(); Don't use query_start_timeval() or query_start_sec_part() unless you _really_ need microsecond precision. They have a side-effect of server remembering that you asked for microseconds, so they'll be logged in a binary log. I've now removed THD::query_start_timeval(), it's too error-prone > + user_table.set_password_last_changed(now.tv_sec); > + > if (unlikely(error= table->file->ha_update_row(table->record[1], > table->record[0])) && > error != HA_ERR_RECORD_IS_THE_SAME) > @@ -3937,6 +4062,19 @@ static bool update_user_table_password(THD *thd, const User_table& user_table, > table->file->print_error(error,MYF(0)); /* purecov: deadcode */ > DBUG_RETURN(1); > } > + > + /* Update the acl password expired state of user */ > + user.password_last_changed= now.tv_sec; > + user.password_expired= false; > + > + /* If user is the connected user, reset the password expired field on sctx > + and allow the user to exit sandbox mode */ > + if (!strcmp(user.user.str, thd->security_ctx->priv_user) && > + !my_strcasecmp(system_charset_info, > + user.host.hostname, > + thd->security_ctx->priv_host)) this should be a method in ACL_USER or in Security_context - it's used too often. > + thd->security_ctx->set_password_expired(false); > + > DBUG_RETURN(0); > } > > @@ -4153,6 +4291,68 @@ static int replace_user_table(THD *thd, const User_table &user_table, > user_table.set_account_locked(lock_value); > new_acl_user.account_locked= lock_value; > } > + > + thd->set_time(); > + Timeval now= thd->query_start_timeval(); same as above > + if (!old_row_exists) > + { > + user_table.set_password_last_changed(now.tv_sec); > + new_acl_user.password_last_changed= now.tv_sec; > + new_acl_user.use_default_pass_lifetime= true; > + } > + > + /* Unexpire the user password */ > + if (combo->is_changing_password) Is it needed? Couldn't you just check if the password was specified? > + { > + user_table.set_password_expired(false); > + user_table.set_password_last_changed(now.tv_sec); > + new_acl_user.password_last_changed= now.tv_sec; > + new_acl_user.password_expired= false; > + > + /* If combo is the connected user, reset the password expired field on sctx > + and allow the user to exit sandbox mode */ > + if (!strcmp(combo->user.str, thd->security_ctx->priv_user) && > + !my_strcasecmp(system_charset_info, > + combo->host.str, > + thd->security_ctx->priv_host)) > + thd->security_ctx->set_password_expired(false); I tend to think (again) that ALTER USER is for DBA's, and the user should use SET PASSWORD. So, let's not allow ALTER USER in the sandbox mode and, consequently, you shouldn't need the check above either. For example, ALTER USER can specify MAX_QUERIES_PER_HOUR. Surely, it's for DBA to specify the limit, it doesn't make any sense to allow every user to overwrite his own limits. And even for authentication, think of PAM. A user can change his own password, but not /etc/pam.d/login for example. Similarly here a user should be able to change his password, but not the authentication plugin. > + } > + > + switch (lex->account_options.password_expire) { > + case PASSWORD_EXPIRE_UNSPECIFIED: > + break; > + case PASSWORD_EXPIRE_NOW: > + user_table.set_password_expired(true); > + new_acl_user.password_expired= true; > + new_acl_user.use_default_pass_lifetime= false; why do you change use_default_pass_lifetime? > + break; > + case PASSWORD_EXPIRE_NEVER: > + user_table.set_password_lifetime(0); > + new_acl_user.password_lifetime= 0; > + new_acl_user.use_default_pass_lifetime= false; > + break; > + case PASSWORD_EXPIRE_DEFAULT: > + user_table.set_password_lifetime_null(); > + new_acl_user.use_default_pass_lifetime= true; > + new_acl_user.password_lifetime= -1; why two variables to say the same thing? > + break; > + case PASSWORD_EXPIRE_INTERVAL: > + longlong interval= lex->account_options.num_expiration_days; > + Timestamp exp_tstamp(new_acl_user.password_last_changed, 0); > + int warn= 0; > + exp_tstamp.add(3600 * 24 * interval, &warn); I'm not sure why you're doing it. If the user sets too many days here - fine, why would that be a problem? > + if (warn & MYSQL_TIME_WARN_OUT_OF_RANGE) > + { > + char num[MAX_BIGINT_WIDTH + 1]; > + my_snprintf(num, sizeof(num), "%lu", interval); > + my_error(ER_WRONG_VALUE, MYF(0), "DAY", num); > + goto end; > + } > + user_table.set_password_lifetime(interval); > + new_acl_user.password_lifetime= interval; > + new_acl_user.use_default_pass_lifetime= false; > + break; > + } > } > > if (old_row_exists) > @@ -10548,6 +10764,39 @@ int mysql_alter_user(THD* thd, List<LEX_USER> &users_list) > > LEX_USER *tmp_lex_user; > List_iterator<LEX_USER> users_list_iterator(users_list); > + bool password_expired= thd->security_ctx->password_expired(); > + bool own_password_changed= false; > + > + /* Precheck the users list to see if the password of the connected > + user is being changed */ > + if (password_expired) > + { // TODO: check bug with identified by '' not blanking the password > + Security_context *sctx= thd->security_ctx; > + while ((tmp_lex_user= users_list_iterator++)) > + { > + LEX_USER* lex_user= get_current_user(thd, tmp_lex_user, false); > + > + bool is_self= !strcmp(lex_user->user.str, sctx->priv_user) && > + !my_strcasecmp(system_charset_info, > + lex_user->host.str, > + sctx->priv_host); > + if (lex_user->is_changing_password && is_self) > + { > + own_password_changed= true; > + break; > + } > + } > + > + if (!own_password_changed) > + { > + mysql_mutex_unlock(&acl_cache->lock); > + mysql_rwlock_unlock(&LOCK_grant); > + my_error(ER_MUST_CHANGE_PASSWORD, MYF(0)); already commented earlier. Just do ER_MUST_CHANGE_PASSWORD always if in a sandbox mode > + DBUG_RETURN(1); > + } > + users_list_iterator.init(users_list); > + } > + > while ((tmp_lex_user= users_list_iterator++)) > { > LEX_USER* lex_user= get_current_user(thd, tmp_lex_user, false); > @@ -13294,6 +13543,44 @@ static void handle_password_errors(const char *user, const char *hostname, PASSW > #endif > } > > +bool check_password_lifetime(THD *thd, const ACL_USER *acl_user) > +{ > + /* the password should never expire */ > + if (!acl_user->password_lifetime) > + return false; > + > + longlong interval= acl_user->password_lifetime; > + if (acl_user->use_default_pass_lifetime) > + { > + mysql_mutex_lock(&LOCK_global_system_variables); > + interval= default_password_lifetime; > + mysql_mutex_unlock(&LOCK_global_system_variables); I'd skip the mutex lock here. The worst that can happen, if someone would be logging in when default_password_lifetime is just being modified - that this someone might be classified incorrectly, allowed in with a expired password or not allowed with a non-expired. Which is a small price to pay to avoid every single user (because most users never change default settings) taking yet another mutex lock on every single login. > + > + /* default global policy applies, and that is password never expires */ > + if (!interval) > + return false; > + } > + > + thd->set_time(); > + Timeval tval_now= thd->query_start_timeval(); and again > + Timestamp tstamp_now(tval_now); > + Timestamp expiration_tstamp(acl_user->password_last_changed, 0); > + int interval_sec= 3600 * 24 * interval; > + int warn= 0; > + > + /* this helps test set a testable password lifetime in seconds not days */ > + DBUG_EXECUTE_IF("password_expiration_interval_sec", { interval_sec= interval; }); > + > + expiration_tstamp.add(interval_sec, &warn); > + > + DBUG_ASSERT(!(warn & MYSQL_TIME_WARN_OUT_OF_RANGE)); > + > + if (tstamp_now.cmp(expiration_tstamp) >= 0) { > + return true; > + } why not to write simply if ((thd->query_start() - acl_user->password_last_changed)/24/3600 > interval) this keeps all the code in one (or two) lines, instead of spreading it over two Timestamp::Timestamp constructors, Timestamp::add() and Timestamp::cmp() methods. And it doesn't have a problem with large interval values, as a bonus. > + > + return false; > +} > > /** > Perform the handshake, authorize the client and update thd sctx variables. > @@ -13512,6 +13799,25 @@ bool acl_authenticate(THD *thd, uint com_change_user_pkt_len) > DBUG_RETURN(1); > } > > + bool disconnect_password_expired= true; > + bool client_can_handle_exp_pass= thd->client_capabilities & > + CLIENT_CAN_HANDLE_EXPIRED_PASSWORDS; > + bool password_lifetime_due= check_password_lifetime(thd, acl_user); > + mysql_mutex_lock(&LOCK_global_system_variables); > + disconnect_password_expired= disconnect_on_expired_password; > + mysql_mutex_unlock(&LOCK_global_system_variables); same as above > + > + if (!client_can_handle_exp_pass && disconnect_password_expired && > + (acl_user->password_expired || password_lifetime_due)) by now you've locked the mutex twice, did some calculations to see if the password expired... and only then you've checked acl_user->password_expired. > + { > + status_var_increment(denied_connections); > + my_error(ER_MUST_CHANGE_PASSWORD_LOGIN, MYF(0)); > + DBUG_RETURN(1); > + } > + > + sctx->set_password_expired(acl_user->password_expired || > + password_lifetime_due); > + > /* > Don't allow the user to connect if he has done too many queries. > As we are testing max_user_connections == 0 here, it means that we > diff --git a/sql/sql_class.h b/sql/sql_class.h > index 69fabee708c..57b043fb8e4 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -1340,6 +1342,11 @@ class Security_context { > restore_security_context(THD *thd, Security_context *backup); > #endif > bool user_matches(Security_context *); > + > + bool password_expired() > + { return pass_expired; } > + void set_password_expired(bool val) > + { pass_expired= val; } Better remove them, this class doesn't have other trivial setters/getters either. > }; > > > diff --git a/sql/sql_const.h b/sql/sql_const.h > index 82f3b9c21f2..3d888c25a9f 100644 > --- a/sql/sql_const.h > +++ b/sql/sql_const.h > @@ -275,6 +275,7 @@ > #define DELAYED_QUEUE_SIZE 1000 > #define DELAYED_WAIT_TIMEOUT (5*60) /**< Wait for delayed insert */ > #define MAX_CONNECT_ERRORS 100 ///< errors before disabling host > +#define DEFAULT_PASSWORD_LIFETIME 0 ///< days before passwords expire no need to create a name for that > > #define LONG_TIMEOUT ((ulong) 3600L*24L*365L) > > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > index 65b52b5b5da..35e9e869b49 100644 > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -1687,6 +1687,17 @@ bool dispatch_command(enum enum_server_command command, THD *thd, > thd->get_stmt_da()->set_skip_flush(); > } > > + if (unlikely(thd->security_ctx->password_expired() && > + command != COM_QUERY && > + command != COM_STMT_CLOSE && > + command != COM_STMT_SEND_LONG_DATA && > + command != COM_PING && > + command != COM_QUIT)) Why do you allow COM_STMT_CLOSE and COM_STMT_SEND_LONG_DATA? > + { > + my_error(ER_MUST_CHANGE_PASSWORD, MYF(0)); > + goto dispatch_end; > + } > + > switch (command) { > case COM_INIT_DB: > { > @@ -3249,6 +3260,14 @@ mysql_execute_command(THD *thd) > #endif > DBUG_ENTER("mysql_execute_command"); > > + if (thd->security_ctx->password_expired() && > + lex->sql_command != SQLCOM_SET_OPTION && > + lex->sql_command != SQLCOM_ALTER_USER) SQLCOM_SET_OPTION would be enough > + { > + my_error(ER_MUST_CHANGE_PASSWORD, MYF(0)); > + DBUG_RETURN(1); > + } > + > DBUG_ASSERT(thd->transaction.stmt.is_empty() || thd->in_sub_stmt); > /* > Each statement or replication event which might produce deadlock > diff --git a/sql/sql_prepare.cc b/sql/sql_prepare.cc > index cb822fc2e98..e621bd6519e 100644 > --- a/sql/sql_prepare.cc > +++ b/sql/sql_prepare.cc > @@ -4188,14 +4188,11 @@ Prepared_statement::execute_loop(String *expanded_query, > if (set_parameters(expanded_query, packet, packet_end)) > return TRUE; > > -#ifdef NOT_YET_FROM_MYSQL_5_6 > - if (unlikely(thd->security_ctx->password_expired && > - !lex->is_change_password)) > + if (unlikely(thd->security_ctx->password_expired())) Really? How can one end up here, if you don't allow COM_STMT_PREPARE and COM_STMT_EXECUTE and SQLCOM_PREPARE/SQLCOM_EXECUTE ? > { > my_error(ER_MUST_CHANGE_PASSWORD, MYF(0)); > return true; > } > -#endif > > reexecute: > // Make sure that reprepare() did not create any new Items. > @@ -4361,15 +4358,12 @@ Prepared_statement::execute_bulk_loop(String *expanded_query, > } > read_types= FALSE; > > -#ifdef NOT_YET_FROM_MYSQL_5_6 > - if (unlikely(thd->security_ctx->password_expired && > - !lex->is_change_password)) > + if (unlikely(thd->security_ctx->password_expired())) Same > { > my_error(ER_MUST_CHANGE_PASSWORD, MYF(0)); > thd->set_bulk_execution(0); > return true; > } > -#endif > > // iterations changed by set_bulk_parameters > while ((iterations || start_param) && !error && !thd->is_error()) > diff --git a/sql/sql_type.cc b/sql/sql_type.cc > index 96a73a85267..e06272e4564 100644 > --- a/sql/sql_type.cc > +++ b/sql/sql_type.cc > @@ -618,20 +618,35 @@ Sec9 & Sec9::round(uint dec) > return *this; > } > > +void Timestamp::set_out_of_range(uint sec, uint usec, int *warn) > +{ > + *warn|= MYSQL_TIME_WARN_OUT_OF_RANGE; > + tv_sec= sec; > + tv_usec= usec; > +} > > void Timestamp::round_or_set_max(uint dec, int *warn) > { > DBUG_ASSERT(dec <= TIME_SECOND_PART_DIGITS); > if (add_nanoseconds_usec(msec_round_add[dec]) && > tv_sec++ >= TIMESTAMP_MAX_VALUE) > - { > - tv_sec= TIMESTAMP_MAX_VALUE; > - tv_usec= TIME_MAX_SECOND_PART; > - *warn|= MYSQL_TIME_WARN_OUT_OF_RANGE; > - } > + set_out_of_range(TIMESTAMP_MAX_VALUE, TIME_MAX_SECOND_PART, warn); > + > my_timeval_trunc(this, dec); > } > > +Timestamp& Timestamp::add(int sec, int *warn) > +{ > + longlong res= tv_sec + sec; > + if (res >= TIMESTAMP_MAX_VALUE) > + set_out_of_range(TIMESTAMP_MAX_VALUE, TIME_MAX_SECOND_PART, warn); > + else if (res < 0) > + set_out_of_range(0, 0, warn); > + else > + tv_sec= (my_time_t) res; > + > + return *this; > +} you don't need Timestamp class at all, nor any changes here. > > bool Temporal::add_nanoseconds_with_round(THD *thd, int *warn, > date_conv_mode_t mode, > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index 8f42b18c176..919aae6858f 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -8030,6 +8032,34 @@ opt_account_option: > { > Lex->account_options.account_locked= ACCOUNTLOCK_UNLOCKED; > } > + | PASSWORD_SYM EXPIRE_SYM opt_password_expire > + { } > + ; > +opt_password_expire: > + /* Nothing */ > + { > + Lex->account_options.password_expire= PASSWORD_EXPIRE_NOW; > + } > + | NEVER_SYM > + { > + Lex->account_options.password_expire= PASSWORD_EXPIRE_NEVER; > + } > + | DEFAULT > + { > + Lex->account_options.password_expire= PASSWORD_EXPIRE_DEFAULT; > + } > + | INTERVAL_SYM real_ulong_num DAY_SYM use NUM instead of real_ulong_num it'll make a number within UINT_MAX32 range which is what you need here anyway, no reason to limit it to UINT_MAX16. > + { > + if (unlikely($2 == 0 || $2 > UINT_MAX16)) > + { > + char num[MAX_BIGINT_WIDTH + 1]; > + my_snprintf(num, sizeof(num), "%lu", $2); > + my_yyabort_error((ER_WRONG_VALUE, MYF(0), "DAY", num)); > + } > + > + Lex->account_options.password_expire= PASSWORD_EXPIRE_INTERVAL; > + Lex->account_options.num_expiration_days= $2; > + } > ; > > ev_alter_on_schedule_completion: > diff --git a/sql/sys_vars.cc b/sql/sys_vars.cc > index 37b3d65e20a..ff521c10d9b 100644 > --- a/sql/sys_vars.cc > +++ b/sql/sys_vars.cc > @@ -1507,6 +1507,25 @@ static Sys_var_ulong Sys_max_connections( > DEFAULT(MAX_CONNECTIONS_DEFAULT), BLOCK_SIZE(1), NO_MUTEX_GUARD, > NOT_IN_BINLOG, ON_CHECK(0), ON_UPDATE(fix_max_connections)); > > +static Sys_var_ulong Sys_default_password_lifetime( > + "default_password_lifetime", > + "This defines the global password expiration policy. Defaults to 0, " > + "meaning automatic password expiration is disabled. If the value is a " > + "positive integer N, the passwords must be changed every N days. This " > + "behavior can be overriden using the password expiration options in " > + "ALTER USER.", Don't say "Defaults to 0", Just "0 means password expiration is disabled". > + GLOBAL_VAR(default_password_lifetime), CMD_LINE(REQUIRED_ARG), > + VALID_RANGE(0, UINT_MAX), DEFAULT(DEFAULT_PASSWORD_LIFETIME), > + BLOCK_SIZE(1)); > + > +static Sys_var_mybool Sys_disconnect_on_expired_password( > + "disconnect_on_expired_password", > + "This variable controls how the server handles clients with expired " > + "passwords. If enabled, the server disconnects the client, otherwise " > + "the server puts the client in sandbox mode.", Not exactly. "How the server handles clients _that don't know about the sandbox mode_" > + GLOBAL_VAR(disconnect_on_expired_password), CMD_LINE(OPT_ARG), > + DEFAULT(FALSE)); > + > static Sys_var_ulong Sys_max_connect_errors( > "max_connect_errors", > "If there is more than this number of interrupted connections from " Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] 4f070b59005: 29 Jan, MDEV-371 Unique indexes for blobs
by Sergei Golubchik 12 Feb '19

12 Feb '19

Hi, Sachin! That's generally pretty good. But I still don't understand what you store in frm and how you manipulate TABLE and TABLE_SHARE key lists. Could you please explain that? On Jan 29, Sachin Setiya wrote: > diff --git a/mysql-test/main/long_unique.result b/mysql-test/main/long_unique.result > new file mode 100644 > --- /dev/null > +++ b/mysql-test/main/long_unique.result > @@ -0,0 +1,1392 @@ > +#Structure of tests > +#First we will check all option for > +#table containing single unique column > +#table containing keys like unique(a,b,c,d) etc > +#then table containing 2 blob unique etc > +set @allowed_packet= @@max_allowed_packet; > +#table with single long blob column; > +create table t1(a blob unique ); > +insert into t1 values(1),(2),(3),(56),('sachin'),('maria'),(123456789034567891),(null),(null),(123456789034567890); > +#blob with primary key not allowed > +create table t2(a blob,primary key(a(10000))); > +ERROR 42000: Specified key was too long; max key length is 1000 bytes > +create table t3(a varchar(10000) primary key); > +ERROR 42000: Specified key was too long; max key length is 1000 bytes > +insert into t1 values(2); > +ERROR 23000: Duplicate entry '2' for key 'a' > +#table structure; > +desc t1; > +Field Type Null Key Default Extra > +a blob YES UNI NULL > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + UNIQUE KEY `a` (`a`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table t1 > +Non_unique 0 > +Key_name a > +Seq_in_index 1 > +Column_name a > +Collation A > +Cardinality NULL > +Sub_part NULL > +Packed NULL > +Null YES > +Index_type HASH Good. But this also mean that HASH is a valid index type, and that one can write CREATE TABLE t1 (a blob, unique (a) USING HASH) generally, this should work. And one can write CREATE TABLE t1 (a blob, unique (a) USING BTREE) and this should fail, "key too long; max key length is 1000 bytes" Note, the syntax is already possible. You just need to handle these cases correctly in mysql_prepare_create_table(). And if neither HASH nor BTREE is specified, it's auto-detection, basically. For short keys it's BTREE, for long keys it's HASH. This is already implemented by this patch :) > +Comment > +Index_comment > + > +MyISAM file: DATADIR/test/t1 > +Record format: Packed > +Character set: latin1_swedish_ci (8) > +Data records: 10 Deleted blocks: 0 > +Recordlength: 20 > + > +table description: > +Key Start Len Index Type > +1 12 8 multip. ulonglong NULL > +select * from information_schema.columns where table_schema = 'test' and table_name = 't1'; > +TABLE_CATALOG TABLE_SCHEMA TABLE_NAME COLUMN_NAME ORDINAL_POSITION COLUMN_DEFAULT IS_NULLABLE DATA_TYPE CHARACTER_MAXIMUM_LENGTH CHARACTER_OCTET_LENGTH NUMERIC_PRECISION NUMERIC_SCALE DATETIME_PRECISION CHARACTER_SET_NAME COLLATION_NAME COLUMN_TYPE COLUMN_KEY EXTRA PRIVILEGES COLUMN_COMMENT IS_GENERATED GENERATION_EXPRESSION > +def test t1 a 1 NULL YES blob 65535 65535 NULL NULL NULL NULL NULL blob UNI select,insert,update,references NEVER NULL > +select * from information_schema.statistics where table_schema = 'test' and table_name = 't1'; > +TABLE_CATALOG TABLE_SCHEMA TABLE_NAME NON_UNIQUE INDEX_SCHEMA INDEX_NAME SEQ_IN_INDEX COLUMN_NAME COLLATION CARDINALITY SUB_PART PACKED NULLABLE INDEX_TYPE COMMENT INDEX_COMMENT > +def test t1 0 test a 1 a A NULL NULL NULL YES HASH > +select * from information_schema.key_column_usage where table_schema= 'test' and table_name= 't1'; > +CONSTRAINT_CATALOG def > +CONSTRAINT_SCHEMA test > +CONSTRAINT_NAME a > +TABLE_CATALOG def > +TABLE_SCHEMA test > +TABLE_NAME t1 > +COLUMN_NAME a > +ORDINAL_POSITION 1 > +POSITION_IN_UNIQUE_CONSTRAINT NULL > +REFERENCED_TABLE_SCHEMA NULL > +REFERENCED_TABLE_NAME NULL > +REFERENCED_COLUMN_NAME NULL > +# table select we should not be able to see db_row_hash_column; > +select * from t1 order by a; > +a > +NULL > +NULL > +1 > +123456789034567890 > +123456789034567891 > +2 > +3 > +56 > +maria > +sachin > +select db_row_hash_1 from t1; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 'field list' > +#duplicate entry test; > +insert into t1 values(2); > +ERROR 23000: Duplicate entry '2' for key 'a' > +insert into t1 values('sachin'); > +ERROR 23000: Duplicate entry 'sachin' for key 'a' > +insert into t1 values(123456789034567891); > +ERROR 23000: Duplicate entry '123456789034567891' for key 'a' > +select * from t1 order by a; > +a > +NULL > +NULL > +1 > +123456789034567890 > +123456789034567891 > +2 > +3 > +56 > +maria > +sachin > +insert into t1 values(11),(22),(33); > +insert into t1 values(12),(22); > +ERROR 23000: Duplicate entry '22' for key 'a' > +select * from t1 order by a; > +a > +NULL > +NULL > +1 > +11 > +12 > +123456789034567890 > +123456789034567891 > +2 > +22 > +3 > +33 > +56 > +maria > +sachin > +insert into t1 values(repeat('s',4000*10)),(repeat('s',4001*10)); > +insert into t1 values(repeat('m',4000*10)),(repeat('m',4000*10)); > +ERROR 23000: Duplicate entry 'mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm' for key 'a' > +insert into t1 values(repeat('m',4001)),(repeat('m',4002)); > +truncate table t1; > +insert into t1 values(1),(2),(3),(4),(5),(8),(7); > + > +MyISAM file: DATADIR/test/t1 > +Record format: Packed > +Character set: latin1_swedish_ci (8) > +Data records: 7 Deleted blocks: 0 > +Recordlength: 20 > + > +table description: > +Key Start Len Index Type > +1 12 8 multip. ulonglong NULL > +#now some alter commands; > +alter table t1 add column b int; > +desc t1; > +Field Type Null Key Default Extra > +a blob YES UNI NULL > +b int(11) YES NULL > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + `b` int(11) DEFAULT NULL, > + UNIQUE KEY `a` (`a`(65535)) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +insert into t1 values(1,2); > +ERROR 23000: Duplicate entry '1' for key 'a' > +insert into t1 values(2,2); > +ERROR 23000: Duplicate entry '2' for key 'a' > +select db_row_hash_1 from t1; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 'field list' > +#now try to change db_row_hash_1 column; > +alter table t1 drop column db_row_hash_1; > +ERROR 42000: Can't DROP COLUMN `db_row_hash_1`; check that it exists > +alter table t1 add column d int , add column e int , drop column db_row_hash_1; > +ERROR 42000: Can't DROP COLUMN `db_row_hash_1`; check that it exists > +alter table t1 modify column db_row_hash_1 int ; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 't1' > +alter table t1 add column a int , add column b int, modify column db_row_hash_1 int ; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 't1' > +alter table t1 change column db_row_hash_1 dsds int; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 't1' > +alter table t1 add column asd int, change column db_row_hash_1 dsds int; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 't1' > +alter table t1 drop column b , add column c int; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + `c` int(11) DEFAULT NULL, > + UNIQUE KEY `a` (`a`(65535)) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +#now add some column with name db_row_hash; > +alter table t1 add column db_row_hash_1 int unique; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + `c` int(11) DEFAULT NULL, > + `db_row_hash_1` int(11) DEFAULT NULL, > + UNIQUE KEY `a` (`a`(65535)), > + UNIQUE KEY `db_row_hash_1` (`db_row_hash_1`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +insert into t1 values(45,1,55),(46,1,55); > +ERROR 23000: Duplicate entry '55' for key 'db_row_hash_1' > +insert into t1 values(45,1,55),(45,1,55); > +ERROR 23000: Duplicate entry '45' for key 'a' > +alter table t1 add column db_row_hash_2 int, add column db_row_hash_3 int; > +desc t1; > +Field Type Null Key Default Extra > +a blob YES UNI NULL > +c int(11) YES NULL > +db_row_hash_1 int(11) YES UNI NULL > +db_row_hash_2 int(11) YES NULL > +db_row_hash_3 int(11) YES NULL > +#this should also drop the unique index ; > +alter table t1 drop column a; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `c` int(11) DEFAULT NULL, > + `db_row_hash_1` int(11) DEFAULT NULL, > + `db_row_hash_2` int(11) DEFAULT NULL, > + `db_row_hash_3` int(11) DEFAULT NULL, > + UNIQUE KEY `db_row_hash_1` (`db_row_hash_1`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 db_row_hash_1 1 db_row_hash_1 A NULL NULL NULL YES BTREE > +#add column with unique index on blob ; > +alter table t1 add column a blob unique; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `c` int(11) DEFAULT NULL, > + `db_row_hash_1` int(11) DEFAULT NULL, > + `db_row_hash_2` int(11) DEFAULT NULL, > + `db_row_hash_3` int(11) DEFAULT NULL, > + `a` blob DEFAULT NULL, > + UNIQUE KEY `db_row_hash_1` (`db_row_hash_1`), > + UNIQUE KEY `a` (`a`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +# try to change the blob unique name; > +#this will change index to b tree; > +alter table t1 change column a aa blob ; eh, the comment is wrong. Your old code changed the column to int, that was indeed changing the index to a btree. But now you simply rename the blob. the index is not changed to btree anymore. > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `c` int(11) DEFAULT NULL, > + `db_row_hash_1` int(11) DEFAULT NULL, > + `db_row_hash_2` int(11) DEFAULT NULL, > + `db_row_hash_3` int(11) DEFAULT NULL, > + `aa` blob DEFAULT NULL, > + UNIQUE KEY `db_row_hash_1` (`db_row_hash_1`), > + UNIQUE KEY `a` (`aa`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 db_row_hash_1 1 db_row_hash_1 A NULL NULL NULL YES BTREE > +t1 0 a 1 aa A NULL NULL NULL YES HASH > +# try to change the blob unique datatype; > +#this will change index to b tree; > +alter table t1 modify column aa int ; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `c` int(11) DEFAULT NULL, > + `db_row_hash_1` int(11) DEFAULT NULL, > + `db_row_hash_2` int(11) DEFAULT NULL, > + `db_row_hash_3` int(11) DEFAULT NULL, > + `aa` int(11) DEFAULT NULL, > + UNIQUE KEY `db_row_hash_1` (`db_row_hash_1`), > + UNIQUE KEY `a` (`aa`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 db_row_hash_1 1 db_row_hash_1 A NULL NULL NULL YES BTREE > +t1 0 a 1 aa A NULL NULL NULL YES BTREE > +alter table t1 add column clm blob unique; > +#try changing the name ; > +alter table t1 change column clm clm_changed blob; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `c` int(11) DEFAULT NULL, > + `db_row_hash_1` int(11) DEFAULT NULL, > + `db_row_hash_2` int(11) DEFAULT NULL, > + `db_row_hash_3` int(11) DEFAULT NULL, > + `aa` int(11) DEFAULT NULL, > + `clm_changed` blob DEFAULT NULL, > + UNIQUE KEY `db_row_hash_1` (`db_row_hash_1`), > + UNIQUE KEY `a` (`aa`), > + UNIQUE KEY `clm` (`clm_changed`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 db_row_hash_1 1 db_row_hash_1 A NULL NULL NULL YES BTREE > +t1 0 a 1 aa A NULL NULL NULL YES BTREE > +t1 0 clm 1 clm_changed A NULL NULL NULL YES HASH > +#now drop the unique key; > +alter table t1 drop key clm; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `c` int(11) DEFAULT NULL, > + `db_row_hash_1` int(11) DEFAULT NULL, > + `db_row_hash_2` int(11) DEFAULT NULL, > + `db_row_hash_3` int(11) DEFAULT NULL, > + `aa` int(11) DEFAULT NULL, > + `clm_changed` blob DEFAULT NULL, > + UNIQUE KEY `db_row_hash_1` (`db_row_hash_1`), > + UNIQUE KEY `a` (`aa`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 db_row_hash_1 1 db_row_hash_1 A NULL NULL NULL YES BTREE > +t1 0 a 1 aa A NULL NULL NULL YES BTREE > +drop table t1; > +create table t1 (a TEXT CHARSET latin1 COLLATE latin1_german2_ci unique); > +desc t1; > +Field Type Null Key Default Extra > +a text YES UNI NULL > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 a 1 a A NULL NULL NULL YES HASH > +insert into t1 values ('ae'); > +insert into t1 values ('AE'); > +ERROR 23000: Duplicate entry 'AE' for key 'a' > +insert into t1 values ('Ä'); > +drop table t1; > +create table t1 (a int primary key, b blob unique); > +desc t1; > +Field Type Null Key Default Extra > +a int(11) NO PRI NULL > +b blob YES UNI NULL > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 PRIMARY 1 a A 0 NULL NULL BTREE > +t1 0 b 1 b A NULL NULL NULL YES HASH > +insert into t1 values(1,1),(2,2),(3,3); > +insert into t1 values(1,1); > +ERROR 23000: Duplicate entry '1' for key 'b' > +insert into t1 values(7,1); > +ERROR 23000: Duplicate entry '1' for key 'b' > +drop table t1; > +#table with multiple long blob column and varchar text column ; > +create table t1(a blob unique, b int , c blob unique , d text unique , e varchar(3000) unique); > +insert into t1 values(1,2,3,4,5),(2,11,22,33,44),(3111,222,333,444,555),(5611,2222,3333,4444,5555), > +('sachin',341,'fdf','gfgfgfg','hghgr'),('maria',345,'frter','dasd','utyuty'), > +(123456789034567891,353534,53453453453456,64565464564564,45435345345345), > +(123456789034567890,43545,657567567567,78967657567567,657567567567567676); > +#table structure; > +desc t1; > +Field Type Null Key Default Extra > +a blob YES UNI NULL > +b int(11) YES NULL > +c blob YES UNI NULL > +d text YES UNI NULL > +e varchar(3000) YES UNI NULL > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + `b` int(11) DEFAULT NULL, > + `c` blob DEFAULT NULL, > + `d` text DEFAULT NULL, > + `e` varchar(3000) DEFAULT NULL, > + UNIQUE KEY `a` (`a`), > + UNIQUE KEY `c` (`c`), > + UNIQUE KEY `d` (`d`), > + UNIQUE KEY `e` (`e`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 a 1 a A NULL NULL NULL YES HASH > +t1 0 c 1 c A NULL NULL NULL YES HASH > +t1 0 d 1 d A NULL NULL NULL YES HASH > +t1 0 e 1 e A NULL NULL NULL YES HASH > + > +MyISAM file: DATADIR/test/t1 > +Record format: Packed > +Character set: latin1_swedish_ci (8) > +Data records: 8 Deleted blocks: 0 > +Recordlength: 3072 > + > +table description: > +Key Start Len Index Type > +1 3063 8 multip. ulonglong NULL > +2 3055 8 multip. ulonglong NULL > +3 3047 8 multip. ulonglong NULL > +4 3039 8 multip. ulonglong NULL > +select * from information_schema.columns where table_schema = 'test' and table_name = 't1'; > +TABLE_CATALOG TABLE_SCHEMA TABLE_NAME COLUMN_NAME ORDINAL_POSITION COLUMN_DEFAULT IS_NULLABLE DATA_TYPE CHARACTER_MAXIMUM_LENGTH CHARACTER_OCTET_LENGTH NUMERIC_PRECISION NUMERIC_SCALE DATETIME_PRECISION CHARACTER_SET_NAME COLLATION_NAME COLUMN_TYPE COLUMN_KEY EXTRA PRIVILEGES COLUMN_COMMENT IS_GENERATED GENERATION_EXPRESSION > +def test t1 a 1 NULL YES blob 65535 65535 NULL NULL NULL NULL NULL blob UNI select,insert,update,references NEVER NULL > +def test t1 b 2 NULL YES int NULL NULL 10 0 NULL NULL NULL int(11) select,insert,update,references NEVER NULL > +def test t1 c 3 NULL YES blob 65535 65535 NULL NULL NULL NULL NULL blob UNI select,insert,update,references NEVER NULL > +def test t1 d 4 NULL YES text 65535 65535 NULL NULL NULL latin1 latin1_swedish_ci text UNI select,insert,update,references NEVER NULL > +def test t1 e 5 NULL YES varchar 3000 3000 NULL NULL NULL latin1 latin1_swedish_ci varchar(3000) UNI select,insert,update,references NEVER NULL > +select * from information_schema.statistics where table_schema = 'test' and table_name = 't1'; > +TABLE_CATALOG TABLE_SCHEMA TABLE_NAME NON_UNIQUE INDEX_SCHEMA INDEX_NAME SEQ_IN_INDEX COLUMN_NAME COLLATION CARDINALITY SUB_PART PACKED NULLABLE INDEX_TYPE COMMENT INDEX_COMMENT > +def test t1 0 test a 1 a A NULL NULL NULL YES HASH > +def test t1 0 test c 1 c A NULL NULL NULL YES HASH > +def test t1 0 test d 1 d A NULL NULL NULL YES HASH > +def test t1 0 test e 1 e A NULL NULL NULL YES HASH > +select * from information_schema.key_column_usage where table_schema= 'test' and table_name= 't1'; > +CONSTRAINT_CATALOG CONSTRAINT_SCHEMA CONSTRAINT_NAME TABLE_CATALOG TABLE_SCHEMA TABLE_NAME COLUMN_NAME ORDINAL_POSITION POSITION_IN_UNIQUE_CONSTRAINT REFERENCED_TABLE_SCHEMA REFERENCED_TABLE_NAME REFERENCED_COLUMN_NAME > +def test a def test t1 a 1 NULL NULL NULL NULL > +def test c def test t1 c 1 NULL NULL NULL NULL > +def test d def test t1 d 1 NULL NULL NULL NULL > +def test e def test t1 e 1 NULL NULL NULL NULL > +#table select we should not be able to see db_row_hash_1 column; > +select * from t1 order by a; > +a b c d e > +1 2 3 4 5 > +123456789034567890 43545 657567567567 78967657567567 657567567567567676 > +123456789034567891 353534 53453453453456 64565464564564 45435345345345 > +2 11 22 33 44 > +3111 222 333 444 555 > +5611 2222 3333 4444 5555 > +maria 345 frter dasd utyuty > +sachin 341 fdf gfgfgfg hghgr > +select db_row_hash_1 from t1; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 'field list' > +select db_row_hash_2 from t1; > +ERROR 42S22: Unknown column 'db_row_hash_2' in 'field list' > +select db_row_hash_3 from t1; > +ERROR 42S22: Unknown column 'db_row_hash_3' in 'field list' > +#duplicate entry test; > +insert into t1 values(21,2,3,42,51); > +ERROR 23000: Duplicate entry '3' for key 'c' > +insert into t1 values('sachin',null,null,null,null); > +ERROR 23000: Duplicate entry 'sachin' for key 'a' > +insert into t1 values(1234567890345671890,4353451,6575675675617,789676575675617,657567567567567676); > +ERROR 23000: Duplicate entry '657567567567567676' for key 'e' > +select * from t1 order by a; > +a b c d e > +1 2 3 4 5 > +123456789034567890 43545 657567567567 78967657567567 657567567567567676 > +123456789034567891 353534 53453453453456 64565464564564 45435345345345 > +2 11 22 33 44 > +3111 222 333 444 555 > +5611 2222 3333 4444 5555 > +maria 345 frter dasd utyuty > +sachin 341 fdf gfgfgfg hghgr > +insert into t1 values(repeat('s',4000*10),100,repeat('s',4000*10),repeat('s',4000*10), > +repeat('s',400)),(repeat('s',4001*10),1000,repeat('s',4001*10),repeat('s',4001*10), > +repeat('s',2995)); > +insert into t1 values(repeat('m',4000*11),10,repeat('s',4000*11),repeat('s',4000*11),repeat('s',2995)); > +ERROR 23000: Duplicate entry 'ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss' for key 'e' > +truncate table t1; > +insert into t1 values(1,2,3,4,5),(2,11,22,33,44),(3111,222,333,444,555),(5611,2222,3333,4444,5555); > +#now some alter commands; > +alter table t1 add column f int; > +desc t1; > +Field Type Null Key Default Extra > +a blob YES UNI NULL > +b int(11) YES NULL > +c blob YES UNI NULL > +d text YES UNI NULL > +e varchar(3000) YES UNI NULL > +f int(11) YES NULL > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + `b` int(11) DEFAULT NULL, > + `c` blob DEFAULT NULL, > + `d` text DEFAULT NULL, > + `e` varchar(3000) DEFAULT NULL, > + `f` int(11) DEFAULT NULL, > + UNIQUE KEY `a` (`a`(65535)), > + UNIQUE KEY `c` (`c`(65535)), > + UNIQUE KEY `d` (`d`(65535)), > + UNIQUE KEY `e` (`e`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +#unique key should not break; > +insert into t1 values(1,2,3,4,5,6); > +ERROR 23000: Duplicate entry '1' for key 'a' > +select db_row_hash_1 , db_row_hash_2, db_row_hash_3 from t1; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 'field list' > +#now try to change db_row_hash_1 column; > +alter table t1 drop column db_row_hash_1, drop column db_row_hash_2, drop column db_row_hash_3; > +ERROR 42000: Can't DROP COLUMN `db_row_hash_1`; check that it exists > +alter table t1 add column dg int , add column ef int , drop column db_row_hash_1; > +ERROR 42000: Can't DROP COLUMN `db_row_hash_1`; check that it exists > +alter table t1 modify column db_row_hash_1 int , modify column db_row_hash_2 int, modify column db_row_hash_3 int; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 't1' > +alter table t1 add column ar int , add column rb int, modify column db_row_hash_1 int , modify column db_row_hash_3 int; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 't1' > +alter table t1 change column db_row_hash_1 dsds int , change column db_row_hash_2 dfdf int , change column db_row_hash_3 gdfg int ; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 't1' > +alter table t1 add column asd int, drop column a, change column db_row_hash_1 dsds int, change db_row_hash_3 fdfdfd int; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 't1' > +alter table t1 drop column b , add column g int; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + `c` blob DEFAULT NULL, > + `d` text DEFAULT NULL, > + `e` varchar(3000) DEFAULT NULL, > + `f` int(11) DEFAULT NULL, > + `g` int(11) DEFAULT NULL, > + UNIQUE KEY `a` (`a`(65535)), > + UNIQUE KEY `c` (`c`(65535)), > + UNIQUE KEY `d` (`d`(65535)), > + UNIQUE KEY `e` (`e`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +#now add some column with name db_row_hash; > +alter table t1 add column db_row_hash_1 int unique; > +alter table t1 add column db_row_hash_2 int unique; > +alter table t1 add column db_row_hash_3 int unique; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + `c` blob DEFAULT NULL, > + `d` text DEFAULT NULL, > + `e` varchar(3000) DEFAULT NULL, > + `f` int(11) DEFAULT NULL, > + `g` int(11) DEFAULT NULL, > + `db_row_hash_1` int(11) DEFAULT NULL, > + `db_row_hash_2` int(11) DEFAULT NULL, > + `db_row_hash_3` int(11) DEFAULT NULL, > + UNIQUE KEY `a` (`a`(65535)), > + UNIQUE KEY `c` (`c`(65535)), > + UNIQUE KEY `d` (`d`(65535)), > + UNIQUE KEY `e` (`e`), > + UNIQUE KEY `db_row_hash_1` (`db_row_hash_1`), > + UNIQUE KEY `db_row_hash_2` (`db_row_hash_2`), > + UNIQUE KEY `db_row_hash_3` (`db_row_hash_3`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +alter table t1 add column db_row_hash_7 int, add column db_row_hash_5 int , add column db_row_hash_4 int ; > +alter table t1 drop column db_row_hash_7,drop column db_row_hash_3, drop column db_row_hash_4; > +desc t1; > +Field Type Null Key Default Extra > +a blob YES UNI NULL > +c blob YES UNI NULL > +d text YES UNI NULL > +e varchar(3000) YES UNI NULL > +f int(11) YES NULL > +g int(11) YES NULL > +db_row_hash_1 int(11) YES UNI NULL > +db_row_hash_2 int(11) YES UNI NULL > +db_row_hash_5 int(11) YES NULL > +#this should not break anything; > +insert into t1 values(1,2,3,4,5,6,23,5,6); > +ERROR 23000: Duplicate entry '1' for key 'a' > +#this should also drop the unique index; > +alter table t1 drop column a, drop column c; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `d` text DEFAULT NULL, > + `e` varchar(3000) DEFAULT NULL, > + `f` int(11) DEFAULT NULL, > + `g` int(11) DEFAULT NULL, > + `db_row_hash_1` int(11) DEFAULT NULL, > + `db_row_hash_2` int(11) DEFAULT NULL, > + `db_row_hash_5` int(11) DEFAULT NULL, > + UNIQUE KEY `d` (`d`(65535)), > + UNIQUE KEY `e` (`e`), > + UNIQUE KEY `db_row_hash_1` (`db_row_hash_1`), > + UNIQUE KEY `db_row_hash_2` (`db_row_hash_2`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 d 1 d A NULL 65535 NULL YES HASH > +t1 0 e 1 e A NULL NULL NULL YES HASH > +t1 0 db_row_hash_1 1 db_row_hash_1 A NULL NULL NULL YES BTREE > +t1 0 db_row_hash_2 1 db_row_hash_2 A NULL NULL NULL YES BTREE > +#add column with unique index on blob; > +alter table t1 add column a blob unique; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `d` text DEFAULT NULL, > + `e` varchar(3000) DEFAULT NULL, > + `f` int(11) DEFAULT NULL, > + `g` int(11) DEFAULT NULL, > + `db_row_hash_1` int(11) DEFAULT NULL, > + `db_row_hash_2` int(11) DEFAULT NULL, > + `db_row_hash_5` int(11) DEFAULT NULL, > + `a` blob DEFAULT NULL, > + UNIQUE KEY `d` (`d`(65535)), > + UNIQUE KEY `e` (`e`), > + UNIQUE KEY `db_row_hash_1` (`db_row_hash_1`), > + UNIQUE KEY `db_row_hash_2` (`db_row_hash_2`), > + UNIQUE KEY `a` (`a`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 d 1 d A NULL 65535 NULL YES HASH > +t1 0 e 1 e A NULL NULL NULL YES HASH > +t1 0 db_row_hash_1 1 db_row_hash_1 A NULL NULL NULL YES BTREE > +t1 0 db_row_hash_2 1 db_row_hash_2 A NULL NULL NULL YES BTREE > +t1 0 a 1 a A NULL NULL NULL YES HASH > +#try to change the blob unique column name; > +#this will change index to b tree; > +alter table t1 modify column a int , modify column e int; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `d` text DEFAULT NULL, > + `e` int(11) DEFAULT NULL, > + `f` int(11) DEFAULT NULL, > + `g` int(11) DEFAULT NULL, > + `db_row_hash_1` int(11) DEFAULT NULL, > + `db_row_hash_2` int(11) DEFAULT NULL, > + `db_row_hash_5` int(11) DEFAULT NULL, > + `a` int(11) DEFAULT NULL, > + UNIQUE KEY `d` (`d`(65535)), > + UNIQUE KEY `e` (`e`), > + UNIQUE KEY `db_row_hash_1` (`db_row_hash_1`), > + UNIQUE KEY `db_row_hash_2` (`db_row_hash_2`), > + UNIQUE KEY `a` (`a`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 d 1 d A NULL 65535 NULL YES HASH > +t1 0 e 1 e A NULL NULL NULL YES BTREE > +t1 0 db_row_hash_1 1 db_row_hash_1 A NULL NULL NULL YES BTREE > +t1 0 db_row_hash_2 1 db_row_hash_2 A NULL NULL NULL YES BTREE > +t1 0 a 1 a A NULL NULL NULL YES BTREE > +alter table t1 add column clm1 blob unique,add column clm2 blob unique; > +#try changing the name; > +alter table t1 change column clm1 clm_changed1 blob, change column clm2 clm_changed2 blob; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `d` text DEFAULT NULL, > + `e` int(11) DEFAULT NULL, > + `f` int(11) DEFAULT NULL, > + `g` int(11) DEFAULT NULL, > + `db_row_hash_1` int(11) DEFAULT NULL, > + `db_row_hash_2` int(11) DEFAULT NULL, > + `db_row_hash_5` int(11) DEFAULT NULL, > + `a` int(11) DEFAULT NULL, > + `clm_changed1` blob DEFAULT NULL, > + `clm_changed2` blob DEFAULT NULL, > + UNIQUE KEY `d` (`d`(65535)), > + UNIQUE KEY `e` (`e`), > + UNIQUE KEY `db_row_hash_1` (`db_row_hash_1`), > + UNIQUE KEY `db_row_hash_2` (`db_row_hash_2`), > + UNIQUE KEY `a` (`a`), > + UNIQUE KEY `clm1` (`clm_changed1`), > + UNIQUE KEY `clm2` (`clm_changed2`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 d 1 d A NULL 65535 NULL YES HASH > +t1 0 e 1 e A NULL NULL NULL YES BTREE > +t1 0 db_row_hash_1 1 db_row_hash_1 A NULL NULL NULL YES BTREE > +t1 0 db_row_hash_2 1 db_row_hash_2 A NULL NULL NULL YES BTREE > +t1 0 a 1 a A NULL NULL NULL YES BTREE > +t1 0 clm1 1 clm_changed1 A NULL NULL NULL YES HASH > +t1 0 clm2 1 clm_changed2 A NULL NULL NULL YES HASH > +#now drop the unique key; > +alter table t1 drop key clm1, drop key clm2; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `d` text DEFAULT NULL, > + `e` int(11) DEFAULT NULL, > + `f` int(11) DEFAULT NULL, > + `g` int(11) DEFAULT NULL, > + `db_row_hash_1` int(11) DEFAULT NULL, > + `db_row_hash_2` int(11) DEFAULT NULL, > + `db_row_hash_5` int(11) DEFAULT NULL, > + `a` int(11) DEFAULT NULL, > + `clm_changed1` blob DEFAULT NULL, > + `clm_changed2` blob DEFAULT NULL, > + UNIQUE KEY `d` (`d`(65535)), > + UNIQUE KEY `e` (`e`), > + UNIQUE KEY `db_row_hash_1` (`db_row_hash_1`), > + UNIQUE KEY `db_row_hash_2` (`db_row_hash_2`), > + UNIQUE KEY `a` (`a`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 d 1 d A NULL 65535 NULL YES HASH > +t1 0 e 1 e A NULL NULL NULL YES BTREE > +t1 0 db_row_hash_1 1 db_row_hash_1 A NULL NULL NULL YES BTREE > +t1 0 db_row_hash_2 1 db_row_hash_2 A NULL NULL NULL YES BTREE > +t1 0 a 1 a A NULL NULL NULL YES BTREE > +drop table t1; > +#now the table with key on multiple columns; the ultimate test; > +create table t1(a blob, b int , c varchar(2000) , d text , e varchar(3000) , f longblob , g int , h text , > +unique(a,b,c), unique(c,d,e),unique(e,f,g,h), unique(b,d,g,h)); > +insert into t1 values(1,1,1,1,1,1,1,1),(2,2,2,2,2,2,2,2),(3,3,3,3,3,3,3,3),(4,4,4,4,4,4,4,4),(5,5,5,5,5,5,5,5), > +('maria',6,'maria','maria','maria','maria',6,'maria'),('mariadb',7,'mariadb','mariadb','mariadb','mariadb',8,'mariadb') > +,(null,null,null,null,null,null,null,null),(null,null,null,null,null,null,null,null); > +#table structure; > +desc t1; > +Field Type Null Key Default Extra > +a blob YES MUL NULL > +b int(11) YES MUL NULL > +c varchar(2000) YES MUL NULL > +d text YES NULL > +e varchar(3000) YES MUL NULL > +f longblob YES NULL > +g int(11) YES NULL > +h text YES NULL > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + `b` int(11) DEFAULT NULL, > + `c` varchar(2000) DEFAULT NULL, > + `d` text DEFAULT NULL, > + `e` varchar(3000) DEFAULT NULL, > + `f` longblob DEFAULT NULL, > + `g` int(11) DEFAULT NULL, > + `h` text DEFAULT NULL, > + UNIQUE KEY `a` (`a`,`b`,`c`), > + UNIQUE KEY `c` (`c`,`d`,`e`), > + UNIQUE KEY `e` (`e`,`f`,`g`,`h`), > + UNIQUE KEY `b` (`b`,`d`,`g`,`h`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 a 1 a A NULL NULL NULL YES HASH > +t1 0 a 2 b A NULL NULL NULL YES HASH > +t1 0 a 3 c A NULL NULL NULL YES HASH > +t1 0 c 1 c A NULL NULL NULL YES HASH > +t1 0 c 2 d A 0 NULL NULL YES HASH > +t1 0 c 3 e A 0 NULL NULL YES HASH > +t1 0 e 1 e A 0 NULL NULL YES HASH > +t1 0 e 2 f A 0 NULL NULL YES HASH > +t1 0 e 3 g A 0 NULL NULL YES HASH > +t1 0 e 4 h A 0 NULL NULL YES HASH > +t1 0 b 1 b A 0 NULL NULL YES HASH > +t1 0 b 2 d A 0 NULL NULL YES HASH > +t1 0 b 3 g A 0 NULL NULL YES HASH > +t1 0 b 4 h A 0 NULL NULL YES HASH > + > +MyISAM file: DATADIR/test/t1 > +Record format: Packed > +Character set: latin1_swedish_ci (8) > +Data records: 9 Deleted blocks: 0 > +Recordlength: 5092 > + > +table description: > +Key Start Len Index Type > +1 5081 8 multip. ulonglong NULL > +2 5073 8 multip. ulonglong NULL > +3 5065 8 multip. ulonglong NULL > +4 5057 8 multip. ulonglong NULL > +select * from information_schema.columns where table_schema = 'test' and table_name = 't1'; > +TABLE_CATALOG TABLE_SCHEMA TABLE_NAME COLUMN_NAME ORDINAL_POSITION COLUMN_DEFAULT IS_NULLABLE DATA_TYPE CHARACTER_MAXIMUM_LENGTH CHARACTER_OCTET_LENGTH NUMERIC_PRECISION NUMERIC_SCALE DATETIME_PRECISION CHARACTER_SET_NAME COLLATION_NAME COLUMN_TYPE COLUMN_KEY EXTRA PRIVILEGES COLUMN_COMMENT IS_GENERATED GENERATION_EXPRESSION > +def test t1 a 1 NULL YES blob 65535 65535 NULL NULL NULL NULL NULL blob MUL select,insert,update,references NEVER NULL > +def test t1 b 2 NULL YES int NULL NULL 10 0 NULL NULL NULL int(11) MUL select,insert,update,references NEVER NULL > +def test t1 c 3 NULL YES varchar 2000 2000 NULL NULL NULL latin1 latin1_swedish_ci varchar(2000) MUL select,insert,update,references NEVER NULL > +def test t1 d 4 NULL YES text 65535 65535 NULL NULL NULL latin1 latin1_swedish_ci text select,insert,update,references NEVER NULL > +def test t1 e 5 NULL YES varchar 3000 3000 NULL NULL NULL latin1 latin1_swedish_ci varchar(3000) MUL select,insert,update,references NEVER NULL > +def test t1 f 6 NULL YES longblob 4294967295 4294967295 NULL NULL NULL NULL NULL longblob select,insert,update,references NEVER NULL > +def test t1 g 7 NULL YES int NULL NULL 10 0 NULL NULL NULL int(11) select,insert,update,references NEVER NULL > +def test t1 h 8 NULL YES text 65535 65535 NULL NULL NULL latin1 latin1_swedish_ci text select,insert,update,references NEVER NULL > +select * from information_schema.statistics where table_schema = 'test' and table_name = 't1'; > +TABLE_CATALOG TABLE_SCHEMA TABLE_NAME NON_UNIQUE INDEX_SCHEMA INDEX_NAME SEQ_IN_INDEX COLUMN_NAME COLLATION CARDINALITY SUB_PART PACKED NULLABLE INDEX_TYPE COMMENT INDEX_COMMENT > +def test t1 0 test a 1 a A NULL NULL NULL YES HASH > +def test t1 0 test a 2 b A NULL NULL NULL YES HASH > +def test t1 0 test a 3 c A NULL NULL NULL YES HASH > +def test t1 0 test c 1 c A NULL NULL NULL YES HASH > +def test t1 0 test c 2 d A 0 NULL NULL YES HASH > +def test t1 0 test c 3 e A 0 NULL NULL YES HASH > +def test t1 0 test e 1 e A 0 NULL NULL YES HASH > +def test t1 0 test e 2 f A 0 NULL NULL YES HASH > +def test t1 0 test e 3 g A 0 NULL NULL YES HASH > +def test t1 0 test e 4 h A 0 NULL NULL YES HASH > +def test t1 0 test b 1 b A 0 NULL NULL YES HASH > +def test t1 0 test b 2 d A 0 NULL NULL YES HASH > +def test t1 0 test b 3 g A 0 NULL NULL YES HASH > +def test t1 0 test b 4 h A 0 NULL NULL YES HASH > +select * from information_schema.key_column_usage where table_schema= 'test' and table_name= 't1'; > +CONSTRAINT_CATALOG CONSTRAINT_SCHEMA CONSTRAINT_NAME TABLE_CATALOG TABLE_SCHEMA TABLE_NAME COLUMN_NAME ORDINAL_POSITION POSITION_IN_UNIQUE_CONSTRAINT REFERENCED_TABLE_SCHEMA REFERENCED_TABLE_NAME REFERENCED_COLUMN_NAME > +def test a def test t1 a 1 NULL NULL NULL NULL > +def test a def test t1 b 2 NULL NULL NULL NULL > +def test a def test t1 c 3 NULL NULL NULL NULL > +def test c def test t1 c 1 NULL NULL NULL NULL > +def test c def test t1 d 2 NULL NULL NULL NULL > +def test c def test t1 e 3 NULL NULL NULL NULL > +def test e def test t1 e 1 NULL NULL NULL NULL > +def test e def test t1 f 2 NULL NULL NULL NULL > +def test e def test t1 g 3 NULL NULL NULL NULL > +def test e def test t1 h 4 NULL NULL NULL NULL > +def test b def test t1 b 1 NULL NULL NULL NULL > +def test b def test t1 d 2 NULL NULL NULL NULL > +def test b def test t1 g 3 NULL NULL NULL NULL > +def test b def test t1 h 4 NULL NULL NULL NULL > +# table select we should not be able to see db_row_hash_1 column; > +select * from t1 order by a; > +a b c d e f g h > +NULL NULL NULL NULL NULL NULL NULL NULL > +NULL NULL NULL NULL NULL NULL NULL NULL > +1 1 1 1 1 1 1 1 > +2 2 2 2 2 2 2 2 > +3 3 3 3 3 3 3 3 > +4 4 4 4 4 4 4 4 > +5 5 5 5 5 5 5 5 > +maria 6 maria maria maria maria 6 maria > +mariadb 7 mariadb mariadb mariadb mariadb 8 mariadb > +select db_row_hash_1 from t1; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 'field list' > +select db_row_hash_2 from t1; > +ERROR 42S22: Unknown column 'db_row_hash_2' in 'field list' > +select db_row_hash_3 from t1; > +ERROR 42S22: Unknown column 'db_row_hash_3' in 'field list' > +#duplicate entry test; > +#duplicate keys entry; > +insert into t1 values(1,1,1,0,0,0,0,0); > +ERROR 23000: Duplicate entry '1-1-1' for key 'a' > +insert into t1 values(0,0,1,1,1,0,0,0); > +ERROR 23000: Duplicate entry '1-1-1' for key 'c' > +insert into t1 values(0,0,0,0,1,1,1,1); > +ERROR 23000: Duplicate entry '1-1-1-1' for key 'e' > +insert into t1 values(1,1,1,1,1,0,0,0); > +ERROR 23000: Duplicate entry '1-1-1' for key 'a' > +insert into t1 values(0,0,0,0,1,1,1,1); > +ERROR 23000: Duplicate entry '1-1-1-1' for key 'e' > +insert into t1 values(1,1,1,1,1,1,1,1); > +ERROR 23000: Duplicate entry '1-1-1' for key 'a' > +select db_row_hash_1,db_row_hash_2,db_row_hash_3,db_row_hash_4,db_row_hash_5 from t1; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 'field list' > +alter table t1 drop column db_row_hash_1, drop column db_row_hash_2, drop column db_row_hash_3; > +ERROR 42000: Can't DROP COLUMN `db_row_hash_1`; check that it exists > +alter table t1 add column dg int , add column ef int , drop column db_row_hash_1; > +ERROR 42000: Can't DROP COLUMN `db_row_hash_1`; check that it exists > +alter table t1 modify column db_row_hash_1 int , modify column db_row_hash_2 int, modify column db_row_hash_3 int; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 't1' > +alter table t1 add column ar int , add column rb int, modify column db_row_hash_1 int , modify column db_row_hash_3 int; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 't1' > +alter table t1 change column db_row_hash_1 dsds int , change column db_row_hash_2 dfdf int , change column db_row_hash_3 gdfg int ; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 't1' > +alter table t1 add column asd int, drop column a, change column db_row_hash_1 dsds int, change db_row_hash_3 fdfdfd int; > +ERROR 42S22: Unknown column 'db_row_hash_1' in 't1' > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + `b` int(11) DEFAULT NULL, > + `c` varchar(2000) DEFAULT NULL, > + `d` text DEFAULT NULL, > + `e` varchar(3000) DEFAULT NULL, > + `f` longblob DEFAULT NULL, > + `g` int(11) DEFAULT NULL, > + `h` text DEFAULT NULL, > + UNIQUE KEY `a` (`a`,`b`,`c`), > + UNIQUE KEY `c` (`c`,`d`,`e`), > + UNIQUE KEY `e` (`e`,`f`,`g`,`h`), > + UNIQUE KEY `b` (`b`,`d`,`g`,`h`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +# add column named db_row_hash_*; > +alter table t1 add column db_row_hash_7 int , add column db_row_hash_5 int, > +add column db_row_hash_1 int, add column db_row_hash_2 int; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + `b` int(11) DEFAULT NULL, > + `c` varchar(2000) DEFAULT NULL, > + `d` text DEFAULT NULL, > + `e` varchar(3000) DEFAULT NULL, > + `f` longblob DEFAULT NULL, > + `g` int(11) DEFAULT NULL, > + `h` text DEFAULT NULL, > + `db_row_hash_7` int(11) DEFAULT NULL, > + `db_row_hash_5` int(11) DEFAULT NULL, > + `db_row_hash_1` int(11) DEFAULT NULL, > + `db_row_hash_2` int(11) DEFAULT NULL, > + UNIQUE KEY `a` (`a`(65535),`b`,`c`), > + UNIQUE KEY `c` (`c`,`d`(65535),`e`), > + UNIQUE KEY `e` (`e`,`f`(65535),`g`,`h`(65535)), > + UNIQUE KEY `b` (`b`,`d`(65535),`g`,`h`(65535)) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 a 1 a A NULL 65535 NULL YES HASH > +t1 0 a 2 b A NULL NULL NULL YES HASH > +t1 0 a 3 c A NULL NULL NULL YES HASH > +t1 0 c 1 c A NULL NULL NULL YES HASH > +t1 0 c 2 d A 0 65535 NULL YES HASH > +t1 0 c 3 e A 0 NULL NULL YES HASH > +t1 0 e 1 e A 0 NULL NULL YES HASH > +t1 0 e 2 f A 0 65535 NULL YES HASH > +t1 0 e 3 g A 0 NULL NULL YES HASH > +t1 0 e 4 h A 0 65535 NULL YES HASH > +t1 0 b 1 b A 0 NULL NULL YES HASH > +t1 0 b 2 d A 0 65535 NULL YES HASH > +t1 0 b 3 g A 0 NULL NULL YES HASH > +t1 0 b 4 h A 0 65535 NULL YES HASH > +alter table t1 drop column db_row_hash_7 , drop column db_row_hash_5 , > +drop column db_row_hash_1, drop column db_row_hash_2 ; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + `b` int(11) DEFAULT NULL, > + `c` varchar(2000) DEFAULT NULL, > + `d` text DEFAULT NULL, > + `e` varchar(3000) DEFAULT NULL, > + `f` longblob DEFAULT NULL, > + `g` int(11) DEFAULT NULL, > + `h` text DEFAULT NULL, > + UNIQUE KEY `a` (`a`(65535),`b`,`c`), > + UNIQUE KEY `c` (`c`,`d`(65535),`e`), > + UNIQUE KEY `b` (`b`,`d`(65535),`g`,`h`(65535)), > + UNIQUE KEY `e` (`e`,`f`(65535),`g`,`h`(65535)) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 a 1 a A NULL 65535 NULL YES HASH > +t1 0 a 2 b A NULL NULL NULL YES HASH > +t1 0 a 3 c A NULL NULL NULL YES HASH > +t1 0 c 1 c A NULL NULL NULL YES HASH > +t1 0 c 2 d A 0 65535 NULL YES HASH > +t1 0 c 3 e A 0 NULL NULL YES HASH > +t1 0 b 1 b A 0 NULL NULL YES HASH > +t1 0 b 2 d A 0 65535 NULL YES HASH > +t1 0 b 3 g A 0 NULL NULL YES HASH > +t1 0 b 4 h A 0 65535 NULL YES HASH > +t1 0 e 1 e A 0 NULL NULL YES HASH > +t1 0 e 2 f A 0 65535 NULL YES HASH > +t1 0 e 3 g A 0 NULL NULL YES HASH > +t1 0 e 4 h A 0 65535 NULL YES HASH > +#try to change column names; > +alter table t1 change column a aa blob , change column b bb blob , change column d dd blob; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `aa` blob DEFAULT NULL, > + `bb` blob DEFAULT NULL, > + `c` varchar(2000) DEFAULT NULL, > + `dd` blob DEFAULT NULL, > + `e` varchar(3000) DEFAULT NULL, > + `f` longblob DEFAULT NULL, > + `g` int(11) DEFAULT NULL, > + `h` text DEFAULT NULL, > + UNIQUE KEY `a` (`aa`(65535),`bb`,`c`), > + UNIQUE KEY `c` (`c`,`dd`(65535),`e`), > + UNIQUE KEY `b` (`bb`,`dd`(65535),`g`,`h`(65535)), > + UNIQUE KEY `e` (`e`,`f`(65535),`g`,`h`(65535)) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 a 1 aa A NULL 65535 NULL YES HASH > +t1 0 a 2 bb A NULL NULL NULL YES HASH > +t1 0 a 3 c A NULL NULL NULL YES HASH > +t1 0 c 1 c A NULL NULL NULL YES HASH > +t1 0 c 2 dd A 0 65535 NULL YES HASH > +t1 0 c 3 e A 0 NULL NULL YES HASH > +t1 0 b 1 bb A 0 NULL NULL YES HASH > +t1 0 b 2 dd A 0 65535 NULL YES HASH > +t1 0 b 3 g A 0 NULL NULL YES HASH > +t1 0 b 4 h A 0 65535 NULL YES HASH > +t1 0 e 1 e A 0 NULL NULL YES HASH > +t1 0 e 2 f A 0 65535 NULL YES HASH > +t1 0 e 3 g A 0 NULL NULL YES HASH > +t1 0 e 4 h A 0 65535 NULL YES HASH > +alter table t1 change column aa a blob , change column bb b blob , change column dd d blob; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + `b` blob DEFAULT NULL, > + `c` varchar(2000) DEFAULT NULL, > + `d` blob DEFAULT NULL, > + `e` varchar(3000) DEFAULT NULL, > + `f` longblob DEFAULT NULL, > + `g` int(11) DEFAULT NULL, > + `h` text DEFAULT NULL, > + UNIQUE KEY `a` (`a`(65535),`b`,`c`), > + UNIQUE KEY `c` (`c`,`d`(65535),`e`), > + UNIQUE KEY `b` (`b`,`d`(65535),`g`,`h`(65535)), > + UNIQUE KEY `e` (`e`,`f`(65535),`g`,`h`(65535)) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 a 1 a A NULL 65535 NULL YES HASH > +t1 0 a 2 b A NULL NULL NULL YES HASH > +t1 0 a 3 c A NULL NULL NULL YES HASH > +t1 0 c 1 c A NULL NULL NULL YES HASH > +t1 0 c 2 d A 0 65535 NULL YES HASH > +t1 0 c 3 e A 0 NULL NULL YES HASH > +t1 0 b 1 b A 0 NULL NULL YES HASH > +t1 0 b 2 d A 0 65535 NULL YES HASH > +t1 0 b 3 g A 0 NULL NULL YES HASH > +t1 0 b 4 h A 0 65535 NULL YES HASH > +t1 0 e 1 e A 0 NULL NULL YES HASH > +t1 0 e 2 f A 0 65535 NULL YES HASH > +t1 0 e 3 g A 0 NULL NULL YES HASH > +t1 0 e 4 h A 0 65535 NULL YES HASH > +#now we will change the data type to int and varchar limit so that we no longer require hash_index; > +#on key a_b_c; > +alter table t1 modify column a varchar(20) , modify column b varchar(20) , modify column c varchar(20); > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` varchar(20) DEFAULT NULL, > + `b` varchar(20) DEFAULT NULL, > + `c` varchar(20) DEFAULT NULL, > + `d` blob DEFAULT NULL, > + `e` varchar(3000) DEFAULT NULL, > + `f` longblob DEFAULT NULL, > + `g` int(11) DEFAULT NULL, > + `h` text DEFAULT NULL, > + UNIQUE KEY `a` (`a`,`b`,`c`), > + UNIQUE KEY `c` (`c`,`d`(65535),`e`), > + UNIQUE KEY `b` (`b`,`d`(65535),`g`,`h`(65535)), > + UNIQUE KEY `e` (`e`,`f`(65535),`g`,`h`(65535)) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 a 1 a A NULL NULL NULL YES BTREE > +t1 0 a 2 b A NULL NULL NULL YES BTREE > +t1 0 a 3 c A NULL NULL NULL YES BTREE > +t1 0 c 1 c A NULL NULL NULL YES HASH > +t1 0 c 2 d A NULL 65535 NULL YES HASH > +t1 0 c 3 e A NULL NULL NULL YES HASH > +t1 0 b 1 b A 0 NULL NULL YES HASH > +t1 0 b 2 d A 0 65535 NULL YES HASH > +t1 0 b 3 g A 0 NULL NULL YES HASH > +t1 0 b 4 h A 0 65535 NULL YES HASH > +t1 0 e 1 e A 0 NULL NULL YES HASH > +t1 0 e 2 f A 0 65535 NULL YES HASH > +t1 0 e 3 g A 0 NULL NULL YES HASH > +t1 0 e 4 h A 0 65535 NULL YES HASH > +#change it back; > +alter table t1 modify column a blob , modify column b blob , modify column c blob; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + `b` blob DEFAULT NULL, > + `c` blob DEFAULT NULL, > + `d` blob DEFAULT NULL, > + `e` varchar(3000) DEFAULT NULL, > + `f` longblob DEFAULT NULL, > + `g` int(11) DEFAULT NULL, > + `h` text DEFAULT NULL, > + UNIQUE KEY `a` (`a`,`b`,`c`), > + UNIQUE KEY `c` (`c`,`d`(65535),`e`), > + UNIQUE KEY `b` (`b`,`d`(65535),`g`,`h`(65535)), > + UNIQUE KEY `e` (`e`,`f`(65535),`g`,`h`(65535)) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 a 1 a A NULL NULL NULL YES HASH > +t1 0 a 2 b A NULL NULL NULL YES HASH > +t1 0 a 3 c A NULL NULL NULL YES HASH > +t1 0 c 1 c A NULL NULL NULL YES HASH > +t1 0 c 2 d A 0 65535 NULL YES HASH > +t1 0 c 3 e A 0 NULL NULL YES HASH > +t1 0 b 1 b A 0 NULL NULL YES HASH > +t1 0 b 2 d A 0 65535 NULL YES HASH > +t1 0 b 3 g A 0 NULL NULL YES HASH > +t1 0 b 4 h A 0 65535 NULL YES HASH > +t1 0 e 1 e A 0 NULL NULL YES HASH > +t1 0 e 2 f A 0 65535 NULL YES HASH > +t1 0 e 3 g A 0 NULL NULL YES HASH > +t1 0 e 4 h A 0 65535 NULL YES HASH > +#try to delete blob column in unique; > +truncate table t1; > +#now try to delete keys; > +alter table t1 drop key c, drop key e; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + `b` blob DEFAULT NULL, > + `c` blob DEFAULT NULL, > + `d` blob DEFAULT NULL, > + `e` varchar(3000) DEFAULT NULL, > + `f` longblob DEFAULT NULL, > + `g` int(11) DEFAULT NULL, > + `h` text DEFAULT NULL, > + UNIQUE KEY `a` (`a`(65535),`b`(65535),`c`(65535)), > + UNIQUE KEY `b` (`b`(65535),`d`(65535),`g`,`h`(65535)) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 a 1 a A NULL 65535 NULL YES HASH > +t1 0 a 2 b A NULL 65535 NULL YES HASH > +t1 0 a 3 c A 0 65535 NULL YES HASH > +t1 0 b 1 b A 0 65535 NULL YES HASH > +t1 0 b 2 d A 0 65535 NULL YES HASH > +t1 0 b 3 g A 0 NULL NULL YES HASH > +t1 0 b 4 h A 0 65535 NULL YES HASH > +drop table t1; > +#now alter table containing some data basically some tests with ignore; > +create table t1 (a blob); > +insert into t1 values(1),(2),(3); > +#normal alter table; > +alter table t1 add unique key(a); > +alter table t1 drop key a; > +truncate table t1; > +insert into t1 values(1),(1),(2),(2),(3); > +alter table t1 add unique key(a); > +ERROR 23000: Duplicate entry '1' for key 'a' > +alter ignore table t1 add unique key(a); > +select * from t1 order by a; > +a > +1 > +2 > +3 > +insert into t1 values(1); > +ERROR 23000: Duplicate entry '1' for key 'a' > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + UNIQUE KEY `a` (`a`(65535)) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 a 1 a A NULL 65535 NULL YES HASH > +drop table t1; > +#Now with multiple keys; > +create table t1(a blob , b blob, c blob , d blob , e int); > +insert into t1 values (1,1,1,1,1); > +insert into t1 values (1,1,1,1,1); > +insert into t1 values (2,1,1,1,1); > +insert into t1 values (2,2,2,2,2); > +insert into t1 values (3,3,4,4,4); > +insert into t1 values (4,4,4,4,4); > +alter table t1 add unique key(a,c), add unique key(b,d), add unique key(e); > +ERROR 23000: Duplicate entry '1-1' for key 'a' > +alter ignore table t1 add unique key(a,c), add unique key(b,d), add unique key(e); > +select * from t1 order by a; > +a b c d e > +1 1 1 1 1 > +2 2 2 2 2 > +3 3 4 4 4 > +insert into t1 values (1,12,1,13,14); > +ERROR 23000: Duplicate entry '1-1' for key 'a' > +insert into t1 values (12,1,14,1,14); > +ERROR 23000: Duplicate entry '1-1' for key 'b' > +insert into t1 values (13,12,13,14,4); > +ERROR 23000: Duplicate entry '4' for key 'e' > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + `b` blob DEFAULT NULL, > + `c` blob DEFAULT NULL, > + `d` blob DEFAULT NULL, > + `e` int(11) DEFAULT NULL, > + UNIQUE KEY `a` (`a`(65535),`c`(65535)), > + UNIQUE KEY `b` (`b`(65535),`d`(65535)), > + UNIQUE KEY `e` (`e`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 a 1 a A NULL 65535 NULL YES HASH > +t1 0 a 2 c A NULL 65535 NULL YES HASH > +t1 0 b 1 b A NULL 65535 NULL YES HASH > +t1 0 b 2 d A 0 65535 NULL YES HASH > +t1 0 e 1 e A 0 NULL NULL YES BTREE > +drop table t1; > +#visibility of db_row_hash > +create table t1 (a blob unique , b blob unique); > +desc t1; > +Field Type Null Key Default Extra > +a blob YES UNI NULL > +b blob YES UNI NULL > +insert into t1 values(1,19); > +insert into t1 values(2,29); > +insert into t1 values(3,39); > +insert into t1 values(4,49); > +create table t2 (DB_ROW_HASH_1 int, DB_ROW_HASH_2 int); > +insert into t2 values(11,1); > +insert into t2 values(22,2); > +insert into t2 values(33,3); > +insert into t2 values(44,4); > +select * from t1 order by a; > +a b > +1 19 > +2 29 > +3 39 > +4 49 > +select * from t2 order by DB_ROW_HASH_1; > +DB_ROW_HASH_1 DB_ROW_HASH_2 > +11 1 > +22 2 > +33 3 > +44 4 > +select DB_ROW_HASH_1, DB_ROW_HASH_2 from t1; > +ERROR 42S22: Unknown column 'DB_ROW_HASH_1' in 'field list' > +#bug > +select DB_ROW_HASH_1, DB_ROW_HASH_2 from t1,t2; > +DB_ROW_HASH_1 DB_ROW_HASH_2 > +11 1 > +11 1 > +11 1 > +11 1 > +22 2 > +22 2 > +22 2 > +22 2 > +33 3 > +33 3 > +33 3 > +33 3 > +44 4 > +44 4 > +44 4 > +44 4 > +select * from t1 where DB_ROW_HASH_1 in (select DB_ROW_HASH_1 from t2); > +ERROR 42S22: Unknown column 'DB_ROW_HASH_1' in 'IN/ALL/ANY subquery' > +select DB_ROW_HASH_1, DB_ROW_HASH_2 from t1,t2 where DB_ROW_HASH_1 in (select DB_ROW_HASH_1 from t2); > +DB_ROW_HASH_1 DB_ROW_HASH_2 > +11 1 > +22 2 > +33 3 > +44 4 > +11 1 > +22 2 > +33 3 > +44 4 > +11 1 > +22 2 > +33 3 > +44 4 > +11 1 > +22 2 > +33 3 > +44 4 > +select * from t2 where DB_ROW_HASH_1 in (select DB_ROW_HASH_1 from t1); > +DB_ROW_HASH_1 DB_ROW_HASH_2 > +11 1 > +22 2 > +33 3 > +44 4 > +select DB_ROW_HASH_1 from t1,t2 where t1.DB_ROW_HASH_1 = t2.DB_ROW_HASH_2; > +ERROR 42S22: Unknown column 't1.DB_ROW_HASH_1' in 'where clause' > +select DB_ROW_HASH_1 from t1 inner join t2 on t1.a = t2.DB_ROW_HASH_2; > +DB_ROW_HASH_1 > +11 > +22 > +33 > +44 > +drop table t1,t2; > +#very long blob entry; > +SET @@GLOBAL.max_allowed_packet=67108864; > +connect 'newcon', localhost, root,,; > +connection newcon; > +show variables like 'max_allowed_packet'; > +Variable_name Value > +max_allowed_packet 67108864 > +create table t1(a longblob unique, b longblob , c longblob , unique(b,c)); > +desc t1; > +Field Type Null Key Default Extra > +a longblob YES UNI NULL > +b longblob YES MUL NULL > +c longblob YES NULL > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` longblob DEFAULT NULL, > + `b` longblob DEFAULT NULL, > + `c` longblob DEFAULT NULL, > + UNIQUE KEY `a` (`a`), > + UNIQUE KEY `b` (`b`,`c`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 a 1 a A NULL NULL NULL YES HASH > +t1 0 b 1 b A NULL NULL NULL YES HASH > +t1 0 b 2 c A 0 NULL NULL YES HASH > +insert into t1 values(concat(repeat('sachin',10000000),'1'),concat(repeat('sachin',10000000),'1'), > +concat(repeat('sachin',10000000),'1')); > +insert into t1 values(concat(repeat('sachin',10000000),'2'),concat(repeat('sachin',10000000),'2'), > +concat(repeat('sachin',10000000),'1')); > +insert into t1 values(concat(repeat('sachin',10000000),'2'),concat(repeat('sachin',10000000),'2'), > +concat(repeat('sachin',10000000),'4')); > +ERROR 23000: Duplicate entry 'sachinsachinsachinsachinsachinsachinsachinsachinsachinsachinsach' for key 'a' > +insert into t1 values(concat(repeat('sachin',10000000),'3'),concat(repeat('sachin',10000000),'1'), > +concat(repeat('sachin',10000000),'1')); > +ERROR 23000: Duplicate entry 'sachinsachinsachinsachinsachinsachinsachinsachinsachinsachinsach' for key 'b' > +drop table t1; > +#long key unique with different key length > +create table t1(a blob, unique(a(3000))); > +desc t1; > +Field Type Null Key Default Extra > +a blob YES UNI NULL > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 a 1 a A NULL 3000 NULL YES HASH > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob DEFAULT NULL, > + UNIQUE KEY `a` (`a`(3000)) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +insert into t1 value(concat(repeat('s',3000),'1')); > +insert into t1 value(concat(repeat('s',3000),'2')); > +ERROR 23000: Duplicate entry 'ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss' for key 'a' > +insert into t1 value(concat(repeat('a',3000),'2')); > +drop table t1; > +create table t1(a varchar(4000), b longblob , c varchar(5000), d longblob, > +unique(a(3500), b), unique(c(4500), d)); > +desc t1; > +Field Type Null Key Default Extra > +a varchar(4000) YES MUL NULL > +b longblob YES NULL > +c varchar(5000) YES MUL NULL > +d longblob YES NULL > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` varchar(4000) DEFAULT NULL, > + `b` longblob DEFAULT NULL, > + `c` varchar(5000) DEFAULT NULL, > + `d` longblob DEFAULT NULL, > + UNIQUE KEY `a` (`a`(3500),`b`), > + UNIQUE KEY `c` (`c`(4500),`d`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +show keys from t1; > +Table Non_unique Key_name Seq_in_index Column_name Collation Cardinality Sub_part Packed Null Index_type Comment Index_comment > +t1 0 a 1 a A NULL 3500 NULL YES HASH > +t1 0 a 2 b A NULL NULL NULL YES HASH > +t1 0 c 1 c A 0 4500 NULL YES HASH > +t1 0 c 2 d A 0 NULL NULL YES HASH > +drop table t1; > +disconnect newcon; > +connection default; > +SET @@GLOBAL.max_allowed_packet=4194304; > +#ext bug > +create table t1(a int primary key, b blob unique, c int, d blob , index(c)); > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` int(11) NOT NULL, > + `b` blob DEFAULT NULL, > + `c` int(11) DEFAULT NULL, > + `d` blob DEFAULT NULL, > + PRIMARY KEY (`a`), > + UNIQUE KEY `b` (`b`), > + KEY `c` (`c`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +insert into t1 values(1,23,1,33); > +insert into t1 values(2,23,1,33); > +ERROR 23000: Duplicate entry '23' for key 'b' > +drop table t1; > +create table t2 (a blob unique , c int , index(c)); > +show create table t2; > +Table Create Table > +t2 CREATE TABLE `t2` ( > + `a` blob DEFAULT NULL, > + `c` int(11) DEFAULT NULL, > + UNIQUE KEY `a` (`a`), > + KEY `c` (`c`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +insert into t2 values(1,1); > +insert into t2 values(2,1); > +drop table t2; > +#not null test //todo solve warnings > +create table t1(a blob unique not null); > +desc t1; > +Field Type Null Key Default Extra > +a blob NO UNI NULL > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob NOT NULL, > + UNIQUE KEY `a` (`a`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +insert into t1 values(1); > +insert into t1 values(3); > +insert into t1 values(1); > +ERROR 23000: Duplicate entry '1' for key 'a' > +drop table t1; > +create table t1(a int primary key, b blob unique , c blob unique not null); > +insert into t1 values(1,1,1); > +insert into t1 values(2,1,2); > +ERROR 23000: Duplicate entry '1' for key 'b' > +insert into t1 values(3,3,1); > +ERROR 23000: Duplicate entry '1' for key 'c' > +drop table t1; > +create table t1 (a blob unique not null, b blob not null, c blob not null, unique(b,c)); > +desc t1; > +Field Type Null Key Default Extra > +a blob NO UNI NULL > +b blob NO MUL NULL > +c blob NO NULL > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` blob NOT NULL, > + `b` blob NOT NULL, > + `c` blob NOT NULL, > + UNIQUE KEY `a` (`a`), > + UNIQUE KEY `b` (`b`,`c`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +insert into t1 values (1, 2, 3); > +insert into t1 values (2, 1, 3); > +insert into t1 values (2, 1, 3); > +ERROR 23000: Duplicate entry '2' for key 'a' > +drop table t1; > +#partition > +create table t1(a blob unique) partition by hash(a); > +ERROR HY000: A BLOB field is not allowed in partition function > +#key length > 2^15 -1 > +create table t1(a blob, unique(a(100001))); > +ERROR 42000: Specified key was too long; max key length is 1000 bytes > +set @@GLOBAL.max_allowed_packet= @allowed_packet; > diff --git a/mysql-test/main/long_unique_debug.result b/mysql-test/main/long_unique_debug.result > new file mode 100644 > --- /dev/null > +++ b/mysql-test/main/long_unique_debug.result > @@ -0,0 +1,32 @@ > +#In this test case we will check what will happen in the case of hash collusion collision, not collusion :) > +SET debug_dbug="d,same_long_unique_hash"; > +create table t1(a blob unique); > +insert into t1 values('xyz'); > +insert into t1 values('abc'); > +insert into t1 values('sachin'); > +insert into t1 values('sachin'); > +ERROR 23000: Duplicate entry 'sachin' for key 'a' > +insert into t1 values('maria'); > +insert into t1 values('maria'); > +ERROR 23000: Duplicate entry 'maria' for key 'a' > +drop table t1; > +create table t1(a blob unique, b blob unique); > +insert into t1 values('xyz', 11); > +insert into t1 values('abc', 22); > +insert into t1 values('sachin', 1); > +insert into t1 values('sachin', 4); > +ERROR 23000: Duplicate entry 'sachin' for key 'a' > +insert into t1 values('maria', 2); > +insert into t1 values('maria', 3); > +ERROR 23000: Duplicate entry 'maria' for key 'a' > +drop table t1; > +create table t1(a blob , b blob , unique(a,b)); > +insert into t1 values('xyz', 11); > +insert into t1 values('abc', 22); > +insert into t1 values('sachin', 1); > +insert into t1 values('sachin', 1); > +ERROR 23000: Duplicate entry 'sachin-1' for key 'a' > +insert into t1 values('maria', 2); > +insert into t1 values('maria', 2); > +ERROR 23000: Duplicate entry 'maria-2' for key 'a' > +drop table t1; > diff --git a/mysql-test/main/mdev-504.test b/mysql-test/main/mdev-504.test > --- a/mysql-test/main/mdev-504.test > +++ b/mysql-test/main/mdev-504.test > @@ -44,7 +43,7 @@ while ($trial) > --connect (con2,localhost,root,,) > --send CALL p_analyze() > > - --let $run = 100 > + --let $run = 1 Why? > > while ($run) > { > diff --git a/mysql-test/main/type_blob.result b/mysql-test/main/type_blob.result > --- a/mysql-test/main/type_blob.result > +++ b/mysql-test/main/type_blob.result > @@ -370,7 +370,7 @@ a 1 > hello 1 > drop table t1; > create table t1 (a text, unique (a(2100))); > -ERROR 42000: Specified key was too long; max key length is 1000 bytes > +drop table t1; or remove the create table instead, it's not a useful test anymore > create table t1 (a text, key (a(2100))); > Warnings: > Note 1071 Specified key was too long; max key length is 1000 bytes > diff --git a/sql/field.h b/sql/field.h > --- a/sql/field.h > +++ b/sql/field.h > @@ -687,7 +687,7 @@ class Field: public Value_source > GEOM_MULTIPOINT = 4, GEOM_MULTILINESTRING = 5, GEOM_MULTIPOLYGON = 6, > GEOM_GEOMETRYCOLLECTION = 7 > }; > - enum imagetype { itRAW, itMBR}; > + enum imagetype { itRAW, itMBR, itHASH}; unused > > utype unireg_check; > uint32 field_length; // Length of field > diff --git a/sql/handler.cc b/sql/handler.cc > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -3686,9 +3686,19 @@ void handler::print_error(int error, myf errflag) > uint key_nr=get_dup_key(error); > if ((int) key_nr >= 0 && key_nr < table->s->keys) > { > + KEY *long_key= NULL; > + if (table->key_info[key_nr].algorithm > + == HA_KEY_ALG_LONG_HASH) > + { > + long_key= table->key_info + key_nr; > + setup_keyinfo_hash(long_key); > + } > print_keydup_error(table, &table->key_info[key_nr], errflag); > + if (long_key) > + re_setup_keyinfo_hash(long_key); > DBUG_VOID_RETURN; > } > + table->re_setup_table(); > } > textno=ER_DUP_KEY; > break; > @@ -6256,6 +6268,162 @@ int handler::ha_reset() > DBUG_RETURN(reset()); > } > > +static int check_duplicate_long_entry_key(TABLE *table, handler *h, uchar *new_rec, > + uint key_no) > +{ > + Field *hash_field; > + int result, error= 0; > + KEY *key_info= table->key_info + key_no; > + hash_field= key_info->key_part->field; > + DBUG_ASSERT((table->key_info[key_no].flags & HA_NULL_PART_KEY && > + table->key_info[key_no].key_length == HA_HASH_KEY_LENGTH_WITH_NULL) > + || table->key_info[key_no].key_length == HA_HASH_KEY_LENGTH_WITHOUT_NULL); > + uchar ptr[HA_HASH_KEY_LENGTH_WITH_NULL]; > + > + if (hash_field->is_real_null()) > + return 0; > + remember, you have `key_info= table->key_info + key_no` above :) you don't need to write `table->key_info[key_no]` anymore here > + key_copy(ptr, new_rec, &table->key_info[key_no], > + table->key_info[key_no].key_length, false); > + > + if (!table->check_unique_buf) > + table->check_unique_buf= (uchar *)alloc_root(&table->mem_root, > + table->s->reclength*sizeof(uchar)); C99 standard says 6.5.3.4 The sizeof operator When applied to an operand that has type char, unsigned char, or signed char, (or a qualified version thereof) the result is 1. so don't bother with `*sizeof(uchar)` > + > + result= h->ha_index_init(key_no, 0); > + if (result) > + return result; > + result= h->ha_index_read_map(table->check_unique_buf, > + ptr, HA_WHOLE_KEY, HA_READ_KEY_EXACT); > + if (!result) > + { > + bool is_same; > + Field * t_field; > + Item_func_hash * temp= (Item_func_hash *)hash_field->vcol_info->expr; > + Item ** arguments= temp->arguments(); > + uint arg_count= temp->argument_count(); > + do > + { > + long diff= table->check_unique_buf - new_rec; > + is_same= true; > + for (uint j=0; j < arg_count; j++) you can add `is_same && ` here to the for() condition too. > + { > + DBUG_ASSERT(arguments[j]->type() == Item::FIELD_ITEM || > + // this one for left(fld_name,length) > + arguments[j]->type() == Item::FUNC_ITEM); > + if (arguments[j]->type() == Item::FIELD_ITEM) > + { > + t_field= static_cast<Item_field *>(arguments[j])->field; > + if (t_field->cmp_offset(diff)) > + is_same= false; > + } > + else > + { > + Item_func_left *fnc= static_cast<Item_func_left *>(arguments[j]); > + DBUG_ASSERT(!my_strcasecmp(system_charset_info, "left", fnc->func_name())); > + DBUG_ASSERT(fnc->arguments()[0]->type() == Item::FIELD_ITEM); > + t_field= static_cast<Item_field *>(fnc->arguments()[0])->field; > + uint length= fnc->arguments()[1]->val_int(); > + if (t_field->cmp_max(t_field->ptr, t_field->ptr + diff, length)) > + is_same= false; > + } > + } > + } > + while (!is_same && !(result= table->file->ha_index_next_same(table->check_unique_buf, > + ptr, table->key_info[key_no].key_length))); > + if (is_same) > + { > + table->dupp_hash_key= key_no; > + error= HA_ERR_FOUND_DUPP_KEY; > + goto exit; > + } > + else > + goto exit; > + } > + if (result == HA_ERR_LOCK_WAIT_TIMEOUT) > + { > + table->dupp_hash_key= key_no; > + //TODO check if this is the only case > + error= HA_ERR_FOUND_DUPP_KEY; why not to keep HA_ERR_LOCK_WAIT_TIMEOUT? > + } > + exit: > + h->ha_index_end(); > + return error; > +} empty line between functions, please > +/** @brief > + check whether inserted records breaks the > + unique constraint on long columns. > + @returns 0 if no duplicate else returns error > + */ > +static int check_duplicate_long_entries(TABLE *table, handler *h, uchar *new_rec) > +{ > + table->dupp_hash_key= -1; > + int result; > + for (uint i= 0; i < table->s->keys; i++) > + { > + if (table->key_info[i].algorithm == HA_KEY_ALG_LONG_HASH && > + (result= check_duplicate_long_entry_key(table, h, new_rec, i))) > + return result; > + } > + return 0; > +} > + > +/** @brief > + check whether updated records breaks the > + unique constraint on long columns. > + In the case of update we just need to check the specic key > + reason for that is consider case > + create table t1(a blob , b blob , x blob , y blob ,unique(a,b) > + ,unique(x,y)) > + and update statement like this > + update t1 set a=23+a; in this case if we try to scan for > + whole keys in table then index scan on x_y will return 0 > + because data is same so in the case of update we take > + key as a parameter in normal insert key should be -1 > + @returns 0 if no duplicate else returns error > + */ > +static int check_duplicate_long_entries_update(TABLE *table, handler *h, uchar *new_rec) > +{ > + Field *field; > + uint key_parts; > + int error= 0; > + KEY *keyinfo; > + KEY_PART_INFO *keypart; > + /* > + Here we are comparing whether new record and old record are same > + with respect to fields in hash_str > + */ > + long reclength= table->record[1]-table->record[0]; > + if (!table->update_handler) > + table->clone_handler_for_update(); > + for (uint i= 0; i < table->s->keys; i++) > + { > + keyinfo= table->key_info + i; > + if (keyinfo->algorithm == HA_KEY_ALG_LONG_HASH) > + { > + key_parts= fields_in_hash_keyinfo(keyinfo); > + keypart= keyinfo->key_part - key_parts; > + for (uint j= 0; j < key_parts; j++, keypart++) > + { > + field= keypart->field; > + /* Compare fields if they are different then check for duplicates*/ > + if(field->cmp_binary_offset(reclength)) > + { > + if((error= check_duplicate_long_entry_key(table, table->update_handler, > + new_rec, i))) > + goto exit; > + /* > + break beacuse check_duplicate_long_entrie_key will > + take care of remaning fields three typos in two lines :) > + */ > + break; > + } > + } > + } > + } > + exit: > + return error; > +} > > int handler::ha_write_row(uchar *buf) > { > diff --git a/sql/item.cc b/sql/item.cc > --- a/sql/item.cc > +++ b/sql/item.cc > @@ -356,7 +356,7 @@ Item::Item(THD *thd): > /* Initially this item is not attached to any JOIN_TAB. */ > join_tab_idx= MAX_TABLES; > > - /* Put item in free list so that we can free all items at end */ > + /* Put item in free list so that we can free all items at end only if*/ Huh? > next= thd->free_list; > thd->free_list= this; > /* > diff --git a/sql/item_func.cc b/sql/item_func.cc > --- a/sql/item_func.cc > +++ b/sql/item_func.cc > @@ -1718,6 +1718,48 @@ bool Item_func_mod::fix_length_and_dec() > DBUG_RETURN(FALSE); > } > > +inline void calc_hash_for_unique(ulong &nr1, ulong &nr2, String *str) static inline (or simply static, a compiler will inline it anyway) > +{ > + CHARSET_INFO *cs; > + uchar l[4]; > + int4store(l, str->length()); > + cs= str->charset(); > + cs->coll->hash_sort(cs, l, sizeof(l), &nr1, &nr2); > + cs= str->charset(); > + cs->coll->hash_sort(cs, (uchar *)str->ptr(), str->length(), &nr1, &nr2); > +} > + > +longlong Item_func_hash::val_int() > +{ > + DBUG_EXECUTE_IF("same_long_unique_hash", return 9;); good that you've added tests for collisions! > + unsigned_flag= true; > + ulong nr1= 1,nr2= 4; > + CHARSET_INFO *cs; > + String * str; > + for(uint i= 0;i<arg_count;i++) > + { > + str = args[i]->val_str(); > + if(args[i]->null_value) > + { > + null_value= 1; > + return 0; > + } > + calc_hash_for_unique(nr1, nr2, str); > + } > + null_value= 0; > + return (longlong)nr1; > +} > + > + > +bool Item_func_hash::fix_length_and_dec() > +{ > + maybe_null= 1; Why are you forcing maybe_null=1? Most functions are "normal" - they can return NULL if one of the arguments is NULL, otherwise they don't return NULL. And Item_func::fix_fields() sets maybe_null accordingly, it'll be 1 if any of the arguments has maybe_null==1, otherwise maybe_null will be 0. You only need to set maybe_null explicitly if your function is special, for ISNULL(x) # never returns NULL, and NULLIF(x,y) can return NULL when both arguments are not NULL. Looks like in your case the default maybe_null beahavior is correct, though. > + decimals= 0; > + max_length= 8; > + return false; > +} > + > + > > double Item_func_neg::real_op() > { > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -994,6 +994,8 @@ void close_thread_table(THD *thd, TABLE **table_ptr) > table->file->update_global_table_stats(); > table->file->update_global_index_stats(); > } > + if (table->update_handler) > + table->delete_update_handler(); you're doing it in sql_update.cc, isn't it enough? > > /* > This look is needed to allow THD::notify_shared_lock() to > diff --git a/sql/sql_show.cc b/sql/sql_show.cc > --- a/sql/sql_show.cc > +++ b/sql/sql_show.cc > @@ -2306,7 +2307,7 @@ int show_create_table(THD *thd, TABLE_LIST *table_list, String *packet, > */ > packet->append(STRING_WITH_LEN("PRIMARY KEY")); > } > - else if (key_info->flags & HA_NOSAME) > + else if (key_info->flags & HA_NOSAME || key_info->algorithm == HA_KEY_ALG_LONG_HASH) I'd think that after setup_table_hash(), HA_NOSAME should be set and you wouldn't need extra checks. > packet->append(STRING_WITH_LEN("UNIQUE KEY ")); > else if (key_info->flags & HA_FULLTEXT) > packet->append(STRING_WITH_LEN("FULLTEXT KEY ")); > @@ -6546,7 +6548,7 @@ static int get_schema_stat_record(THD *thd, TABLE_LIST *tables, > else if (!tables->view) > { > TABLE *show_table= tables->table; > - KEY *key_info=show_table->s->key_info; > + KEY *key_info=show_table->key_info; you mean, TABLE_SHARE::key_info shows your HASH definition? and but you change (with setup_table_hash) TABLE::key_info to store the original user's UNIQUE definition? Why is TABLE_SHARE::key_info changed? > if (show_table->file) > { > show_table->file->info(HA_STATUS_VARIABLE | > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -4106,12 +4177,28 @@ mysql_prepare_create_table(THD *thd, HA_CREATE_INFO *create_info, > if (key->type == Key::UNIQUE && !(key_info->flags & HA_NULL_PART_KEY)) > unique_key=1; > key_info->key_length=(uint16) key_length; > - if (key_length > max_key_length && key->type != Key::FULLTEXT) > + if (key_length > max_key_length && key->type != Key::FULLTEXT && > + !is_hash_field_needed) > { > my_error(ER_TOO_LONG_KEY,MYF(0),max_key_length); > DBUG_RETURN(TRUE); > } > > + if (is_hash_field_needed) > + { > + Create_field *hash_fld= add_hash_field(thd, &alter_info->create_list, > + create_info->default_table_charset, > + key_info); > + hash_fld->offset= record_offset; > + record_offset+= hash_fld->pack_length; > + if (key_info->flags & HA_NULL_PART_KEY) > + null_fields++; > + else > + { > + hash_fld->flags|= NOT_NULL_FLAG; > + hash_fld->pack_flag&= ~FIELDFLAG_MAYBE_NULL; > + } > + } Hmm, haven't we agreed *not* to write the virtual hash field into the frm, not show it to create/alter table and add it automatically when the table is opened? > if (validate_comment_length(thd, &key->key_create_info.comment, > INDEX_COMMENT_MAXLEN, > ER_TOO_LONG_INDEX_COMMENT, > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -10820,6 +10820,12 @@ function_call_conflict: > if (unlikely($$ == NULL)) > MYSQL_YYABORT; > } > + | HASH_SYM '(' expr_list ')' > + { > + $$= new (thd->mem_root)Item_func_hash(thd,*$3); > + if (unlikely($$ == NULL)) > + MYSQL_YYABORT; > + } Haven't we agreed to make this HASH() function fully internal, not visible to the parser? > /* LAST_VALUE here conflicts with the definition for window functions. > We have these 2 separate rules to remove the shift/reduce conflict. > */ > diff --git a/sql/table.cc b/sql/table.cc > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -747,7 +746,7 @@ static bool create_key_infos(const uchar *strpos, const uchar *frm_image_end, > if (i == 0) > { > ext_key_parts+= (share->use_ext_keys ? first_keyinfo->user_defined_key_parts*(keys-1) : 0); > - n_length=keys * sizeof(KEY) + ext_key_parts * sizeof(KEY_PART_INFO); > + n_length=keys * sizeof(KEY) + (ext_key_parts) * sizeof(KEY_PART_INFO); Another left-over from an intermediate implementation? :) > if (!(keyinfo= (KEY*) alloc_root(&share->mem_root, > n_length + len))) > return 1; > @@ -798,6 +797,12 @@ static bool create_key_infos(const uchar *strpos, const uchar *frm_image_end, > } > key_part->store_length=key_part->length; > } > + if (keyinfo->algorithm == HA_KEY_ALG_LONG_HASH) > + { > + keyinfo->key_length= HA_HASH_KEY_LENGTH_WITHOUT_NULL; > + //Storing key hash > + key_part++; > + } I don't understand what you do here > > /* > Add primary key to end of extended keys for non unique keys for > @@ -831,7 +836,9 @@ static bool create_key_infos(const uchar *strpos, const uchar *frm_image_end, > if (j == first_key_parts) > keyinfo->ext_key_flags= keyinfo->flags | HA_EXT_NOSAME; > } > - share->ext_key_parts+= keyinfo->ext_key_parts; > + if (keyinfo->algorithm == HA_KEY_ALG_LONG_HASH) > + share->ext_key_parts++; > + share->ext_key_parts+= keyinfo->ext_key_parts; I don't understand what you do here > } > keynames=(char*) key_part; > strpos+= strnmov(keynames, (char *) strpos, frm_image_end - strpos) - keynames; > @@ -3322,6 +3421,36 @@ enum open_frm_error open_table_from_share(THD *thd, TABLE_SHARE *share, > > key_part_end= key_part + (share->use_ext_keys ? key_info->ext_key_parts : > key_info->user_defined_key_parts) ; > + if (key_info->algorithm == HA_KEY_ALG_LONG_HASH) > + { > + /* > + Either it can be first time opening the table share or it can be second time > + of more. The difference is when it is first time key_part[0]->fieldnr points > + to blob/long field, but when it is 2nd time there will bw always one key_part > + and it will point to hash_field. > + So in the case of 2nd time we will make key_info->key_part point to start of long > + field. you're overcomplicating here. if you would *always* create your HASH key_def in parse_vcol_defs(), you'll have TABLE_SHARE to be always "like the first time", you simply never modify TABLE_SHARE and always modify TABLEin parse_vcol_defs(). and you won't need setup_keyinfo_hash/setup_table_hash functions either. > + For example we have unique(a,b,c) > + In first share opening key_part will point to a field > + but in parse_vcol_defs it will be changed to point to db_row_hash field > + in Second or later opening key_part will be pointing to db_row_hash > + We will chnage it back to point to field a, because in this way we can create > + vcol_info for hash field in parse_vcol_defs. > + */ > + //Second or more time share opening > + key_info->user_defined_key_parts= 0; > + key_part_end= key_part; > + while(!(share->field[key_part_end->fieldnr -1 ]->flags & LONG_UNIQUE_HASH_FIELD)) > + { > + key_part_end++; > + key_info->user_defined_key_parts++; > + } > + key_info->usable_key_parts= key_info->ext_key_parts= key_info->user_defined_key_parts; > + key_part_end++; > + share_keyinfo= share->key_info + key_no; > + if (share_keyinfo->key_part->field->flags & LONG_UNIQUE_HASH_FIELD) > + share_keyinfo->key_part-= key_info->user_defined_key_parts; > + } > for ( ; key_part < key_part_end; key_part++) > { > Field *field= key_part->field= outparam->field[key_part->fieldnr - 1]; > @@ -8498,6 +8630,120 @@ double KEY::actual_rec_per_key(uint i) > } > > > +/* > + find out the field positoin in hash_str() > + position starts from 0 > + else return -1; > +*/ > +int find_field_pos_in_hash(Item *hash_item, const char * field_name) > +{ > + Item_func_or_sum * temp= static_cast<Item_func_or_sum *>(hash_item); > + Item_args * t_item= static_cast<Item_args *>(temp); > + uint arg_count= t_item->argument_count(); > + Item ** arguments= t_item->arguments(); > + Field * t_field; > + > + for (uint j=0; j < arg_count; j++) > + { > + DBUG_ASSERT(arguments[j]->type() == Item::FIELD_ITEM || > + arguments[j]->type() == Item::FUNC_ITEM); > + if (arguments[j]->type() == Item::FIELD_ITEM) > + { > + t_field= static_cast<Item_field *>(arguments[j])->field; > + } > + else > + { > + Item_func_left *fnc= static_cast<Item_func_left *>(arguments[j]); > + t_field= static_cast<Item_field *>(fnc->arguments()[0])->field; > + } > + if (!my_strcasecmp(system_charset_info, t_field->field_name.str, field_name)) > + return j; > + } > + return -1; > +} > + > +/* > + find total number of field in hash expr > +*/ > +int fields_in_hash_keyinfo(KEY *keyinfo) > +{ > + Item_func_hash * temp= (Item_func_hash *) > + keyinfo->key_part->field->vcol_info->expr; > + return temp->argument_count(); > +} > + > +void setup_keyinfo_hash(KEY *key_info) > +{ > + uint no_of_keyparts= fields_in_hash_keyinfo(key_info); > + key_info->key_part-= no_of_keyparts; > + key_info->user_defined_key_parts= key_info->usable_key_parts= > + key_info->ext_key_parts= no_of_keyparts; add DBUG_ASSERT to make sure it's your HA_KEY_ALG_LONG_HASH > +} > + > +void re_setup_keyinfo_hash(KEY *key_info) > +{ > + while(!(key_info->key_part->field->flags & LONG_UNIQUE_HASH_FIELD)) > + key_info->key_part++; > + key_info->user_defined_key_parts= key_info->usable_key_parts= > + key_info->ext_key_parts= 1; > +} > +/** > + @brief clone of current handler. > + Creates a clone of handler used in update for > + unique hash key. > +*/ > +void TABLE::clone_handler_for_update() > +{ > + handler *update_handler= NULL; > + if (!s->long_unique_table) > + return; > + update_handler= file->clone(s->normalized_path.str, > + in_use->mem_root); > + update_handler->ha_external_lock(in_use, F_RDLCK); > + this->update_handler= update_handler; > + return; > +} > + > +/** > + @brief Deletes update handler object > +*/ > +void TABLE::delete_update_handler() > +{ > + update_handler->ha_external_lock(in_use, F_UNLCK); > + update_handler->ha_close(); > + delete update_handler; > + this->update_handler= NULL; > +} > +/** > + @brief This function makes table object with > + long unique keys ready for optimizer and alter table Looks like a bad function name and a bad explanation. I think - but not sure! - that it restores the original UNIQUE definition, with columns as specified by the user. Is that so? > + @param table Table object > + */ > +void TABLE::setup_table_hash() > +{ > + > + if (!this->s->long_unique_table) > + return; > + KEY *keyinfo= key_info; > + for (uint i= 0; i < this->s->keys; i++, keyinfo++) > + if (keyinfo->algorithm == HA_KEY_ALG_LONG_HASH) > + setup_keyinfo_hash(keyinfo); > +} > + > +/** > + @brief Revert the effect of setup_table_hash > + @param table Table Object > + */ > +void TABLE::re_setup_table() > +{ > + if (!s->long_unique_table) > + return; > + KEY *keyinfo= key_info; > + for (uint i= 0; i < s->keys; i++, keyinfo++) > + if (keyinfo->algorithm == HA_KEY_ALG_LONG_HASH) > + re_setup_keyinfo_hash(keyinfo); > +} > + > LEX_CSTRING *fk_option_name(enum_fk_option opt) > { > static LEX_CSTRING names[]= > diff --git a/sql/table.h b/sql/table.h > --- a/sql/table.h > +++ b/sql/table.h > @@ -1105,6 +1120,17 @@ struct TABLE > THD *in_use; /* Which thread uses this */ > > uchar *record[3]; /* Pointer to records */ > + /* record buf to resolve hash collisions for long UNIQUE constraints */ > + uchar *check_unique_buf; > + handler *update_handler; /* Handler used in case of update */ > + /* > + In the case of write row for long unique we are unable to find > + which key is violated. Because we in case of duplicate hash we never reach > + handler write_row function. So print_error will always print that > + key 0 is violated. We store which key is violated in this variable > + by default this should be initialized to -1 > + */ > + int dupp_hash_key; couldn't you simply store it in table->file->errkey ? > uchar *write_row_record; /* Used as optimisation in > THD::write_row */ > uchar *insert_values; /* used by INSERT ... UPDATE */ > diff --git a/sql/table_cache.cc b/sql/table_cache.cc > --- a/sql/table_cache.cc > +++ b/sql/table_cache.cc > @@ -822,13 +822,10 @@ TABLE_SHARE *tdc_acquire_share(THD *thd, TABLE_LIST *tl, uint flags, > > if (res == -1) > DBUG_RETURN(0); > - else if (res == 1) > - continue; Why? > > element= (TDC_element*) lf_hash_search_using_hash_value(&tdc_hash, > thd->tdc_hash_pins, hash_value, (uchar*) key, key_length); > lf_hash_search_unpin(thd->tdc_hash_pins); > - DBUG_ASSERT(element); > > if (!(share= alloc_table_share(tl->db.str, tl->table_name.str, key, key_length))) > { > diff --git a/storage/myisam/myisamchk.c b/storage/myisam/myisamchk.c > --- a/storage/myisam/myisamchk.c > +++ b/storage/myisam/myisamchk.c > @@ -1333,7 +1333,8 @@ static void descript(HA_CHECK *param, register MI_INFO *info, char * name) > key++,keyinfo++) > { > keyseg=keyinfo->seg; > - if (keyinfo->flag & HA_NOSAME) text="unique "; > + if (keyinfo->flag & HA_NOSAME) > + text="unique "; I presume you can revert this change, looks like some left-over from when you tried to make myisamchk to recognize your uniques and then later reverted it. > else if (keyinfo->flag & HA_FULLTEXT) text="fulltext "; > else text="multip."; > > Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 2

Re: [Maria-developers] 4b01d3aee60: MDEV-17082 Application-time periods: CREATE
by Sergei Golubchik 12 Feb '19

12 Feb '19

Hi, Nikita! On Jan 31, Nikita Malyavin wrote: > revision-id: 4b01d3aee60 (versioning-1.0.7-2-g4b01d3aee60) > parent(s): 44144533e50 > author: Nikita Malyavin <nikitamalyavin(a)gmail.com> > committer: Nikita Malyavin <nikitamalyavin(a)gmail.com> > timestamp: 2019-01-30 22:53:43 +1000 > message: > > MDEV-17082 Application-time periods: CREATE > > * add syntax `CREATE TABLE ... PERIOD FOR <apptime>` > * add table period entity > > diff --git a/mysql-test/suite/period/r/create.result b/mysql-test/suite/period/r/create.result > --- /dev/null > +++ b/mysql-test/suite/period/r/create.result > @@ -0,0 +1,82 @@ > +create or replace table t (id int primary key, s date, e date, > +period for mytime(s,e)); > +show create table t; > +Table Create Table > +t CREATE TABLE `t` ( > + `id` int(11) NOT NULL, > + `s` date NOT NULL, > + `e` date NOT NULL, > + PRIMARY KEY (`id`), > + PERIOD FOR `mytime` (`s`, `e`), > + CONSTRAINT `CONSTRAINT_1` CHECK (`s` < `e`) as discussed, let's give the constraint the same name as the period. but don't forget to add a test where a user explicitly creates a constraint with this name. Like create or replace table t (id int primary key, s date, e date, period for mytime(s,e), constraint mytime (id < 100)); > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +create or replace table t (id int primary key, s timestamp(6), e timestamp(6), > +period for mytime(s,e)); > +show create table t; > +Table Create Table > +t CREATE TABLE `t` ( > + `id` int(11) NOT NULL, > + `s` timestamp(6) NOT NULL DEFAULT '0000-00-00 00:00:00.000000', > + `e` timestamp(6) NOT NULL DEFAULT '0000-00-00 00:00:00.000000', > + PRIMARY KEY (`id`), > + PERIOD FOR `mytime` (`s`, `e`), > + CONSTRAINT `CONSTRAINT_1` CHECK (`s` < `e`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +# SQL16, Part 2, 11.3 <table definition>, Syntax Rules, 2)a) > +# 2) If a <table period definition> TPD is specified, then: > +# a) <table scope> shall not be specified. > +create or replace temporary table t (s date, e date, period for mytime(s,e)); > +ERROR HY000: Temporary tables with application-time period not allowed _are_ not allowed. or, better, Application-time period table cannot be temporary > +# SQL16, Part 2, 11.3 <table definition>, Syntax Rules, 2)e)iii) > +# The <data type or domain name> contained in CD1 is either DATE or a > +# timestamp type and it is equivalent to the <data type or domain name> > +# contained in CD2. > +create or replace table t (id int primary key, s datetime, e date, > +period for mytime(s,e)); > +ERROR HY000: PERIOD FOR `mytime` fields types mismatch _field_ type mismatch or Fields `s` and `t` of PERIOD FOR `mytime` have different types. or Fields of PERIOD FOR `mytime` have different types. > +create or replace table t (s timestamp(2), e timestamp(6), > +period for mytime(s,e)); > +ERROR HY000: PERIOD FOR `mytime` fields types mismatch > +create or replace table t (id int primary key, s int, e date, > +period for mytime(s,e)); > +ERROR 42000: Incorrect column specifier for column 's' incorrect column _type_? > +create or replace table t (id int primary key, s date, e date, > +period for mytime(s,x)); > +ERROR 42S22: Unknown column 'x' in 'mytime' > +create or replace table t (id int primary key, s date, e date, > +period for mytime(s,e), > +period for mytime2(s,e)); > +ERROR HY000: Cannot specify more than one application-time period > +# SQL16, Part 2, 11.3 <table definition>, Syntax Rules, 2)d) > +# No <column name> in any <column definition> shall be equivalent to PN. > +create or replace table t (mytime int, s date, e date, > +period for mytime(s,e)); > +ERROR HY000: Could not specify name `mytime` for field. It is already used by period. Better "Column and period `mytime` have the same name". Because "already used" implies that period got the name first, while in the table definition the field was specified first. > +# SQL16, Part 2, 11.3 <table definition>, Syntax Rules, 2)e)v)2)A) > +# Neither CD1 nor CD2 shall contain an <identity column specification>, a > +# <generation clause>, a <system time period start column specification>, > +# or a <system time period end column specification>. > +create or replace table t (id int primary key, > +s date, > +e date generated always as (s+1), > +period for mytime(s,e)); > +ERROR HY000: Period field `e` cannot be GENERATED ALWAYS AS > +create or replace table t (id int primary key, > +s date, > +e date as (s+1) VIRTUAL, > +period for mytime(s,e)); > +ERROR HY000: Period field `e` cannot be GENERATED ALWAYS AS > +create or replace table t (id int primary key, s timestamp(6), e timestamp(6), > +st timestamp(6) as row start, > +en timestamp(6) as row end, > +period for system_time (st, en), > +period for mytime(st,e)) with system versioning; > +ERROR HY000: Period field `st` cannot be GENERATED ALWAYS AS > +# SQL16, Part 2, 11.3 <table definition>, Syntax Rules, 2) > +# Let IDCN be an implementation-dependent <constraint name> that is not > +# equivalent to the <constraint name> of any table constraint descriptor > +# included in S. > +create or replace table t (x int, s date, e date, > +period for mytime(s, e), > +constraint mytime check (x > 1)); please, add SHOW CREATE TABLE here (add it after every successful CREATE TABLE). > +create or replace database test; > diff --git a/sql/datadict.cc b/sql/datadict.cc > --- a/sql/datadict.cc > +++ b/sql/datadict.cc > @@ -292,6 +292,12 @@ bool dd_read_extra2(const uchar *frm_image, size_t len, extra2_fields *fields) We've discussed that TRUNCATE TABLE bug fix, it won't need dd_read_extra2(). You still want to have a separate function for reading extra2, fine, but as dd_frm_type() won't needed it, I'd rather move it into table.cc and declared it static. In fact, if you'd like to keep extra2 code separate, you could move everything extra2 related to a separate object (like your extra2_fields) with methods for reading and writing it. But, please, don't do this refactoring before March. > fields->field_flags.str= extra2; > fields->field_flags.length= length; > break; > + case EXTRA2_APPLICATION_TIME_PERIOD: > + if (fields->application_period.str) > + DBUG_RETURN(true); > + fields->application_period.str= extra2; > + fields->application_period.length= length; > + break; > default: > /* abort frm parsing if it's an unknown but important extra2 value */ > if (type >= EXTRA2_ENGINE_IMPORTANT) > diff --git a/sql/handler.cc b/sql/handler.cc > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -7352,3 +7352,99 @@ bool Vers_parse_info::check_sys_fields(const Lex_table_name &table_name, > > +bool Table_scope_and_contents_source_st::check_fields( > + THD *thd, Alter_info *alter_info, TABLE_LIST &create_table) > +{ > + bool res= vers_check_system_fields(thd, alter_info, create_table); > + if (res) > + return true; > + > + if (!period_info.name) > + return false; > + > + if (tmp_table()) > + { > + my_error(ER_PERIOD_TEMPORARY_NOT_ALLOWED, MYF(0)); > + return true; > + } > + > + Table_period_info::start_end_t &period= period_info.period; > + const Create_field *row_start= NULL; > + const Create_field *row_end= NULL; > + List_iterator<Create_field> it(alter_info->create_list); > + while (const Create_field *f= it++) > + { > + if (period.start.streq(f->field_name)) row_start= f; > + else if (period.end.streq(f->field_name)) row_end= f; > + > + if (period_info.name.streq(f->field_name)) > + { > + // TODO this stub should be removed by MDEV-16976 why removed? > + my_error(ER_PERIOD_NAME_IS_NOT_ALLOWED_FOR_FIELD, MYF(0), f->field_name.str); > + return true; > + } > + } > > diff --git a/sql/table.h b/sql/table.h > --- a/sql/table.h > +++ b/sql/table.h > @@ -1730,6 +1747,9 @@ class IS_table_read_plan; > /** The threshold size a blob field buffer before it is freed */ > #define MAX_TDC_BLOB_SIZE 65536 > > +/** number of bytes read by uint2korr and sint2korr */ > +#define korr2size 2 No, that's silly. It's like #define TWO 2 Of course korr2size is 2. And korr3size is 3, and korr4size is 4. If you want to get rid of the magic number, don't put it into the name of the macro :) That's why I suggested something like #define fieldno_size 2 #define fieldno_korr uint2korr #define fieldno_store int2store Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 5

[Maria-developers] MariaDB Developers Unconference, New York
by Ian Gilfillan 11 Feb '19

11 Feb '19

A final reminder about the MariaDB Developers Unconference taking place 23 - 24 February in New York, just before OpenWorks. We're almost at capacity, so if you haven't signed up yet, please do so soon. Details about where to sign up and how to suggest sessions are at https://mariadb.org/2019-developers-unconference-new-york/ Hope to see you there!

1 0

[Maria-developers] Review for ANALYZE TABLE with Sampling
by Vicențiu Ciorbaru 10 Feb '19

10 Feb '19

Hi Sergei! Can you review that you are happy with the storage engine API changes? I'veustructured the commit to be as small as possible to achieve the desired outcome. In my tests, we are now twice as fast as MySQL for a 10 mil row table with 13 columns. Vicențiu -------- Forwarded Message --------From: vicentiu(a)mariadb.orgTo: commits(a)mariadb.orgSubject: 53730224efd: Improve histogram collection performance by samplingDate: Sun, 10 Feb 2019 20:09:49 +0200 (EET) revision-id: 53730224efd987f97a6cc968ff5214ee499d84e0 (mariadb-10.4.1- 163-g53730224efd)parent(s): 3c305d3f1951f1667f84e48ddd98674c6318c39dauthor: Vicențiu Ciorbarucommitter: Vicențiu Ciorbarutimestamp: 2019-02-10 19:54:50 +0200message: Improve histogram collection performance by sampling Histogram collection is done by sampling a percentage of rows from the table,not looking at all individual ones. The default implementation, to keep the server's Engine IndepenentStatistics component working uses Bernoulli sampling. It does a simpletable scan, but only accepts rows that pass a dice roll. Thisimplementation is done as a storage engine interface method, so as toallow faster and/or more efficient implementations for storage enginesinternally. The number of rows collected is capped to a minimum of 50000 andincreases logarithmically with a coffecient of 4096. The coffecient ischosen so that we expect an error of less than 3% in our estimationsaccording to the paper:"Random Sampling for Histogram Construction: How much is enough?”– Surajit Chaudhuri, Rajeev Motwani, Vivek Narasayya, ACM SIGMOD, 1998. This interface is also a precursor to allowing SELECT ... FROM tablewith sampling to work. Performance wise, for a table of 10 million rows and 13 columns, 6 int,6 doubles, one string, the previous analyze table statistics took1 minute 20 seconds to collect all data. Post implementation, thetime is less than 6 seconds. This was run on an InnoDB table, NVME SSD withapproximately 2GB/s read speed and 500MB/s write speed. --- mysql-test/main/selectivity.result | 8 +++---- mysql- test/main/selectivity_innodb.result | 8 +++-- -- sql/handler.cc | 14 +++++++++++ sql/handler.h | 40 ++++++++++++++++++++++++++++++- sql/sql_statistics.cc | 32 +++++++++++++++++++------ 5 files changed, 86 insertions(+), 16 deletions(-) diff --git a/mysql-test/main/selectivity.result b/mysql- test/main/selectivity.resultindex 00907235ecc..6d09c1c9b62 100644--- a/mysql-test/main/selectivity.result+++ b/mysql- test/main/selectivity.result@@ -1290,9 +1290,9 @@ explain extended select * from t1, t2, t1 as t3 where t1.b=t2.c and t2.d=t3.a and t3.b<5 and t1.a < 2000; id select_type table type possibl e_keys key key_len ref rows filtered Extra-1 SIMPLE t1 ALL NULL NULL NULL NULL 262144 100.00 Using where+1 SIMPLE t1 ALL NULL NULL NULL NULL 262117 100.00 Using where 1 SIMPLE t2 ref c,d c 5 test.t1.b 5 100.00 -1 SIMPLE t3 ALL NULL NULL NULL NULL 262144 100.00 Using where; Using join buffer (flat, BNL join)+1 SIMPLE t3 ALL NULL NULL NULL NULL 262117 100.00 Using where; Using join buffer (flat, BNL join) Warnings: Note 1003 select `test`.`t1`.`a` AS `a`,`test`.`t1`.`b` AS `b`,`test`.`t2`.`c` AS `c`,`test`.`t2`.`d` AS `d`,`test`.`t3`.`a` AS `a`,`test`.`t3`.`b` AS `b` from `test`.`t1` join `test`.`t2` join `test`.`t1` `t3` where `test`.`t2`.`c` = `test`.`t1`.`b` and `test`.`t3`.`a` = `test`.`t2`.`d` and `test`.`t3`.`b` < 5 and `test`.`t1`.`a` < 2000 select * from t1, t2, t1 as t3@@ -1307,9 +1307,9 @@ explain extended select * from t1, t2, t1 as t3 where t1.b=t2.c and t2.d=t3.a and t3.b<5 and t1.a < 2000; id select_ type table type possible_keys key key_len ref rows filtered Extra-1 SIMPLE t3 ALL NULL NULL NULL NULL 262144 0.00 Using where+1 SIMPLE t3 ALL NULL NULL NULL NULL 262117 0.00 Using where 1 SIMPLE t2 ref c,d d 5 test.t3.a 7 100.00 -1 SIMPLE t1 ALL NULL NULL NULL NULL 262144 2.00 Using where; Using join buffer (flat, BNL join)+1 SIMPLE t1 ALL NULL NULL NULL NULL 262117 2.00 Using where; Using join buffer (flat, BNL join) Warnings: Note 1003 select `test`.`t1`.`a` AS `a`,`test`.`t1`.`b` AS `b`,`test`.`t2`.`c` AS `c`,`test`.`t2`.`d` AS `d`,`test`.`t3`.`a` AS `a`,`test`.`t3`.`b` AS `b` from `test`.`t1` join `test`.`t2` join `test`.`t1` `t3` where `test`.`t1`.`b` = `test`.`t2`.`c` and `test`.`t2`.`d` = `test`.`t3`.`a` and `test`.`t3`.`b` < 5 and `test`.`t1`.`a` < 2000 select * from t1, t2, t1 as t3diff --git a/mysql-test/main/selectivity_innodb.result b/mysql-test/main/selectivity_innodb.resultindex 93917065722..0b20a40f69f 100644--- a/mysql- test/main/selectivity_innodb.result+++ b/mysql- test/main/selectivity_innodb.result@@ -1300,9 +1300,9 @@ explain extended select * from t1, t2, t1 as t3 where t1.b=t2.c and t2.d=t3.a and t3.b<5 and t1.a < 2000; id select_type table type possibl e_keys key key_len ref rows filtered Extra-1 SIMPLE t1 ALL NULL NULL NULL NULL 262144 100.00 Using where+1 SIMPLE t1 ALL NULL NULL NULL NULL 262117 100.00 Using where 1 SIMPLE t2 ref c,d c 5 test.t1.b 5 100.00 -1 SIMPLE t3 ALL NULL NULL NULL NULL 262144 100.00 Using where; Using join buffer (flat, BNL join)+1 SIMPLE t3 ALL NULL NULL NULL NULL 262117 100.00 Using where; Using join buffer (flat, BNL join) Warnings: Note 1003 select `test`.`t1`.`a` AS `a`,`test`.`t1`.`b` AS `b`,`test`.`t2`.`c` AS `c`,`test`.`t2`.`d` AS `d`,`test`.`t3`.`a` AS `a`,`test`.`t3`.`b` AS `b` from `test`.`t1` join `test`.`t2` join `test`.`t1` `t3` where `test`.`t2`.`c` = `test`.`t1`.`b` and `test`.`t3`.`a` = `test`.`t2`.`d` and `test`.`t3`.`b` < 5 and `test`.`t1`.`a` < 2000 select * from t1, t2, t1 as t3@@ -1317,9 +1317,9 @@ explain extended select * from t1, t2, t1 as t3 where t1.b=t2.c and t2.d=t3.a and t3.b<5 and t1.a < 2000; id select_ type table type possible_keys key key_len ref rows filtered Extra-1 SIMPLE t3 ALL NULL NULL NULL NULL 262144 0.00 Using where+1 SIMPLE t3 ALL NULL NULL NULL NULL 262117 0.00 Using where 1 SIMPLE t2 ref c,d d 5 test.t3.a 7 100.00 -1 SIMPLE t1 ALL NULL NULL NULL NULL 262144 2.00 Using where; Using join buffer (flat, BNL join)+1 SIMPLE t1 ALL NULL NULL NULL NULL 262117 2.00 Using where; Using join buffer (flat, BNL join) Warnings: Note 1003 select `test`.`t1`.`a` AS `a`,`test`.`t1`.`b` AS `b`,`test`.`t2`.`c` AS `c`,`test`.`t2`.`d` AS `d`,`test`.`t3`.`a` AS `a`,`test`.`t3`.`b` AS `b` from `test`.`t1` join `test`.`t2` join `test`.`t1` `t3` where `test`.`t1`.`b` = `test`.`t2`.`c` and `test`.`t2`.`d` = `test`.`t3`.`a` and `test`.`t3`.`b` < 5 and `test`.`t1`.`a` < 2000 select * from t1, t2, t1 as t3diff --git a/sql/handler.cc b/sql/handler.ccindex 2e45b5883ea..16fb126918e 100644--- a/sql/handler.cc+++ b/sql/handler.cc@@ -7547,3 +7547,17 @@ bool Vers_parse_info::check_sys_fields(const Lex_table_name &table_name, "ROW END" : found_flag ? "ROW START" : "ROW START/END"); return true; }++int handler::random_sample(uchar *buf)+{+ int rc;+ THD *thd= ha_thd();+ do+ {+ if (table->in_use- >check_killed(1))+ return HA_ERR_ABORTED_BY_USER;+ rc= rnd_next(buf);+ } while (rc == HA_ERR_RECORD_DELETED || thd_rnd(thd) > sample_fraction);++ return rc;+}diff --git a/sql/handler.h b/sql/handler.hindex dfb2333b24e..fcdadbeb42b 100644--- a/sql/handler.h+++ b/sql/handler.h@@ -1913,6 +1913,11 @@ enum enum_stats_auto_recalc { HA_STATS_AUTO_RECALC_DEFAULT= 0, HA_STATS_AUTO_RECALC_ON, HA_STATS_AUTO_RECALC_OFF }; +enum sample_mode {+ HA_SAMPLE_BERNOULLI= 0,+ HA_SAMPLE_SYSTEM+};+ /** A helper struct for schema DDL statements: CREATE SCHEMA [IF NOT EXISTS] name [ schema_specification... ]@@ -2947,9 +2952,11 @@ class handler :public Sql_alloc /** Length of ref (1-8 or the clustered key length) */ uint ref_length; FT_INFO *ft_handler;- enum init_stat { NONE=0, INDEX, RND };+ enum init_stat { NONE=0, INDEX, RND, SAMPLE }; init_stat inited, pre_inited; + double sample_fraction= 0;+ enum sample_mode sample_mode; const COND *pushed_cond; /** next_insert_id is the next value which should be inserted into the@@ -3112,6 +3119,31 @@ class handler :public Sql_alloc virtual int prepare_range_scan(const key_range *start_key, const key_range *end_key) { return 0; } + int ha_random_sample_init(THD *thd, enum sample_mode mode, double fraction)+ __attribute__((warn_unused_result))+ {+ DBUG_ENTER("h a_random_sample_init");+ DBUG_ASSERT(inited==NONE);+ int result;+ sample_mode= mode;+ sample_fraction= fraction;+ inited= (result= random_sample_init(mode, fraction)) ? NONE : SAMPLE;+ DBUG_RETURN(result);+ }+ int ha_random_sample(uchar *buf)+ __attribute__((warn_unused_result))+ {+ DBUG_ENTER("ha_ra ndom_sample");+ DBUG_ASSERT(inited == SAMPLE);+ DBUG_RETURN(random_sample(buf));+ }+ int ha_random_sample_end()+ {+ DBUG_ENTER("ha_random_sample_end");+ inited= NONE;+ DBUG_RETURN(random_sample_end());+ }+ int ha_rnd_init(bool scan) __attribute__ ((warn_unused_result)) { DBUG_EXECUTE_IF("ha_rnd_init_fail", return HA_ERR_TABLE_DEF_CHANGED;);@@ -4425,6 +4457,12 @@ class handler :public Sql_alloc /* Note: ha_index_read_idx_map() may bypass index_init() */ virtual int index_init(uint idx, bool sorted) { return 0; } virtual int index_end() { return 0; }+ virtual int random_sample_init(enum sample_mode mode, double fraction)+ {+ return rnd_init(TRUE);+ }+ virtual int random_sample(uchar *buf);+ virtual int random_sample_end() { return rnd_end(); } /** rnd_init() can be called two times without rnd_end() in between (it only makes sense if scan=1).diff --git a/sql/sql_statistics.cc b/sql/sql_statistics.ccindex db214a1fe28..22a015821de 100644--- a/sql/sql_statistics.cc+++ b/sql/sql_statistics.cc@@ -2727,12 +2727,16 @@ int collect_statistics_for_table(THD *thd, TABLE *table) Field *table_field; ha_rows rows= 0; handler *file=table->file;+ double sample_fraction;+ const ha_rows MIN_THRESHOLD_FOR_SAMPLING= 50000; DBUG_ENTER("collect_statistics_for_table"); table- >collected_stats->cardinality_is_null= TRUE; table->collected_stats- >cardinality= 0; + table->file->info(HA_STATUS_VARIABLE);+ for (field_ptr= table->field; *field_ptr; field_ptr++) { table_field= *field_ptr; @@ -2743,12 +2747,27 @@ int collect_statistics_for_table(THD *thd, TABLE *table) restore_record(table, s->default_values); - /* Perform a full table scan to collect statistics on 'table's columns */- if (!(rc= file->ha_rnd_init(TRUE)))- { ++ if (file->records() < MIN_THRESHOLD_FOR_SAMPLING)+ {+ sample_fraction= 1;+ }+ else+ {+ sample_fraction= std::fmin(+ (MIN_THRESHOLD_FOR_SAMPLING + 4096 *+ log(200 * file->records())) / file->records(), 1);+ }+++ /* Fetch samples from the table to collect statistics on table's columns */++ if (!(rc= file->ha_random_sample_init(thd, HA_SAMPLE_BERNOULLI,+ sample_fra ction)))+ { DEBUG_SYNC(table->in_use, "statistics_collection_start"); - while ((rc= file- >ha_rnd_next(table->record[0])) != HA_ERR_END_OF_FILE)+ while ((rc= file->ha_random_sample(table->record[0])) != HA_ERR_END_OF_FILE) { if (thd->killed) break;@@ -2768,10 +2787,9 @@ int collect_statistics_for_table(THD *thd, TABLE *table) break; rows++; }- file- >ha_rnd_end();+ file->ha_random_sample_end(); } rc= (rc == HA_ERR_END_OF_FILE && !thd->killed) ? 0 : 1;- /* Calculate values for all statistical characteristics on columns and and for each field f of 'table' save them in the write_stat structure@@ -2780,7 +2798,7 @@ int collect_statistics_for_table(THD *thd, TABLE *table) if (!rc) { table->collected_stats->cardinality_is_null= FALSE;- table->collected_stats->cardinality= rows;+ table- >collected_stats->cardinality= rows / sample_fraction; } bitmap_clear_all(table->write_set);

1 0

[Maria-developers] Some more input on optimizer_trace
by Sergey Petrunia 10 Feb '19

10 Feb '19

Hi Varun, I've did some adjustments MDEV-18489 and pushed the patch into 10.4-optimizer-trace, please check it out. Also I have filed MDEV-18527 and MDEV-18528. Some input on the code: > --- 10.4-optimizer-trace-orig/sql/sql_select.cc > +++ 10.4-optimizer-trace-cl/sql/sql_select.cc > @@ -15983,12 +16250,26 @@ optimize_cond(JOIN *join, COND *conds, > that occurs in a function set a pointer to the multiple equality > predicate. Substitute a constant instead of this field if the > multiple equality contains a constant. > - */ > - DBUG_EXECUTE("where", print_where(conds, "original", QT_ORDINARY);); > + */ > + > + Opt_trace_context *const trace = &thd->opt_trace; > + Json_writer *writer= trace->get_current_json(); > + Json_writer_object trace_wrapper(writer); > + Json_writer_object trace_cond(writer, "condition_processing"); > + trace_cond.add("condition", join->conds == conds ? "WHERE" : "HAVING") > + .add("original_condition", conds); > + > + Json_writer_array trace_steps(writer, "steps"); > + DBUG_EXECUTE("where", print_where(conds, "original", QT_ORDINARY);); Small question: Why was DBUG_EXECUTE shifted right? A bigger question: the code seems unnecesarily verbose: 1. > + Opt_trace_context *const trace = &thd->opt_trace; 2. > + Json_writer *writer= trace->get_current_json(); 3. > + Json_writer_object trace_wrapper(writer); 4. > + Json_writer_object trace_cond(writer, "condition_processing"); Can we save the space by just calling the constructors: Json_writer_object trace_wrapper(thd); Json_writer_object trace_cond(thd, "condition_processing"); ? This applies here and in many other places. Alternative, we could use: Json_writer_object trace_wrapper(thd); Json_writer_object trace_cond(trace_wrapper, "condition_processing"); .. which makes the nesting clearer (and we could also add debug safety check: it is invalid to operate on trace_wrapper until trace_cond hasn't been end()'ed) > --- 10.4-optimizer-trace-orig/sql/opt_range.cc > +++ 10.4-optimizer-trace-cl/sql/opt_range.cc > > +void TRP_ROR_UNION::trace_basic_info(const PARAM *param, > + Json_writer_object *trace_object) const > +{ > + Opt_trace_context *const trace = &param->thd->opt_trace; > + Json_writer* writer= trace->get_current_json(); > + trace_object->add("type", "index_roworder_union"); > + Json_writer_array ota(writer, "union_of"); The name 'ota' makes sense in MySQL codebase (where it is a contraction of Optimizer_trace_array), but is really confusing in MariaDB codebase. Please change everywhere to "smth_trace" or something like that (jwa in the worst case). > @@ -2654,12 +2833,18 @@ int SQL_SELECT::test_quick_select(THD *t > > if (cond) > { > - if ((tree= cond->get_mm_tree(&param, &cond))) > + { > + Json_writer_array trace_range_summary(writer, > + "setup_range_conditions"); > + tree= cond->get_mm_tree(&param, &cond); > + } Does this ever produce anything meaningful, other than empty: "setup_range_conditions": [], In MySQL, the possible contents of this array are: "impossible_condition": { "cause": "comparison_with_null_always_false" } /* impossible_condition */ "impossible_condition": { "cause": "null_field_in_non_null_column" } /* impossible_condition */ But in MariaDB we don't have this (should we?) Also, why_use_underscores_in_value_of_cause? It is a quoted string where spaces are allowed. MySQL seems to have figured this out at some point and have a few cause strings using spaces. > --- 10.4-optimizer-trace-orig/sql/opt_table_elimination.cc > +++ 10.4-optimizer-trace-cl/sql/opt_table_elimination.cc > @@ -522,7 +524,8 @@ eliminate_tables_for_list(JOIN *join, > List<TABLE_LIST> *join_list, > table_map tables_in_list, > Item *on_expr, > - table_map tables_used_elsewhere); > + table_map tables_used_elsewhere, > + Json_writer_array* eliminate_tables); Please change the name to something indicating it's just a trace. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 1

Re: [Maria-developers] b1a4d1e4937: MDEV-16973 Application-time periods: DELETE
by Sergei Golubchik 08 Feb '19

08 Feb '19

Hi, Nikita! On Jan 31, Nikita Malyavin wrote: > revision-id: b1a4d1e4937 (versioning-1.0.7-3-gb1a4d1e4937) > parent(s): 4b01d3aee60 > author: Nikita Malyavin <nikitamalyavin(a)gmail.com> > committer: Nikita Malyavin <nikitamalyavin(a)gmail.com> > timestamp: 2019-01-30 22:54:00 +1000 > message: > > MDEV-16973 Application-time periods: DELETE > > * inject portion of time updates into mysql_delete main loop > * triggered case emits delete+insert, no updates > * PORTION OF `SYSTEM_TIME` is forbidden > * `DELETE HISTORY .. FOR PORTION OF ...` is forbidden as well > diff --git a/mysql-test/suite/period/r/delete.result b/mysql-test/suite/period/r/delete.result > --- /dev/null > +++ b/mysql-test/suite/period/r/delete.result > @@ -0,0 +1,244 @@ > +create or replace table t (id int, s date, e date, period for apptime(s,e)); > +insert into t values(1, '1999-01-01', '2018-12-12'); > +insert into t values(1, '1999-01-01', '2017-01-01'); > +insert into t values(1, '2017-01-01', '2019-01-01'); > +insert into t values(2, '1998-01-01', '2018-12-12'); > +insert into t values(3, '1997-01-01', '2015-01-01'); > +insert into t values(4, '2016-01-01', '2020-01-01'); > +insert into t values(5, '2010-01-01', '2015-01-01'); > +create or replace table t1 (id int, s date, e date, period for apptime(s,e)); > +insert t1 select * from t; > +create or replace table t2 (id int, s date, e date, period for apptime(s,e)); > +insert t2 select * from t; > +delete from t for portion of apptime from '2000-01-01' to '2018-01-01'; > +delete from t1 for portion of APPTIME from '2000-01-01' to '2018-01-01'; > +select * from t; > +id s e > +1 1999-01-01 2000-01-01 > +1 1999-01-01 2000-01-01 > +1 2018-01-01 2018-12-12 > +1 2018-01-01 2019-01-01 > +2 1998-01-01 2000-01-01 > +2 2018-01-01 2018-12-12 > +3 1997-01-01 2000-01-01 > +4 2018-01-01 2020-01-01 > +select * from t1; > +id s e > +1 1999-01-01 2000-01-01 > +1 1999-01-01 2000-01-01 > +1 2018-01-01 2018-12-12 > +1 2018-01-01 2019-01-01 > +2 1998-01-01 2000-01-01 > +2 2018-01-01 2018-12-12 > +3 1997-01-01 2000-01-01 > +4 2018-01-01 2020-01-01 > +select * from log_tbl; > +id log here (and everywhere selecting from log_tbl), better do ORDER BY id it's quite difficult to follow the sequence of events otherwise > +1 >DEL: 1, 1999-01-01, 2018-12-12 > +10 <INS: 1, 1999-01-01, 2000-01-01 > +11 >DEL: 1, 2017-01-01, 2019-01-01 > +12 <DEL: 1, 2017-01-01, 2019-01-01 > +13 >INS: 1, 2018-01-01, 2019-01-01 > +14 <INS: 1, 2018-01-01, 2019-01-01 > +15 >DEL: 2, 1998-01-01, 2018-12-12 > +16 <DEL: 2, 1998-01-01, 2018-12-12 > +17 >INS: 2, 1998-01-01, 2000-01-01 > +18 <INS: 2, 1998-01-01, 2000-01-01 > +19 >INS: 2, 2018-01-01, 2018-12-12 > +2 <DEL: 1, 1999-01-01, 2018-12-12 > +20 <INS: 2, 2018-01-01, 2018-12-12 > +21 >DEL: 3, 1997-01-01, 2015-01-01 > +22 <DEL: 3, 1997-01-01, 2015-01-01 > +23 >INS: 3, 1997-01-01, 2000-01-01 > +24 <INS: 3, 1997-01-01, 2000-01-01 > +25 >DEL: 4, 2016-01-01, 2020-01-01 > +26 <DEL: 4, 2016-01-01, 2020-01-01 > +27 >INS: 4, 2018-01-01, 2020-01-01 > +28 <INS: 4, 2018-01-01, 2020-01-01 > +29 >DEL: 5, 2010-01-01, 2015-01-01 > +3 >INS: 1, 1999-01-01, 2000-01-01 > +30 <DEL: 5, 2010-01-01, 2015-01-01 > +4 <INS: 1, 1999-01-01, 2000-01-01 > +5 >INS: 1, 2018-01-01, 2018-12-12 > +6 <INS: 1, 2018-01-01, 2018-12-12 > +7 >DEL: 1, 1999-01-01, 2017-01-01 > +8 <DEL: 1, 1999-01-01, 2017-01-01 > +9 >INS: 1, 1999-01-01, 2000-01-01 > +# INSERT trigger only also works > +drop trigger tr1del_t2; > +drop trigger tr2del_t2; > +delete from t2 for portion of APPTIME from '2000-01-01' to '2018-01-01'; > +select * from log_tbl; > +id log > +1 >INS: 1, 1999-01-01, 2000-01-01 > +10 <INS: 2, 1998-01-01, 2000-01-01 > +11 >INS: 2, 2018-01-01, 2018-12-12 > +12 <INS: 2, 2018-01-01, 2018-12-12 > +13 >INS: 3, 1997-01-01, 2000-01-01 > +14 <INS: 3, 1997-01-01, 2000-01-01 > +15 >INS: 4, 2018-01-01, 2020-01-01 > +16 <INS: 4, 2018-01-01, 2020-01-01 > +2 <INS: 1, 1999-01-01, 2000-01-01 > +3 >INS: 1, 2018-01-01, 2018-12-12 > +4 <INS: 1, 2018-01-01, 2018-12-12 > +5 >INS: 1, 1999-01-01, 2000-01-01 > +6 <INS: 1, 1999-01-01, 2000-01-01 > +7 >INS: 1, 2018-01-01, 2019-01-01 > +8 <INS: 1, 2018-01-01, 2019-01-01 > +9 >INS: 2, 1998-01-01, 2000-01-01 > +# multi-table DELETE is not possible > +delete t, t1 from t1, t for portion of apptime from '2000-01-01' to '2018-01-01'; > +ERROR 42000: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'for portion of apptime from '2000-01-01' to '2018-01-01'' at line 1 > +delete t for portion of apptime from '2000-01-01' to '2018-01-01', t1 from t, t1; > +ERROR 42000: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'for portion of apptime from '2000-01-01' to '2018-01-01', t1 from t, t1' at line 1 > +# Here another check fails before parsing ends > +delete t, t1 from t for portion of apptime from '2000-01-01' to '2018-01-01', t1; > +ERROR 42S02: Unknown table 't1' in MULTI DELETE > +delete history from t2 for portion of apptime from '2000-01-01' to '2018-01-01'; > +ERROR 42000: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'for portion of apptime from '2000-01-01' to '2018-01-01'' at line 1 > +delete from t for portion of othertime from '2000-01-01' to '2018-01-01'; > +ERROR HY000: Period `othertime` is not found in table > +delete from t for portion of system_time from '2000-01-01' to '2018-01-01'; > +ERROR 42000: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'system_time' at line 1 > +create or replace table t (id int, str text, s date, e date, > +period for apptime(s,e)); > +insert into t values(1, 'data', '1999-01-01', '2018-12-12'); > +insert into t values(1, 'other data', '1999-01-01', '2018-12-12'); > +insert into t values(1, 'deleted', '2000-01-01', '2018-01-01'); > +delete from t for portion of apptime from '2000-01-01' to '2018-01-01'; > +show warnings; > +Level Code Message > +select * from t; > +id str s e > +1 data 1999-01-01 2000-01-01 > +1 data 2018-01-01 2018-12-12 > +1 other data 1999-01-01 2000-01-01 > +1 other data 2018-01-01 2018-12-12 > +drop table t1; > +# SQL16, Part 2, 15.7 <Effect of deleting rows from base tables>, > +# General rules, 8)b)i) > +# If the column descriptor that corresponds to the i-th field of BR > +# describes an identity column, a generated column, a system-time period > +# start column, or a system-time period end column, then let V i be > +# DEFAULT. > +# auto_increment field is updated > +create or replace table t (id int primary key auto_increment, s date, e date, > +period for apptime(s, e)); > +insert into t values (default, '1999-01-01', '2018-12-12'); > +select * from t; > +id s e > +1 1999-01-01 2018-12-12 > +delete from t for portion of apptime from '2000-01-01' to '2018-01-01'; > +select * from t; > +id s e > +2 1999-01-01 2000-01-01 > +3 2018-01-01 2018-12-12 > +truncate t; > +# same for trigger case > +insert into t values (default, '1999-01-01', '2018-12-12'); > +delete from t for portion of apptime from '2000-01-01' to '2018-01-01'; > +select * from t; > +id s e > +2 1999-01-01 2000-01-01 > +3 2018-01-01 2018-12-12 > +select * from log_tbl; > +id log > +1 >DEL: 1999-01-01, 2018-12-12 > +2 <DEL: 1999-01-01, 2018-12-12 > +3 >INS: 1999-01-01, 2000-01-01 > +4 <INS: 1999-01-01, 2000-01-01 > +5 >INS: 2018-01-01, 2018-12-12 > +6 <INS: 2018-01-01, 2018-12-12 > +# auto_increment field overflow > +create or replace table t (id tinyint auto_increment primary key, > +s date, e date, period for apptime(s,e)); > +insert into t values(127, '1999-01-01', '2018-12-12'); > +delete from t for portion of apptime from '2000-01-01' to '2018-01-01'; > +ERROR 22003: Out of range value for column 'id' at row 1 add select * from t here, please > +# same for trigger case > +delete from t for portion of apptime from '2000-01-01' to '2018-01-01'; > +ERROR 22003: Out of range value for column 'id' at row 1 and here and select from the log_tbl too > +# generated columns are updated > +create or replace table t (s date, e date, > +xs date as (s) stored, xe date as (e) stored, > +period for apptime(s, e)); > +insert into t values('1999-01-01', '2018-12-12', default, default); > +select * from t; > +s e xs xe > +1999-01-01 2018-12-12 1999-01-01 2018-12-12 > +delete from t for portion of apptime from '2000-01-01' to '2018-01-01'; > +select * from t; > +s e xs xe > +1999-01-01 2000-01-01 1999-01-01 2000-01-01 > +2018-01-01 2018-12-12 2018-01-01 2018-12-12 > +truncate t; > +# same for trigger case > +insert into t values('1999-01-01', '2018-12-12', default, default); > +delete from t for portion of apptime from '2000-01-01' to '2018-01-01'; > +select * from t; > +s e xs xe > +1999-01-01 2000-01-01 1999-01-01 2000-01-01 > +2018-01-01 2018-12-12 2018-01-01 2018-12-12 > +select * from log_tbl; > +id log > +1 >DEL: 1999-01-01, 2018-12-12 > +2 <DEL: 1999-01-01, 2018-12-12 > +3 >INS: 1999-01-01, 2000-01-01 > +4 <INS: 1999-01-01, 2000-01-01 > +5 >INS: 2018-01-01, 2018-12-12 > +6 <INS: 2018-01-01, 2018-12-12 > +# View can't be used > +create or replace view v as select * from t; > +delete from v for portion of p from '2000-01-01' to '2018-01-01'; > +ERROR 42S02: 'v' is a view > +# system_time columns are updated > +create or replace table t ( > +s date, e date, > +row_start SYS_TYPE as row start invisible, > +row_end SYS_TYPE as row end invisible, > +period for apptime(s, e), > +period for system_time (row_start, row_end)) with system versioning; > +insert into t values('1999-01-01', '2018-12-12'), > +('1999-01-01', '1999-12-12'); > +select row_start into @ins_time from t limit 1; > +select * from t order by s, e; > +s e > +1999-01-01 1999-12-12 > +1999-01-01 2018-12-12 > +delete from t for portion of apptime from '2000-01-01' to '2018-01-01'; > +select *, if(row_start = @ins_time, "OLD", "NEW"), check_row(row_start, row_end) > +from t for system_time all > +order by s, e, row_start; > +s e if(row_start = @ins_time, "OLD", "NEW") check_row(row_start, row_end) > +1999-01-01 1999-12-12 OLD CURRENT ROW > +1999-01-01 2000-01-01 NEW CURRENT ROW > +1999-01-01 2018-12-12 OLD HISTORICAL ROW > +2018-01-01 2018-12-12 NEW CURRENT ROW > +# same for trigger case > +delete from t; > +delete history from t; > +insert into t values('1999-01-01', '2018-12-12'), > +('1999-01-01', '1999-12-12'); > +select row_start into @ins_time from t limit 1; > +select * from t order by s, e; > +s e > +1999-01-01 1999-12-12 > +1999-01-01 2018-12-12 > +delete from t for portion of apptime from '2000-01-01' to '2018-01-01'; > +select *, if(row_start = @ins_time, "OLD", "NEW"), check_row(row_start, row_end) > +from t for system_time all > +order by s, e, row_start; > +s e if(row_start = @ins_time, "OLD", "NEW") check_row(row_start, row_end) > +1999-01-01 1999-12-12 OLD CURRENT ROW > +1999-01-01 2000-01-01 NEW CURRENT ROW > +1999-01-01 2018-12-12 OLD HISTORICAL ROW > +2018-01-01 2018-12-12 NEW CURRENT ROW > +select * from log_tbl; > +id log > +1 >DEL: 1999-01-01, 2018-12-12 > +2 <DEL: 1999-01-01, 2018-12-12 > +3 >INS: 1999-01-01, 2000-01-01 > +4 <INS: 1999-01-01, 2000-01-01 > +5 >INS: 2018-01-01, 2018-12-12 > +6 <INS: 2018-01-01, 2018-12-12 > +create or replace database test; > diff --git a/mysql-test/suite/versioning/t/select.test b/mysql-test/suite/versioning/t/select.test > --- a/mysql-test/suite/versioning/t/select.test > +++ b/mysql-test/suite/versioning/t/select.test > @@ -107,6 +107,32 @@ for system_time as of timestamp @t0 as t; > drop table t1; > drop table t2; > > +# Query conditions check > + > +create or replace table t1(x int) with system versioning; > +insert into t1 values (1); > +delete from t1; > +insert into t1 values (2); > +delete from t1; > +insert into t1 values (3); > +delete from t1; > + > +select row_start into @start1 from t1 for system_time all where x = 1; > +select row_end into @end1 from t1 for system_time all where x = 1; > +select row_start into @start2 from t1 for system_time all where x = 2; > +select row_end into @end2 from t1 for system_time all where x = 2; > +select row_start into @start3 from t1 for system_time all where x = 3; > +select row_end into @end3 from t1 for system_time all where x = 3; > + > +select x as ASOF_x from t1 for system_time as of @start2; > +select x as ASOF_x from t1 for system_time as of @end2; > +select x as FROMTO_x from t1 for system_time from @start1 to @end3; > +select x as FROMTO_x from t1 for system_time from @end1 to @start2; > +select x as BETWAND_x from t1 for system_time between @start1 and @end3; > +select x as BETWAND_x from t1 for system_time between @end1 and @start2; > + > +drop table t1; what does that have to do with MDEV-16973? > + > # Wildcard expansion on hidden fields > > create table t1( > @@ -233,9 +259,9 @@ select x from t1 for system_time as of @trx_start; > --echo ### Issue #365, bug 4 (related to #226, optimized fields) > create or replace table t1 (i int, b int) with system versioning; > insert into t1 values (0, 0), (0, 0); > -select min(i) over (partition by b) as f > -from (select i + 0 as i, b from t1) as tt > -order by i; > +#select min(i) over (partition by b) as f > +#from (select i + 0 as i, b from t1) as tt > +#order by i; why is this? > > --echo ### Issue #365, bug 5 (dangling AND) > create or replace table t1 (a int); > diff --git a/sql/sql_delete.cc b/sql/sql_delete.cc > --- a/sql/sql_delete.cc > +++ b/sql/sql_delete.cc > @@ -245,6 +245,48 @@ static bool record_should_be_deleted(THD *thd, TABLE *table, SQL_SELECT *sel, > return false; > } > > +inline > +int TABLE::update_portion_of_time(THD *thd, > + const vers_select_conds_t &period_conds, > + bool *inside_period) I don't understand why you want to keep this very much DELETE-only functionality in the TABLE class which is used everywhere. And what's the point of pretending it's in a common TABLE class, if it can only be used in sql_delete.cc? I find it quite confusing :( > +{ > + bool lcond= period_conds.field_start->val_datetime_packed(thd) > + < period_conds.start.item->val_datetime_packed(thd); > + bool rcond= period_conds.field_end->val_datetime_packed(thd) > + > period_conds.end.item->val_datetime_packed(thd); > + > + *inside_period= !lcond && !rcond; > + if (*inside_period) > + return 0; > + > + DBUG_ASSERT(!triggers || (!triggers->has_triggers(TRG_EVENT_INSERT, > + TRG_ACTION_BEFORE) > + && !triggers->has_triggers(TRG_EVENT_INSERT, > + TRG_ACTION_AFTER) > + && !triggers->has_delete_triggers())); > + > + int res= 0; > + Item *src= lcond ? period_conds.start.item : period_conds.end.item; > + uint dst_fieldno= lcond ? s->period.end_fieldno : s->period.start_fieldno; > + > + store_record(this, record[1]); > + if (likely(!res)) > + res= src->save_in_field(field[dst_fieldno], true); > + > + if (likely(!res)) > + res= update_generated_fields(); > + > + if(likely(!res)) > + res= file->ha_update_row(record[1], record[0]); > + > + restore_record(this, record[1]); > + > + if (likely(!res) && lcond && rcond) > + res= period_make_insert(period_conds.end.item, > + field[s->period.start_fieldno]); > + > + return res; > +} > > inline > int TABLE::delete_row() > @@ -672,6 +736,9 @@ bool mysql_delete(THD *thd, TABLE_LIST *table_list, COND *conds, > > table->mark_columns_needed_for_delete(); > > + if (table_list->has_period()) > + table->use_all_columns(); may be even if (table_list->has_period()) table->use_all_columns() else table->mark_columns_needed_for_delete(); > + > if ((table->file->ha_table_flags() & HA_CAN_FORCE_BULK_DELETE) && > !table->prepare_triggers_for_delete_stmt_or_event()) > will_batch= !table->file->start_bulk_delete(); > @@ -727,6 +794,16 @@ bool mysql_delete(THD *thd, TABLE_LIST *table_list, COND *conds, > delete_record= true; > } > > + /* > + From SQL2016, Part 2, 15.7 <Effect of deleting rows from base table>, > + General Rules, 8), we can conclude that DELETE FOR PORTTION OF time performs > + 0-2 INSERTS + DELETE. We can substitute INSERT+DELETE with one UPDATE, but > + only if there are no triggers set. > + It is also meaningless for system-versioned table > + */ > + portion_of_time_through_update= !has_triggers > + && !table->versioned(VERS_TIMESTAMP); I still don't understand why you disable portion_of_time_through_update for VERS_TIMESTAMP, but not for VERS_TRX_ID. > + > THD_STAGE_INFO(thd, stage_updating); > while (likely(!(error=info.read_record())) && likely(!thd->killed) && > likely(!thd->is_error())) > diff --git a/sql/sql_lex.cc b/sql/sql_lex.cc > --- a/sql/sql_lex.cc > +++ b/sql/sql_lex.cc > @@ -3576,6 +3577,20 @@ void LEX::set_trg_event_type_for_tables() > break; > } > > + if (period_conditions.is_set()) > + { > + switch (sql_command) > + { > + case SQLCOM_DELETE: > + case SQLCOM_UPDATE: > + case SQLCOM_REPLACE: > + new_trg_event_map |= static_cast<uint8> > + (1 << static_cast<int>(TRG_EVENT_INSERT)); I've added a helper for this recently, use new_trg_event_map |= trg2bit(TRG_EVENT_INSERT); > + default: > + break; > + } > + } > + > > /* > Do not iterate over sub-selects, only the tables in the outermost > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -9295,6 +9296,22 @@ history_point: > $$= Vers_history_point($1, $2); > } > ; > +opt_for_portion_of_time_clause: > + /* empty */ > + { > + $$= false; > + } > + | FOR_SYM PORTION_SYM OF_SYM ident FROM history_point TO_SYM history_point history_point allows TIMESTAMP '2010-10-10 10:10:10' and TRANSACTION 1234. You don't need any of that, just use a corresponding expression rule. E.g. bit_expr, like history_point does. > + { > + if (unlikely(0 == strcasecmp($4.str, "SYSTEM_TIME"))) > + { > + thd->parse_error(ER_SYNTAX_ERROR, $4.str); no, for the error message to look correct you need to pass the pointer into the query text. It's usually done like this: FOR_SYM PORTION_SYM OF_SYM remember_tok_start ident FROM ... { ... thd->parse_error(ER_SYNTAX_ERROR, $4); > + MYSQL_YYABORT; > + } > + $$= true; > + Lex->period_conditions.init(SYSTEM_TIME_FROM_TO, $6, $8, $4); > + } > + ; > > opt_for_system_time_clause: > /* empty */ Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 3

[Maria-developers] eq_range_index_dives_limit
by Sergey Petrunia 06 Feb '19

06 Feb '19

Hello, So, MDEV-16934 introduced eq_range_index_dives_limit into 10.2.8 and 10.3.0. The default was set to 0 (which means no limit) in order to not introduce optimizer behavior change into stable versions. The question is: should 10.4 also have 0 by default or we can set it to some finite limit? MySQL's default value is 10. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

3 2

Re: [Maria-developers] d1ccc60: MDEV-14005 Remove need for Partition Key to be part of Primary Key.
by Sergei Golubchik 05 Feb '19

05 Feb '19

Hi, Alexey! On Feb 04, Alexey Botchkov wrote: > Hi, Sergei. > > Below are my reply on your comments. > I think it's better get these agreed before I produce the new patch. > > > > + if ((sav_inited= f->inited) == RND) > > > + f->ha_rnd_end(); > > > > When can this happen? > UPDATE statements do the rnd_init() in order to access through rnd_pos, > and when they need to do the table scan. So, UPDATE does rnd_init(), then rnd_pos(), then update_row(), and at that point you need to look up the key in all partitions, so you need index_read()? Could you, please, add a comment here to explain this sequence of events? > > > +int ha_partition::check_uniques_insert(uchar *buf, > > > ... > > > + { > > > > can you use key_copy() here? > Difficult to use here key_copy() as it is. > Here i need to get the record from the arbitrary buffer, > while key_copy() gets it from the table->record[0]. > It seems weird that the key_copy() actually has the 'from_record > argument, but only uses it to check the null bit. May be, fix it? Meaning, adding 'offset' argument to Field::get_key_image(), I suppose. > > > @@ -283,6 +283,14 @@ class partition_info : public Sql_alloc > .. > > > + uint n_uniques_to_check; > > > + uchar *unique_key_buf[2]; > > > > Is it safe? Isn't partition_info shared between many handlers > > so that every handler can only use it read-only? > > it's not quite clear to me. it seems that every TABLE has its own > > partition_info. But cloned handlers all share the same partition_info. > > You're right, cloned handlers all share the same part_info. I didn't notice > this. > So i guess i should move the uniue_key_buf[2] to the class ha_partition. > Do you know BTW if we ever use cloned handlers to write/update? I don't know, sorry. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] f93af1f93e2: MDEV-12836 Avoid table rebuild when adding or removing of auto_increment settings
by Sergei Golubchik 05 Feb '19

05 Feb '19

Hi, Eugene! On Feb 04, Eugene Kosov wrote: > revision-id: f93af1f93e2 (mariadb-10.3.12-19-gf93af1f93e2) > parent(s): d4144c8e010 > author: Eugene Kosov <claprix(a)yandex.ru> > committer: Eugene Kosov <claprix(a)yandex.ru> > timestamp: 2019-01-23 19:32:02 +0300 > message: > > MDEV-12836 Avoid table rebuild when adding or removing of auto_increment settings > > ALTER_AUTO_INC: this flag indicates that ALTER operation > adds or removes AUTO_INCREMENT from some field > > Field::is_equal(): now sometime returns IS_EQUAL_BUT_AUTO_INC > which means that only AUTO_INCREMENT was changed for that field You enable instant alter for adding auto_inc. This is not what Valerii asked for in MDEV-12836, and for a reason. Consider this test case: create table t1 (a int unique); insert t1 values (NULL),(NULL),(NULL); alter table t1 modify a int auto_increment; here `alter table` modifies the data, actually generating auto-inc values for all NULL columns. I don't think you can do that instantly. You can only remove auto-inc instantly. And for that you don't need any big changes, just modify Field::is_equal() to allow it: diff --git a/sql/field.cc b/sql/field.cc --- a/sql/field.cc +++ b/sql/field.cc @@ -3507,7 +3507,7 @@ uint Field_new_decimal::is_equal(Create_field *new_field) return ((new_field->type_handler() == type_handler()) && ((new_field->flags & UNSIGNED_FLAG) == (uint) (flags & UNSIGNED_FLAG)) && - ((new_field->flags & AUTO_INCREMENT_FLAG) == + ((new_field->flags & AUTO_INCREMENT_FLAG) <= (uint) (flags & AUTO_INCREMENT_FLAG)) && (new_field->length == max_display_length()) && (new_field->decimals == dec)); @@ -9508,7 +9508,7 @@ uint Field_num::is_equal(Create_field *new_field) return ((new_field->type_handler() == type_handler()) && ((new_field->flags & UNSIGNED_FLAG) == (uint) (flags & UNSIGNED_FLAG)) && - ((new_field->flags & AUTO_INCREMENT_FLAG) == + ((new_field->flags & AUTO_INCREMENT_FLAG) <= (uint) (flags & AUTO_INCREMENT_FLAG)) && (new_field->pack_length == pack_length())); } And this should be enough. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 3

Re: [Maria-developers] c39f74ce0d9: MDEV-16974 Application period tables: UPDATE
by Sergei Golubchik 05 Feb '19

05 Feb '19

Hi, Nikita! See comments below On Jan 06, Nikita Malyavin wrote: > revision-id: c39f74ce0d9 (versioning-1.0.7-4-gc39f74ce0d9) > parent(s): fd18ed07d65 > author: Nikita Malyavin <nikitamalyavin(a)gmail.com> > committer: Nikita Malyavin <nikitamalyavin(a)gmail.com> > timestamp: 2018-12-03 13:19:18 +1000 > message: > > MDEV-16974 Application period tables: UPDATE > > diff --git a/mysql-test/suite/period/r/delete.result b/mysql-test/suite/period/r/delete.result > index 30c9220d6f9..ccb8663bf72 100644 > --- a/mysql-test/suite/period/r/delete.result > +++ b/mysql-test/suite/period/r/delete.result > @@ -83,6 +83,9 @@ log > <INS: 3, 1997-01-01, 2000-01-01 > >INS: 4, 2018-01-01, 2020-01-01 > <INS: 4, 2018-01-01, 2020-01-01 > +# multi-table DELETE is prohibited > +delete t, t1 from t for portion of apptime from '2000-01-01' to '2018-01-01', t1; > +ERROR HY000: PORTION OF time is not allowed here why not ER_PARSE_ERROR? > delete history from t2 for portion of apptime from '2000-01-01' to '2018-01-01'; > ERROR 42000: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '' at line 1 > delete from t for portion of othertime from '2000-01-01' to '2018-01-01'; > diff --git a/mysql-test/suite/period/r/update.result b/mysql-test/suite/period/r/update.result > new file mode 100644 > index 00000000000..1f59102c131 > --- /dev/null > +++ b/mysql-test/suite/period/r/update.result > @@ -0,0 +1,244 @@ > +create or replace table t (id int, s date, e date, period for apptime(s,e)); > +insert into t values(1, '1999-01-01', '2018-12-12'); > +insert into t values(1, '1999-01-01', '2017-01-01'); > +insert into t values(1, '2017-01-01', '2019-01-01'); > +insert into t values(2, '1998-01-01', '2018-12-12'); > +insert into t values(3, '1997-01-01', '2015-01-01'); > +insert into t values(4, '2016-01-01', '2020-01-01'); I'd insert also, say, (5, '2010-01-01', '2015-01-01'). > +create or replace table t1 (id int, s date, e date, period for apptime(s,e)); > +insert t1 select * from t; > +create or replace table t2 (id int, s date, e date, period for apptime(s,e)); > +insert t2 select * from t; > +update t for portion of apptime from '2000-01-01' to '2018-01-01' > + set id=id + 5; > +select * from t; same comment about unsorted selects > +id s e > +6 2000-01-01 2018-01-01 > +6 2000-01-01 2017-01-01 > +6 2017-01-01 2018-01-01 > +7 2000-01-01 2018-01-01 > +8 2000-01-01 2015-01-01 > +9 2016-01-01 2018-01-01 > +1 1999-01-01 2000-01-01 > +1 2018-01-01 2018-12-12 > +1 1999-01-01 2000-01-01 > +1 2018-01-01 2019-01-01 > +2 1998-01-01 2000-01-01 > +2 2018-01-01 2018-12-12 > +3 1997-01-01 2000-01-01 > +4 2018-01-01 2020-01-01 > +# Check triggers > +update t1 for portion of apptime from '2000-01-01' to '2018-01-01' > + set id=id + 5; > +select * from t1 order by id, s, e; > +id s e > +1 1999-01-01 2000-01-01 > +1 1999-01-01 2000-01-01 > +1 2018-01-01 2018-12-12 > +1 2018-01-01 2019-01-01 > +2 1998-01-01 2000-01-01 > +2 2018-01-01 2018-12-12 > +3 1997-01-01 2000-01-01 > +4 2018-01-01 2020-01-01 > +6 2000-01-01 2017-01-01 > +6 2000-01-01 2018-01-01 > +6 2017-01-01 2018-01-01 > +7 2000-01-01 2018-01-01 > +8 2000-01-01 2015-01-01 > +9 2016-01-01 2018-01-01 > +select * from log_tbl; > +log > +>UPD: 1, 1999-01-01, 2018-12-12 -> 6, 2000-01-01, 2018-01-01 > +<UPD: 1, 1999-01-01, 2018-12-12 -> 6, 2000-01-01, 2018-01-01 > +>INS: 1, 1999-01-01, 2000-01-01 > +<INS: 1, 1999-01-01, 2000-01-01 > +>INS: 1, 2018-01-01, 2018-12-12 > +<INS: 1, 2018-01-01, 2018-12-12 > +>UPD: 1, 1999-01-01, 2017-01-01 -> 6, 2000-01-01, 2017-01-01 > +<UPD: 1, 1999-01-01, 2017-01-01 -> 6, 2000-01-01, 2017-01-01 > +>INS: 1, 1999-01-01, 2000-01-01 > +<INS: 1, 1999-01-01, 2000-01-01 > +>UPD: 1, 2017-01-01, 2019-01-01 -> 6, 2017-01-01, 2018-01-01 > +<UPD: 1, 2017-01-01, 2019-01-01 -> 6, 2017-01-01, 2018-01-01 > +>INS: 1, 2018-01-01, 2019-01-01 > +<INS: 1, 2018-01-01, 2019-01-01 > +>UPD: 2, 1998-01-01, 2018-12-12 -> 7, 2000-01-01, 2018-01-01 > +<UPD: 2, 1998-01-01, 2018-12-12 -> 7, 2000-01-01, 2018-01-01 > +>INS: 2, 1998-01-01, 2000-01-01 > +<INS: 2, 1998-01-01, 2000-01-01 > +>INS: 2, 2018-01-01, 2018-12-12 > +<INS: 2, 2018-01-01, 2018-12-12 > +>UPD: 3, 1997-01-01, 2015-01-01 -> 8, 2000-01-01, 2015-01-01 > +<UPD: 3, 1997-01-01, 2015-01-01 -> 8, 2000-01-01, 2015-01-01 > +>INS: 3, 1997-01-01, 2000-01-01 > +<INS: 3, 1997-01-01, 2000-01-01 > +>UPD: 4, 2016-01-01, 2020-01-01 -> 9, 2016-01-01, 2018-01-01 > +<UPD: 4, 2016-01-01, 2020-01-01 -> 9, 2016-01-01, 2018-01-01 > +>INS: 4, 2018-01-01, 2020-01-01 > +<INS: 4, 2018-01-01, 2020-01-01 > +# INSERT trigger only also works > +drop trigger tr1upd_t2; > +drop trigger tr2upd_t2; > +update t2 for portion of apptime from '2000-01-01' to '2018-01-01' > + set id=id + 5; > +select * from t2 order by id, s, e; > +id s e > +1 1999-01-01 2000-01-01 > +1 1999-01-01 2000-01-01 > +1 2018-01-01 2018-12-12 > +1 2018-01-01 2019-01-01 > +2 1998-01-01 2000-01-01 > +2 2018-01-01 2018-12-12 > +3 1997-01-01 2000-01-01 > +4 2018-01-01 2020-01-01 > +6 2000-01-01 2017-01-01 > +6 2000-01-01 2018-01-01 > +6 2017-01-01 2018-01-01 > +7 2000-01-01 2018-01-01 > +8 2000-01-01 2015-01-01 > +9 2016-01-01 2018-01-01 > +select * from log_tbl; > +log > +>INS: 1, 1999-01-01, 2000-01-01 > +<INS: 1, 1999-01-01, 2000-01-01 > +>INS: 1, 2018-01-01, 2018-12-12 > +<INS: 1, 2018-01-01, 2018-12-12 > +>INS: 1, 1999-01-01, 2000-01-01 > +<INS: 1, 1999-01-01, 2000-01-01 > +>INS: 1, 2018-01-01, 2019-01-01 > +<INS: 1, 2018-01-01, 2019-01-01 > +>INS: 2, 1998-01-01, 2000-01-01 > +<INS: 2, 1998-01-01, 2000-01-01 > +>INS: 2, 2018-01-01, 2018-12-12 > +<INS: 2, 2018-01-01, 2018-12-12 > +>INS: 3, 1997-01-01, 2000-01-01 > +<INS: 3, 1997-01-01, 2000-01-01 > +>INS: 4, 2018-01-01, 2020-01-01 > +<INS: 4, 2018-01-01, 2020-01-01 > +select * from t for portion of apptime from 0 to 1 for system_time all; > +ERROR 42000: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '' at line 1 should be "right syntax to use near 'for portion of apptime from ...'" > +update t for portion of apptime from 0 to 1 for system_time all set id=1; > +ERROR 42000: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near 'set id=1' at line 1 why not "right syntax to use near 'for system_time all ...'" ? > +# Modifying period start/end fields is forbidden. > +# SQL17: fix the reference, please (SQL:2016, Part 2, section ... paragraph ...) > +# Neither BSTARTCOL nor BENDCOL shall be an explicit <object column> > +# contained in the <set clause list>. > +update t for portion of apptime from '2000-01-01' to '2018-01-01' > + set id= id + 5, s=subdate(s, 5), e=adddate(e, 5); > +ERROR HY000: Column `s` used in period `apptime` specified in update SET list > +# Precision timestamps > +create or replace table t (id int, s timestamp(5), e timestamp(5), > +period for apptime(s,e)); > +insert into t values(1, '1999-01-01', '2018-12-12'); > +insert into t values(1, '1999-01-01', '2017-01-01'); > +update t for portion of apptime from '2000-01-01 00:00:00.00015' > + to '2018-01-01 12:34:56.31415' > + set id= id + 5; > +select * from t; > +id s e > +6 2000-01-01 00:00:00.00015 2018-01-01 12:34:56.31415 > +6 2000-01-01 00:00:00.00015 2017-01-01 00:00:00.00000 > +1 1999-01-01 00:00:00.00000 2000-01-01 00:00:00.00015 > +1 2018-01-01 12:34:56.31415 2018-12-12 00:00:00.00000 > +1 1999-01-01 00:00:00.00000 2000-01-01 00:00:00.00015 > +# Strings > +create or replace table t (id int, str text, s date, e date, > +period for apptime(s,e)); > +insert into t values(1, 'data', '1999-01-01', '2018-12-12'); > +insert into t values(1, 'other data', '1999-01-01', '2018-12-12'); > +update t for portion of apptime from '2000-01-01' to '2018-01-01' > + set id= id + 5; > +select * from t; > +id str s e > +6 data 2000-01-01 2018-01-01 > +6 other data 2000-01-01 2018-01-01 > +1 data 1999-01-01 2000-01-01 > +1 data 2018-01-01 2018-12-12 > +1 other data 1999-01-01 2000-01-01 > +1 other data 2018-01-01 2018-12-12 > +# multi-table UPDATE is prohibited > +create or replace table t1(x int); > +update t for portion of apptime from '2000-01-01' to '2018-01-01', t1 > +set t.id= t.id + 5; > +ERROR HY000: PORTION OF time is not allowed here why not ER_PARSE_ERROR? > +update t1 set x= (select id from t for portion of apptime from '2000-01-01' to '2018-01-01'); > +ERROR HY000: PORTION OF time is not allowed here why not ER_PARSE_ERROR? > +# SQL17: fix the reference, please > +# Let FROMVAL be <point in time 1>. FROMVAL shall not generally contain a > +# reference to a column of T or a <routine invocation> > +# whose subject routine is an SQL-invoked routine that > +# is possibly non-deterministic or that possibly modifies SQL-data. > +# ...Same for <point in time 2> (TOVAL) > +update t for portion of apptime from 5*(5+s) to 1 set t.id= t.id + 5; > +ERROR HY000: Values in range FOR PORTION OF `apptime` should be constant expressions > +update t for portion of apptime from 1 to e set t.id= t.id + 5; > +ERROR HY000: Values in range FOR PORTION OF `apptime` should be constant expressions > +set @s= '2000-01-01'; > +set @e= '2018-01-01'; > +create or replace function f() returns date return @e; > +create or replace function g() returns date not deterministic return @e; > +create or replace function h() returns date deterministic return @e; > +update t for portion of apptime from @s to f() set t.id= t.id + 5; > +ERROR HY000: Values in range FOR PORTION OF `apptime` should be constant expressions > +update t for portion of apptime from @s to g() set t.id= t.id + 5; > +ERROR HY000: Values in range FOR PORTION OF `apptime` should be constant expressions > +# success > +update t for portion of apptime from @s to h() set t.id= t.id + 5; > +# select value is cached > +update t for portion of apptime from (select s from t2 limit 1) to h() set t.id= t.id + 5; > +# auto_inrement field is updated > +create or replace table t (id int primary key auto_increment, x int, > +s date, e date, period for apptime(s, e)); > +insert into t values (default, 1, '1999-01-01', '2018-12-12'); > +update t for portion of apptime from '2000-01-01' to '2018-01-01' set x= x + 5; > +select * from t; > +id x s e > +1 6 2000-01-01 2018-01-01 > +2 1 1999-01-01 2000-01-01 > +3 1 2018-01-01 2018-12-12 > +truncate t; > +insert into t values (default, 1, '1999-01-01', '2018-12-12'); > +update t for portion of apptime from '2000-01-01' to '2018-01-01' set x= 1; > +select * from t; > +id x s e > +1 1 2000-01-01 2018-01-01 > +2 1 1999-01-01 2000-01-01 > +3 1 2018-01-01 2018-12-12 > +# generated columns are updated > +create or replace table t (x int, s date, e date, > +xs date as (s) stored, xe date as (e) stored, > +period for apptime(s, e)); > +insert into t values(1, '1999-01-01', '2018-12-12', default, default); > +select * from t; > +x s e xs xe > +1 1999-01-01 2018-12-12 1999-01-01 2018-12-12 > +update t for portion of apptime from '2000-01-01' to '2018-01-01' set x= x + 5; > +select *, xs=s and xe=e from t; > +x s e xs xe xs=s and xe=e > +6 2000-01-01 2018-01-01 2000-01-01 2018-01-01 1 > +1 1999-01-01 2000-01-01 1999-01-01 2000-01-01 1 > +1 2018-01-01 2018-12-12 2018-01-01 2018-12-12 1 > +# system_time columns are updated > +create or replace table t (x int, s date, e date, > +row_start SYS_TYPE as row start invisible, > +row_end SYS_TYPE as row end invisible, a bit strange to use invisible in tests for strict standard compliance :) but ok, whatever > +period for apptime(s, e), > +period for system_time(row_start, row_end)) with system versioning; > +insert into t values(1, '1999-01-01', '2018-12-12'), > +(2, '1999-01-01', '1999-12-12'); > +select row_start into @ins_time from t limit 1; > +select * from t order by x, s, e; > +x s e > +1 1999-01-01 2018-12-12 > +2 1999-01-01 1999-12-12 > +update t for portion of apptime from '2000-01-01' to '2018-01-01' set x= x + 5; > +select *, if(row_start = @ins_time, "OLD", "NEW"), check_row(row_start, row_end) > +from t for system_time all > +order by x, s, e, row_start; > +x s e if(row_start = @ins_time, "OLD", "NEW") check_row(row_start, row_end) > +1 1999-01-01 2000-01-01 NEW CURRENT ROW > +1 1999-01-01 2018-12-12 OLD HISTORICAL ROW > +1 2018-01-01 2018-12-12 NEW CURRENT ROW > +2 1999-01-01 1999-12-12 OLD CURRENT ROW > +6 2000-01-01 2018-01-01 NEW CURRENT ROW > +create or replace database test; > diff --git a/mysql-test/suite/period/t/delete.test b/mysql-test/suite/period/t/delete.test > index b34de6c37e7..39a68959523 100644 > --- a/mysql-test/suite/period/t/delete.test > +++ b/mysql-test/suite/period/t/delete.test > @@ -34,6 +34,10 @@ drop trigger tr2del_t2; > delete from t2 for portion of APPTIME from '2000-01-01' to '2018-01-01'; > select * from log_tbl; > > +--echo # multi-table DELETE is prohibited > +--error ER_PORTION_OF_TIME_NOT_ALLOWED > +delete t, t1 from t for portion of apptime from '2000-01-01' to '2018-01-01', t1; move this into your MDEV-16973 commit is possible, please, > + > --error ER_PARSE_ERROR > delete history from t2 for portion of apptime from '2000-01-01' to '2018-01-01'; > > diff --git a/mysql-test/suite/period/t/update.opt b/mysql-test/suite/period/t/update.opt > new file mode 100644 > index 00000000000..92a881bef67 > --- /dev/null > +++ b/mysql-test/suite/period/t/update.opt > @@ -0,0 +1 @@ > +--explicit_defaults_for_timestamp Why? it's best to avoid non-default command line options, unless you actually test this particular option. If you just need it for convenience - don't. every non-default command line options means a server restart, and they're slow. write DEFAULT NULL explicitly when needed. > diff --git a/sql/share/errmsg-utf8.txt b/sql/share/errmsg-utf8.txt > index 7ceff1a215e..c5a359b09e8 100644 > --- a/sql/share/errmsg-utf8.txt > +++ b/sql/share/errmsg-utf8.txt > @@ -7951,3 +7951,12 @@ ER_PERIOD_NOT_FOUND > > ER_PERIOD_SYSTEM_TIME_NOT_ALLOWED > eng "period SYSTEM_TIME is not allowed here" > + > +ER_PORTION_OF_TIME_NOT_ALLOWED > + eng "PORTION OF time is not allowed here" > + > +ER_PERIOD_PORTION_OF_TIME_CONSTANT > + eng "Values in range FOR PORTION OF %`s should be constant expressions" Reuse ER_NO_CONST_EXPR_IN_RANGE_OR_LIST_ERROR please. Like ER_NO_CONST_EXPR_IN_RANGE_OR_LIST_ERROR -> ER_NOT_CONSTANT_EXPRESSION eng "Expression in RANGE/LIST VALUES must be constant" -> eng "Expression in %s must be constant" and in mysql.h #define ER_NO_CONST_EXPR_IN_RANGE_OR_LIST_ERROR ER_NOT_CONSTANT_EXPRESSION > + > +ER_PERIOD_COLUMNS_UPDATED > + eng "Column %`s used in period %`s specified in update SET list" > diff --git a/sql/sql_lex.cc b/sql/sql_lex.cc > index 7a18ac82236..93d2f417a50 100644 > --- a/sql/sql_lex.cc > +++ b/sql/sql_lex.cc > @@ -1343,13 +1343,16 @@ int Lex_input_stream::lex_token(YYSTYPE *yylval, THD *thd) > /* > * Additional look-ahead to resolve doubtful cases like: > * SELECT ... FOR UPDATE > - * SELECT ... FOR SYSTEM_TIME ... . > + * SELECT ... FOR SYSTEM_TIME ... > + * SELECT ... FOR PORTION OF ... . Can you show an example of the statement where you need a second look-ahead to distinguish between FOR PORTION OF and FOR SYSTEM_TIME or FOR UPDATE ? > */ > token= lex_one_token(yylval, thd); > add_digest_token(token, yylval); > switch(token) { > case SYSTEM_TIME_SYM: > return FOR_SYSTEM_TIME_SYM; > + case PORTION: > + return FOR_PORTION_SYM; > default: > /* > Save the token following 'FOR_SYM' > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > index 220fcf6cc34..44989b20fae 100644 > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -8121,6 +8121,12 @@ int setup_conds(THD *thd, TABLE_LIST *tables, List<TABLE_LIST> &leaves, > > for (table= tables; table; table= table->next_local) > { > + if (table->has_period() && !thd->lex->portion_of_time_applicable()) > + { > + my_error(ER_PORTION_OF_TIME_NOT_ALLOWED, MYF(0)); > + goto err_no_arena; > + } I don't understand that either, why do you need to check the correct syntax in setup_conds() ? > + > if (select_lex == thd->lex->first_select_lex() && > select_lex->first_cond_optimization && > table->merged_for_insert && > diff --git a/sql/sql_delete.cc b/sql/sql_delete.cc > index e76eb729536..4726557d762 100644 > --- a/sql/sql_delete.cc > +++ b/sql/sql_delete.cc > @@ -752,6 +752,10 @@ bool mysql_delete(THD *thd, TABLE_LIST *table_list, COND *conds, > delete_record= true; > } > > + /* SQL standard defines DELETE FOR PORTTION OF time as 0-2 INSERTS + DELETE > + * We can substitute INSERT+DELETE with one UPDATE, but only if there are > + * no triggers set. It is also meaningless for system-versioned table > + */ 1. format multi-line comments like everywhere in the code, please 2. every time you refer to the standard, say where the standard says that 3. move this comment in your MDEV-16973 commit, if possible > portion_of_time_through_update= !has_period_triggers > && !table->versioned(VERS_TIMESTAMP); > > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index 0c368b40836..10535326ce5 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -11910,12 +11911,13 @@ join_table_parens: > > > table_primary_ident: > - table_ident opt_use_partition opt_for_system_time_clause > + table_ident opt_use_partition > + opt_for_portion_of_time_clause opt_for_system_time_clause Oh, no. FOR PORTION OF has very few well defined places where the standard allows it. Don't put it into the generic used-everywhere table_primary_ident. That's why you needed FOR_PORTION_SYM and a second look-ahead. > opt_table_alias_clause opt_key_definition > { > SELECT_LEX *sel= Select; > sel->table_join_options= 0; > - if (!($$= Select->add_table_to_list(thd, $1, $4, > + if (!($$= Select->add_table_to_list(thd, $1, $5, > Select->get_table_join_options(), > YYPS->m_lock_type, > YYPS->m_mdl_type, > diff --git a/sql/table.h b/sql/table.h > index 1b897c1b4c6..f104f4fc99c 100644 > --- a/sql/table.h > +++ b/sql/table.h > @@ -2420,7 +2423,7 @@ struct TABLE_LIST > > bool has_period() const > { > - return period_conditions.name; > + return period_conditions.is_set(); why? > } > > /** > > diff --git a/sql/table.cc b/sql/table.cc > index 8f1c94dd633..93908a9e2e3 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -6456,7 +6456,7 @@ void TABLE::mark_columns_needed_for_delete(bool with_period) > retrieve the row again. > */ > > -void TABLE::mark_columns_needed_for_update() > +void TABLE::mark_columns_needed_for_update(bool with_period) just like in the delete case, I think this is the caller's problem > { > DBUG_ENTER("TABLE::mark_columns_needed_for_update"); > bool need_signal= false; > @@ -7870,19 +7871,41 @@ static int period_make_insert(TABLE *table, Item *src, Field *dst) > } > > +int TABLE::cut_fields_for_portion_of_time(THD *thd, const vers_select_conds_t &period_conds) looks like something that belongs to sql_update.cc, not TABLE > +{ > + bool lcond= period_conds.field_start->val_datetime_packed(thd) > + < period_conds.start.item->val_datetime_packed(thd); > + bool rcond= period_conds.field_end->val_datetime_packed(thd) > + > period_conds.end.item->val_datetime_packed(thd); > + > + Field *start_field= field[s->period.start_fieldno]; > + Field *end_field= field[s->period.end_fieldno]; > + > + int res= 0; > + if (lcond && !start_field->has_explicit_value()) > + res= period_conds.start.item->save_in_field(start_field, true); > + > + if (likely(!res) && rcond && !end_field->has_explicit_value()) > + res= period_conds.end.item->save_in_field(end_field, true); > + > + return res; > +} > + > -int TABLE::update_portion_of_time(THD *thd, vers_select_conds_t &period_conds, > +int TABLE::update_portion_of_time(THD *thd, const vers_select_conds_t &period_conds, > bool *inside_period) looks like something that belongs to sql_delete.cc, not TABLE > { > bool lcond= period_conds.field_start->val_datetime_packed(thd) > @@ -7917,7 +7940,8 @@ int TABLE::update_portion_of_time(THD *thd, vers_select_conds_t &period_conds, > return res; > } > > -int TABLE::insert_portion_of_time(THD *thd, vers_select_conds_t &period_conds) > +int TABLE::insert_portion_of_time(THD *thd, const vers_select_conds_t &period_conds, > + ha_rows &rows_inserted) pointers or _constant_ references. Here, rows_inserted must be a pointer. > { > bool lcond= period_conds.field_start->val_datetime_packed(thd) > < period_conds.start.item->val_datetime_packed(thd); > diff --git a/sql/sql_update.cc b/sql/sql_update.cc > index 27bd6f04670..d5240f5524e 100644 > --- a/sql/sql_update.cc > +++ b/sql/sql_update.cc > @@ -177,6 +178,21 @@ static bool check_fields(THD *thd, List<Item> &items, bool update_view) > f->set_has_explicit_value(); > } > } > + > + if (thd->lex->sql_command == SQLCOM_UPDATE && table->has_period()) what else can sql_command be here? SQLCOM_UPDATE_MULTI? SQLCOM_UPDATE_MULTI cannot have table->has_period(). > + { > + for (List_iterator_fast<Item> it(items); (item=it++);) > + { > + Lex_ident name(item->name); > + vers_select_conds_t &period= table->period_conditions; > + if (name.streq(period.field_start->name) > + || name.streq(period.field_end->name)) Don't compare names, compare fields. The field to be updated is, I guess, item->field_for_view_update()->field > + { > + my_error(ER_PERIOD_COLUMNS_UPDATED, MYF(0), name.str, period.name.str); > + return true; > + } > + } > + } > return FALSE; > } > > @@ -323,6 +339,7 @@ int mysql_update(THD *thd, > List<Item> all_fields; > killed_state killed_status= NOT_KILLED; > bool has_triggers, binlog_is_row, do_direct_update= FALSE; > + bool has_period_triggers= false; same as for sql_delete.cc, one has_triggers is enough, second has_period_triggers is quite confusing. > Update_plan query_plan(thd->mem_root); > Explain_update *explain; > TABLE_LIST *update_source_table; > @@ -880,14 +905,25 @@ int mysql_update(THD *thd, > explain->tracker.on_record_after_where(); > store_record(table,record[1]); > > + if (table_list->has_period()) > + table->cut_fields_for_portion_of_time(thd, table_list->period_conditions); 1. this should be done after BEFORE triggers (see SQL:2016, part 2, 15.13 "Effect of replacing rows in base tables") 2. why in cut_fields_for_portion_of_time() you check for has_explicit_value()? fields cannot have explicit values there, you've checked that in check_fields(). > + > if (fill_record_n_invoke_before_triggers(thd, table, fields, values, 0, > TRG_EVENT_UPDATE)) > break; /* purecov: inspected */ > > found++; > > - if (!can_compare_record || compare_record(table)) > + bool record_was_same= false; > + bool need_update= !can_compare_record || compare_record(table) || > + thd->lex->sql_command == SQLCOM_DELETE; why sql_command would be SQLCOM_DELETE here? > + > + if (need_update) > { > + if (table->versioned(VERS_TIMESTAMP) && > + thd->lex->sql_command == SQLCOM_DELETE) > + table->vers_update_end(); > + > if (table->default_field && table->update_default_fields(1, ignore)) > { > error= 1; > @@ -994,6 +1032,13 @@ int mysql_update(THD *thd, > break; > } > > + if (need_update && !record_was_same && table_list->has_period()) I suspect this should happen even if record_was_same. The standard never says "if new values are the same as old values, don't update anything" > + { > + restore_record(table, record[1]); > + table->insert_portion_of_time(thd, table_list->period_conditions, > + rows_inserted); > + } > + > if (!--limit && using_limit) > { > /* Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 4

Re: [Maria-developers] 83444b747f1: MDEV-16975 Application period tables: ALTER TABLE
by Sergei Golubchik 05 Feb '19

05 Feb '19

Hi, Nikita! Please, see my comments below On Dec 28, Nikita Malyavin wrote: > revision-id: 83444b747f1 (versioning-1.0.7-5-g83444b747f1) > parent(s): c39f74ce0d9 > author: Nikita Malyavin <nikitamalyavin(a)gmail.com> > committer: Nikita Malyavin <nikitamalyavin(a)gmail.com> > timestamp: 2018-12-03 13:19:18 +1000 > message: > > MDEV-16975 Application period tables: ALTER TABLE > > * implicit period constraint is hidden and cannot be dropped independently > * create...like and create...select support > > diff --git a/mysql-test/suite/period/r/alter.result b/mysql-test/suite/period/r/alter.result > new file mode 100644 > index 00000000000..6bb9f905358 > --- /dev/null > +++ b/mysql-test/suite/period/r/alter.result > @@ -0,0 +1,101 @@ > +set @s= '1992-01-01'; > +set @e= '1999-12-31'; > +create or replace table t (s date, e date); > +alter table t add period for a(s, e); > +ERROR HY000: Period field `s` cannot be NULL > +alter table t change s s date, add period for a(s, e); > +ERROR HY000: Period field `s` cannot be NULL > +alter table t change s s date, change e e date, add period for a(s, e); > +ERROR HY000: Period field `s` cannot be NULL > +alter table t change s s date not null, change e e date not null, > +add period for a(s, e); > +show create table t; > +Table Create Table > +t CREATE TABLE `t` ( > + `s` date NOT NULL, > + `e` date NOT NULL, > + PERIOD FOR `a` (`s`, `e`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +alter table t add id int; > +alter table t drop id; > +insert t values(@e, @s); > +ERROR 23000: CONSTRAINT `CONSTRAINT_1` failed for `test`.`t` Hmm. The standard, indeed, says "there's an implicit constraint". But it does not say "CHECK contraint". UNIQUE is also a constraint, Foreign key is also a constraint. So, I think here it means really just *a* constraint (a limitation, in other words), meaning only that start value must be before the end value. It doesn't mean that a CHECK constraint must be created. In other words, it should not be a CHECK constraint, it should not produce a `CONSTRAINT xxx failed` message, and, of course, it should never ever be shown as a CHECK constraint. > +alter table t drop constraint constraint_1; > +ERROR HY000: Can't DROP CONSTRAINT `CONSTRAINT_1`. Use DROP PERIOD `a` for this > +alter table t drop period for a; > +# Constraint is dropped > +insert t values(@e, @s); > +alter table t drop period for a; > +ERROR 42000: Can't DROP PERIOD `a`; check that it exists > +alter table t add period for a(s, e), drop period for a; > +ERROR 42000: Can't DROP PERIOD `a`; check that it exists > +truncate t; > +alter table t add period for a(s, e); > +insert t values(@e, @s); > +ERROR 23000: CONSTRAINT `CONSTRAINT_1` failed for `test`.`t` > +alter table t add period for a(s, e), drop period for a; > +insert t values(@e, @s); > +ERROR 23000: CONSTRAINT `CONSTRAINT_1` failed for `test`.`t` > +alter table t add s1 date not null, add period for b(s1, e), drop period for a; > +show create table t; > +Table Create Table > +t CREATE TABLE `t` ( > + `s` date NOT NULL, > + `e` date NOT NULL, > + `s1` date NOT NULL, > + PERIOD FOR `b` (`s1`, `e`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +insert t(s, s1, e) values(@e, @s, @e); > +insert t(s, s1, e) values(@e, @e, @s); > +ERROR 23000: CONSTRAINT `CONSTRAINT_1` failed for `test`.`t` > +alter table t add constraint check (s < s1 and s1 < e); > +ERROR HY000: CONSTRAINT `(null)` uses fields from PERIOD `b` huh? looks like a null-pointer dereference with a defensive crash-safe printf. and why is it an error? > +create table t1 like t; > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `s` date NOT NULL, > + `e` date NOT NULL, > + `s1` date NOT NULL, > + PERIOD FOR `b` (`s1`, `e`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +create table t2 (period for b(s,e)) select * from t; > +ERROR 23000: CONSTRAINT `CONSTRAINT_1` failed for `test`.`t2` > +create table t2 (period for b(s1,e)) select * from t; > +# SQL17 p.895 please, fix all the references to the standard to mention the part number, section number and paragraph number instead of the page number. > +# The declared type of BC1 shall be either DATE or a timestamp type > +# and shall be equivalent to the declared type of BC2. > +create or replace table t (s timestamp not null, e timestamp(6) not null); > +alter table t add period for a(s, e); > +ERROR HY000: PERIOD FOR `a` fields types mismatch > +# SQL17 p.895 > +# No column of T shall have a column name that is equivalent to ATPN. > +create or replace table t (a int, s date, e date); > +alter table t add period for a(s, e); > +ERROR HY000: Period and field `a` cannot have same name. > +# SQL17 p.895 > +# Neither BC1 nor BC2 shall be an identity column, a generated column, > +# a system-time period start column, or a system-time period end column. > +create or replace table t (id int primary key, > +s date, > +e date generated always as (s+1)); > +alter table t add period for a(s, e); > +ERROR HY000: Period field `s` cannot be NULL > +create or replace table t (id int primary key, > +s date, > +e date as (s+1) VIRTUAL); > +alter table t add period for a(s, e); > +ERROR HY000: Period field `s` cannot be NULL > +create or replace table t (id int primary key, s timestamp(6), e timestamp(6), > +st timestamp(6) as row start, > +en timestamp(6) as row end, > +period for system_time (st, en)) with system versioning; > +alter table t add period for a(s, en); > +ERROR HY000: Period field `en` cannot be ROW END > +# SQL17 p.895 > +# The table descriptor of T shall not include a period descriptor other > +# than a system-time period descriptor. > +alter table t add period for a(s, e); > +alter table t add period for b(s, e); > +ERROR HY000: cannot specify more than one application-time period > +create or replace database test; > diff --git a/mysql-test/suite/period/r/create.result b/mysql-test/suite/period/r/create.result > index 420eceb2e9a..8c06a0cc134 100644 > --- a/mysql-test/suite/period/r/create.result > +++ b/mysql-test/suite/period/r/create.result > @@ -51,7 +51,7 @@ ERROR HY000: cannot specify more than one application-time period > # No <column name> in any <column definition> shall be equivalent to PN. > create or replace table t (mytime int, s date, e date, > period for mytime(s,e)); > -ERROR HY000: Could not specify name `mytime` for field. It is already used by period. > +ERROR HY000: Period and field `mytime` cannot have same name. This is better. Although I'd just use an existing ER_DUP_FIELDNAME > # CD1 and CD2 both contain either an explicit or an implicit > # <column constraint definition> that specifies > # NOT NULL NOT DEFERRABLE ENFORCED. > diff --git a/sql/field.cc b/sql/field.cc > index d7214687e2d..e065d61e3a5 100644 > --- a/sql/field.cc > +++ b/sql/field.cc > @@ -10545,6 +10545,8 @@ Column_definition::Column_definition(THD *thd, Field *old_field, > unireg_check= Field::TMYSQL_COMPRESSED; > compression_method_ptr= zlib_compression_method; > } > + if (orig_field->maybe_null()) > + flags|= EXPLICIT_NULL_FLAG; This shouldn't be needed, as I wrote in a CREATE TABLE review, NOT NULL should be implicit, not an error. > } > else > { > diff --git a/sql/sql_alter.h b/sql/sql_alter.h > index 108b98afdd7..d917793dd90 100644 > --- a/sql/sql_alter.h > +++ b/sql/sql_alter.h > @@ -95,6 +95,8 @@ class Alter_info > CHECK_CONSTRAINT_IF_NOT_EXISTS= 1 > }; > List<Virtual_column_info> check_constraint_list; > + List<Table_period_info> period_list; looks unused (in this patch, at least) > + > // Type of ALTER TABLE operation. > alter_table_operations flags; > ulong partition_flags; > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index 4189131f801..211969647d4 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -6152,6 +6152,11 @@ handle_if_exists_options(THD *thd, TABLE *table, Alter_info *alter_info) > } > } > } > + else if (drop->type == Alter_drop::PERIOD) > + { > + if (table->s->period.name.streq(drop->name)) > + remove_drop= FALSE; you don't seem to have a single test for DROP PERIOD IF EXISTS > + } > else /* Alter_drop::KEY and Alter_drop::FOREIGN_KEY */ > { > uint n_key; > @@ -8406,6 +8412,34 @@ mysql_prepare_alter_table(THD *thd, TABLE *table, > } > } > > + if (table->s->period.name) > + { > + drop_it.rewind(); > + Alter_drop *drop; > + for (bool found= false; !found && (drop= drop_it++); ) > + { > + found= table->s->period.name.streq(drop->name); > + } > + > + if (drop) > + { > + drop_period= true; > + drop_it.remove(); > + } > + else if (create_info->period_info.is_set() && table->s->period.name) > + { > + my_error(ER_PERIOD_MAX_COUNT_EXCEEDED, MYF(0), "application"); > + goto err; > + } > + else > + { > + Field *s= table->s->period.start_field(table->s); > + Field *e= table->s->period.end_field(table->s); > + create_info->period_info.set_period(s->field_name, e->field_name); > + create_info->period_info.name= table->s->period.name; why couldn't you remember Field's, why only names? is it because period_info is also used on CREATE TABLE and you don't have Field's there? > + } > + } > + > /* Add all table level constraints which are not in the drop list */ > if (table->s->table_check_constraints) > { > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index 10535326ce5..545c8f831b9 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -8190,6 +8195,10 @@ alter_list_item: > { > Lex->alter_info.flags|= ALTER_ADD_PERIOD; > } > + | ADD period_for_application_time where's ADD PERIOD if_not_exists? > + { > + Lex->alter_info.flags|= ALTER_ADD_CHECK_CONSTRAINT; > + } > | add_column '(' create_field_list ')' > { > LEX *lex=Lex; Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 3

Re: [Maria-developers] ce91482: MDEV-17599 ALTER TABLE DROP CONSTRAINT does not work for foreign keys.
by Sergei Golubchik 04 Feb '19

04 Feb '19

Hi, Alexey! Looks good, one comment below Ok to push On Feb 04, Alexey Botchkov wrote: > revision-id: ce91482df1af6935f81a2142d0ee9391feab2454 (mariadb-10.2.21-45-gce91482) > parent(s): 97930df13c0e403940969ebb47398760b59f753c > committer: Alexey Botchkov > timestamp: 2019-02-04 12:57:57 +0400 > message: > > MDEV-17599 ALTER TABLE DROP CONSTRAINT does not work for foreign keys. > > The list of table constraints doesn't include foreign keys and uniques. > So we replace DROP CONSTRAINT with DROP [FOREIGN] KEY in this case. > > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index 1b426f8..bd6ab9e 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -8993,6 +8993,64 @@ bool mysql_alter_table(THD *thd,char *new_db, char *new_name, > > THD_STAGE_INFO(thd, stage_setup); > > + > + if (alter_info->flags & Alter_info::ALTER_DROP_CHECK_CONSTRAINT) > + { > + /* > + ALTER TABLE DROP CONSTRAINT > + should be replaced with ... DROP FOREIGN KEY > + if the constraint is the FOREIGN KEY one. > + */ > + > + List_iterator<Alter_drop> drop_it(alter_info->drop_list); > + Alter_drop *drop; > + alter_info->flags&= ~Alter_info::ALTER_DROP_CHECK_CONSTRAINT; > + > + while ((drop= drop_it++)) > + { > + if (drop->type == Alter_drop::CHECK_CONSTRAINT) > + { > + { > + /* Test if there is a FOREIGN KEY with this name. */ > + List <FOREIGN_KEY_INFO> fk_child_key_list; > + FOREIGN_KEY_INFO *f_key; > + table->file->get_foreign_key_list(thd, &fk_child_key_list); move this out of the loop, perhaps? > + List_iterator<FOREIGN_KEY_INFO> fk_key_it(fk_child_key_list); > + > + while ((f_key= fk_key_it++)) > + { > + if (my_strcasecmp(system_charset_info, f_key->foreign_id->str, > + drop->name) == 0) > + { > + drop->type= Alter_drop::FOREIGN_KEY; > + alter_info->flags|= Alter_info::DROP_FOREIGN_KEY; > + goto do_continue; > + } > + } > + } > + > + { > + /* Test if there is an UNIQUE with this name. */ > + uint n_key; > + > + for (n_key=0; n_key < table->s->keys; n_key++) > + { > + if ((table->key_info[n_key].flags & HA_NOSAME) && > + my_strcasecmp(system_charset_info, > + drop->name, table->key_info[n_key].name) == 0) > + { > + drop->type= Alter_drop::KEY; > + alter_info->flags|= Alter_info::ALTER_DROP_INDEX; > + goto do_continue; > + } > + } > + } > + } > + alter_info->flags|= Alter_info::ALTER_DROP_CHECK_CONSTRAINT; > +do_continue:; > + } > + } > + > handle_if_exists_options(thd, table, alter_info); > > /* > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index b2b9cb4..cc6670b 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -1741,7 +1741,7 @@ bool my_yyoverflow(short **a, YYSTYPE **b, ulong *yystacksize); > IDENT_sys TEXT_STRING_sys TEXT_STRING_literal > NCHAR_STRING opt_component key_cache_name > sp_opt_label BIN_NUM label_ident TEXT_STRING_filesystem ident_or_empty > - opt_constraint constraint opt_ident ident_table_alias > + opt_constraint opt_constraint_no_id constraint opt_ident ident_table_alias > > %type <lex_str_ptr> > opt_table_alias > @@ -6123,6 +6123,11 @@ check_constraint: > } > ; > > +opt_constraint_no_id: > + /* Empty */ {} > + | CONSTRAINT {} > + ; > + > opt_constraint: > /* empty */ { $$= null_lex_str; } > | constraint { $$= $1; } > @@ -7653,7 +7658,7 @@ alter_list_item: > lex->alter_info.drop_list.push_back(ad, thd->mem_root); > lex->alter_info.flags|= Alter_info::DROP_FOREIGN_KEY; > } > - | DROP PRIMARY_SYM KEY_SYM > + | DROP opt_constraint_no_id PRIMARY_SYM KEY_SYM > { > LEX *lex=Lex; > Alter_drop *ad= (new (thd->mem_root) > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] 43296f4: MDEV-17599 ALTER TABLE DROP CONSTRAINT does not work for foreign keys.
by Sergei Golubchik 31 Jan '19

31 Jan '19

Hi, Alexey! On Jan 31, Alexey Botchkov wrote: > > can a CHECK constraint have the same name as an existing FK? > > Yes, it can. > So in this case the 'ALTER ... DROP CONSTRAINT c' drops the FOREIGN KEY. On > the second call this ALTER > removes the CHECK. > How do you like that? > I think it's ok as the very idea of same names for constraints seems sick. Let's handle it separately. Could you please report it as a bug? A table shouldn't have two constraints with the same name. > > what about UNIQUE constraint? > > The 'ALTER TABLE ADD CONSTRAINT UNIQUE unique_key' just adds UNIQUE > key. So i'm in doubt how we should handle the 'DROP CONSTRAINT > unique_key' statement. By dropping the unique key, why, is there another option? UNIQUE index is a constraint, not just an optimization helper, like a non-unique index. You already have a loop over all drop constraints, checking for unique shouldn't be a problem. It should drop PRIMARY KEY too, I suppose. We allow ADD CONSTRAINT PRIMARY KEY. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] 43296f4: MDEV-17599 ALTER TABLE DROP CONSTRAINT does not work for foreign keys.
by Sergei Golubchik 31 Jan '19

31 Jan '19

Hi, Alexey! Few questions first: * can a CHECK constraint have the same name as an existing FK? * what about UNIQUE constraint? it's also a constraint. The standard says that DROP CONSTRAINT should work for uniques, and ALTER already supports ADD CONSTRAINT UNIQUE (and ADD CONSTRAINT FOREIGN KEY) Otherwise it's fine, a couple of comments see below. On Jan 31, Alexey Botchkov wrote: > revision-id: 43296f4a90cc9281b4ddad498bfab4a4e43131b9 (mariadb-10.2.21-45-g43296f4) > parent(s): 97930df13c0e403940969ebb47398760b59f753c > committer: Alexey Botchkov > timestamp: 2019-01-31 16:23:12 +0400 > message: > > MDEV-17599 ALTER TABLE DROP CONSTRAINT does not work for foreign keys. > > The list of table constraints doesn't include foreign keys. > So we replace DROP CONSTRAINT with DROP FOREIGN KEY in this case. > > diff --git a/mysql-test/t/alter_table.test b/mysql-test/t/alter_table.test > index df077c8..659cae1 100644 > --- a/mysql-test/t/alter_table.test > +++ b/mysql-test/t/alter_table.test > @@ -2013,5 +2013,16 @@ DROP TABLE t1; > > > --echo # > +--echo # MDEV-17599 ALTER TABLE DROP CONSTRAINT does not work for foreign keys. > +--echo # > + > +CREATE TABLE t1(id INT PRIMARY KEY, c1 INT) ENGINE= INNODB; > +CREATE TABLE t2(id INT PRIMARY KEY, c1 INT, CONSTRAINT sid FOREIGN KEY (`c1`) REFERENCES t1 (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION) ENGINE= INNODB; > +SHOW CREATE TABLE t2; > +ALTER TABLE t2 DROP CONSTRAINT sid; please, add a test that drops FK and CHECK constraints in one statement. > +SHOW CREATE TABLE t2; > +DROP TABLE t2, t1; > + > +--echo # > --echo # End of 10.2 tests > --echo # > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index 1b426f8..c080e5a 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -8993,6 +8993,44 @@ bool mysql_alter_table(THD *thd,char *new_db, char *new_name, > > THD_STAGE_INFO(thd, stage_setup); > > + > + if (alter_info->flags & Alter_info::ALTER_DROP_CHECK_CONSTRAINT) > + { > + /* > + ALTER TABLE DROP CONSTRAINT > + should be replaced with ... DROP FOREIGN KEY > + if the constraint is the FOREIGN KEY one. > + */ > + > + List_iterator<Alter_drop> drop_it(alter_info->drop_list); > + Alter_drop *drop; > + alter_info->flags&= ~Alter_info::ALTER_DROP_CHECK_CONSTRAINT; > + > + while ((drop= drop_it++)) > + { > + if (drop->type == Alter_drop::CHECK_CONSTRAINT) > + { > + List <FOREIGN_KEY_INFO> fk_child_key_list; > + FOREIGN_KEY_INFO *f_key; > + table->file->get_foreign_key_list(thd, &fk_child_key_list); better do this outside of the loop > + List_iterator<FOREIGN_KEY_INFO> fk_key_it(fk_child_key_list); > + > + while ((f_key= fk_key_it++)) > + { > + if (my_strcasecmp(system_charset_info, f_key->foreign_id->str, > + drop->name) == 0) > + { > + drop->type= Alter_drop::FOREIGN_KEY; > + alter_info->flags|= Alter_info::DROP_FOREIGN_KEY; > + goto do_continue; > + } > + } > + } > + alter_info->flags|= Alter_info::ALTER_DROP_CHECK_CONSTRAINT; > +do_continue:; > + } > + } > + > handle_if_exists_options(thd, table, alter_info); Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] Review for follow up fix of MDEV-17323: Backport INFORMATION_SCHEMA.CHECK_CONSTRAINTS to 10.2
by Vicențiu Ciorbaru 31 Jan '19

31 Jan '19

Hi Anel! This is not a terribly bad patch, however you have forgotten the following things: You removed the misspelled is_check_constraint.test file, but forgot to remove the is_check_constraint.result file. Now we would be left with a dangling result file. Please be more careful next time. You modified the logic of is_check_constraints.test file from the one in 10.3 I suggest you use git cherry-pick next time when backporting a patch to get the correct changes. This also helps with merges later, as a change in 10.2 will be easily reconcileable in 10.3. Moving onwards with the patch, you will find other comments inline. > > > commit c94f53a34e0b6bf313ecb36a13d5d138150a8a8d > Author: Anel Husakovic <anel(a)mariadb.org> > Date: Thu Jan 24 03:06:56 2019 -0800 > > diff --git a/mysql-test/suite/funcs_1/t/is_check_constraint.test b/mysql-test/suite/funcs_1/t/is_check_constraint.test > deleted file mode 100644 > index 30a72d02b34..00000000000 > --- a/mysql-test/suite/funcs_1/t/is_check_constraint.test > +++ /dev/null > @@ -1,92 +0,0 @@ > ---source include/have_innodb.inc > ---source include/not_embedded.inc > ---echo # > ---echo # MDEV-17323: Backport INFORMATION_SCHEMA.CHECK_CONSTRAINTS to 10.2 > ---echo # > -CREATE user boo1; > -GRANT select,create,alter,drop on foo.* to boo1; > -SHOW GRANTS for boo1; > -CREATE user boo2; > -create database foo; > -# Connect with user boo1 > -CONNECT(con1,localhost, boo1,, foo); > - > -SET check_constraint_checks=1; > -CREATE TABLE t0 > -( > - t int, check (t>32) # table constraint > -) ENGINE=myisam; > ---sorted_result > -SELECT * from information_schema.check_constraints; > - > -ALTER TABLE t0 > -ADD CONSTRAINT CHK_t0_t CHECK(t<100); > ---sorted_result > -SELECT * from information_schema.check_constraints; > - > -ALTER TABLE t0 > -DROP CONSTRAINT CHK_t0_t; > ---sorted_result > -SELECT * from information_schema.check_constraints; > - > -ALTER TABLE t0 > -ADD CONSTRAINT CHECK(t<50); > ---sorted_result > -SELECT * from information_schema.check_constraints; > - > -CREATE TABLE t1 > -( t int CHECK(t>2), # field constraint > - tt int, > - CONSTRAINT CHECK (tt > 32), CONSTRAINT CHECK (tt <50),# autogenerated names table constraints > - CONSTRAINT CHK_tt CHECK(tt<100) # named table constraint > -) ENGINE=InnoDB; > - --sorted_result > -SELECT * from information_schema.check_constraints; > - > -ALTER TABLE t1 > -DROP CONSTRAINT CHK_tt; > ---sorted_result > -SELECT * from information_schema.check_constraints; > - > -CREATE TABLE t2 > -( > -name VARCHAR(30) CHECK(CHAR_LENGTH(name)>2), #field constraint > -start_date DATE, > -end_date DATE, > -CONSTRAINT CHK_dates CHECK(start_date IS NULL) #table constraint > -)ENGINE=Innodb; > - --sorted_result > -SELECT * from information_schema.check_constraints; > - > -ALTER TABLE t1 > -ADD CONSTRAINT CHK_new_ CHECK(t>tt); > ---sorted_result > -SELECT * from information_schema.check_constraints; > - > -# Create table with same field and table check constraint name > -CREATE TABLE t3 > -( > -a int, > -b int check (b>0), # field constraint named 'b' > -CONSTRAINT b check (b>10) # table constraint > -) ENGINE=InnoDB; > - --sorted_result > -SELECT * from information_schema.check_constraints; > - > -DISCONNECT con1; > -CONNECT(con2, localhost, boo2,, test); > - --sorted_result > -SELECT * from information_schema.check_constraints; > - > -DISCONNECT con2; > -CONNECT(con1, localhost, boo1,,foo); > -DROP TABLE t0; > -DROP TABLE t1; > -DROP TABLE t2; > -DROP TABLE t3; > -DROP DATABASE foo; > - > -DISCONNECT con1; > ---CONNECTION default > -DROP USER boo1; > -DROP USER boo2; This updated version of a test covers less functionality than the previous version, I don't like that. Please keep the old version as is, just rename it properly. Serg has already added the include to skip embedded tests in the most up-to-date 10.2. Please rebase on top of that version before you perform the rename. > > diff --git a/sql/sql_show.cc b/sql/sql_show.cc > index 39763451dfb..e951c69d009 100644 > --- a/sql/sql_show.cc > +++ b/sql/sql_show.cc > @@ -6474,33 +6474,35 @@ static int get_check_constraints_record(THD *thd, TABLE_LIST *tables, > LEX_STRING *table_name) > { > DBUG_ENTER("get_check_constraints_record"); > - if(res) > + if (res) Nice catch with the missing whitespace. > { > if (thd->is_error()) > push_warning(thd, Sql_condition::WARN_LEVEL_WARN, > thd->get_stmt_da()->sql_errno(), > thd->get_stmt_da()->message()); > - thd->clear_error(); > - DBUG_RETURN(0); > + thd->clear_error(); > + DBUG_RETURN(0); Great change, making sure the code is indented appropriately. > } > - if(!tables->view) > + else if (!tables->view) Again, nice catch with the missing whitespace, but there is no need for the else statement here. The DBUG_RETURN from above covers it. > { > - StringBuffer<MAX_FIELD_WIDTH> str(system_charset_info); > - for (uint i= 0; i < tables->table->s->table_check_constraints; i++) > + if (tables->table->s->table_check_constraints) There is no need for the if statement here. The for loop would already catch the condition and not execute any bit. There is also a subtle difference with initialising the StringBuffer object only once, instead of once every loop. "Smart" compilers might be good enough to realise that the object should be reused, but I don't think that is the general case. Lets make the compiler's job easier and do things properly. Keep the StringBuffer initialisation before the for loop and just reset it's length. If you think that str.length(0) is confusing (I agree that the method name reflects a getter, not a setter), put a comment after it saying something like: "Clear the string." Somehow I vaguely remember we discussed this before, but somehow the wrong version ended up in 10.3. We will change that too in a subsequent patch. Also, ouch, I missed during the review that you switched the columns around compared to 10.3. The current version in 10.2 seems like the correct one for me. Which one is the correct version? I remember we discussed this but I can't find the logs. Furthermore, when we do such changes, please explain *everything you modify* in the commit message, don't make me hunt for differences. > > { > - Virtual_column_info *check= tables->table->check_constraints[i]; > - table->field[0]->store(STRING_WITH_LEN("def"), system_charset_info); > - table->field[3]->store(check->name.str, check->name.length, > - system_charset_info); > - str.length(0); > - check->print(&str); > - table->field[4]->store(str.ptr(), str.length(), system_charset_info); > - if (schema_table_store_record(thd, table)) > - DBUG_RETURN(1); > + for (uint i= 0; i < tables->table->s->table_check_constraints; i++) > + { > + StringBuffer<MAX_FIELD_WIDTH> str(system_charset_info); > + Virtual_column_info *check= tables->table->check_constraints[i]; > + restore_record(table, s->default_values); > + table->field[0]->store(STRING_WITH_LEN("def"), system_charset_info); > + table->field[1]->store(db_name->str, db_name->length, system_charset_info); > + table->field[2]->store(check->name.str, check->name.length, system_charset_info); > + table->field[3]->store(table_name->str, table_name->length, system_charset_info); > + check->print(&str); > + table->field[4]->store(str.ptr(), str.length(), system_charset_info); > + schema_table_store_record(thd, table); > + } > } > } > - > - DBUG_RETURN(0); > + DBUG_RETURN(res); > } > > static int get_schema_constraints_record(THD *thd, TABLE_LIST *tables, > @@ -9342,8 +9346,8 @@ ST_SCHEMA_TABLE schema_tables[]= > fill_schema_applicable_roles, 0, 0, -1, -1, 0, 0}, > {"CHARACTER_SETS", charsets_fields_info, 0, > fill_schema_charsets, make_character_sets_old_format, 0, -1, -1, 0, 0}, > - {"CHECK_CONSTRAINTS", check_constraints_fields_info, 0, get_all_tables, 0, > - get_check_constraints_record, 1, 2, 0, OPTIMIZE_I_S_TABLE|OPEN_TABLE_ONLY}, > + {"CHECK_CONSTRAINTS", check_constraints_fields_info, 0, > + get_all_tables, 0, get_check_constraints_record, 1, 2, 0, OPTIMIZE_I_S_TABLE|OPEN_TABLE_ONLY}, I don't think this particular change was necessary. > {"COLLATIONS", collation_fields_info, 0, > fill_schema_collation, make_old_format, 0, -1, -1, 0, 0}, > {"COLLATION_CHARACTER_SET_APPLICABILITY", coll_charset_app_fields_info, Vicențiu

2 1

Re: [Maria-developers] ce4f8955d8d: MDEV-17082 Application period tables: CREATE
by Sergei Golubchik 30 Jan '19

30 Jan '19

Hi, Nikita! On Jan 18, Nikita Malyavin wrote: > > > diff --git a/sql/table.cc b/sql/table.cc > > > index ab1a04ccae4..87fa1ba4d61 100644 > > > --- a/sql/table.cc > > > +++ b/sql/table.cc > > > @@ -1228,6 +1228,14 @@ static const Type_handler *old_frm_type_handler(uint pack_flag, > > > } > > > > > > > > > +bool TABLE_SHARE::init_period_from_extra2(period_info_t &period, > > > + const uchar *data) > > > +{ > > > + period.start_fieldno= uint2korr(data); > > > + period.end_fieldno= uint2korr(data + sizeof(uint16)); > > > > not "sizeof(uint16)" but 2. I don't think it's guaranteed that > > sizeof(uint16) == 2, but as you do uint2korr, you only read 2 bytes, > > so data + 2 is always correct. > > > Yes, it's correct, and I've fixed it, but it wasn't about coorrectness > -- but readability. > Now what 4, or what is 8 -- is the question that the reader could > сonsider. This numbers look magical. > With using sizeof(uint16) the one could at least understand, that this > is about several double-byte words read. In a similar case I used something like #define SIZE 2 #define uintSIZEkorr(X) uint2korr(X) This way there are no magically-looking numbers. You can even write `uint ## SIZE ## korr`, if you're feeling adventurous. > The another problem is that sizeof(uint16) is already used in system > versioning, few lines below, so the resulting code will look patchy > here. > > > > + my_error(ER_PERIOD_FIELD_WRONG_ATTRIBUTES, MYF(0), > > > + f->field_name.str, "VIRTUAL or GENERATED"); > > > > just use "GENERATED", no English in the error message parameters, please. > > > VIRTUAL/GENERATED? What'd You say? You cannot have parts of an English phrase as a parameter. And "VIRTUAL" is just a subset of "GENERATED" anyway. So, just write my_error(ER_PERIOD_FIELD_WRONG_ATTRIBUTES, MYF(0), f->field_name.str, "GENERATED"); or, better, my_error(ER_PERIOD_FIELD_WRONG_ATTRIBUTES, MYF(0), f->field_name.str, "GENERATED ALWAYS AS"); > > > + else if (f->flags & VERS_SYSTEM_FIELD) > > > + { > > > + my_error(ER_PERIOD_FIELD_WRONG_ATTRIBUTES, MYF(0), f->field_name.str, > > > + f->flags & VERS_SYS_START_FLAG ? "ROW START" : "ROW END"); > > > > No need to have a special case for that. Use the same error as for vcol_info: > > > > if (f->vcol_info || f->flags & VERS_SYSTEM_FIELD) > > { > > my_error(ER_PERIOD_FIELD_WRONG_ATTRIBUTES, MYF(0), > > f->field_name.str, "GENERATED ALWAYS AS"); > > > Looks not very much user-friendly I think. Let's leave the verbosity as it was No, it's illogical. You have columns, like a TIMESTAMP(6) GENERATED ALWAYS AS (NOW() + INTERVAL 1 DAY), b TIMESTAMP(6) GENERATED ALWAYS AS ROW START for the first column the error is Period field %`s cannot be GENERATED for the second column the error is Period field %`s cannot be ROW START Why? They're both GENERATED. You don't say Period field %`s cannot be NOW() + INTERVAL 1 DAY See? So just say in both cases that the period cannot be "GENERATED ALWAYS AS". This is quite sufficient. > > > +ER_PERIOD_TYPES_MISMATCH > > > + eng "PERIOD FOR %`s fields types mismatch" > > > + > > > +ER_PERIOD_MAX_COUNT_EXCEEDED > > > > better say ER_MORE_THAN_ONE_PERIOD or something... > > "max count exceeded" suggests there is some not-trivial > > (and possibly variable) max count > > > > Or ER_PERIOD_ALREADY_DEFINED or ER_DUPLICATE_PERIOD etc. > > "already defined" looks like a good one. > > > > And I've already commented (in another patch) on using English > > words as arguments - it will look like a bug when the error message is > > translated > > > I like ER_MORE_THAN_ONE_PERIOD more. IMO application-time periods > designed in the way to support a possibility of several periods > definition, so i like to treat it as "N periods max supported, but now > N=1" The standard clearly says there can be at most one application-time period. So "Application-time period already defined" is just as fine. When the new standard will allow for many application-time periods, we can rewrite the error message. But don't hold your breath. And, again, you cannot have "application" as a parameter. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 3

Re: [Maria-developers] 81768c1: MDEV-5313 Improving audit plugin API.
by Sergei Golubchik 29 Jan '19

29 Jan '19

Hi, Alexey! On Jan 20, Alexey Botchkov wrote: > +typedef struct st_mysql MYSQL; > +typedef struct st_mysql_res MYSQL_RES; > +typedef char **MYSQL_ROW; > + > +extern struct sql_service_st { > + MYSQL *(*sqls_init)(MYSQL *mysql); > + void (*sqls_close)(MYSQL *mysql); > + int (*sqls_real_query)(MYSQL *mysql, const char *q, > + unsigned long length); > + unsigned long (*sqls_affected_rows)(MYSQL *mysql); > + uint (*sqls_errno)(MYSQL *mysql); > + const char *(*sqls_error)(MYSQL *mysql); > + MYSQL_RES *(*sqls_store_result)(MYSQL *mysql); > + void (*sqls_free_result)(MYSQL_RES *result); > + unsigned long (*sqls_num_rows)(MYSQL_RES *res); > + unsigned int (*sqls_num_fields)(MYSQL_RES *res); > + MYSQL_ROW (*sqls_fetch_row)(MYSQL_RES *result); > + unsigned long * (*sqls_fetch_lengths)(MYSQL_RES *result); > +} *sql_service; I'd say, if you're emulating client C API, you don't need all these sqls* functions. You basically only need one function, let's call it mysql_internal_connect() which is a variant of mysql_real_connect() for running SQL internally in the server. And it'll do mysql->methods= &internal_sql_methods; which will be your callback table, you won't need a second one. And after that mysql_* API functions will work normally. At least those that you decide to support. Alternatively you don't even need mysql_internal_connect(), you can integrate it into CLI_MYSQL_REAL_CONNECT (just like you've done with libmysqld.c:mysql_real_connect() calling cli_mysql_real_connect()). Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] 322301e2434: MDEV-16849 Extending indexed VARCHAR column should be instantaneous
by Sergei Golubchik 29 Jan '19

29 Jan '19

Hi, Thirunarayanan! On Jan 29, Thirunarayanan Balathandayuthapani wrote: > > > +ALTER TABLE t1 MODIFY f2 VARCHAR(50), DROP INDEX idx1; > > > +CALL get_table_id("test/t1", @tbl1_id); > > > +SELECT @tbl1_id = @tbl_id; > > > +@tbl1_id = @tbl_id > > > +0 > > > +SELECT @idx1_id = @idx_id; > > > +@idx1_id = @idx_id > > > +0 > > > > Why `DROP INDEX idx1` ? > > > > As far as I remember, the point was "make VARCHAR shorter, but still > > longer than a prefix". In this case there is no need to rebuild the > > index, why was it rebuilt? > > > If we shorten the column length then whole table must be rebuild. So there > is no need for index comparison over there. Oh, I see. Ok to push, then. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] fd18ed07d65: MDEV-16973 Application period tables: DELETE
by Sergei Golubchik 28 Jan '19

28 Jan '19

Hi, Nikita! Thanks! Please, see review comments below On Jan 03, Nikita Malyavin wrote: > revision-id: fd18ed07d65 (versioning-1.0.7-3-gfd18ed07d65) > parent(s): a643ca815af > author: Nikita Malyavin <nikitamalyavin(a)gmail.com> > committer: Nikita Malyavin <nikitamalyavin(a)gmail.com> > timestamp: 2018-12-03 13:19:18 +1000 > message: > > MDEV-16973 Application period tables: DELETE > > * inject portion of time updates into mysql_delete main loop > * triggered case emits delete+insert, no updates > * PORTION OF `SYSTEM_TIME` is forbidden > * `DELETE HISTORY .. FOR PORTION OF ...` is forbidden as well > > diff --git a/mysql-test/suite/period/r/delete.result b/mysql-test/suite/period/r/delete.result > new file mode 100644 > index 00000000000..30c9220d6f9 > --- /dev/null > +++ b/mysql-test/suite/period/r/delete.result > @@ -0,0 +1,216 @@ > +create or replace table t (id int, s date, e date, period for apptime(s,e)); > +insert into t values(1, '1999-01-01', '2018-12-12'); > +insert into t values(1, '1999-01-01', '2017-01-01'); > +insert into t values(1, '2017-01-01', '2019-01-01'); > +insert into t values(2, '1998-01-01', '2018-12-12'); > +insert into t values(3, '1997-01-01', '2015-01-01'); > +insert into t values(4, '2016-01-01', '2020-01-01'); I'd insert also, say, (5, '2010-01-01', '2015-01-01'). > +create or replace table t1 (id int, s date, e date, period for apptime(s,e)); > +insert t1 select * from t; > +create or replace table t2 (id int, s date, e date, period for apptime(s,e)); > +insert t2 select * from t; > +delete from t for portion of apptime from '2000-01-01' to '2018-01-01'; > +delete from t1 for portion of APPTIME from '2000-01-01' to '2018-01-01'; > +select * from t; try to avoid unsorted selects, the row order is undefined in these cases use --sorted_results command before the select or the order by. --sorted_results is generally better, because it's done on the client, so it doesn't change the execution plan on the server. > +id s e > +1 1999-01-01 2000-01-01 > +1 1999-01-01 2000-01-01 > +1 2018-01-01 2019-01-01 > +2 1998-01-01 2000-01-01 > +3 1997-01-01 2000-01-01 > +4 2018-01-01 2020-01-01 > +1 2018-01-01 2018-12-12 > +2 2018-01-01 2018-12-12 > +select * from t1 order by id, s, e; > +id s e > +1 1999-01-01 2000-01-01 > +1 1999-01-01 2000-01-01 > +1 2018-01-01 2018-12-12 > +1 2018-01-01 2019-01-01 > +2 1998-01-01 2000-01-01 > +2 2018-01-01 2018-12-12 > +3 1997-01-01 2000-01-01 > +4 2018-01-01 2020-01-01 > +select * from log_tbl; > +log > +>DEL: 1, 1999-01-01, 2018-12-12 > +<DEL: 1, 1999-01-01, 2018-12-12 > +>INS: 1, 1999-01-01, 2000-01-01 > +<INS: 1, 1999-01-01, 2000-01-01 > +>INS: 1, 2018-01-01, 2018-12-12 > +<INS: 1, 2018-01-01, 2018-12-12 > +>DEL: 1, 1999-01-01, 2017-01-01 > +<DEL: 1, 1999-01-01, 2017-01-01 > +>INS: 1, 1999-01-01, 2000-01-01 > +<INS: 1, 1999-01-01, 2000-01-01 > +>DEL: 1, 2017-01-01, 2019-01-01 > +<DEL: 1, 2017-01-01, 2019-01-01 > +>INS: 1, 2018-01-01, 2019-01-01 > +<INS: 1, 2018-01-01, 2019-01-01 > +>DEL: 2, 1998-01-01, 2018-12-12 > +<DEL: 2, 1998-01-01, 2018-12-12 > +>INS: 2, 1998-01-01, 2000-01-01 > +<INS: 2, 1998-01-01, 2000-01-01 > +>INS: 2, 2018-01-01, 2018-12-12 > +<INS: 2, 2018-01-01, 2018-12-12 > +>DEL: 3, 1997-01-01, 2015-01-01 > +<DEL: 3, 1997-01-01, 2015-01-01 > +>INS: 3, 1997-01-01, 2000-01-01 > +<INS: 3, 1997-01-01, 2000-01-01 > +>DEL: 4, 2016-01-01, 2020-01-01 > +<DEL: 4, 2016-01-01, 2020-01-01 > +>INS: 4, 2018-01-01, 2020-01-01 > +<INS: 4, 2018-01-01, 2020-01-01 > +# INSERT trigger only also works > +drop trigger tr1del_t2; > +drop trigger tr2del_t2; > +delete from t2 for portion of APPTIME from '2000-01-01' to '2018-01-01'; > +select * from log_tbl; > +log > +>INS: 1, 1999-01-01, 2000-01-01 > +<INS: 1, 1999-01-01, 2000-01-01 > +>INS: 1, 2018-01-01, 2018-12-12 > +<INS: 1, 2018-01-01, 2018-12-12 > +>INS: 1, 1999-01-01, 2000-01-01 > +<INS: 1, 1999-01-01, 2000-01-01 > +>INS: 1, 2018-01-01, 2019-01-01 > +<INS: 1, 2018-01-01, 2019-01-01 > +>INS: 2, 1998-01-01, 2000-01-01 > +<INS: 2, 1998-01-01, 2000-01-01 > +>INS: 2, 2018-01-01, 2018-12-12 > +<INS: 2, 2018-01-01, 2018-12-12 > +>INS: 3, 1997-01-01, 2000-01-01 > +<INS: 3, 1997-01-01, 2000-01-01 > +>INS: 4, 2018-01-01, 2020-01-01 > +<INS: 4, 2018-01-01, 2020-01-01 > +delete history from t2 for portion of apptime from '2000-01-01' to '2018-01-01'; > +ERROR 42000: You have an error in your SQL syntax; check the manual that corresponds to your MariaDB server version for the right syntax to use near '' at line 1 The error is good, ER_PARSE_ERROR is very appropriate here. But it should say "right syntax to use near 'for portion of' at line 1" > +delete from t for portion of othertime from '2000-01-01' to '2018-01-01'; > +ERROR HY000: period `othertime` is not found in table > +delete from t for portion of system_time from '2000-01-01' to '2018-01-01'; > +ERROR HY000: period SYSTEM_TIME is not allowed here could be just ER_PARSE_ERROR too, I suppose > +create or replace table t (id int, str text, s date, e date, > +period for apptime(s,e)); > +insert into t values(1, 'data', '1999-01-01', '2018-12-12'); > +insert into t values(1, 'other data', '1999-01-01', '2018-12-12'); > +insert into t values(1, 'deleted', '2000-01-01', '2018-01-01'); > +delete from t for portion of apptime from '2000-01-01' to '2018-01-01'; > +show warnings; > +Level Code Message > +select * from t; --sorted_results or order by > +id str s e > +1 data 1999-01-01 2000-01-01 > +1 other data 1999-01-01 2000-01-01 > +1 data 2018-01-01 2018-12-12 > +1 other data 2018-01-01 2018-12-12 > +drop table t1; > +# SQL17 Case 15.7-8.b.i good! you forgot "part 2" and "general rules", but even without them it only takes a couple of seconds to find this quote. > +# If the column descriptor that corresponds to the i-th field of BR > +# describes an identity column, a generated column, a system-time period > +# start column, or a system-time period end column, then let V i be > +# DEFAULT. > +# auto_inrement field is updated "increment" > +create or replace table t (id int primary key auto_increment, s date, e date, > +period for apptime(s, e)); > +insert into t values (default, '1999-01-01', '2018-12-12'); please, add a select * here. > +delete from t for portion of apptime from '2000-01-01' to '2018-01-01'; > +select * from t; > +id s e > +2 1999-01-01 2000-01-01 > +3 2018-01-01 2018-12-12 > +truncate t; > +# same for trigger case > +insert into t values (default, '1999-01-01', '2018-12-12'); > +delete from t for portion of apptime from '2000-01-01' to '2018-01-01'; > +select * from t; > +id s e > +2 1999-01-01 2000-01-01 > +3 2018-01-01 2018-12-12 > +select * from log_tbl; > +log > +>DEL: 1999-01-01, 2018-12-12 > +<DEL: 1999-01-01, 2018-12-12 > +>INS: 1999-01-01, 2000-01-01 > +<INS: 1999-01-01, 2000-01-01 > +>INS: 2018-01-01, 2018-12-12 > +<INS: 2018-01-01, 2018-12-12 > +# generated columns are updated > +create or replace table t (s date, e date, > +xs date as (s) stored, xe date as (e) stored, > +period for apptime(s, e)); > +insert into t values('1999-01-01', '2018-12-12', default, default); > +select * from t; > +s e xs xe > +1999-01-01 2018-12-12 1999-01-01 2018-12-12 > +delete from t for portion of apptime from '2000-01-01' to '2018-01-01'; > +select * from t; > +s e xs xe > +1999-01-01 2000-01-01 1999-01-01 2000-01-01 > +2018-01-01 2018-12-12 2018-01-01 2018-12-12 > +truncate t; > +# same for trigger case > +insert into t values('1999-01-01', '2018-12-12', default, default); > +delete from t for portion of apptime from '2000-01-01' to '2018-01-01'; > +select * from t; > +s e xs xe > +1999-01-01 2000-01-01 1999-01-01 2000-01-01 > +2018-01-01 2018-12-12 2018-01-01 2018-12-12 > +select * from log_tbl; > +log > +>DEL: 1999-01-01, 2018-12-12 > +<DEL: 1999-01-01, 2018-12-12 > +>INS: 1999-01-01, 2000-01-01 > +<INS: 1999-01-01, 2000-01-01 > +>INS: 2018-01-01, 2018-12-12 > +<INS: 2018-01-01, 2018-12-12 > +# system_time columns are updated > +create or replace table t ( > +s date, e date, > +row_start SYS_TYPE as row start invisible, > +row_end SYS_TYPE as row end invisible, > +period for apptime(s, e), > +period for system_time (row_start, row_end)) with system versioning; > +insert into t values('1999-01-01', '2018-12-12'), > +('1999-01-01', '1999-12-12'); > +select row_start into @ins_time from t limit 1; > +select * from t order by s, e; > +s e > +1999-01-01 1999-12-12 > +1999-01-01 2018-12-12 > +delete from t for portion of apptime from '2000-01-01' to '2018-01-01'; > +select *, if(row_start = @ins_time, "OLD", "NEW"), check_row(row_start, row_end) > +from t for system_time all > +order by s, e, row_start; > +s e if(row_start = @ins_time, "OLD", "NEW") check_row(row_start, row_end) > +1999-01-01 1999-12-12 OLD CURRENT ROW > +1999-01-01 2000-01-01 NEW CURRENT ROW > +1999-01-01 2018-12-12 OLD HISTORICAL ROW > +2018-01-01 2018-12-12 NEW CURRENT ROW > +# same for trigger case > +delete from t; > +delete history from t; > +insert into t values('1999-01-01', '2018-12-12'), > +('1999-01-01', '1999-12-12'); > +select row_start into @ins_time from t limit 1; > +select * from t order by s, e; > +s e > +1999-01-01 1999-12-12 > +1999-01-01 2018-12-12 > +delete from t for portion of apptime from '2000-01-01' to '2018-01-01'; > +select *, if(row_start = @ins_time, "OLD", "NEW"), check_row(row_start, row_end) > +from t for system_time all > +order by s, e, row_start; > +s e if(row_start = @ins_time, "OLD", "NEW") check_row(row_start, row_end) > +1999-01-01 1999-12-12 OLD CURRENT ROW > +1999-01-01 2000-01-01 NEW CURRENT ROW > +1999-01-01 2018-12-12 OLD HISTORICAL ROW > +2018-01-01 2018-12-12 NEW CURRENT ROW > +select * from log_tbl; > +log > +>DEL: 1999-01-01, 2018-12-12 > +<DEL: 1999-01-01, 2018-12-12 > +>INS: 1999-01-01, 2000-01-01 > +<INS: 1999-01-01, 2000-01-01 > +>INS: 2018-01-01, 2018-12-12 > +<INS: 2018-01-01, 2018-12-12 > +create or replace database test; > diff --git a/sql/lex.h b/sql/lex.h > index d336c273a18..883316ab988 100644 > --- a/sql/lex.h > +++ b/sql/lex.h > @@ -476,6 +476,7 @@ static SYMBOL symbols[] = { > { "POINT", SYM(POINT_SYM)}, > { "POLYGON", SYM(POLYGON)}, > { "PORT", SYM(PORT_SYM)}, > + { "PORTION", SYM(PORTION)}, PORTION_SYM, please > { "PRECEDES", SYM(PRECEDES_SYM)}, > { "PRECEDING", SYM(PRECEDING_SYM)}, > { "PRECISION", SYM(PRECISION)}, > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index 4f2c3fe7611..0c368b40836 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -1033,6 +1033,7 @@ bool my_yyoverflow(short **a, YYSTYPE **b, size_t *yystacksize); > %token PERCENT_RANK_SYM > %token PERCENTILE_CONT_SYM > %token PERCENTILE_DISC_SYM > +%token PORTION add /* SQL-2016-R */ comment and it seems to be 2016 standard, not 2017 standard, https://en.wikipedia.org/wiki/SQL please change it everywhere, it's too confusing otherwise > %token POSITION_SYM /* SQL-2003-N */ > %token PRECISION /* SQL-2003-R */ > %token PRIMARY_SYM /* SQL-2003-R */ > @@ -13437,23 +13449,26 @@ delete_part2: > opt_delete_options single_multi {} > | HISTORY_SYM delete_single_table opt_delete_system_time > { > + MYSQL_YYABORT_UNLESS(!Lex->last_table()->has_period()); > Lex->last_table()->vers_conditions= Lex->vers_conditions; > Lex->pop_select(); //main select > } > ; > > delete_single_table: > - FROM table_ident opt_use_partition > + FROM table_ident opt_for_portion_of_time_clause opt_use_partition better put opt_for_portion_of_time_clause after opt_use_partition, I think it's more logical that way. FOR PORTION OF is logical specification, while opt_use_partition is physical one, just like the table name, so it makes sense to keep them together > { > if (unlikely(!Select-> > add_table_to_list(thd, $2, NULL, TL_OPTION_UPDATING, > YYPS->m_lock_type, > YYPS->m_mdl_type, > NULL, > - $3))) > + $4))) > MYSQL_YYABORT; > YYPS->m_lock_type= TL_READ_DEFAULT; > YYPS->m_mdl_type= MDL_SHARED_READ; > + if ($3) > + Lex->last_table()->period_conditions= Lex->period_conditions; > } > ; > > diff --git a/sql/share/errmsg-utf8.txt b/sql/share/errmsg-utf8.txt > index ceb3a58a427..7ceff1a215e 100644 > --- a/sql/share/errmsg-utf8.txt > +++ b/sql/share/errmsg-utf8.txt > @@ -7945,3 +7945,9 @@ ER_PERIOD_FIELD_WRONG_ATTRIBUTES > > ER_PERIOD_FIELD_HOLDS_MORE_THAN_ONE_PERIOD > eng "Field %`s is specified for more than one period" > + > +ER_PERIOD_NOT_FOUND > + eng "period %`s is not found in table" first letter capitalized, please > + > +ER_PERIOD_SYSTEM_TIME_NOT_ALLOWED > + eng "period SYSTEM_TIME is not allowed here" same here, or, better, use ER_PARSE_ERROR > diff --git a/sql/sql_delete.cc b/sql/sql_delete.cc > index 3ae113a4595..e76eb729536 100644 > --- a/sql/sql_delete.cc > +++ b/sql/sql_delete.cc > @@ -287,7 +287,8 @@ bool mysql_delete(THD *thd, TABLE_LIST *table_list, COND *conds, > bool return_error= 0; > ha_rows deleted= 0; > bool reverse= FALSE; > - bool has_triggers; > + bool has_triggers= false; > + bool has_period_triggers= false; I don't see why you need has_triggers and has_period_triggers, I find it confusing. just make `has_triggers=true` if there's `FOR PORTION OF` and there are insert triggers. In other words, has_triggers should be true if there are any triggers that this statement might need to fire. > ORDER *order= (ORDER *) ((order_list && order_list->elements) ? > order_list->first : NULL); > SELECT_LEX *select_lex= thd->lex->first_select_lex(); > @@ -334,6 +338,21 @@ bool mysql_delete(THD *thd, TABLE_LIST *table_list, COND *conds, > table_list->on_expr= NULL; > } > } > + if (table_list->has_period()) > + { > + if (table_list->is_view_or_derived()) > + { > + my_error(ER_IT_IS_A_VIEW, MYF(0), table_list->table_name.str); > + DBUG_RETURN(true); > + } > + > + int err= select_lex->period_setup_conds(thd, table_list, conds); > + if (err) > + DBUG_RETURN(true); > + > + conds= table_list->on_expr; > + table_list->on_expr= NULL; Why did you define SELECT_LEX::period_setup_conds to put the condition in TABLE_LIST::on_expr only to hack it up from there? Just make it to return the new condition or NULL in case of an error. > + } > > if (mysql_handle_list_of_derived(thd->lex, table_list, DT_MERGE_FOR_INSERT)) > DBUG_RETURN(TRUE); > @@ -727,6 +752,9 @@ bool mysql_delete(THD *thd, TABLE_LIST *table_list, COND *conds, > delete_record= true; > } > > + portion_of_time_through_update= !has_period_triggers > + && !table->versioned(VERS_TIMESTAMP); I suspect that with VERS_TRX_ID you'd also need to use delete/insert. Add a test, please. > + > THD_STAGE_INFO(thd, stage_updating); > while (likely(!(error=info.read_record())) && likely(!thd->killed) && > likely(!thd->is_error())) > @@ -969,7 +1013,8 @@ int mysql_prepare_delete(THD *thd, TABLE_LIST *table_list, > DBUG_RETURN(TRUE); > } > > - if (unique_table(thd, table_list, table_list->next_global, 0)) > + if (table_list->has_period() Really? Why? > + || unique_table(thd, table_list, table_list->next_global, 0)) > *delete_while_scanning= false; > > if (select_lex->inner_refs_list.elements && > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > index 5f76f36983b..de8ed2f14b3 100644 > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -8302,6 +8302,7 @@ TABLE_LIST *st_select_lex::add_table_to_list(THD *thd, > ptr->ignore_leaves= MY_TEST(table_options & TL_OPTION_IGNORE_LEAVES); > ptr->sequence= MY_TEST(table_options & TL_OPTION_SEQUENCE); > ptr->derived= table->sel; > + ptr->vers_conditions.name= "SYSTEM_TIME"; why? > if (!ptr->derived && is_infoschema_db(&ptr->db)) > { > ST_SCHEMA_TABLE *schema_table; > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index 19cea4d6a15..ea1c7834f57 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -717,12 +717,152 @@ void vers_select_conds_t::print(String *str, enum_query_type query_type) const > } > } > > +void period_update_condition(THD *thd, TABLE_LIST *table, SELECT_LEX *select, static > + vers_select_conds_t &conds, bool timestamp) please, either use pointers or _const_ references. in other words, `conds` here should be a pointer > +{ > + DBUG_ASSERT(table); > + DBUG_ASSERT(table->table); > +#define newx new (thd->mem_root) > + TABLE_SHARE *share= table->table->s; > + const TABLE_SHARE::period_info_t *period= conds.period; > + > + const LEX_CSTRING &fstart= period->start_field(share)->field_name; > + const LEX_CSTRING &fend= period->end_field(share)->field_name; > + > + conds.field_start= newx Item_field(thd, &select->context, > + table->db.str, table->alias.str, > + thd->make_clex_string(fstart)); > + conds.field_end= newx Item_field(thd, &select->context, > + table->db.str, table->alias.str, > + thd->make_clex_string(fend)); > + > + Item *cond1= NULL, *cond2= NULL, *cond3= NULL, *curr= NULL; > + if (timestamp) > + { > + MYSQL_TIME max_time; > + switch (conds.type) > + { > + case SYSTEM_TIME_UNSPECIFIED: > + thd->variables.time_zone->gmt_sec_to_TIME(&max_time, TIMESTAMP_MAX_VALUE); > + max_time.second_part= TIME_MAX_SECOND_PART; > + curr= newx Item_datetime_literal(thd, &max_time, TIME_SECOND_PART_DIGITS); > + cond1= newx Item_func_eq(thd, conds.field_end, curr); > + break; > + case SYSTEM_TIME_AS_OF: > + cond1= newx Item_func_le(thd, conds.field_start, conds.start.item); > + cond2= newx Item_func_gt(thd, conds.field_end, conds.start.item); > + break; > + case SYSTEM_TIME_FROM_TO: > + cond1= newx Item_func_lt(thd, conds.field_start, conds.end.item); > + cond2= newx Item_func_gt(thd, conds.field_end, conds.start.item); > + cond3= newx Item_func_lt(thd, conds.start.item, conds.end.item); > + break; > + case SYSTEM_TIME_BETWEEN: > + cond1= newx Item_func_le(thd, conds.field_start, conds.end.item); > + cond2= newx Item_func_gt(thd, conds.field_end, conds.start.item); > + cond3= newx Item_func_le(thd, conds.start.item, conds.end.item); > + break; > + case SYSTEM_TIME_BEFORE: > + cond1= newx Item_func_lt(thd, conds.field_end, conds.start.item); > + break; > + default: > + DBUG_ASSERT(0); > + } > + } > + else > + { > + DBUG_ASSERT(table->table->s && table->table->s->db_plugin); > + > + Item *trx_id0= conds.start.item; > + Item *trx_id1= conds.end.item; > + if (conds.start.item && conds.start.unit == VERS_TIMESTAMP) > + { > + bool backwards= conds.type != SYSTEM_TIME_AS_OF; > + trx_id0= newx Item_func_trt_id(thd, conds.start.item, > + TR_table::FLD_TRX_ID, backwards); > + } > + if (conds.end.item && conds.end.unit == VERS_TIMESTAMP) > + { > + trx_id1= newx Item_func_trt_id(thd, conds.end.item, > + TR_table::FLD_TRX_ID, false); > + } > + > + switch (conds.type) > + { > + case SYSTEM_TIME_UNSPECIFIED: > + curr= newx Item_int(thd, ULONGLONG_MAX); > + cond1= newx Item_func_eq(thd, conds.field_end, curr); add here DBUG_ASSERT(!conds.start.item); DBUG_ASSERT(!conds.end.item); > + break; > + case SYSTEM_TIME_AS_OF: > + cond1= newx Item_func_trt_trx_sees_eq(thd, trx_id0, conds.field_start); > + cond2= newx Item_func_trt_trx_sees(thd, conds.field_end, trx_id0); DBUG_ASSERT(!conds.end.item); > + break; > + case SYSTEM_TIME_FROM_TO: > + cond1= newx Item_func_trt_trx_sees(thd, trx_id1, conds.field_start); > + cond3= newx Item_func_lt(thd, conds.start.item, conds.end.item); > + break; hmm, you don't use trx_id0 here at all. did you forget cond2= newx Item_func_trt_trx_sees_eq(thd, conds.field_end, trx_id0); > + case SYSTEM_TIME_BETWEEN: > + cond1= newx Item_func_trt_trx_sees_eq(thd, trx_id1, conds.field_start); > + cond2= newx Item_func_trt_trx_sees_eq(thd, conds.field_end, trx_id0); > + cond3= newx Item_func_le(thd, conds.start.item, conds.end.item); > + break; > + case SYSTEM_TIME_BEFORE: > + cond1= newx Item_func_trt_trx_sees(thd, trx_id0, conds.field_end); > + break; > + default: > + DBUG_ASSERT(0); > + } > + } > + > + if (cond1) > + { > + cond1= and_items(thd, cond2, cond1); > + cond1= and_items(thd, cond3, cond1); > + table->on_expr= and_items(thd, table->on_expr, cond1); > + } > +#undef newx > +} > + > +int SELECT_LEX::period_setup_conds(THD *thd, TABLE_LIST *tables, Item *where) > +{ > + DBUG_ENTER("SELECT_LEX::period_setup_conds"); > + > + if (!thd->stmt_arena->is_conventional() && > + !thd->stmt_arena->is_stmt_prepare_or_first_sp_execute()) > + { > + // statement is already prepared > + DBUG_RETURN(0); > + } > + > + if (thd->lex->is_view_context_analysis()) > + DBUG_RETURN(0); I see that you've copied these two if's from SELECT_LEX::vers_setup_conds. I suspect that the first if() is not quite correct here, and I plan to take a closer look at it. so, it'd be great if you could come up with some way to avoid copying these two if()'s. So that there will be only one place to fix. May be, extract them into a function or something, I don't know. > + > + for (TABLE_LIST *table= tables; table; table= table->next_local) 1. can here be more than one table? you only allow FOR PORTION OF in delete_single_table parser rule. 2. if no, add an assert here that there's only one table in the list 3. and please add a test for DELETE FROM view FOR PORTION OF > + { > + if (!table->table) > + continue; > + vers_select_conds_t &conds= table->period_conditions; > + if (0 == strcasecmp(conds.name.str, "SYSTEM_TIME")) why not to check it in the parser? > + { > + my_error(ER_PERIOD_SYSTEM_TIME_NOT_ALLOWED, MYF(0)); > + DBUG_RETURN(-1); > + } > + if (!table->table->s->period.name.streq(conds.name)) > + { > + my_error(ER_PERIOD_NOT_FOUND, MYF(0), conds.name.str); > + DBUG_RETURN(-1); > + } > + > + conds.period= &table->table->s->period; > + period_update_condition(thd, table, this, conds, true); > + table->on_expr= and_items(thd, where, table->on_expr); > + } > + DBUG_RETURN(0); > +} > + > int SELECT_LEX::vers_setup_conds(THD *thd, TABLE_LIST *tables) > { > DBUG_ENTER("SELECT_LEX::vers_setup_cond"); > -#define newx new (thd->mem_root) > - > - TABLE_LIST *table; > > if (!thd->stmt_arena->is_conventional() && > !thd->stmt_arena->is_stmt_prepare_or_first_sp_execute()) > diff --git a/sql/table.cc b/sql/table.cc > index 87fa1ba4d61..8f1c94dd633 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -6426,9 +6426,14 @@ void TABLE::mark_columns_needed_for_delete() > > if (s->versioned) > { > - bitmap_set_bit(read_set, s->vers.start_field(s)->field_index); > - bitmap_set_bit(read_set, s->vers.end_field(s)->field_index); > - bitmap_set_bit(write_set, s->vers.end_field(s)->field_index); > + bitmap_set_bit(read_set, s->vers.start_fieldno); > + bitmap_set_bit(read_set, s->vers.end_fieldno); > + bitmap_set_bit(write_set, s->vers.end_fieldno); > + } > + > + if (with_period) > + { > + use_all_columns(); > } Nope, I think this is conceptually wrong. TABLE::mark_columns_needed_for_delete marks columns that are needed to *delete* a row. The fact that you might need to *insert* if there's a FOR PORTION OF, is irrelevant here, it's something the caller should bother about. > } > > @@ -7834,6 +7839,101 @@ int TABLE::update_default_fields(bool update_command, bool ignore_errors) > DBUG_RETURN(res); > } > > +static int table_update_generated_fields(TABLE *table) > +{ > + int res= 0; > + if (table->found_next_number_field) > + { > + table->next_number_field= table->found_next_number_field; > + res= table->found_next_number_field->set_default(); > + table->file->update_auto_increment(); > + } > + > + if (table->default_field) > + res= table->update_default_fields(0, false); why is that? > + > + if (likely(!res) && table->vfield) > + res= table->update_virtual_fields(table->file, VCOL_UPDATE_FOR_WRITE); > + if (likely(!res) && table->versioned()) > + table->vers_update_fields(); I believe you need to verify CHECK constraints too. and, please, add a test where they fail here. > + return res; > +} > + > +static int period_make_insert(TABLE *table, Item *src, Field *dst) > +{ > + THD *thd= table->in_use; > + > + store_record(table, record[1]); > + int res= src->save_in_field(dst, true); > + > + if (likely(!res)) > + table_update_generated_fields(table); perhaps, move store_record() and save_in_field() into table_update_generated_fields() ? less duplicated code. But the function will need to be renamed, like prepare_for_period_insert or something > + > + if (likely(!res) && table->triggers) > + res= table->triggers->process_triggers(thd, TRG_EVENT_INSERT, TRG_ACTION_BEFORE, true); > + > + if (likely(!res)) > + res = table->file->ha_write_row(table->record[0]); > + > + if (likely(!res) && table->triggers) > + res= table->triggers->process_triggers(thd, TRG_EVENT_INSERT, TRG_ACTION_AFTER, true); > + > + restore_record(table, record[1]); > + return res; > +} > + > +int TABLE::update_portion_of_time(THD *thd, vers_select_conds_t &period_conds, > + bool *inside_period) is it used for UPDATE too? or only for DELETE? > +{ > + bool lcond= period_conds.field_start->val_datetime_packed(thd) > + < period_conds.start.item->val_datetime_packed(thd); > + bool rcond= period_conds.field_end->val_datetime_packed(thd) > + > period_conds.end.item->val_datetime_packed(thd); > + > + *inside_period= !lcond && !rcond; > + if (*inside_period) > + return 0; > + > + int res= 0; > + Item *src= lcond ? period_conds.start.item : period_conds.end.item; > + uint dst_fieldno= lcond ? s->period.end_fieldno : s->period.start_fieldno; > + > + store_record(this, record[1]); > + if (likely(!res)) > + res= src->save_in_field(field[dst_fieldno], true); > + > + if (likely(!res)) > + res= table_update_generated_fields(this); > + may be, an assert that there're no delete/insert triggers in a table? > + if(likely(!res)) > + res= file->ha_update_row(record[1], record[0]); > + > + restore_record(this, record[1]); > + > + if (likely(!res) && lcond && rcond) > + res= period_make_insert(this, period_conds.end.item, > + field[s->period.start_fieldno]); > + > + return res; > +} > + > +int TABLE::insert_portion_of_time(THD *thd, vers_select_conds_t &period_conds) > +{ > + bool lcond= period_conds.field_start->val_datetime_packed(thd) > + < period_conds.start.item->val_datetime_packed(thd); > + bool rcond= period_conds.field_end->val_datetime_packed(thd) > + > period_conds.end.item->val_datetime_packed(thd); > + > + int res= 0; > + if (lcond) > + res= period_make_insert(this, period_conds.start.item, > + field[s->period.end_fieldno]); > + if (likely(!res) && rcond) > + res= period_make_insert(this, period_conds.end.item, > + field[s->period.start_fieldno]); > + > + return res; > +} > > void TABLE::vers_update_fields() > { Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 5

Re: [Maria-developers] 322301e2434: MDEV-16849 Extending indexed VARCHAR column should be instantaneous
by Sergei Golubchik 28 Jan '19

28 Jan '19

Hi, Thirunarayanan! On Jan 28, Thirunarayanan Balathandayuthapani wrote: > revision-id: 322301e2434 (mariadb-10.2.21-2-g322301e2434) > parent(s): fe4c65db1dc > author: Thirunarayanan Balathandayuthapani <thiru(a)mariadb.com> > committer: Thirunarayanan Balathandayuthapani <thiru(a)mariadb.com> > timestamp: 2019-01-01 11:01:03 +0530 > message: > > MDEV-16849 Extending indexed VARCHAR column should be instantaneous > > - Addressed Serg's review comments. > > --- > diff --git a/mysql-test/suite/innodb/r/alter_varchar_change.result b/mysql-test/suite/innodb/r/alter_varchar_change.result > index 5d2c6403aab..2f412aecbf1 100644 > --- a/mysql-test/suite/innodb/r/alter_varchar_change.result > +++ b/mysql-test/suite/innodb/r/alter_varchar_change.result > @@ -316,6 +316,137 @@ t1 CREATE TABLE `t1` ( > DROP TABLE t1; > CREATE TABLE t1(f1 INT NOT NULL, > f2 VARCHAR(100), > +INDEX idx(f2(10)), > +INDEX idx1(f1))ENGINE=InnoDB; > +CALL get_table_id("test/t1", @tbl_id); > +ALTER TABLE t1 MODIFY f2 VARCHAR(50), DROP INDEX idx1; > +CALL get_table_id("test/t1", @tbl1_id); > +SELECT @tbl1_id = @tbl_id; > +@tbl1_id = @tbl_id > +0 > +SELECT @idx1_id = @idx_id; > +@idx1_id = @idx_id > +0 Why `DROP INDEX idx1` ? As far as I remember, the point was "make VARCHAR shorter, but still longer than a prefix". In this case there is no need to rebuild the index, why was it rebuilt? > +SHOW CREATE TABLE t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `f1` int(11) NOT NULL, > + `f2` varchar(50) DEFAULT NULL, > + KEY `idx` (`f2`(10)) > +) ENGINE=InnoDB DEFAULT CHARSET=latin1 > +DROP TABLE t1; > +CREATE TABLE t1(f1 INT NOT NULL, > +f2 VARCHAR(100), > +INDEX idx(f2(10)), > +INDEX idx1(f1))ENGINE=InnoDB; > +CALL get_table_id("test/t1", @tbl_id); > +ALTER TABLE t1 MODIFY f2 VARCHAR(5), DROP INDEX idx1; > +CALL get_table_id("test/t1", @tbl1_id); > +SELECT @tbl1_id = @tbl_id; > +@tbl1_id = @tbl_id > +0 > +SELECT @idx1_id = @idx_id; > +@idx1_id = @idx_id > +0 Correct. "VARCHAR is changed to be shorter than a prefix" means that the index must be rebuilt. > +SHOW CREATE TABLE t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `f1` int(11) NOT NULL, > + `f2` varchar(5) DEFAULT NULL, > + KEY `idx` (`f2`) > +) ENGINE=InnoDB DEFAULT CHARSET=latin1 > +DROP TABLE t1; ... > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index 2864474a8e8..7eca33812b9 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -6629,6 +6629,7 @@ static bool fill_alter_inplace_info(THD *thd, > Go through keys and check if the original ones are compatible > with new table. > */ > + uint old_field_len= 0; > KEY *table_key; > KEY *table_key_end= table->key_info + table->s->keys; > KEY *new_key; > @@ -6698,8 +6699,16 @@ static bool fill_alter_inplace_info(THD *thd, > Key definition has changed if we are using a different field or > if the user key part length is different. > */ > - if (key_part->length <= new_part->length && > - old_field->pack_length() < new_field->pack_length && > + old_field_len= old_field->pack_length(); > + > + if (old_field->type() == MYSQL_TYPE_VARCHAR) > + { > + old_field_len= (old_field->pack_length() > + - ((Field_varstring*) old_field)->length_bytes); > + } > + > + if (key_part->length == old_field_len && > + key_part->length <= new_part->length && Why <= here? Doesn't it mean that if key_part->length == new_part->length then you'll still set ALTER_COLUMN_INDEX_LENGTH even if _nothing_ has changed? > (key_part->field->is_equal((Create_field*) new_field) > == IS_EQUAL_PACK_LENGTH)) > { > Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] 54411ba: MDEV-17148 DROP DATABASE throw "Directory not empty" after changed lower_case_table_names.
by Sergei Golubchik 28 Jan '19

28 Jan '19

Hi, Alexey! We've already discussed the solution, so ok to push. But it's a bit weird to have "mdev17148" in the std_data, could you rename it to mdev17148.frm? Or, even better, don't check in a binary file, but do instead create table t1 (a int); copy_file $datadir/test/t1.frm $datadir/db1/T1.frm; std_data is for files that cannot be easily created in the test. Like a .frm file from mysql-4.0. On Dec 24, Alexey Botchkov wrote: > revision-id: 54411ba5caee35d0a64e727c117c222b6eef8812 (mariadb-10.2.19-98-g54411ba) > parent(s): 975f4a1295f2b678c5ecea2cf4cfc69e177d88f5 > committer: Alexey Botchkov > timestamp: 2018-12-24 16:41:16 +0400 > message: > > MDEV-17148 DROP DATABASE throw "Directory not empty" after changed lower_case_table_names. > > No need to lowercase table names on case-sensitive file systems, as the > cache won't contain the 'lowercased' table anyway. And it prevents the > UPPERCASE.frm from being deleted. > > mysql-test/r/lowercase_table.result | 2 ++ > mysql-test/std_data/mdev17148 | Bin 0 -> 433 bytes > mysql-test/t/lowercase_table.test | 10 ++++++++++ > sql/sql_db.cc | 8 ++++++-- > 4 files changed, 18 insertions(+), 2 deletions(-) > Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] 3262afc6c50: 5.6.42-84.2
by Marko Mäkelä 24 Jan '19

24 Jan '19

Hi Sanja, The patch is adding many changes from MySQL for things that we have already fixed in a better way in both InnoDB and XtraDB when merging from MySQL 5.6.42. Most of it should not be applied. I checked the following output: cd percona-server git diff Percona-Server-5.6.41-84.1..Percona-Server-5.6.42-84.2 storage/innobase/| diff -I^@@ - <(cd ../mysql-server;git diff mysql-5.6.41..mysql-5.6.42 storage/innobase/) It identified the following files that have useful changes from Percona: storage/xtradb/include/univ.i storage/xtradb/log/log0online.cc storage/xtradb/row/row0mysql.cc Let me identify the acceptable changes below. Please do not apply anything else to storage/xtradb. > storage/xtradb/handler/ha_innodb.cc | 5 ++++ > storage/xtradb/handler/handler0alter.cc | 35 ++++++++++++++++++++++------ > storage/xtradb/include/os0file.h | 6 ++++- > storage/xtradb/row/row0merge.cc | 17 ++++++++++++-- > storage/xtradb/row/row0sel.cc | 4 ++-- Please, no changes to these files. > storage/xtradb/include/univ.i | 2 +- > storage/xtradb/log/log0online.cc | 41 ++++++++++++++++++++++++++++++++- These are OK as such. > storage/xtradb/row/row0mysql.cc | 15 +++--------- This contains too many changes. Let me quote the only acceptable change: > diff --git a/storage/xtradb/row/row0mysql.cc b/storage/xtradb/row/row0mysql.cc > index ddde8b8f730..9fc5c669ba0 100644 > --- a/storage/xtradb/row/row0mysql.cc > +++ b/storage/xtradb/row/row0mysql.cc > @@ -5500,18 +5500,6 @@ row_rename_table_for_mysql( > goto funct_exit; > } > > - /* Wait for background fts sync to finish */ > - for (retry = 1; dict_fts_index_syncing(table); ++retry) { > - DICT_BG_YIELD(trx); > - if (retry % 100 == 0) { > - ib_logf(IB_LOG_LEVEL_INFO, > - "Unable to rename table %s to new name" > - " %s because FTS sync is running on table." > - " Retrying\n", > - old_name, new_name); > - } > - } > - > /* We use the private SQL parser of Innobase to generate the query > graphs needed in updating the dictionary data from system tables. */ > The above is http://github.com/percona/percona-server/commit/a7f9ecef3ed61468eb5a37af4f1… which is removing a Percona workaround for a MySQL bug that was fixed both in MySQL 5.6.42 and in MariaDB. Best regards, Marko

1 0

Re: [Maria-developers] 1a4031d9c8f: MDEV-17297: stats.records=0 for a table of Archive engine when it has rows, when we run ANALYZE command
by Sergei Golubchik 23 Jan '19

23 Jan '19

Hi, Sergei! On Nov 07, Sergei Petrunia wrote: > revision-id: 1a4031d9c8f314896daf8f7c19d6172b9bba313b (mariadb-10.3.10-49-g1a4031d9c8f) > parent(s): e058a251c10350f3727ca1df022dc5786933535b > author: Sergei Petrunia > committer: Sergei Petrunia > timestamp: 2018-11-07 11:58:30 +0300 > message: > > MDEV-17297: stats.records=0 for a table of Archive engine when it has rows, when we run ANALYZE command ... > diff --git a/storage/archive/ha_archive.cc b/storage/archive/ha_archive.cc > index 487c0038239..9d7f7e66b28 100644 > --- a/storage/archive/ha_archive.cc > +++ b/storage/archive/ha_archive.cc > @@ -1737,6 +1721,38 @@ int ha_archive::info(uint flag) > } > > > +int ha_archive::external_lock(THD *thd, int lock_type) > +{ > + if (lock_type == F_RDLCK) > + { > + // We are going to read from the table. Flush any pending writes that we > + // may have > + flush_and_clear_pending_writes(); > + } > + return 0; > +} This is somewhat unexpected. Normally buffers are flushed at the end of a statement, on F_UNLCK. For example, you'll have a better chance of surviving a crash (or kill -9). Otherwise I don't see much of a difference, so do it as you like. Shouldn't it go into 5.5, as the earliest affected version? > +void ha_archive::flush_and_clear_pending_writes() > +{ > + mysql_mutex_lock(&share->mutex); > + if (share->dirty) > + { > + DBUG_PRINT("ha_archive", ("archive flushing out rows for scan")); > + DBUG_ASSERT(share->archive_write_open); > + azflush(&(share->archive_write), Z_SYNC_FLUSH); > + share->dirty= FALSE; > + } > + > + /* > + This should be an accurate number now, though bulk and delayed inserts can > + cause the number to be inaccurate. > + */ > + stats.records= share->rows_recorded; > + mysql_mutex_unlock(&share->mutex); > +} Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 1

Re: [Maria-developers] 1eb364f8b3d: MDEV-17421: mtr does not restart the server whose parameters were changed
by Sergei Golubchik 23 Jan '19

23 Jan '19

Hi, Julius! Looks ok, I'll merge it. But without mtr_restart_t1/mtr_restart_t2 tests. You cannot rely on any specific order of test execution, mtr reorders them as it sees fit. On Jan 23, Julius Goryavsky wrote: > revision-id: 1eb364f8b3d (mariadb-10.1.35-84-g1eb364f8b3d) > parent(s): 3c3c4ae2254 > author: Julius Goryavsky <sysprg(a)gmail.com> > committer: Julius Goryavsky <sysprg(a)gmail.com> > timestamp: 2018-10-10 17:16:34 +0200 > message: > > MDEV-17421: mtr does not restart the server whose parameters were changed > > If a mtr test runs multiple servers and only some of them get > restarted on whatever reason with new command-line parameters, > then subsequent mtr test may fail, because no cleanup is performed. > Replication and Galera test suites are affected. > > In the mtr script, there is a server_need_restart function > that decides whether we need to start a new mysqld process before > the new (next) test. If the mysqld parameters were changed in the > previous test - not necessarily the parameters of the primary mysqld > server, maybe even the secondary server parameters - this function > decides to start a new mysqld process. But since it does not remove > the old (changed) parameters, the new process starts with the > parameters changed by the *previous* test. > > To correct this error, we must delete the modified process > parameters after checking that they have been changed during > the previous test. > > This patch also simplifies and makes more stable the > galera_drop_database test, during debugging of which this > problem was detected. > > https://jira.mariadb.org/browse/MDEV-17421 > > --- > mysql-test/mysql-test-run.pl | 1 + > mysql-test/suite/galera/disabled.def | 1 - > .../suite/galera/r/galera_mtr_restart_t1.result | 1 + > .../suite/galera/r/galera_mtr_restart_t2.result | 1 + > .../suite/galera/t/galera_drop_database.test | 26 +++------------- > .../suite/galera/t/galera_mtr_restart_t1.test | 35 ++++++++++++++++++++++ > .../suite/galera/t/galera_mtr_restart_t2.test | 13 ++++++++ > mysql-test/suite/rpl/r/mtr_restart_t1.result | 5 ++++ > mysql-test/suite/rpl/r/mtr_restart_t2.result | 4 +++ > mysql-test/suite/rpl/t/mtr_restart_t1.test | 31 +++++++++++++++++++ > mysql-test/suite/rpl/t/mtr_restart_t2.test | 18 +++++++++++ > 11 files changed, 113 insertions(+), 23 deletions(-) > > diff --git a/mysql-test/mysql-test-run.pl b/mysql-test/mysql-test-run.pl > index d969d7bf9f6..d3ffd16987d 100755 > --- a/mysql-test/mysql-test-run.pl > +++ b/mysql-test/mysql-test-run.pl > @@ -5282,6 +5282,7 @@ sub server_need_restart { > exists $server->{'restart_opts'}) > { > my $use_dynamic_option_switch= 0; > + delete $server->{'restart_opts'}; > if (!$use_dynamic_option_switch) > { > mtr_verbose_restart($server, "running with different options '" . > diff --git a/mysql-test/suite/galera/disabled.def b/mysql-test/suite/galera/disabled.def > index 603031f52b7..464ed6444f9 100644 > --- a/mysql-test/suite/galera/disabled.def > +++ b/mysql-test/suite/galera/disabled.def > @@ -34,4 +34,3 @@ partition : MDEV-13881 galera.partition failed in buildbot with wrong result > galera_as_slave_replication_budle : MDEV-15785 Test case galera_as_slave_replication_bundle caused debug assertion > galera_wan : MDEV-17259: Test failure on galera.galera_wan > galera_pc_ignore_sb : MDEV-17357 Test failure on galera.galera_pc_ignore_sb > -galera_drop_database : test > diff --git a/mysql-test/suite/galera/r/galera_mtr_restart_t1.result b/mysql-test/suite/galera/r/galera_mtr_restart_t1.result > new file mode 100644 > index 00000000000..c628a99f1e8 > --- /dev/null > +++ b/mysql-test/suite/galera/r/galera_mtr_restart_t1.result > @@ -0,0 +1 @@ > +weight=111 > diff --git a/mysql-test/suite/galera/r/galera_mtr_restart_t2.result b/mysql-test/suite/galera/r/galera_mtr_restart_t2.result > new file mode 100644 > index 00000000000..0d488f3d174 > --- /dev/null > +++ b/mysql-test/suite/galera/r/galera_mtr_restart_t2.result > @@ -0,0 +1 @@ > +weight=1 > diff --git a/mysql-test/suite/galera/t/galera_drop_database.test b/mysql-test/suite/galera/t/galera_drop_database.test > index 47fe8315198..12d9efea2f9 100644 > --- a/mysql-test/suite/galera/t/galera_drop_database.test > +++ b/mysql-test/suite/galera/t/galera_drop_database.test > @@ -9,6 +9,7 @@ > --let $node_2=node_2 > --source include/auto_increment_offset_save.inc > > +# Create test database with two sets of the FTS indexes: > CREATE DATABASE fts; > USE fts; > CREATE TABLE fts_t1 (f1 INT PRIMARY KEY AUTO_INCREMENT, f2 VARCHAR(100), FULLTEXT (f2)) ENGINE=InnoDB; > @@ -23,34 +24,19 @@ DROP TABLE ten; > UPDATE fts_t1 SET f2 = 'abcd'; > UPDATE fts_t2 SET f2 = 'efjh'; > > +# Restart the second node: > --connection node_2 > -let $wsrep_cluster_address = `SELECT @@global.wsrep_node_incoming_address`; > --source include/restart_mysqld.inc > > --connection node_1 > --let $wait_condition = SELECT VARIABLE_VALUE = 2 FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_cluster_size'; > --source include/wait_condition.inc > > ---let $galera_connection_name = node_2a > ---let $galera_server_number = 2 > ---source include/galera_connect.inc > ---connection node_2a > +--connection node_2 > --source include/wait_until_ready.inc > > +# Drop the tables and database after nodes restarted: > --connection node_1 > ---let $restart_parameters = --wsrep-cluster-address=gcomm://$wsrep_cluster_address > ---source include/restart_mysqld.inc > - > ---connection node_2a > ---let $wait_condition = SELECT VARIABLE_VALUE = 2 FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_cluster_size'; > ---source include/wait_condition.inc > - > ---let $galera_connection_name = node_1a > ---let $galera_server_number = 1 > ---source include/galera_connect.inc > ---connection node_1a > ---source include/wait_until_ready.inc > - > USE fts; > DROP TABLE fts_t1; > DROP TABLE fts_t2; > @@ -58,8 +44,4 @@ SHOW TABLES; > DROP DATABASE fts; > > # Restore original auto_increment_offset values. > ---let $node_1=node_1a > ---let $node_2=node_2a > --source include/auto_increment_offset_restore.inc > - > ---source include/galera_end.inc > diff --git a/mysql-test/suite/galera/t/galera_mtr_restart_t1.test b/mysql-test/suite/galera/t/galera_mtr_restart_t1.test > new file mode 100644 > index 00000000000..563d199625a > --- /dev/null > +++ b/mysql-test/suite/galera/t/galera_mtr_restart_t1.test > @@ -0,0 +1,35 @@ > +# This test verifies that mtr will restart the mysqld process, > +# whose parameters were changed during the test. The verification > +# itself is carried out in the following galera_mtr_restart_t2 > +# test. If mtr restart the mysqld process, then the pc.weight > +# value will be reset to the default ("1"), but if there is no > +# restart, then we will see the changed value ("111") during > +# the next test. > +# > +--source include/galera_cluster.inc > +--source include/have_innodb.inc > + > +# Save original auto_increment_offset values. > +--let $node_1=node_1 > +--let $node_2=node_2 > +--source include/auto_increment_offset_save.inc > + > +--connection node_2 > +--let $restart_parameters = --wsrep_provider_options=pc.weight=111;repl.causal_read_timeout=PT90S;evs.suspect_timeout=PT10S;evs.inactive_timeout=PT30S;evs.install_timeout=PT15S > +--source include/restart_mysqld.inc > + > +--connection node_1 > +--let $wait_condition = SELECT VARIABLE_VALUE = 2 FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_cluster_size'; > +--source include/wait_condition.inc > + > +--connection node_2 > +--source include/wait_until_ready.inc > + > +# Check that the parameter value really changed: > +--let $gp = `SELECT SUBSTR(@@wsrep_provider_options, LOCATE('pc.weight =', @@wsrep_provider_options) + LENGTH('pc.weight = '))` > +--let $weight = `SELECT SUBSTR('$gp', 1, LOCATE(';', '$gp') - 1)` > +--echo weight=$weight > + > +# Restore original auto_increment_offset values. > +--connection node_1 > +--source include/auto_increment_offset_restore.inc > diff --git a/mysql-test/suite/galera/t/galera_mtr_restart_t2.test b/mysql-test/suite/galera/t/galera_mtr_restart_t2.test > new file mode 100644 > index 00000000000..fcc1d0515a6 > --- /dev/null > +++ b/mysql-test/suite/galera/t/galera_mtr_restart_t2.test > @@ -0,0 +1,13 @@ > +# This test verifies that mtr will restart the mysqld process, > +# whose parameters were changed during the previous test. If mtr > +# restart the mysqld process, then the pc.weight value will be > +# reset to the default ("1"), but if there is no restart, then > +# we will see the changed value ("111") in this test. > +# > +--source include/galera_cluster.inc > +--source include/have_innodb.inc > + > +--connection node_2 > +--let $gp = `SELECT SUBSTR(@@wsrep_provider_options, LOCATE('pc.weight =', @@wsrep_provider_options) + LENGTH('pc.weight = '))` > +--let $weight = `SELECT SUBSTR('$gp', 1, LOCATE(';', '$gp') - 1)` > +--echo weight=$weight > diff --git a/mysql-test/suite/rpl/r/mtr_restart_t1.result b/mysql-test/suite/rpl/r/mtr_restart_t1.result > new file mode 100644 > index 00000000000..56b64a2fc70 > --- /dev/null > +++ b/mysql-test/suite/rpl/r/mtr_restart_t1.result > @@ -0,0 +1,5 @@ > +include/master-slave.inc > +[connection master] > +include/rpl_stop_server.inc [server_number=1] > +new auto_increment_offset=111 > +include/rpl_end.inc > diff --git a/mysql-test/suite/rpl/r/mtr_restart_t2.result b/mysql-test/suite/rpl/r/mtr_restart_t2.result > new file mode 100644 > index 00000000000..3c8fe59d607 > --- /dev/null > +++ b/mysql-test/suite/rpl/r/mtr_restart_t2.result > @@ -0,0 +1,4 @@ > +include/master-slave.inc > +[connection master] > +auto_increment_offset=1 > +include/rpl_end.inc > diff --git a/mysql-test/suite/rpl/t/mtr_restart_t1.test b/mysql-test/suite/rpl/t/mtr_restart_t1.test > new file mode 100644 > index 00000000000..3003a49c424 > --- /dev/null > +++ b/mysql-test/suite/rpl/t/mtr_restart_t1.test > @@ -0,0 +1,31 @@ > +# This test verifies that mtr will restart the mysqld process, > +# whose parameters were changed during the test. The verification > +# itself is carried out in the following mtr_restart_t2 test. > +# If mtr restart the mysqld process, then the auto_increment_offset > +# parameter value will be reset to the default ("1"), but if there > +# is no restart, then we will see the changed value ("111") during > +# the next test. > +# > +--source include/have_binlog_format_row.inc > +--source include/master-slave.inc > + > +connection master; > + > +let $auto_increment_offset = `SELECT @@global.auto_increment_offset`; > + > +--let $rpl_server_number=1 > +source include/rpl_stop_server.inc; > +--let $rpl_server_parameters=--auto-increment-offset=111 > +--let $keep_include_silent=1 > +source include/rpl_start_server.inc; > +--let $keep_include_silent=0 > + > +let $auto_increment_offset_new = `SELECT @@global.auto_increment_offset`; > +--echo new auto_increment_offset=$auto_increment_offset_new > + > +--disable_query_log > +--eval SET @@global.auto_increment_offset = $auto_increment_offset; > +--enable_query_log > + > +--connection master > +--source include/rpl_end.inc > diff --git a/mysql-test/suite/rpl/t/mtr_restart_t2.test b/mysql-test/suite/rpl/t/mtr_restart_t2.test > new file mode 100644 > index 00000000000..ab246af6b44 > --- /dev/null > +++ b/mysql-test/suite/rpl/t/mtr_restart_t2.test > @@ -0,0 +1,18 @@ > +# This test verifies that mtr will restart the mysqld process, > +# whose parameters were changed during the previous test. If mtr > +# restart the mysqld process, then the auto_increment_offsert > +# parameter value will be reset to the default ("1"), but if there > +# is no restart, then we will see the changed value ("111") in > +# this test. > +# > +--source include/have_binlog_format_row.inc > +--source include/master-slave.inc > + > +connection master; > + > +let $auto_increment_offset = `SELECT @@global.auto_increment_offset`; > + > +--echo auto_increment_offset=$auto_increment_offset > + > +--connection master > +--source include/rpl_end.inc > Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] c8248ca9a22: MDEV-17376 Server fails to set ADD_PK_INDEX, DROP_PK_INDEX if unique index nominated as PK
by Sergei Golubchik 23 Jan '19

23 Jan '19

Hi, Thiru! Thanks! This is, indeed, much easier to follow. Nicely done. Ok to push. On Jan 23, Thirunarayanan Balathandayuthapani wrote: > revision-id: c8248ca9a22 (mariadb-10.0.37-45-gc8248ca9a22) > parent(s): 12f362c3338 > author: Thirunarayanan Balathandayuthapani <thiru(a)mariadb.com> > committer: Thirunarayanan Balathandayuthapani <thiru(a)mariadb.com> > timestamp: 2019-01-18 12:01:18 +0530 > message: > > MDEV-17376 Server fails to set ADD_PK_INDEX, DROP_PK_INDEX if unique index nominated as PK > > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index 2302026b18b..645746a7ab0 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -6513,6 +6519,17 @@ static bool fill_alter_inplace_info(THD *thd, > new_field->field->field_index != key_part->fieldnr - 1) > goto index_changed; > } > + > + /* > + Rebuild the index if following condition get satisfied: > + > + (i) Old table doesn't have primary key, new table has it and vice-versa > + (ii) Primary key changed to another existing index > + */ > + if ((new_key == new_pk) != > + (table_key == &table->key_info[table->s->primary_key])) I would suggest to rewrite the condition as table_key - table->key_info == table->s->primary_key it means the same thing, but doesn't confuse the reader with a question "what's with a buffer overrun if table->s->primary_key == MAX_KEY?" > + goto index_changed; > + > continue; > > index_changed: > Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] c3ef531aaff: MDEV-10963 Fragmented BINLOG query
by Sergei Golubchik 22 Jan '19

22 Jan '19

Hi, Andrei! Thanks! We're almost there. just a couple of minor comments re. format strings, see below. On Dec 18, andrei.elkin(a)pp.inet.fi wrote: > revision-id: c3ef531aaff39d3262882213157d36a545b600dd (mariadb-10.1.37-31-gc3ef531aaff) > parent(s): 541500295abdba7fa619065069291ac9c1dd6e83 > author: Andrei Elkin > committer: Andrei Elkin > timestamp: 2018-12-18 18:28:59 +0200 > message: > > MDEV-10963 Fragmented BINLOG query ... > diff --git a/sql/log_event.cc b/sql/log_event.cc > --- a/sql/log_event.cc > +++ b/sql/log_event.cc > @@ -10491,12 +10505,128 @@ void Rows_log_event::pack_info(Protocol *protocol) > #endif > > #ifdef MYSQL_CLIENT > +/** > + Print an event "body" cache to @c file possibly in two fragments. > + Each fragement is optionally per @c do_wrap to produce an SQL statement. > + > + @param file a file to print to > + @param body the "body" IO_CACHE of event > + @param do_wrap whether to wrap base64-encoded strings with > + SQL cover. > + @param delimiter delimiter string > + > + The function signals on any error through setting @c body->error to -1. > +*/ > +void copy_cache_to_file_wrapped(FILE *file, > + IO_CACHE *body, > + bool do_wrap, > + const char *delimiter) > +{ > + const char str_binlog[]= "\nBINLOG '\n"; > + const char fmt_delim[]= "'%s\n"; > + const char fmt_n_delim[]= "\n'%s\n"; > + const my_off_t cache_size= my_b_tell(body); > + > + if (reinit_io_cache(body, READ_CACHE, 0L, FALSE, FALSE)) > + { > + body->error= -1; > + goto end; > + } > + > + if (!do_wrap) > + { > + my_b_copy_to_file(body, file, SIZE_T_MAX); > + } > + else if (4 + sizeof(str_binlog) + cache_size + sizeof(fmt_delim) > > + opt_binlog_rows_event_max_encoded_size) > + { > + /* > + 2 fragments can always represent near 1GB row-based > + base64-encoded event as two strings each of size less than > + max(max_allowed_packet). Greater number of fragments does not > + save from potential need to tweak (increase) @@max_allowed_packet > + before to process the fragments. So 2 is safe and enough. > + > + Split the big query when its packet size's estimation exceeds a > + limit. The estimate includes the maximum packet header > + contribution of non-compressed packet. > + */ > + const char fmt_frag[]= "%sSET @binlog_fragment_%d ='\n"; > + > + my_fprintf(file, fmt_frag, "\n", 0); > + if (my_b_copy_to_file(body, file, cache_size/2 + 1)) > + { > + body->error= -1; > + goto end; > + } > + my_fprintf(file, fmt_n_delim, delimiter); see, you use fmt_n_delim only once. here. but you end it with \n. And it forced you to start fmt_frag with %s. please, remove \n from the end of fmt_n_delim and add it to the beginning of fmt_frag > + > + my_fprintf(file, fmt_frag, "", 1); > + if (my_b_copy_to_file(body, file, SIZE_T_MAX)) > + { > + body->error= -1; > + goto end; > + } > + my_fprintf(file, fmt_delim, delimiter); and here - no \n before ' ? I've got lost in all these new lines and format constants. can you please show what the binlog dump will look like? > + > + my_fprintf(file, "BINLOG @binlog_fragment_%d, @binlog_fragment_%d%s\n", > + 0, 1, delimiter); don't pass 0 and 1 as parameters, it's not like they can ever change :) > + } > + else > + { > + my_fprintf(file, str_binlog); > + if (my_b_copy_to_file(body, file, SIZE_T_MAX)) > + { > + body->error= -1; > + goto end; > + } > + my_fprintf(file, fmt_delim, delimiter); > + } > + reinit_io_cache(body, WRITE_CACHE, 0, FALSE, TRUE); > + > +end: > + return; > +} > + > +/** > + The function invokes base64 encoder to run on the current > + event string and store the result into two caches. > + When the event ends the current statement the caches are is copied into > + the argument file. > + Copying is also concerned how to wrap the event, specifically to produce > + a valid SQL syntax. > + When the encoded data size is within max(MAX_ALLOWED_PACKET) > + a regular BINLOG query is composed. Otherwise it is build as fragmented > + > + SET @binlog_fragment_0='...'; > + SET @binlog_fragment_1='...'; > + BINLOG @binlog_fragment_0, @binlog_fragment_1; > + > + where fragments are represented by a pair of indexed user > + "one shot" variables. > + > + NOTE. still @note > + If any changes made don't forget to duplicate them to > + Old_rows_log_event as long as it's supported. > + > + @param file pointer to IO_CACHE > + @param print_event_info pointer to print_event_info specializing > + what out of and how to print the event > + @param name the name of a table that the event operates on > + > + The function signals on any error of cache access through setting > + that cache's @c error to -1. > +*/ > void Rows_log_event::print_helper(FILE *file, > PRINT_EVENT_INFO *print_event_info, > char const *const name) > { > IO_CACHE *const head= &print_event_info->head_cache; > IO_CACHE *const body= &print_event_info->body_cache; > + bool do_print_encoded= > + print_event_info->base64_output_mode != BASE64_OUTPUT_DECODE_ROWS && > + !print_event_info->short_form; > + > if (!print_event_info->short_form) > { > bool const last_stmt_event= get_flags(STMT_END_F); Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] 02587f61f40: MDEV-13095 Implement User Account locking
by Sergei Golubchik 21 Jan '19

21 Jan '19

Hi, Robert! On Jan 21, Robert Bindar wrote: > Hi Sergei! Thanks for reviewing this :-) > > > > You could simply add accessors like > > > > longlong get_password_lifetime() const > > { > > Field *f= get_field(end_priv_columns + 11, MYSQL_TYPE_LONG); > > return f ? f->val_int() : 0; > > } > > > > without changing anything else in User_table_tabular. > > Correct me if I'm mistaken. Let's assume we index like that, > > then get_is_role will become: > > bool get_is_role() const > { > Field *f = get_field(end_priv_columns + 14, MYSQL_TYPE_ENUM); > return f ? f->val_int() - 1 : 0; > } > > If we'll start the server with a 10.4.1 datadir for instance, > get_is_role will go beyond num_fields() and thus will always return > false. From my understanding so far, this means we won't be able to > support previous MariaDB datadirs without upgrading. Is this the > intended behavior? You've missed that I also suggested _not_ to add any more columns to the mysql.user table :) So you can never have is_role and password_lifetime in the same table. It's either 5.7 structure or 10.3 structure. And get_is_role() stays as it was: bool get_is_role() const { Field *f = get_field(end_priv_columns + 11, MYSQL_TYPE_ENUM); return f ? f->val_int() - 1 : 0; } That's exactly what I meant by "without changing anything else in User_table_tabular" Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] 02587f61f40: MDEV-13095 Implement User Account locking
by Sergei Golubchik 21 Jan '19

21 Jan '19

Hi, Robert! On Jan 20, Robert Bindar wrote: > revision-id: 02587f61f40 (mariadb-10.4.1-100-g02587f61f40) > parent(s): 937c90ce2d3 > author: Robert Bindar <robert(a)mariadb.org> > committer: Sergei Golubchik <serg(a)mariadb.com> > timestamp: 2019-01-20 00:33:44 +0100 > message: > > MDEV-13095 Implement User Account locking > > Add server support for user account locking. > This patch adds two new statements for denying a user's subsequent login > attempts: > LOCK USER[S] > user_name [, user2_name ] ... > UNLOCK USER[S] > user_name [, user2_name ] ... This doesn't seem to be correct anymore :) > The SHOW GRANTS command was updated to support locking state > information. > > diff --git a/mysql-test/include/switch_to_mysql_user.inc b/mysql-test/include/switch_to_mysql_user.inc > index f5801db6114..eff1d98c2df 100644 > --- a/mysql-test/include/switch_to_mysql_user.inc > +++ b/mysql-test/include/switch_to_mysql_user.inc > @@ -45,6 +45,9 @@ CREATE TABLE mysql.user ( > plugin char(64) CHARACTER SET latin1 DEFAULT '' NOT NULL, > authentication_string TEXT NOT NULL, > password_expired ENUM('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, > + password_last_changed datetime NULL, > + password_lifetime int(20) unsigned NULL, > + account_locked enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, No-no. mysql.user structure was frozen in time the moment I pushed the commit with the new structure. It doesn't change anymore. To test 5.7 support, just add ALTER TABLE in one specific test file. See main/system_mysql_db_507.test > is_role enum('N','Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL, > default_role char(80) binary DEFAULT '' NOT NULL, > max_statement_time decimal(12,6) DEFAULT 0 NOT NULL, > diff --git a/mysql-test/main/lock_user.result b/mysql-test/main/lock_user.result > --- /dev/null > +++ b/mysql-test/main/lock_user.result > @@ -0,0 +1,132 @@ > +connect con2,localhost,user2; > +disconnect con2; > +connection default; > +alter user user1@localhost account unlock; > +# > +# A user shouldn't be able to lock itself out "lock himself out" And why not? A user can drop himself or change the password to some garbage. It's the same as locking or worse. > +# > +create user newuser@localhost; > +grant CREATE USER on *.* to newuser@localhost; > +connect con1,localhost,newuser; > +connection con1; > +alter user newuser@localhost account lock; > +ERROR HY000: Operation ALTER USER failed for 'newuser'@'localhost' > +disconnect con1; > +connection default; > +drop user newuser@localhost; ... > +alter user user1@localhost account lock; > +show grants for user1@localhost; > +Grants for user1@localhost > +GRANT USAGE ON *.* TO 'user1'@'localhost' > +ALTER USER 'user1'@'localhost' ACCOUNT LOCK No, I don't think so. It's MySQL compatibility feature, make it MySQL compatible. And it's just not very logical to put ALTER USER in SHOW GRANTS. So, SHOW GRANTS should not show ACCOUNT LOCK, but SHOW CREATE USER should. > diff --git a/scripts/mysql_system_tables.sql b/scripts/mysql_system_tables.sql > index f788f5d67d5..01d686ac345 100644 > --- a/scripts/mysql_system_tables.sql > +++ b/scripts/mysql_system_tables.sql > @@ -79,7 +79,10 @@ CREATE DEFINER=root@localhost SQL SECURITY DEFINER VIEW IF NOT EXISTS user AS SE > CAST(IFNULL(JSON_VALUE(Priv, '$.max_user_connections'), 0) AS SIGNED) AS max_user_connections, > IFNULL(JSON_VALUE(Priv, '$.plugin'), '') AS plugin, > IFNULL(JSON_VALUE(Priv, '$.authentication_string'), '') AS authentication_string, > - 'N' AS password_expired, > + ELT(IFNULL(JSON_VALUE(Priv, '$.password_expired'), 0) + 1, 'N', 'Y') AS password_expired, > + CAST(JSON_UNQUOTE(JSON_EXTRACT(Priv, '$.password_last_changed')) AS DATETIME) AS password_last_changed, > + CAST(JSON_VALUE(Priv, '$.password_lifetime') AS UNSIGNED) AS password_lifetime, > + ELT(IFNULL(JSON_VALUE(Priv, '$.account_locked'), 0) + 1, 'N', 'Y') AS account_locked, No-no. mysql.user is frozen, it doesn't change anymore. It stays always as it was in 10.3. This was the whole point of moving to JSON, to never ever add new columns to mysql.user table. > ELT(IFNULL(JSON_VALUE(Priv, '$.is_role'), 0) + 1, 'N', 'Y') AS is_role, > IFNULL(JSON_VALUE(Priv, '$.default_role'), '') AS default_role, > CAST(IFNULL(JSON_VALUE(Priv, '$.max_statement_time'), 0.0) AS DECIMAL(12,6)) AS max_statement_time > diff --git a/scripts/mysql_system_tables_fix.sql b/scripts/mysql_system_tables_fix.sql > index 82ec2faa12d..5442e5fee73 100644 > --- a/scripts/mysql_system_tables_fix.sql > +++ b/scripts/mysql_system_tables_fix.sql > @@ -643,12 +643,16 @@ ALTER TABLE user ADD plugin char(64) CHARACTER SET latin1 DEFAULT '' NOT NULL, > ALTER TABLE user MODIFY plugin char(64) CHARACTER SET latin1 DEFAULT '' NOT NULL, > MODIFY authentication_string TEXT NOT NULL; > ALTER TABLE user ADD password_expired ENUM('N', 'Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL; > +ALTER TABLE user ADD password_last_changed datetime NULL after password_expired; > +ALTER TABLE user ADD password_lifetime int(20) unsigned NULL after password_last_changed; > +ALTER TABLE user ADD account_locked enum('N', 'Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL after password_lifetime; But that's okay, because otherwise CREATE ... SELECT below won't work. > ALTER TABLE user ADD is_role enum('N', 'Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL; > ALTER TABLE user ADD default_role char(80) binary DEFAULT '' NOT NULL; > ALTER TABLE user ADD max_statement_time decimal(12,6) DEFAULT 0 NOT NULL; > -- Somewhere above, we ran ALTER TABLE user .... CONVERT TO CHARACTER SET utf8 COLLATE utf8_bin. > -- we want password_expired column to have collation utf8_general_ci. > ALTER TABLE user MODIFY password_expired ENUM('N', 'Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL; > +ALTER TABLE user MODIFY account_locked enum('N', 'Y') COLLATE utf8_general_ci DEFAULT 'N' NOT NULL; I don't think it's needed, we're dropping the table anyway. > -- Checking for any duplicate hostname and username combination are exists. > @@ -804,6 +808,10 @@ IF 'BASE TABLE' = (select table_type from information_schema.tables where table_ > 'max_statement_time', max_statement_time, > 'plugin', if(plugin>'',plugin,if(length(password)=16,'mysql_old_password','mysql_native_password')), > 'authentication_string', if(plugin>'',authentication_string,password), > + 'password_expired', 'Y'=password_expired, > + 'password_last_changed', password_last_changed, > + 'password_lifetime', password_lifetime, It would be better to put password expiration changes in the password expiration commit. > + 'account_locked', 'Y'=account_locked, > 'default_role', default_role, > 'is_role', 'Y'=is_role)) as Priv > FROM user; > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index fe6fc9148bd..e4fc96b61cd 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -798,12 +806,25 @@ class Grant_table_base > else if (start_priv_columns) > break; > } > + > + end_security_columns= end_priv_columns; > + for (; end_security_columns < num_fields(); end_security_columns++) > + { > + Field *field= m_table->field[end_security_columns]; > + const char *colname= field->field_name.str; > + if (field->real_type() == MYSQL_TYPE_ENUM && > + !strcmp(colname, "is_role")) > + break; > + } Are you trying to make it super-generic, like with privileges, for an arbitrary number of security columns? Don't bother, 5.7 was the last MySQL version with privilege tables, they won't be adding more "security columns" to it. You could simply add accessors like longlong get_password_lifetime() const { Field *f= get_field(end_priv_columns + 11, MYSQL_TYPE_LONG); return f ? f->val_int() : 0; } without changing anything else in User_table_tabular. > } > > /* The index at which privilege columns start. */ > uint start_priv_columns; > /* The index after the last privilege column */ > uint end_priv_columns; > + /* The index after the last security column, i.e. index of is_role. This index > + is equal to num_fields if the is_role column does not exist */ > + uint end_security_columns; > > TABLE *m_table; > }; > @@ -8758,7 +8838,6 @@ static ROLE_GRANT_PAIR *find_role_grant_pair(const LEX_CSTRING *u, > return (ROLE_GRANT_PAIR *) > my_hash_search(&acl_roles_mappings, (uchar*)pair_key.ptr(), key_length); > } > - > static bool show_role_grants(THD *thd, const char *username, restore the empty line please > const char *hostname, ACL_USER_BASE *acl_entry, > char *buff, size_t buffsize) > @@ -10555,6 +10634,52 @@ int mysql_alter_user(THD* thd, List<LEX_USER> &users_list) > DBUG_RETURN(result); > } > > +/* > + Mark an user account as locked or unlocked. > + > + SYNOPSIS Doxygen comments. This style that you've copied is from (iirc) before 2005:) > + lock_user_account() > + thd The current thread. > + user_table A TL_WRITE opened User_table > + lex_user The user to lock/unlock. > + acl_user The acl user to be updated > + > + RETURN > + > 0 Error. Error message already sent. > + 0 OK. > +*/ > +static int lock_user_account(THD* thd, const User_table &user_table, > + LEX_USER *lex_user, ACL_USER *acl_user) > +{ > + bool update_account_locking= thd->lex->account_options.update_account_locking; > + bool account_locked_value= thd->lex->account_options.account_locked_value; > + > + DBUG_ENTER("lock_user_account"); > + > + if (!update_account_locking) > + DBUG_RETURN(0); > + > + mysql_mutex_assert_owner(&acl_cache->lock); > + > + /* > + Do not allow the current user to lock itself out. > + */ > + ACL_USER *current_acl_user= find_user_wild(thd->security_ctx->priv_host, > + thd->security_ctx->priv_user); > + if (!current_acl_user || > + (!strcmp(lex_user->user.str, current_acl_user->user.str) && > + !my_strcasecmp(system_charset_info, lex_user->host.str, > + current_acl_user->host.hostname))) > + { > + DBUG_RETURN(1); > + } > + > + acl_user->account_locked= account_locked_value; > + > + user_table.set_account_locked(account_locked_value); > + > + DBUG_RETURN(0); > +} > > static bool > mysql_revoke_sp_privs(THD *thd, Grant_tables *tables, const Sp_handler *sph, > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index 8f62dca4aec..6faa5966d29 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -1148,6 +1148,7 @@ bool my_yyoverflow(short **a, YYSTYPE **b, size_t *yystacksize); > Non-reserved keywords > */ > > +%token <kwd> ACCOUNT_SYM comment /* MYSQL */ > %token <kwd> ACTION /* SQL-2003-N */ > %token <kwd> ADMIN_SYM /* SQL-2003-N */ > %token <kwd> ADDDATE_SYM /* MYSQL-FUNC */ > +opt_account_option: > + ACCOUNT_SYM LOCK_SYM > + { > + Lex->account_options.update_account_locking= true; > + Lex->account_options.account_locked_value= true; > + } > + | ACCOUNT_SYM UNLOCK_SYM > + { > + Lex->account_options.update_account_locking= true; > + Lex->account_options.account_locked_value= false; Why wouldn't you make it a 3-state enum { NOT_SPECIFIED, LOCKED, UNLOCKED } ? looks more natural here. > + } > + ; > + > ev_alter_on_schedule_completion: > /* empty */ { $$= 0;} Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 2

[Maria-developers] Fwd: ce4f8955d8d: MDEV-17082 Application period tables: CREATE
by Nikita Malyavin 17 Jan '19

17 Jan '19

---------- Forwarded message --------- From: Nikita Malyavin <nikitamalyavin(a)gmail.com> Date: пт, 18 янв. 2019 г. в 01:46 Subject: Re: ce4f8955d8d: MDEV-17082 Application period tables: CREATE To: Sergei Golubchik <serg(a)mariadb.org> Hi, Sergei! > > +# SQL17 p.832 > > +# Let IDCN be an implementation-dependent <constraint name> that is not > > +# equivalent to the <constraint name> of any table constraint descriptor > > +# included in S. > > Not enough context. It was quite difficult to find this quote. > > When you're quoting the standard, please, specify the part name and the > section name. And there's no need to specify a page. For example: > > SQL2016, Part 2, 11.27 <add table period definition>, General Rules, 2)a) > Fixed, as well as all the rest cites > > + Table_period_info("SYSTEM_TIME", sizeof "SYSTEM_TIME" - 1), > > We have a macro for that, use > > Table_period_info(STRING_WITH_LEN("SYSTEM_TIME")) > ✅ > > + then no promition is done. > > promotion ✅ > > + Table_period_info& get_table_period_info() > > + { > > + return create_info.period_info; > > + } > > really? :) you've introduced a new method get_table_period_info > only to use it two lines below and nowhere else? > Not really, of course. But agree, it's not worth to have such a shortcut. > > + > > + int add_period(Lex_ident name, Lex_ident_sys_st start, Lex_ident_sys_st end) > > + { > > + Table_period_info &info= get_table_period_info(); > > + if (info.is_set()) > > + { > > + my_error(ER_PERIOD_MAX_COUNT_EXCEEDED, MYF(0), "application"); > > + return 1; > > + } > > + info.set_period(start, end); > > + info.name= name; > > + > > + Virtual_column_info *constr= new Virtual_column_info(); > > + constr->expr= lt_creator.create(thd, create_item_ident_nosp(thd, &start), > > + create_item_ident_nosp(thd, &end)); > > + add_constraint(&null_clex_str, constr, false); > > + return 0; > > + } > > + > > sp_package *get_sp_package() const; > > > > /** > > diff --git a/sql/table.cc b/sql/table.cc > > index ab1a04ccae4..87fa1ba4d61 100644 > > --- a/sql/table.cc > > +++ b/sql/table.cc > > @@ -1228,6 +1228,14 @@ static const Type_handler *old_frm_type_handler(uint pack_flag, > > } > > > > > > +bool TABLE_SHARE::init_period_from_extra2(period_info_t &period, > > + const uchar *data) > > +{ > > + period.start_fieldno= uint2korr(data); > > + period.end_fieldno= uint2korr(data + sizeof(uint16)); > > not "sizeof(uint16)" but 2. I don't think it's guaranteed that > sizeof(uint16) == 2, but as you do uint2korr, you only read 2 bytes, > so data + 2 is always correct. > Yes, it's correct, and I've fixed it, but it wasn't about coorrectness -- but readability. Now what 4, or what is 8 -- is the question that the reader could сonsider. This numbers look magical. With using sizeof(uint16) the one could at least understand, that this is about several double-byte words read. The another problem is that sizeof(uint16) is already used in system versioning, few lines below, so the resulting code will look patchy here. > > + my_error(ER_PERIOD_FIELD_WRONG_ATTRIBUTES, MYF(0), > > + f->field_name.str, "VIRTUAL or GENERATED"); > > just use "GENERATED", no English in the error message parameters, please. > VIRTUAL/GENERATED? What'd You say? > > + my_error(ER_PERIOD_FIELD_WRONG_ATTRIBUTES, MYF(0), > > + f->field_name.str, "NULL"); > > why is that? it's unrelated and inconsistent with cases like > > create table t1 (a int null, primary key (a)); > > where a column is silently created NOT NULL. > So, you should ignore explicit NULL and create columns NOT NULL. > > If you want to change this, it should be consistent in all cases. > create an MDEV, suggest there that the statement above should be an error. > but I don't think it's worth spending time on now. > Ok, I understand. Fixed. > > + else if (f->flags & VERS_SYSTEM_FIELD) > > + { > > + my_error(ER_PERIOD_FIELD_WRONG_ATTRIBUTES, MYF(0), f->field_name.str, > > + f->flags & VERS_SYS_START_FLAG ? "ROW START" : "ROW END"); > > No need to have a special case for that. Use the same error as for vcol_info: > > if (f->vcol_info || f->flags & VERS_SYSTEM_FIELD) > { > my_error(ER_PERIOD_FIELD_WRONG_ATTRIBUTES, MYF(0), > f->field_name.str, "GENERATED ALWAYS AS"); > Looks not very much user-friendly I think. Let's leave the verbosity as it was > What does this error have to do with MDEV-16976? > SQL standard clearly says (part 2, 11.3 <table definition>, syntax rules, 2)d) > > d) No <column name> in any <column definition> shall be equivalent to PN. > > (here PN is "period name"). So MDEV-16976 or not, a period name should > never match a column name. > I was thinking to implement it through virtual column. Then this check will become redundant, and we'll get duplicate field error > > + eng "Temporary tables with application period not allowed" > > "application time period" > ✅ > > + > > +ER_PERIOD_FIELD_NOT_FOUND > > + eng "Field %`s in PERIOD FOR %`s is not found in table" > > I don't think this needs a new error message, just use ER_BAD_FIELD_ERROR > ✅ > > + > > +ER_PERIOD_TYPES_MISMATCH > > + eng "PERIOD FOR %`s fields types mismatch" > > + > > +ER_PERIOD_MAX_COUNT_EXCEEDED > > better say ER_MORE_THAN_ONE_PERIOD or something... > "max count exceeded" suggests there is some not-trivial > (and possibly variable) max count > > Or ER_PERIOD_ALREADY_DEFINED or ER_DUPLICATE_PERIOD etc. > "already defined" looks like a good one. > > And I've already commented (in another patch) on using English > words as arguments - it will look like a bug when the error message is > translated > I like ER_MORE_THAN_ONE_PERIOD more. IMO application-time periods designed in the way to support a possibility of several periods definition, so i like to treat it as "N periods max supported, but now N=1" > > + eng "cannot specify more than one %s-time period" > > + > > +ER_PERIOD_NAME_IS_NOT_ALLOWED_FOR_FIELD > > + eng "Could not specify name %`s for field. It is already used by period." > > I don't think this needs a new error message, just use ER_DUP_FIELDNAME > ✅ > > + > > +ER_PERIOD_FIELD_WRONG_ATTRIBUTES > > + eng "Period field %`s cannot be %s" > > + > > +ER_PERIOD_FIELD_HOLDS_MORE_THAN_ONE_PERIOD > > + eng "Field %`s is specified for more than one period" > > This looks unused ✅ Sincerely Nikita Malyavin

1 0

[Maria-developers] Please review MDEV-18195 ASAN use-after-poison in my_strcasecmp_utf8 / Item::eq upon prepared statement with ORDER BY NAME_CONST
by Alexander Barkov 17 Jan '19

17 Jan '19

Hi Sanja, Please review a patch for MDEV-18195. Thanks.

1 0

Re: [Maria-developers] d08031dfae2: MDEV-17376 Server fails to set ADD_PK_INDEX, DROP_PK_INDEX if unique index nominated as PK
by Sergei Golubchik 17 Jan '19

17 Jan '19

Hi, Thirunarayanan! On Jan 16, Thirunarayanan Balathandayuthapani wrote: > revision-id: d08031dfae2 (mariadb-10.0.37-45-gd08031dfae2) > parent(s): 12f362c3338 > author: Thirunarayanan Balathandayuthapani <thiru(a)mariadb.com> > committer: Thirunarayanan Balathandayuthapani <thiru(a)mariadb.com> > timestamp: 2019-01-16 15:46:28 +0530 > message: > > MDEV-17376 Server fails to set ADD_PK_INDEX, DROP_PK_INDEX if unique index nominated as PK > > Problem: > ======== > Server fails to notify the engine by not setting the ADD_PK_INDEX and > DROP_PK_INDEX When there is a > i) Change in candidate for primary key. > ii) New candidate for primary key. > > Fix: > ==== > Server sets the ADD_PK_INDEX and DROP_PK_INDEX while doing alter for the > above problematic case. Sorry, that's way too complex. I mean the original code with candidate_key_count, is_candidate_key, and all that. And you're making it more complex. If a unique key is promoted to a primary key, this unique key is always be the first key in the table. So what you need to check is * if the old table didn't have a primary key and the new table has it, then a primary key was added. As easy as if (table->s->primary_key >= MAX_KEY && altered_table->s->primary_key != MAX_KEY) * The opposite case, a primary key was removed * if both tables had a primary key, but the key number 0 in the new table was not key number 0 in the old table - a primary key was changed: if (key_info->usable_key_parts) ... And that should cover all possible cases, as far as I can see. May need some tweaking for key_info->usable_key_parts of newly added indexes, I'm not quite sure what number they get. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 2

[Maria-developers] (no subject)
by varsha kukreja 12 Jan '19

12 Jan '19

Hey this is Varsha Kukreaj a third year student from SPITMumbai ..I would like to contribute to the open source community .Can someone help me get started?..I have majorly worked on backend on 2 college sponsored projects and working currently on a project by Government Organization. I have fair knowledge in Javascript, NodeJs, Typescript, Spring Boot, Laravel , Docker and apache thrift. It woukd be great if could if someone could help me get started

2 1

Re: [Maria-developers] MDEV 18092
by Anel Husakovic 10 Jan '19

10 Jan '19

Hi Alexander, I was on vacation too, just came back, thank you for teaching me. Indeed it works, here is a patch with quicker solution and the test case *https://github.com/an3l/server/commit/252ca79493e18fa8fd364992f5022af8a57b79d5 <https://github.com/an3l/server/commit/252ca79493e18fa8fd364992f5022af8a57b7…>* I added the PR also, so if something needs to be improved please let me know. In addition I will try later also to add `store_schema_params()` function to handler. Thanks, Anel On Mon, Dec 31, 2018 at 7:02 AM Alexander Barkov <bar(a)mariadb.com> wrote: > Hi Anel, > > > > On 12/28/2018 02:12 PM, Anel Husakovic wrote: > > Hi Alexander, > > my name is Anel, and I'm working for Foundation. > > I would like to take/ https://jira.mariadb.org/browse/MDEV-18092/ if > > you allow me, and would appreciate any hint in order to learn more about > > this codebase :). > > Nice to meet you! Thanks for working on this! > > > I debugged already a bit and have seen that there is parser error, > > where /*parser_state = m_buf = 0x7fff6403ec90 "CREATE DEFINER=\"\" > > PACKAGE BODY \"db1\".\"employee_tools\" BEGIN END"/ > > is raised because the incorrect body of /Sp_handler_package_spec > > /and/or /Sp_handler_package_body /I suppose ? > > (/empty_body_lex_cstring/() for *Sp_handler_package_spec *and > > *Sp_handler_package_body*), > > > > I tried to change the body of the above methods, but either my syntax is > > not good ("/AS BEGIN NULL; END/") or this is incorrect conclusion :). > > > It seems Sp_handler::sp_load_for_information_schema() > creates a wrong CREATE statement that cannot be parsed. > > This should be fixed. sp_load_for_information_schema() is > supposed to build a simplified CREATE statement that can be parsed. > > > But for exactly this problem, it's probably not necessarily. > > > I think store_schema_params() should just skip PACKAGE and PACKAGE BODY > records quickly. > > These stored objects do not have any parameters or return values > (only procedures and functions have). > So no needs to build a CREATE statement and parse it: > this won't give us any data useful for INFORMATION_SCHEMA.PARAMETERS > anyway. > > > > Ideally, store_schema_params() should be turned into virtual > methods in Sp_handler, doing: > - empty job for PACKAGE and PACKAGE BODY > - adding a return data type record for FUNCTION > - adding parameters for FUNCTION and PROCEDURE > > > A quicker solution would be just to change this block: > > if (!sph) > DBUG_RETURN(0); > > > to something like this: > > if (!sph || > sph->type() == TYPE_ENUM_PACKAGE || > sph->type() == TYPE_ENUM_PACKAGE_BODY) > DBUG_RETURN(0); > > > <skip> > > > Thanks in advance and happy holidays, > > Thanks! Same to you! > > > I'll be on holidays until the 10th of January. > Sorry for possible slow replies during these days. > > > > > > Anel >

2 1

Re: [Maria-developers] Expiration of user passwords
by Sergei Golubchik 08 Jan '19

08 Jan '19

Hi, Robert! On Jan 07, Robert Bindar wrote: > Hi! > > With this email I intend to kick off a debate on the design of > password expiration in MariaDB. > Given this feature may impact the end user experience, I would love if > the experienced folks here will step in and help us come up with the > best approach possible. Random thoughts are highly appreciated as > well. > > *-- Some thoughts* > > I have been trying to wrap my head around how password management > should look like in MariaDB, my conclusion so far is that the less > intrusive it is the greater are our chances for the common user to > like it. > > MySQL released the first version of password expiration in 5.6.6 > (~2012, didn't change much since then). From my research, it seems the > common user was caught off-guard by this feature, got annoyed by its > restrictiveness and chose to disable it permanently. I believe we > should avoid this situation as much as possible. > I can see the whole story going, nevertheless, in a different > direction for our business clients. I suspect the password expiration is more useful in satisfying various enterprise security checklists and requirements than in actually improving security. For example, https://cryptosmith.com/password-sanity/exp-harmful/ http://blog.axantum.com/2009/12/password-expiration-is-meaningless.html etc, you can google up as many as you'd like > **-- How MySQL does it* > > This is a short summary of what's in the MySQL docs about the password > management system. I'll have it here so people that are unaware of how > this feature works in MySQL are up to speed. > > MySQL introduced 3 new columns in mysql.user: > > /** *//*password_expired* (Y/N)// > // /** If 'Y', the user's password will be considered expired */// > //** **password_last_changed* (timestamp)// > // /** The datetime when the user changed the password last time */// > //** **password_lifetime* (smallint)// > // /** The number of days until the password expires (it's not clear whether// > // the days are counted since password_last_changed or since when the// > // expiration statement was executed) *// > > Given we support MySQL 5.7, we have to have them as well I believe. > > The /CREATE USER/ and /ALTER USER/ statements were extended as follows:** > > *** /[ALTER|CREATE] USER user@host PASSWORD EXPIRE [option];/ > > - If no option is specified, the password is expired with immediate effect. > - If option is /DEFAULT/, the /*password_lifetime*/ column is set to the > number of days specified in the */default_password_lifetime/* system var. > - If option is /NEVER/, the user password_lifetime column is set to 0, the > password is never expired for this account. > - Option can also be /INTERVAL n DAY/, in which case password_lifetime > is set to /'n'/ (see column definition above). > > *Password expiration behavior* > MySQL controls the effect of an expired password with the > */disconnect_on_expired_password/* system var. > For active sessions, when a password is expired, the server either > disconnects the client or puts it in a sandbox mode. > New client connections can specify an option so the server knows not to > disconnect them, otherwise the new connection is handled according to the > system var above. > In sandbox mode, the client is able to execute only /SET PASSWORD/, > /ALTER USER/ and /SET/ statements for vars. > > *Other additions* > - C library API for specifying disconnect/sandbox option this should be already done, CONC-314. > - mysql client new */--connect-expired-password/* option > * > **-- Things to improve* > > Below are a few points I found so far that I believe should be discussed > before moving on to a prototype implementation: > > *1.* Syntax wise > > Do we want to follow the path of MySQL and extend the /ALTER USER/ and > /CREATE USER/ statements for password expiration support? > *Opinion**|* The /ALTER/CREATE USER/ statements seem to be doing quite a lot > of things already. Introducing a new statement such as > > * * */PASSWORD EXPIRE user@host [option];/ > > may be worth exploring. I think ALTER/CREATE USER is perfectly expected to set or modify various properties of an account. So it's quite fitting to use it here. "PASSWORD EXPIRE" looks strange. First, SQL statements start with a verb. Second "PASSWORD EXPIRE" (or "EXPIRE PASSWORD", if you'd like) means to expire it now, not somewhere in the future. And anyway, I don't see why one would need a new statement to modify just one property of an account, if all other properties are modified with ALTER USER. > *2.* Password Expiration behavior > > Is it actually a good idea to overengineer the feature with > sandbox/disconnect modes? > *Opinion|* Quite a lot of people are complaining on online platforms > that their SQL statements stopped working suddenly or that they can't > connect to the MySQL server anymore. What about we just ask for a new > password when the next login is attempted? We have ways to disconnect > a client connection anyway if needed. A good choice for the default > behavior of this feature might also do the trick. I'd disable password expiration by default, see urls above :) > *3.* Having passwords expire on a specific date > > MySQL introduced the concept of expiring the passwords after /N/ days. > I can see a good use-case in wanting to expire an account's password > on a specific date/timestamp as well: > > /ALTER USER user1@host PASSWORD EXPIRE DATE tstamp;/ > or > /PASSWORD EXPIRE user1@host DATE tstamp;/ > > Example use cases: > * Let's assume company X has a security sanity check on 10th of November > and wants everybody to update their passwords that day. > * Company Y wants the password expiration event to happen before the > working day starts and thus not interrupt their employees during their > productive hours. Now, that is overengineering, in my opinion :) Let's postpone it until users will be requesting it. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] 82d7e2094c8: MDEV-7598 Lock user after too many password errors
by Sergei Golubchik 08 Jan '19

08 Jan '19

Hi, Vladislav! On Jan 04, Vladislav Vaintroub wrote: > revision-id: 82d7e2094c8 (mariadb-10.4.1-41-g82d7e2094c8) > parent(s): c4ec6bb69ec > author: Vladislav Vaintroub <wlad(a)mariadb.com> > committer: Vladislav Vaintroub <wlad(a)mariadb.com> > timestamp: 2019-01-04 10:42:05 +0100 > message: > > MDEV-7598 Lock user after too many password errors COM_CHANGE_USER has its own blocking, that works differently. See change_user_notembedded test. 1. Does it also increment acl_user->password_errors counter now? 2. Should COM_CHANGE_USER be changed to behave more like connect? even the answer to the second question is "no", please, add tests for change_user to document the behavior from the first question. > diff --git a/mysql-test/main/max_password_errors.test b/mysql-test/main/max_password_errors.test > new file mode 100644 > index 00000000000..fbb9e699b49 > --- /dev/null > +++ b/mysql-test/main/max_password_errors.test > @@ -0,0 +1,46 @@ > +--source include/not_embedded.inc > +set @old_max_password_errors=@@max_password_errors; > +set global max_password_errors=2; > +create user u identified by 'good_pass'; > + > +# Test that user is blocked after 'max_password_errors' bad passwords > +--replace_result $MASTER_MYSOCK MASTER_SOCKET $MASTER_MYPORT MASTER_PORT > +error ER_ACCESS_DENIED_ERROR; > +connect(con1, localhost, u, bas_pass); > +--replace_result $MASTER_MYSOCK MASTER_SOCKET $MASTER_MYPORT MASTER_PORT > +error ER_ACCESS_DENIED_ERROR; > +connect (con1, localhost, u, bad_pass); > +--replace_result $MASTER_MYSOCK MASTER_SOCKET $MASTER_MYPORT MASTER_PORT > +error ER_USER_IS_BLOCKED; > +connect(con1, localhost, u, good_pass); add another bad_pass connect here, please, to show what error one will get (ER_ACCESS_DENIED_ERROR or ER_USER_IS_BLOCKED). > +# Test that FLUSH PRIVILEGES clears the error > +FLUSH PRIVILEGES; > +connect (con1, localhost, u, good_pass); > +disconnect con1; > diff --git a/mysql-test/main/max_password_errors_auth_named_pipe.opt b/mysql-test/main/max_password_errors_auth_named_pipe.opt I'd better put the test into the plugins suite. That's where all similar tests are. > diff --git a/mysql-test/main/max_password_errors_auth_socket.opt b/mysql-test/main/max_password_errors_auth_socket.opt same > diff --git a/mysql-test/main/max_password_errors_auth_socket.test b/mysql-test/main/max_password_errors_auth_socket.test > new file mode 100644 > index 00000000000..4f31927b02d > --- /dev/null > +++ b/mysql-test/main/max_password_errors_auth_socket.test > @@ -0,0 +1,20 @@ > +# Tests that max_password_errors has no effect on login errors with > +# passwordless plugins (Windows version / auth_named_pipe) > + > +--source include/not_embedded.inc > +--source include/have_unix_socket.inc > + > +set @old_max_password_errors=@@max_password_errors; > +create user nosuchuser identified with 'unix_socket'; Technically, you need to test that $USER != nosuchuser. Otherwise someone someday will surely run it from a nosuchuser account :) May be just include have_unix_socket.inc after you created nosuchuser, it should do the trick. > diff --git a/mysql-test/main/mysqld--help.result b/mysql-test/main/mysqld--help.result > index 8faf332a7dd..3255927d4a6 100644 > --- a/mysql-test/main/mysqld--help.result > +++ b/mysql-test/main/mysqld--help.result > @@ -546,6 +546,11 @@ The following specify which files/extra groups are read (specified before remain > The maximum BLOB length to send to server from > mysql_send_long_data API. Deprecated option; use > max_allowed_packet instead. > + --max-password-errors=# > + If there is more than this number of failed connect > + attempts due to invalid password, user will be blocked > + from further connections until FLUSH_PRIVILEGES.Value 0 > + disables user blocking 1. Space before "Value" 2. Why 0 disables blocking? max_connect_errors doesn't have a special 0 value. > --max-prepared-stmt-count=# > Maximum number of prepared statements in the server > --max-recursive-iterations[=#] > @@ -1518,6 +1523,7 @@ max-heap-table-size 16777216 > max-join-size 18446744073709551615 > max-length-for-sort-data 1024 > max-long-data-size 16777216 > +max-password-errors 0 > max-prepared-stmt-count 16382 > max-recursive-iterations 18446744073709551615 see? max-recursive-iterations doesn't have a special 0 value, it simply uses maxint. So does max-join-size. > max-relay-log-size 1073741824 > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index 85b54009219..95f92688677 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -12323,6 +12324,23 @@ static bool send_plugin_request_packet(MPVIO_EXT *mpvio, > } > > #ifndef NO_EMBEDDED_ACCESS_CHECKS > + > +/** > + Safeguard to avoid blocking the root, when max_password_errors > + limit is reached. > + > + Currently, we allow password errors for superuser on localhost. > + > + @return true, if password errors should be ignored, and user should not be locked. > +*/ > +static bool ignore_max_password_errors(const ACL_USER *acl_user) > +{ > + const char *host= acl_user->host.hostname; > + return (acl_user->access & SUPER_ACL) > + && (!strcasecmp(host, "localhost") || > + !strcmp(host, "127.0.0.1") || > + !strcmp(host, "::1")); > +} I don't like special cases :( May be after implementing `unix_socket OR mysql_native_password` feature, this special case could be removed? > /** > Finds acl entry in user database for authentication purposes. > > @@ -12341,6 +12359,21 @@ static bool find_mpvio_user(MPVIO_EXT *mpvio) > mysql_mutex_lock(&acl_cache->lock); > > ACL_USER *user= find_user_or_anon(sctx->host, sctx->user, sctx->ip); > + > + if (user && max_password_errors && user->password_errors >= max_password_errors) > + { > + if (ignore_max_password_errors(user)) > + user->password_errors= 0; add a comment here, like // skip a second mutex lock in handle_password_errors() > + else > + { > + mysql_mutex_unlock(&acl_cache->lock); > + my_error(ER_USER_IS_BLOCKED, MYF(0)); > + general_log_print(mpvio->auth_info.thd, COM_CONNECT, > + ER_THD(mpvio->auth_info.thd, ER_USER_IS_BLOCKED)); > + DBUG_RETURN(1); > + } > + } > + > if (user) > mpvio->acl_user= user->copy(mpvio->auth_info.thd->mem_root); > > @@ -13305,6 +13370,8 @@ bool acl_authenticate(THD *thd, uint com_change_user_pkt_len) > break; > case CR_AUTH_USER_CREDENTIALS: > errors.m_authentication= 1; > + if (thd->password && max_password_errors) add && !mpvio->make_it_fail here (see how make_it_fail is set). > + handle_password_errors(acl_user->user.str, acl_user->host.hostname, PASSWD_ERROR_INCREMENT); > break; > case CR_ERROR: > default: Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 2

[Maria-developers] Help request: Contribute to MariaDB 10.3 in Debian efforts
by Otto Kekäläinen 07 Jan '19

07 Jan '19

Hello! I've spent the last 3-4 weeks preparing MariaDB 10.3 for Debian. I am almost done, but there are a bunch of smaller issues I need help with. I would be glad for anybody who can look into the issues and give solution ideas - or even file a merge request on Debian's Gitlab instance where the Debian development happens: https://wiki.debian.org/Teams/MySQL/patches Here is the overall tracker for MariadB 10.3 and it shows various issues: https://tracker.debian.org/pkg/mariadb-10.3 Bugs filed against MariaDB in Debian: https://bugs.debian.org/cgi-bin/pkgreport.cgi?repeatmerged=no&src=mariadb-1… Build status in Debian: https://buildd.debian.org/status/package.php?p=mariadb-10.3 Build status on Launchpad (the system Ubuntu uses): https://launchpad.net/~mysql-ubuntu/+archive/ubuntu/mariadb-10.3/+builds?bu… Build/CI status on Debian Salsa (the development system): https://salsa.debian.org/mariadb-team/mariadb-10.3/-/jobs Git repository for the packaging: https://salsa.debian.org/mariadb-team/mariadb-10.3 Please pick somethng that is close to your expertise and help out if you can! :) - Otto

4 5

Re: [Maria-developers] Audit plugin filter expressions.
by Ralf Gebhardt 02 Jan '19

02 Jan '19

And this definition would be used if we are not using tables for users and filters? As with tables we would not have JSON objects for user and role. Alexey Botchkov <holyfoot(a)mariadb.com> schrieb am Mi. 2. Jan. 2019 um 09:33: > Hi, Ralf, Sergei! > > Here below you can see the proposed syntax of filter expressions > for the Audit Plugin. > Similarly to what we see in MySQL, firstly the 'filter'-s are > defined then they are assigned to 'user'-s and 'role'-s. > Role's filters prevail over user's. > > > Toplevel structure is an JSON object with 3 optional keys: > "filters" - an array of 'filter' definitions > "users" - filters assigned to users. > 'roles' - filters assigned to roles. > > Looks like this: > > { > "filters": [ > { > "name" : "all", > "def" : {filter_definition} > }, > { > "name" : "connect_events", > "def" : {filter_definition} > } > ], > "users" : > [ > { > "user" : "root@%", > "filter" : "connect_events" > }, > { > "user" : "hf(a)%.ru", > "filter" : "all" > }, > ], > "roles" : > [ > { > "role" : "sales", > "filter" : "all" > } > ] > } > > User names syntax is same as the used in server. > When looking the 'users' for filter we scan the array and use the first > matching > 'user' and use it's filter. (don't look for other possible matches) > 'role'-s support no wildcards, so are simpler to match. > > > filter_definition: - and array of definitions > filter_array_item [,filter_array_item ...] > > filter_array_item: > filter_name | > simple_filter > > simple_filter: > { > "do_log" : true|false, - an optional key. Supposed to be TRUE if not > specified. > "event_type : [CONNECT, QUERY, TABLE], > "details" : > { > "connect_type" : ["CONNECT, DISCONNECT, CHANGE_USER, SET_ROLE], > "query_type" : [DDL, DML, DCL, SELECT, INSERT ...], > "table_type" : [READ, WRITE], > "query_tables" : ["db.table", ...], > "table_tables" : ["db.table", ...] > } > } > > So examining an event we have the array of simple_filters to > check. If the event matches the simple_filter, we act as the 'do_log" > tells - write it to log or stop looking for match. > If no matches were found, we don't log the event. > > The example of the filter definition: > { > filter_name: "log_ddl_dcl", > filter_def: > { > "event_type": "QUERY", > "details": { "query_type": ["DDL", "DCL"] } > } > } > > { > filter_name: "log_queries_except_test_db", > filter_def: > [ > { > "do_log": FALSE, > "event_type": "QUERY", > "details": { "query_tables": "test.*" } > }, > { > "event_type": "QUERY" > ] > ] > } > > > > That's basically it. > What would be your opinion? :) > > Regards > HF > > -- Ralf Gebhardt, Product Manager MariaDB Corporation Ab | http://www.mariadb.com | m: +49 152-564-823-67 MARIADB OPENWORKS February 25-27, 2019 · NYC · Register Now mariadb.com/openworks <https://mariadb.com/conference/>

1 0

Re: [Maria-developers] ce4f8955d8d: MDEV-17082 Application period tables: CREATE
by Sergei Golubchik 27 Dec '18

27 Dec '18

Hi, Nikita! Looks good. See few minor comments below On Nov 28, Nikita Malyavin wrote: > revision-id: ce4f8955d8d4a556f8d1a7b271da5f7de8a35c42 (versioning-1.0.7-2-gce4f8955d8d) > parent(s): 9661692d3384e22ea6c5be1a8612feaa9605b0aa > author: Nikita Malyavin <nikitamalyavin(a)gmail.com> > committer: Nikita Malyavin <nikitamalyavin(a)gmail.com> > timestamp: 2018-10-29 15:48:01 +1000 > message: > > MDEV-17082 Application period tables: CREATE > > * add syntax `CREATE TABLE ... PERIOD FOR <apptime>` > * add table period entity > > diff --git a/include/mysql_com.h b/include/mysql_com.h > index 902c0ff2706..5451db88b0b 100644 > --- a/include/mysql_com.h > +++ b/include/mysql_com.h > @@ -203,6 +203,7 @@ enum enum_indicator_type > #define VERS_UPDATE_UNVERSIONED_FLAG (1 << 29) /* column that doesn't support > system versioning when table > itself supports it*/ > +#define EXPLICIT_NULL_FLAG (1 << 30) /* Field explicitly marked as NULL */ Not needed, see below > #define REFRESH_GRANT (1ULL << 0) /* Refresh grant tables */ > #define REFRESH_LOG (1ULL << 1) /* Start on new log file */ > diff --git a/mysql-test/suite/period/r/create.result b/mysql-test/suite/period/r/create.result > new file mode 100644 > index 00000000000..420eceb2e9a > --- /dev/null > +++ b/mysql-test/suite/period/r/create.result > @@ -0,0 +1,88 @@ > +create or replace table t (id int primary key, s date, e date, > +period for mytime(s,e)); > +show create table t; > +Table Create Table > +t CREATE TABLE `t` ( > + `id` int(11) NOT NULL, > + `s` date NOT NULL, > + `e` date NOT NULL, > + PRIMARY KEY (`id`), > + PERIOD FOR `mytime` (`s`, `e`), > + CONSTRAINT `CONSTRAINT_1` CHECK (`s` < `e`) I agree it should be hidden. But I'd prefer you did it in this commit, not in ALTER TABLE commit. You could've just committed it separately and then used interactive rebase to fixup the CREATE TABLE commit. Ok for now, but for the future if you realize your older commit had a deficiency, please fixup the original commit, instead of fixing it post-factum as a part of a later feature commit. As long as the old commit isn't pushed, of course :) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +create or replace table t (id int primary key, s timestamp(6), e timestamp(6), > +period for mytime(s,e)); > +show create table t; > +Table Create Table > +t CREATE TABLE `t` ( > + `id` int(11) NOT NULL, > + `s` timestamp(6) NOT NULL DEFAULT '0000-00-00 00:00:00.000000', > + `e` timestamp(6) NOT NULL DEFAULT '0000-00-00 00:00:00.000000', > + PRIMARY KEY (`id`), > + PERIOD FOR `mytime` (`s`, `e`), > + CONSTRAINT `CONSTRAINT_1` CHECK (`s` < `e`) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +# SQL17 p. 832 > +# 2) If a <table period definition> TPD is specified, then: > +# a) <table scope> shall not be specified. > +create or replace temporary table t (s date, e date, period for mytime(s,e)); > +ERROR HY000: Temporary tables with application period not allowed > +# SQL17 p. 831 > +# The <data type or domain name> contained in CD1 is either DATE or a > +# timestamp type and it is equivalent to the <data type or domain name> > +# contained in CD2. > +create or replace table t (id int primary key, s datetime, e date, > +period for mytime(s,e)); > +ERROR HY000: PERIOD FOR `mytime` fields types mismatch > +create or replace table t (s timestamp(2), e timestamp(6), > +period for mytime(s,e)); > +ERROR HY000: PERIOD FOR `mytime` fields types mismatch > +create or replace table t (id int primary key, s int, e date, > +period for mytime(s,e)); > +ERROR 42000: Incorrect column specifier for column 's' > +create or replace table t (id int primary key, s date, e date, > +period for mytime(s,x)); > +ERROR HY000: Field `x` in PERIOD FOR `mytime` is not found in table > +create or replace table t (id int primary key, s date, e date, > +period for mytime(s,e), > +period for mytime2(s,e)); > +ERROR HY000: cannot specify more than one application-time period > +# SQL17 p. 831 > +# No <column name> in any <column definition> shall be equivalent to PN. > +create or replace table t (mytime int, s date, e date, > +period for mytime(s,e)); > +ERROR HY000: Could not specify name `mytime` for field. It is already used by period. > +# CD1 and CD2 both contain either an explicit or an implicit > +# <column constraint definition> that specifies > +# NOT NULL NOT DEFERRABLE ENFORCED. > +create or replace table t (id int primary key, s date NULL, e date, > +period for mytime(s,e)); > +ERROR HY000: Period field `s` cannot be NULL > +# SQL17 p.832 > +# Neither CD1 nor CD2 shall contain an <identity column specification>, a > +# <generation clause>, a <system time period start column specification>, > +# or a <system time period end column specification>. > +create or replace table t (id int primary key, > +s date, > +e date generated always as (s+1), > +period for mytime(s,e)); > +ERROR HY000: Period field `e` cannot be VIRTUAL or GENERATED > +create or replace table t (id int primary key, > +s date, > +e date as (s+1) VIRTUAL, > +period for mytime(s,e)); > +ERROR HY000: Period field `e` cannot be VIRTUAL or GENERATED > +create or replace table t (id int primary key, s timestamp(6), e timestamp(6), > +st timestamp(6) as row start, > +en timestamp(6) as row end, > +period for system_time (st, en), > +period for mytime(st,e)) with system versioning; > +ERROR HY000: Period field `st` cannot be ROW START > +# SQL17 p.832 > +# Let IDCN be an implementation-dependent <constraint name> that is not > +# equivalent to the <constraint name> of any table constraint descriptor > +# included in S. Not enough context. It was quite difficult to find this quote. When you're quoting the standard, please, specify the part name and the section name. And there's no need to specify a page. For example: SQL2016, Part 2, 11.27 <add table period definition>, General Rules, 2)a) > +create or replace table t (x int, s date, e date, > +period for mytime(s, e), > +constraint mytime check (x > 1)); > +create or replace database test; > diff --git a/sql/handler.h b/sql/handler.h > index 8d244f2784e..69419901810 100644 > --- a/sql/handler.h > +++ b/sql/handler.h > @@ -1906,46 +1906,61 @@ enum vers_sys_type_t > VERS_TRX_ID > }; > > -struct Vers_parse_info > +struct Table_period_info > { > - Vers_parse_info() : > - check_unit(VERS_UNDEFINED), > - versioned_fields(false), > - unversioned_fields(false) > - {} > + Table_period_info() {} > + Table_period_info(const char *name_arg, size_t size) : > + name(name_arg, size) {} > + > + Lex_ident name; > > struct start_end_t > { > - start_end_t() > - {} > + start_end_t() {}; > - start_end_t(LEX_CSTRING _start, LEX_CSTRING _end) : > + start_end_t(const LEX_CSTRING& _start, const LEX_CSTRING& _end) : > start(_start), > end(_end) {} > Lex_ident start; > Lex_ident end; > }; > + start_end_t period; > > - start_end_t system_time; > - start_end_t as_row; > - vers_sys_type_t check_unit; > + bool is_set() const > + { > + DBUG_ASSERT(bool(period.start) == bool(period.end)); > + return period.start; > + } > > - void set_system_time(Lex_ident start, Lex_ident end) > + void set_period(const Lex_ident& start, const Lex_ident& end) > { > - system_time.start= start; > - system_time.end= end; > + period.start= start; > + period.end= end; > } > +}; > + > +struct Vers_parse_info: public Table_period_info > +{ > + Vers_parse_info() : > + Table_period_info("SYSTEM_TIME", sizeof "SYSTEM_TIME" - 1), We have a macro for that, use Table_period_info(STRING_WITH_LEN("SYSTEM_TIME")) > + check_unit(VERS_UNDEFINED), > + versioned_fields(false), > + unversioned_fields(false) > + {} > + > + Table_period_info::start_end_t as_row; > + vers_sys_type_t check_unit; > > protected: > friend struct Table_scope_and_contents_source_st; > void set_start(const LEX_CSTRING field_name) > { > as_row.start= field_name; > - system_time.start= field_name; > + period.start= field_name; > } > void set_end(const LEX_CSTRING field_name) > { > as_row.end= field_name; > - system_time.end= field_name; > + period.end= field_name; > } > bool is_start(const char *name) const; > bool is_end(const char *name) const; > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index b04d869bf1d..4189131f801 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -3026,7 +3026,8 @@ CHARSET_INFO* get_sql_field_charset(Column_definition *sql_field, > by adding the features DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP. > > If the first TIMESTAMP column appears to be nullable, or to have an > - explicit default, or to be a virtual column, then no promition is done. > + explicit default, or to be a virtual column, or to be part of table period, > + then no promition is done. promotion > > @param column_definitions The list of column definitions, in the physical > order in which they appear in the table. > diff --git a/sql/sql_lex.h b/sql/sql_lex.h > index edb409d6354..aec83dc6f6a 100644 > --- a/sql/sql_lex.h > +++ b/sql/sql_lex.h > @@ -4185,6 +4185,30 @@ struct LEX: public Query_tables_list > { > return create_info.vers_info; > } > + > + Table_period_info& get_table_period_info() > + { > + return create_info.period_info; > + } really? :) you've introduced a new method get_table_period_info only to use it two lines below and nowhere else? > + > + int add_period(Lex_ident name, Lex_ident_sys_st start, Lex_ident_sys_st end) > + { > + Table_period_info &info= get_table_period_info(); > + if (info.is_set()) > + { > + my_error(ER_PERIOD_MAX_COUNT_EXCEEDED, MYF(0), "application"); > + return 1; > + } > + info.set_period(start, end); > + info.name= name; > + > + Virtual_column_info *constr= new Virtual_column_info(); > + constr->expr= lt_creator.create(thd, create_item_ident_nosp(thd, &start), > + create_item_ident_nosp(thd, &end)); > + add_constraint(&null_clex_str, constr, false); > + return 0; > + } > + > sp_package *get_sp_package() const; > > /** > diff --git a/sql/table.cc b/sql/table.cc > index ab1a04ccae4..87fa1ba4d61 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -1228,6 +1228,14 @@ static const Type_handler *old_frm_type_handler(uint pack_flag, > } > > > +bool TABLE_SHARE::init_period_from_extra2(period_info_t &period, > + const uchar *data) > +{ > + period.start_fieldno= uint2korr(data); > + period.end_fieldno= uint2korr(data + sizeof(uint16)); not "sizeof(uint16)" but 2. I don't think it's guaranteed that sizeof(uint16) == 2, but as you do uint2korr, you only read 2 bytes, so data + 2 is always correct. > + return period.start_fieldno >= fields || period.end_fieldno >= fields; > +} > + > /** > Read data from a binary .frm file image into a TABLE_SHARE > > @@ -1773,27 +1781,36 @@ int TABLE_SHARE::init_from_binary_frm_image(THD *thd, bool write, > } > > /* Set system versioning information. */ > + vers.name= Lex_ident(STRING_WITH_LEN("SYSTEM_TIME")); > if (extra2.system_period == NULL) > { > versioned= VERS_UNDEFINED; > - row_start_field= 0; > - row_end_field= 0; > + vers.start_fieldno= 0; > + vers.end_fieldno= 0; > } > else > { > DBUG_PRINT("info", ("Setting system versioning informations")); > - uint16 row_start= uint2korr(extra2.system_period); > - uint16 row_end= uint2korr(extra2.system_period + sizeof(uint16)); > - if (row_start >= share->fields || row_end >= share->fields) > + if (init_period_from_extra2(vers, extra2.system_period)) > goto err; > - DBUG_PRINT("info", ("Columns with system versioning: [%d, %d]", row_start, row_end)); > + DBUG_PRINT("info", ("Columns with system versioning: [%d, %d]", > + vers.start_fieldno, vers.end_fieldno)); > versioned= VERS_TIMESTAMP; > vers_can_native= plugin_hton(se_plugin)->flags & HTON_NATIVE_SYS_VERSIONING; > - row_start_field= row_start; > - row_end_field= row_end; > status_var_increment(thd->status_var.feature_system_versioning); > } // if (system_period == NULL) > > + if (extra2.application_period.str) > + { > + period.name.length= extra2.application_period.length - 2*sizeof(uint16); same > + period.name.str= strmake_root(&mem_root, > + (char*)extra2.application_period.str, > + period.name.length); > + const uchar *field_pos= extra2.application_period.str + period.name.length; > + if (init_period_from_extra2(period, field_pos)) > + goto err; > + } > + > for (i=0 ; i < share->fields; i++, strpos+=field_pack_length, field_ptr++) > { > uint interval_nr= 0, recpos; > diff --git a/sql/unireg.cc b/sql/unireg.cc > index 4692b2d74d1..ca7e005455d 100644 > --- a/sql/unireg.cc > +++ b/sql/unireg.cc > @@ -176,6 +170,7 @@ LEX_CUSTRING build_frm_image(THD *thd, const LEX_CSTRING *table, > ulong data_offset; > uint options_len; > uint gis_extra2_len= 0; > + uint period_info_len= create_info->period_info.name.length + 2*sizeof(uint16); I won't comment on it any longer, but please everywhere s/sizeof(uint16)/2/g > uchar fileinfo[FRM_HEADER_SIZE],forminfo[FRM_FORMINFO_SIZE]; > const partition_info *part_info= IF_PARTITIONING(thd->work_part_info, 0); > bool error; > diff --git a/sql/handler.cc b/sql/handler.cc > index 2ced4e959a2..a4c1684ccb8 100644 > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -7352,3 +7352,113 @@ bool Vers_parse_info::check_sys_fields(const Lex_table_name &table_name, > "ROW END" : found_flag ? "ROW START" : "ROW START/END"); > return true; > } > + > +static bool check_period_field(const Create_field* f, const char* name, > + const char* period_name) > +{ > + bool res= false; > + if (!f) > + { > + my_error(ER_PERIOD_FIELD_NOT_FOUND, MYF(0), name, period_name); > + res= true; > + } > + else if (f->type_handler()->mysql_timestamp_type() == MYSQL_TIMESTAMP_ERROR) > + { > + my_error(ER_WRONG_FIELD_SPEC, MYF(0), name); > + res= true; > + } > + else if (f->vcol_info) > + { > + my_error(ER_PERIOD_FIELD_WRONG_ATTRIBUTES, MYF(0), > + f->field_name.str, "VIRTUAL or GENERATED"); just use "GENERATED", no English in the error message parameters, please. > + res= true; > + } > + else if (f->flags & EXPLICIT_NULL_FLAG) > + { > + my_error(ER_PERIOD_FIELD_WRONG_ATTRIBUTES, MYF(0), > + f->field_name.str, "NULL"); why is that? it's unrelated and inconsistent with cases like create table t1 (a int null, primary key (a)); where a column is silently created NOT NULL. So, you should ignore explicit NULL and create columns NOT NULL. If you want to change this, it should be consistent in all cases. create an MDEV, suggest there that the statement above should be an error. but I don't think it's worth spending time on now. > + res= true; > + } > + else if (f->flags & VERS_SYSTEM_FIELD) > + { > + my_error(ER_PERIOD_FIELD_WRONG_ATTRIBUTES, MYF(0), f->field_name.str, > + f->flags & VERS_SYS_START_FLAG ? "ROW START" : "ROW END"); No need to have a special case for that. Use the same error as for vcol_info: if (f->vcol_info || f->flags & VERS_SYSTEM_FIELD) { my_error(ER_PERIOD_FIELD_WRONG_ATTRIBUTES, MYF(0), f->field_name.str, "GENERATED ALWAYS AS"); > + res= true; > + } > + > + return res; > +} > + > +bool Table_scope_and_contents_source_st::check_fields( > + THD *thd, Alter_info *alter_info, TABLE_LIST &create_table) > +{ > + bool res= vers_check_system_fields(thd, alter_info, create_table); > + if (res) > + return true; > + > + if (!period_info.name) > + return false; > + > + if (tmp_table()) > + { > + my_error(ER_PERIOD_TEMPORARY_NOT_ALLOWED, MYF(0)); > + return true; > + } > + > + Table_period_info::start_end_t &period= period_info.period; > + const Create_field *row_start= NULL; > + const Create_field *row_end= NULL; > + List_iterator<Create_field> it(alter_info->create_list); > + while (const Create_field *f= it++) > + { > + if (period.start.streq(f->field_name)) row_start= f; > + else if (period.end.streq(f->field_name)) row_end= f; > + > + if (period_info.name.streq(f->field_name)) > + { > + // TODO this stub should be removed by MDEV-16976 > + my_error(ER_PERIOD_NAME_IS_NOT_ALLOWED_FOR_FIELD, MYF(0), f->field_name.str); > + return true; What does this error have to do with MDEV-16976? SQL standard clearly says (part 2, 11.3 <table definition>, syntax rules, 2)d) d) No <column name> in any <column definition> shall be equivalent to PN. (here PN is "period name"). So MDEV-16976 or not, a period name should never match a column name. > + } > + } > + > + res= check_period_field(row_start, period.start.str, period_info.name.str); > + res= res || check_period_field(row_end, period.end.str, period_info.name.str); > + if (res) > + return true; > + > + if (row_start->type_handler() != row_end->type_handler() > + || row_start->length != row_end->length) > + { > + my_error(ER_PERIOD_TYPES_MISMATCH, MYF(0), period_info.name.str); > + res= true; > + } > + > + return res; > +} > + > +bool > +Table_scope_and_contents_source_st::fix_create_fields(THD *thd, > + Alter_info *alter_info, > + const TABLE_LIST &create_table, > + bool create_select) > +{ > + if (vers_fix_system_fields(thd, alter_info, create_table, create_select)) > + return true; > + > + if (!period_info.name) > + return false; > + > + Table_period_info::start_end_t &period= period_info.period; > + List_iterator<Create_field> it(alter_info->create_list); > + while (Create_field *f= it++) > + { > + if (period.start.streq(f->field_name) || period.end.streq(f->field_name)) > + { > + f->period= &period_info; > + if (!(f->flags & EXPLICIT_NULL_FLAG)) > + f->flags|= NOT_NULL_FLAG; > + } > + } > + return false; > +} > diff --git a/sql/share/errmsg-utf8.txt b/sql/share/errmsg-utf8.txt > index 121eb0a0726..ceb3a58a427 100644 > --- a/sql/share/errmsg-utf8.txt > +++ b/sql/share/errmsg-utf8.txt > @@ -7924,3 +7924,24 @@ ER_ALTER_OPERATION_TABLE_OPTIONS_NEED_REBUILD > > ER_TRUNCATE_ILLEGAL_VERS > eng "Cannot truncate a versioned table" > + > +ER_PERIOD_TEMPORARY_NOT_ALLOWED > + eng "Temporary tables with application period not allowed" "application time period" > + > +ER_PERIOD_FIELD_NOT_FOUND > + eng "Field %`s in PERIOD FOR %`s is not found in table" I don't think this needs a new error message, just use ER_BAD_FIELD_ERROR > + > +ER_PERIOD_TYPES_MISMATCH > + eng "PERIOD FOR %`s fields types mismatch" > + > +ER_PERIOD_MAX_COUNT_EXCEEDED better say ER_MORE_THAN_ONE_PERIOD or something... "max count exceeded" suggests there is some not-trivial (and possibly variable) max count Or ER_PERIOD_ALREADY_DEFINED or ER_DUPLICATE_PERIOD etc. "already defined" looks like a good one. And I've already commented (in another patch) on using English words as arguments - it will look like a bug when the error message is translated > + eng "cannot specify more than one %s-time period" > + > +ER_PERIOD_NAME_IS_NOT_ALLOWED_FOR_FIELD > + eng "Could not specify name %`s for field. It is already used by period." I don't think this needs a new error message, just use ER_DUP_FIELDNAME > + > +ER_PERIOD_FIELD_WRONG_ATTRIBUTES > + eng "Period field %`s cannot be %s" > + > +ER_PERIOD_FIELD_HOLDS_MORE_THAN_ONE_PERIOD > + eng "Field %`s is specified for more than one period" This looks unused Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] Fwd: bc7d600a27f: MDEV-15951 system versioning by trx id doesn't work with partitioning
by Nikita Malyavin 20 Dec '18

20 Dec '18

---------- Forwarded message --------- From: Nikita Malyavin <nikitamalyavin(a)gmail.com> Date: пт, 21 дек. 2018 г. в 04:52 Subject: Re: bc7d600a27f: MDEV-15951 system versioning by trx id doesn't work with partitioning To: Sergei Golubchik <serg(a)mariadb.org> вт, 13 нояб. 2018 г. в 04:43, Sergei Golubchik <serg(a)mariadb.org>: > > Hi, Nikita! > Hi, Sergei! I've updated the tests with respect to issues You mentioned. For the rest i'll give my answers. > > -create table t1 (x int) > > +create or replace table t1 (x int, ROW_START, ROW_END, period for system_time(row_start, row_end)) > > ouch. that looks very weird. please, only replace the type, not the > whole column definition. Like everywhere else: > > --replace_result $sys_datatype_expl SYS_DATATYPE > Fixed it, thanks. Did not notice the weirdness in test results reading. > if you have tests with innodb and trx_id, please, put them in a separate > test file, say, partition_innodb.test. partition.test is for common > tests that are repeated three times, with myisam, innodb, timestamps, > and trx_id. There is no need to repeat your test three times. Did it, though i have an opinion that it makes tests run slower at overall. Every test case adds quite a noticeable lag, more than several test(s) runs. > > diff --git a/mysql-test/suite/versioning/t/partition.combinations b/mysql-test/suite/versioning/t/partition.combinations > > index 4d73ef5a5ea..561c5656929 100644 > > --- a/mysql-test/suite/versioning/t/partition.combinations > > +++ b/mysql-test/suite/versioning/t/partition.combinations > > @@ -1,5 +1,8 @@ > > [timestamp] > > default-storage-engine=innodb > > > > +[trx_id] > > +default-storage-engine=innodb > > + > > [myisam] > > default-storage-engine=myisam > > But then you don't need partition.combinations at all, just add > > --source engines.inc > > like other tests do. > -- Fixed > > @@ -4938,6 +4938,7 @@ int ha_create_table(THD *thd, const char *path, > > !create_info->tmp_table(); > > > > share.frm_image= frm; > > + share.orig_table_name= table_name; > > why? if orig_table_name is NULL, error_table_name() will use table_name, > so there's never need to make orig_table_name==table_name. > > and I don't understand all your other manipulations with > orig_table_name. Could you please explain that? > It affects ALTER TABLE error messages. table_name will contain temporary name, and error_table_name() will output "(temporary)", which is rather inconvenient. AFAICS, orig_table_name purpose is exactly for fixing that, but some corner cases were not covered to this moment. > > > > // open an frm image > > if (share.init_from_binary_frm_image(thd, write_frm_now, > > @@ -6952,29 +6953,6 @@ bool Vers_parse_info::fix_implicit(THD *thd, Alter_info *alter_info) > > return false; > > } > > > > -bool Table_scope_and_contents_source_st::vers_native(THD *thd) const > > -{ > > - if (ha_check_storage_engine_flag(db_type, HTON_NATIVE_SYS_VERSIONING)) > > - return true; > > - > > -#ifdef WITH_PARTITION_STORAGE_ENGINE > > - partition_info *info= thd->work_part_info; > > - if (info && !(used_fields & HA_CREATE_USED_ENGINE)) > > - { > > - if (handlerton *hton= info->default_engine_type) > > - return ha_check_storage_engine_flag(hton, HTON_NATIVE_SYS_VERSIONING); > > - > > - List_iterator_fast<partition_element> it(info->partitions); > > - while (partition_element *partition_element= it++) > > - { > > - if (partition_element->find_engine_flag(HTON_NATIVE_SYS_VERSIONING)) > > - return true; > > - } > > - } > > -#endif > > - return false; > > -} > > what was wrong with that (and all other changed code in this file) ? > Mostly I'm ok with handler.cc code 🙂 except it is mixed with various classes implementations. First, find_engine_flag is not the correct function here, as it says "if there's _any_ node in a tree with corresponding flag". We should check _all_ nodes instead of _any_. Second, we need to make this check at TABLE_SHARE::init_from_binary_frm_image to set TABLE_SHARE::versioned properly and handle errors in place. That's why I've introduced handler::vers_can native [a nice thing about it is that wrote it first and then found same-named share's field used exactly where it's needed, just with lack of tree traversal]. But it makes Table_scope_and_contents_source_st::vers_native redundant, as well as all related code in handler.cc. Other important thing is that TABLE_SHARE::vers_can_native value is now also correct with respect to partition tree, but i have not it used anywhere. I have updated my pull request at github. I don't know how it keeps in sync with mailing lists, so here's a link: https://github.com/MariaDB/server/pull/870. Also rebased to latest 10.3 for convenience Regards! Nikita Malyavin Tempesta Technologies https://github.com/tempesta-tech

1 0

Re: [Maria-developers] d282f5c5560: MDEV-10963 Fragmented BINLOG query
by Sergei Golubchik 18 Dec '18

18 Dec '18

Hi, Andrei! Looks better! There are no major problems, but see comments below. There're few suggestions how to simplify the code. On Nov 05, Andrei Elkin wrote: > revision-id: d282f5c55609469cd74d7390f70c7d922c778711 (mariadb-10.1.35-93-gd282f5c5560) > parent(s): 2a576f71c5d3c7aacef564e5b1251f83bde48f51 > author: Andrei Elkin <andrei.elkin(a)mariadb.com> > committer: Andrei Elkin <andrei.elkin(a)mariadb.com> > timestamp: 2018-10-21 23:42:00 +0300 > message: > > MDEV-10963 Fragmented BINLOG query > > diff --git a/mysql-test/suite/binlog/t/binlog_mysqlbinlog_row_frag.test b/mysql-test/suite/binlog/t/binlog_mysqlbinlog_row_frag.test > new file mode 100644 > index 00000000000..bdf41c94c76 > --- /dev/null > +++ b/mysql-test/suite/binlog/t/binlog_mysqlbinlog_row_frag.test > @@ -0,0 +1,50 @@ > +--source include/have_debug.inc > +--source include/have_log_bin.inc > +--source include/have_binlog_format_row.inc you don't need to include have_log_bin, if you include have_binlog_format_row. > + > +--let $MYSQLD_DATADIR= `select @@datadir` > +--let $max_size=1024 > + > +CREATE TABLE t (a TEXT); > +# events of interest are guaranteed to stay in 000001 log > +RESET MASTER; > +--eval INSERT INTO t SET a=repeat('a', $max_size) eh? why did you do it with let/eval instead of a simple sql statement? you don't use $max_size anywhere else. > +SELECT a from t into @a; > +FLUSH LOGS; > +DELETE FROM t; > + > +--exec $MYSQL_BINLOG --debug-binlog-row-event-max-encoded-size=256 $MYSQLD_DATADIR/master-bin.000001 > $MYSQLTEST_VARDIR/tmp/mysqlbinlog.sql > + > +--let $assert_text= BINLOG is fragmented > +--let $assert_select= BINLOG @binlog_fragment_0, @binlog_fragment_1 > +--let $assert_count= 1 > +--let $assert_file= $MYSQLTEST_VARDIR/tmp/mysqlbinlog.sql > +--source include/assert_grep.inc no, please, use search_pattern_in_file.inc instead. > + > +--exec $MYSQL test < $MYSQLTEST_VARDIR/tmp/mysqlbinlog.sql > + > +SELECT a LIKE @a as 'true' FROM t; > +SELECT @binlog_fragment_0, @binlog_fragment_1 as 'NULL'; that makes no sense, @binlog_fragment_0 and _1 were set in a separate client session. You cannot test whether they were cleared or not there, by looking at the values here > + > +# improper syntax error > +--echo BINLOG number-of-fragments must be exactly two > +--error ER_PARSE_ERROR > +BINLOG @binlog_fragment; > +--error ER_PARSE_ERROR > +BINLOG @binlog_fragment, @binlog_fragment, @binlog_fragment; > + > +# corrupted fragments error check (to the expected error code notice, > +# the same error code occurs in a similar unfragmented case) > +SET @binlog_fragment_0='012345'; > +SET @binlog_fragment_1='012345'; > +--error ER_SYNTAX_ERROR > +BINLOG @binlog_fragment_0, @binlog_fragment_1; > + > +# Not existing fragment is not allowed > +SET @binlog_fragment_0='012345'; > +--error ER_WRONG_TYPE_FOR_VAR > +BINLOG @binlog_fragment_0, @binlog_fragment_not_exist; > + > +--echo # Cleanup > +--remove_file $MYSQLTEST_VARDIR/tmp/mysqlbinlog.sql > +DROP TABLE t; > diff --git a/mysys/mf_iocache2.c b/mysys/mf_iocache2.c > --- a/mysys/mf_iocache2.c > +++ b/mysys/mf_iocache2.c > @@ -22,52 +22,53 @@ > #include <stdarg.h> > #include <m_ctype.h> > > -/* > - Copy contents of an IO_CACHE to a file. > - > - SYNOPSIS > - my_b_copy_to_file() > - cache IO_CACHE to copy from > - file File to copy to > - > - DESCRIPTION > - Copy the contents of the cache to the file. The cache will be > - re-inited to a read cache and will read from the beginning of the > - cache. > - > - If a failure to write fully occurs, the cache is only copied > - partially. > +/** > + Copy the cache to the file. Copying can be constrained to @c count > + number of bytes when the parameter is less than SIZE_T_MAX. The > + cache will be optionally re-inited to a read cache and will read > + from the beginning of the cache. If a failure to write fully > + occurs, the cache is only copied partially. > > TODO > - Make this function solid by handling partial reads from the cache > - in a correct manner: it should be atomic. > - > - RETURN VALUE > - 0 All OK > - 1 An error occurred > + Make this function solid by handling partial reads from the cache > + in a correct manner: it should be atomic. > + > + @param cache IO_CACHE to copy from > + @param file File to copy to > + @param do_reinit whether to turn the cache to read mode > + @param count the copied size or the max of the type > + when the whole cache is to be copied. > + @return > + 0 All OK > + 1 An error occurred > */ > int > -my_b_copy_to_file(IO_CACHE *cache, FILE *file) > +my_b_copy_to_file(IO_CACHE *cache, FILE *file, > + my_bool do_reinit, > + size_t count) > { > - size_t bytes_in_cache; > + size_t curr_write, bytes_in_cache; > DBUG_ENTER("my_b_copy_to_file"); > > /* Reinit the cache to read from the beginning of the cache */ > - if (reinit_io_cache(cache, READ_CACHE, 0L, FALSE, FALSE)) > + if (do_reinit && reinit_io_cache(cache, READ_CACHE, 0L, FALSE, FALSE)) generally, when there's a function that is always called with a constant (compile-time) argument, I prefer to split the code compile-time too, if it isn't too much trouble. In this case it would mean a new function like int my_b_copy_all_to_file(IO_CACHE *cache, FILE *file) { if (reinit_io_cache(cache, READ_CACHE, 0L, FALSE, FALSE) return 1; return my_b_copy_to_file(cache, file, SIZE_T_MAX); } and all old code will be changed to use my_b_copy_all_to_file(). Old my_b_copy_to_file() won't need to do reinit_io_cache() anymore and your code will use it directly. > DBUG_RETURN(1); > bytes_in_cache= my_b_bytes_in_cache(cache); > do > { > - if (my_fwrite(file, cache->read_pos, bytes_in_cache, > + curr_write= MY_MIN(bytes_in_cache, count); > + if (my_fwrite(file, cache->read_pos, curr_write, > MYF(MY_WME | MY_NABP)) == (size_t) -1) > DBUG_RETURN(1); > - } while ((bytes_in_cache= my_b_fill(cache))); > + > + cache->read_pos += curr_write; > + count -= curr_write; > + } while (count && (bytes_in_cache= my_b_fill(cache))); > if(cache->error == -1) > DBUG_RETURN(1); > DBUG_RETURN(0); > } > > - > my_off_t my_b_append_tell(IO_CACHE* info) > { > /* > diff --git a/sql/log_event.cc b/sql/log_event.cc > index e07b7002398..aeca794f0cd 100644 > --- a/sql/log_event.cc > +++ b/sql/log_event.cc > @@ -10474,12 +10488,151 @@ void Rows_log_event::pack_info(Protocol *protocol) > #endif > > #ifdef MYSQL_CLIENT > +/** > + Print an event "body" cache to @c file possibly in multiple fragements. > + Each fragement is optionally per @c do_wrap to procude an SQL statement. > + > + @param file a file to print to > + @param body the "body" IO_CACHE of event > + @param do_wrap whether to wrap base64-encoded strings with > + SQL cover. > + The function signals on any error through setting @c body->error to -1. > +*/ > +void copy_cache_to_file_wrapped(FILE *file, > + IO_CACHE *body, > + bool do_wrap, > + const char *delimiter) > +{ > + uint n_frag= 1; > + const char* before_frag= NULL; > + char* after_frag= NULL; > + char* after_last= NULL; > + /* > + 2 fragments can always represent near 1GB row-based > + base64-encoded event as two strings each of size less than > + max(max_allowed_packet). Greater number of fragments does not > + save from potential need to tweak (increase) @@max_allowed_packet > + before to process the fragments. So 2 is safe and enough. > + */ > + const char fmt_last_frag2[]= > + "\nBINLOG @binlog_fragment_0, @binlog_fragment_1%s\n"; > + const char fmt_before_frag[]= "\nSET /* ONE_SHOT */ @binlog_fragment_%d ='\n"; this ONE_SHOT is confusing, even if in a comment. Better not to do it :) > + /* > + Buffer to hold computed formatted strings according to specifiers. > + The sizes may depend on an actual fragment number size in terms of decimal > + signs so its maximum is estimated (not precisely yet safely) below. > + */ > + char buf[sizeof(fmt_before_frag) + sizeof(fmt_last_frag2) > + + ((sizeof(n_frag) * 8)/3 + 1) // max of decimal index > + + sizeof(PRINT_EVENT_INFO::max_delimiter_len) + 3]; // delim, \n and 0 sizeof(max_delimiter_len) ? it's sizeof(uint), right? Did you mean sizeof(PRINT_EVENT_INFO::delimiter) or simply PRINT_EVENT_INFO::max_delimiter_len without sizeof? > + > + if (do_wrap) > + { > + after_frag= (char*) my_malloc(sizeof(buf), MYF(MY_WME)); > + sprintf(after_frag, "'%s\n", delimiter); > + if (my_b_tell(body) > opt_binlog_rows_event_max_encoded_size) > + n_frag= 2; > + if (n_frag > 1) > + { > + before_frag= fmt_before_frag; > + after_last= (char*) my_malloc(sizeof(buf), MYF(MY_WME)); > + sprintf(after_last, fmt_last_frag2, (char*) delimiter); > + } > + else > + { > + before_frag= "\nBINLOG '\n"; // single "fragment" > + } > + } > + > + size_t total_size= my_b_tell(body), total_written= 0; > + size_t frag_size= total_size / n_frag + 1, curr_size; > + > + if (reinit_io_cache(body, READ_CACHE, 0L, FALSE, FALSE)) > + { > + body->error= -1; > + goto err; > + } > + > + for (uint i= 0; i < n_frag; i++, total_written += curr_size) > + { > + curr_size= i < n_frag - 1 ? frag_size : total_size - total_written; > + > + DBUG_ASSERT(i < n_frag - 1 || curr_size <= frag_size); > + > + if (before_frag) > + { > + sprintf(buf, before_frag, i); > + my_fwrite(file, (uchar*) buf, strlen(buf), MYF(MY_WME | MY_NABP)); > + } > + if (my_b_copy_to_file(body, file, FALSE, curr_size)) > + { > + body->error= -1; > + goto err; > + } > + if (after_frag) > + { > + sprintf(buf, after_frag, NULL); > + my_fwrite(file, (uchar*) buf, strlen(buf), MYF(MY_WME | MY_NABP)); > + } > + } Hmm, dunno. I suspect you can do it three times shorter and five times easier to read if you wouldn't try to generalize it for a arbitrary number of fragments with arbitrary prefixes and suffixes. Just if (my_b_tell(body) < opt_binlog_rows_event_max_encoded_size - margin) { my_fprintf(file, "BINLOG '"); my_b_copy_to_file(body, file); my_fprintf(file, "'%s\n", delimiter); } else { my_fprintf(file, "SET @binlog_fragment_0='"); my_b_copy_to_file(body, file, opt_binlog_rows_event_max_encoded_size); my_fprintf(file, "'%s\nSET @binlog_fragment_1='", delimiter); my_b_copy_to_file(body, file, SIZE_T_MAX); my_fprintf(file, "'%s\nBINLOG @binlog_fragment_0, @binlog_fragment_1%s\n", delimiter, delimiter); } See? > + > + if (after_last) > + { > + sprintf(buf, after_last, n_frag); > + my_fwrite(file, (uchar*) buf, strlen(buf), MYF(MY_WME | MY_NABP)); > + } > + reinit_io_cache(body, WRITE_CACHE, 0, FALSE, TRUE); > + > +err: > + my_free(after_frag); > + my_free(after_last); > +} > + > +/** > + The function invokes base64 encoder to run on the current > + event string and store the result into two caches. > + When the event ends the current statement the caches are is copied into > + the argument file. > + Copying is also concerned how to wrap the event, specifically to produce > + a valid SQL syntax. > + When the encoded data size is within max(MAX_ALLOWED_PACKET) > + a regular BINLOG query is composed. Otherwise it is build as fragmented > + > + SET @binlog_fragment_0='...'; > + SET @binlog_fragment_1='...'; > + BINLOG DEFRAGMENT(@binlog_fragment_0, @binlog_fragment_1); > + > + where fragments are represented by a sequence of "indexed" user > + variables. > + Two more statements are composed as well > + > + SET @binlog_fragment_0=NULL; > + SET @binlog_fragment_1=NULL; > + > + to promptly release memory. No, they aren't > + > + NOTE. @note > + If any changes made don't forget to duplicate them to > + Old_rows_log_event as long as it's supported. > + > + @param file pointer to IO_CACHE > + @param print_event_info pointer to print_event_info specializing > + what out of and how to print the event > + @param name the name of a table that the event operates on > + > + The function signals on any error of cache access through setting > + that cache's @c error to -1. > +*/ > void Rows_log_event::print_helper(FILE *file, > PRINT_EVENT_INFO *print_event_info, > char const *const name) > { > IO_CACHE *const head= &print_event_info->head_cache; > IO_CACHE *const body= &print_event_info->body_cache; > + bool do_print_encoded= > + print_event_info->base64_output_mode != BASE64_OUTPUT_DECODE_ROWS && > + !print_event_info->short_form; > + > if (!print_event_info->short_form) > { > bool const last_stmt_event= get_flags(STMT_END_F); > diff --git a/sql/log_event.h b/sql/log_event.h > index 90900f63533..28277e659d2 100644 > --- a/sql/log_event.h > +++ b/sql/log_event.h > @@ -749,6 +749,7 @@ typedef struct st_print_event_info > that was printed. We cache these so that we don't have to print > them if they are unchanged. > */ > + static const uint max_delimiter_len= 16; why did you introduce this max_delimiter_len, if all you use is sizeof(delimiter) anyway? (and even that is not needed) > // TODO: have the last catalog here ?? > char db[FN_REFLEN+1]; // TODO: make this a LEX_STRING when thd->db is > bool flags2_inited; > @@ -798,7 +799,7 @@ typedef struct st_print_event_info > bool printed_fd_event; > my_off_t hexdump_from; > uint8 common_header_len; > - char delimiter[16]; > + char delimiter[max_delimiter_len]; > > uint verbose; > table_mapping m_table_map; > diff --git a/sql/sql_binlog.cc b/sql/sql_binlog.cc > index 91cf038907e..b4e3342d8f3 100644 > --- a/sql/sql_binlog.cc > +++ b/sql/sql_binlog.cc > @@ -28,6 +28,70 @@ > // START_EVENT_V3, > // Log_event_type, > // Log_event > + > +/** > + Copy fragments into the standard placeholder thd->lex->comment.str. > + > + Compute the size of the (still) encoded total, > + allocate and then copy fragments one after another. > + The size can exceed max(max_allowed_packet) which is not a > + problem as no String instance is created off this char array. > + > + @param thd THD handle > + @return > + 0 at success, > + -1 otherwise. > +*/ > +int binlog_defragment(THD *thd) > +{ > + user_var_entry *entry[2]; > + LEX_STRING name[2]= { thd->lex->comment, thd->lex->ident }; > + > + /* compute the total size */ > + thd->lex->comment.str= NULL; > + thd->lex->comment.length= 0; > + for (uint k= 0; k < 2; k++) > + { > + entry[k]= > + (user_var_entry*) my_hash_search(&thd->user_vars, (uchar*) name[k].str, > + name[k].length); > + if (!entry[k] || entry[k]->type != STRING_RESULT) > + { > + my_error(ER_WRONG_TYPE_FOR_VAR, MYF(0), name[k].str); > + return -1; > + } > + thd->lex->comment.length += entry[k]->length; > + } > + > + thd->lex->comment.str= // to be freed by the caller > + (char *) my_malloc(thd->lex->comment.length, MYF(MY_WME)); > + if (!thd->lex->comment.str) > + { > + my_error(ER_OUTOFMEMORY, MYF(ME_FATALERROR), 1); > + return -1; > + } > + > + /* fragments are merged into allocated buf while the user var:s get reset */ > + size_t gathered_length= 0; > + for (uint k=0; k < 2; k++) > + { > + memcpy(thd->lex->comment.str + gathered_length, entry[k]->value, entry[k]->length); > + gathered_length += entry[k]->length; > + if (update_hash(entry[k], true, NULL, 0, STRING_RESULT, &my_charset_bin, 0)) > + { > + my_printf_error(ER_WRONG_TYPE_FOR_VAR, > + "%s: BINLOG fragment user " > + "variable '%s' could not be unset", MYF(0), > + ER_THD(thd, ER_WRONG_TYPE_FOR_VAR), entry[k]->value); > + } I don't see how update_hash(entry[k], true, ...) can ever fail, so there's no need to pretend that it can. > + } > + > + DBUG_ASSERT(gathered_length == thd->lex->comment.length); > + > + return 0; > +} > + > + > /** > Execute a BINLOG statement. > > @@ -119,6 +175,23 @@ void mysql_client_binlog_statement(THD* thd) > rli->sql_driver_thd= thd; > rli->no_storage= TRUE; > > + if (unlikely(is_fragmented= thd->lex->comment.str && thd->lex->ident.str)) > + if (binlog_defragment(thd)) > + goto end; > + > + if (!(coded_len= thd->lex->comment.length)) > + { > + my_error(ER_SYNTAX_ERROR, MYF(0)); > + goto end; > + } > + > + decoded_len= base64_needed_decoded_length(coded_len); > + if (!(buf= (char *) my_malloc(decoded_len, MYF(MY_WME)))) > + { > + my_error(ER_OUTOFMEMORY, MYF(ME_FATALERROR), 1); > + goto end; > + } > + Technically, it should be possible to decode base64 in-place and avoid allocating a second 3GB buffer. But let's not do it in this MDEV :) > for (char const *strptr= thd->lex->comment.str ; > strptr < thd->lex->comment.str + thd->lex->comment.length ; ) > { Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 2

[Maria-developers] New Foundation Buildbot and Github build status push
by Teodor Mircea Ionita 18 Dec '18

18 Dec '18

Hi all, For the past 6 months we have been working on a new Buildbot deployment at: https://buildbot.mariadb.org Currently we have docker x64 builders for all supported Linux platforms (aptly named quick builders, since they all finish a build + default MTR under 40 minutes): https://buildbot.mariadb.org/#/builders We are working on adding builders for all the other supported targets in the near future as well as acquiring more building power to accommodate for --big and more extensive/complex tests. We are open to donations in that regard. Saved package artifacts can be accessed from any individual build log via the "Download" link or directly from here: https://ci.mariadb.org The configuration and associated documentation can be found in the tools repo: https://github.com/MariaDB/mariadb.org-tools/tree/master/buildbot.mariadb.o… We encourage collaboration and are open to pull requests, you can do so for adding new builders, fixing build steps or adjusting the docker images available in the dockerfiles/ sub-directory, etc.. Currently we consider this deployment in beta phase and have enabled Github status push for the main repo for direct code pushes as well as pull requests. The build status shows up in the same Github widget as Travis does and you can follow any individual report to the actual buildbot log via the links. Sample report for a push: https://github.com/MariaDB/server/commits/bb-10.4-statustest Sample PR: https://github.com/MariaDB/server/pull/1034 Reproducing Linux failures are facilitated by the Dockerfiles available in the tool's repo, which are used by the Buildbot master to create the build environment on the docker enabled workers: https://github.com/MariaDB/mariadb.org-tools/tree/master/buildbot.mariadb.o… Ideally, any developer should pay attention to these build failures on Github and attempt to fix any issues before merging into the main branches. While in beta, we are trying to weed out environmental or configuration failures too, so please report those as you come by them so we can fix them. The end goal is to enable protected branches, where any individual change has to pass a certain selection of robust tests before merging, which ultimately, would ensure that the MariaDB code is in a releasable state after any individual commit. We welcome feedback, improvement suggestions and any particular questions you might have about the current setup and future plans. We will check-in from time to time as we have more goodness to show. Best regards, Teodor -- Developer, Infra MariaDB Foundation https://mariadb.org

1 0

Re: [Maria-developers] [Commits] e1b0a8f9622: MDEV-6453: Assertion `inited==NONE || (inited==RND && scan)' failed in handler::ha_rnd_init(bool)
by Sergey Petrunia 15 Dec '18

15 Dec '18

Hi Varun, On Mon, Jan 29, 2018 at 12:27:35PM +0530, Varun wrote: > revision-id: e1b0a8f9622ab8c2bab988cb71225f992fec320a (mariadb-10.0.30-286-ge1b0a8f9622) > parent(s): d01dbe66a8bf9cb6031f95159c49100f9299a768 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2018-01-29 12:23:31 +0530 > message: > > MDEV-6453: Assertion `inited==NONE || (inited==RND && scan)' failed in handler::ha_rnd_init(bool) > with InnoDB, joins, AND/OR conditions > > The inited parameter handler is not initialised when we do a quick_select after a table scan. > As far I understand, the problem only shows with "Range Checked for each record". The failure scenario is as follows: - range checked for each record plan in coonstructed - the first check picks to a full table scan. - the second check picks to do a QUICK_ROR_INTERSECT_SELECT scan - QUICK_ROR_INTERSECT_SELECT starts to initialize the quick select. - and it hits an assertion, because the handle object is already initialized - index merge finds the handler not to be initialized correctly. That is, the cause of handler object not being correctly initialized is "range checked for each record" feature. Because of that, I think it should be fixed in that feature as well. A more suitable location would be in range-checked-for-each-record's code, in test_if_quick_select(): diff --git a/sql/sql_select.cc b/sql/sql_select.cc index db3ed8a1aa9..6634554ee6a 100644 --- a/sql/sql_select.cc +++ b/sql/sql_select.cc @@ -19637,6 +19637,10 @@ test_if_quick_select(JOIN_TAB *tab) delete tab->select->quick; tab->select->quick=0; + + if (tab->table->file->inited != handler::NONE) + tab->table->file->ha_index_or_rnd_end(); + int res= tab->select->test_quick_select(tab->join->thd, tab->keys, (table_map) 0, HA_POS_ERROR, 0, FALSE, /*remove where parts*/FALSE); What do you think? (If you agree, let's use the above variant) > --- > mysql-test/r/range_innodb.result | 18 ++++++++++++++++++ > mysql-test/t/range_innodb.test | 17 +++++++++++++++++ > sql/opt_range.cc | 2 ++ > 3 files changed, 37 insertions(+) > > diff --git a/mysql-test/r/range_innodb.result b/mysql-test/r/range_innodb.result > index 794e6c7b3cc..8bb1c833a56 100644 > --- a/mysql-test/r/range_innodb.result > +++ b/mysql-test/r/range_innodb.result > @@ -37,3 +37,21 @@ id select_type table type possible_keys key key_len ref rows Extra > 1 SIMPLE t0 ALL NULL NULL NULL NULL 10 > 1 SIMPLE t2 range a,b b 5 NULL 201 Using where; Using join buffer (flat, BNL join) > drop table t0,t1,t2; > +CREATE TABLE t1 ( > +pk INT PRIMARY KEY, f1 INT, f2 CHAR(1), f3 CHAR(1), > +KEY(f1), KEY(f2) > +) ENGINE=InnoDB; > +INSERT INTO t1 VALUES > +(1,4,'v',NULL),(2,6,'v',NULL),(3,7,'c',NULL),(4,1,'e',NULL),(5,0,'x',NULL), > +(6,7,'i',NULL),(7,7,'e',NULL),(8,1,'p',NULL),(9,7,'s',NULL),(10,1,'j',NULL), > +(11,5,'z',NULL),(12,2,'c',NULL),(13,0,'a',NULL),(14,1,'q',NULL),(15,8,'y',NULL), > +(16,1,'m',NULL),(17,1,'r',NULL),(18,9,'v',NULL),(19,1,'n',NULL); > +CREATE TABLE t2 (f4 INT, f5 CHAR(1)) ENGINE=InnoDB; > +INSERT INTO t2 VALUES (4,'q'),(NULL,'j'); > +SELECT * FROM t1 AS t1_1, t1 AS t1_2, t2 > +WHERE f5 = t1_2.f2 AND ( t1_1.f1 = 103 AND t1_1.f2 = 'o' OR t1_1.pk < f4 ); > +pk f1 f2 f3 pk f1 f2 f3 f4 f5 > +1 4 v NULL 14 1 q NULL 4 q > +2 6 v NULL 14 1 q NULL 4 q > +3 7 c NULL 14 1 q NULL 4 q > +drop table t1,t2; > diff --git a/mysql-test/t/range_innodb.test b/mysql-test/t/range_innodb.test > index f76794814ef..605006587cc 100644 > --- a/mysql-test/t/range_innodb.test > +++ b/mysql-test/t/range_innodb.test > @@ -45,3 +45,20 @@ explain select * from t0 left join t2 on t2.a <t0.a and t2.b between 50 and 250; > > drop table t0,t1,t2; > > +CREATE TABLE t1 ( > + pk INT PRIMARY KEY, f1 INT, f2 CHAR(1), f3 CHAR(1), > + KEY(f1), KEY(f2) > +) ENGINE=InnoDB; > + > +INSERT INTO t1 VALUES > +(1,4,'v',NULL),(2,6,'v',NULL),(3,7,'c',NULL),(4,1,'e',NULL),(5,0,'x',NULL), > +(6,7,'i',NULL),(7,7,'e',NULL),(8,1,'p',NULL),(9,7,'s',NULL),(10,1,'j',NULL), > +(11,5,'z',NULL),(12,2,'c',NULL),(13,0,'a',NULL),(14,1,'q',NULL),(15,8,'y',NULL), > +(16,1,'m',NULL),(17,1,'r',NULL),(18,9,'v',NULL),(19,1,'n',NULL); > + > +CREATE TABLE t2 (f4 INT, f5 CHAR(1)) ENGINE=InnoDB; > +INSERT INTO t2 VALUES (4,'q'),(NULL,'j'); > + > +SELECT * FROM t1 AS t1_1, t1 AS t1_2, t2 > +WHERE f5 = t1_2.f2 AND ( t1_1.f1 = 103 AND t1_1.f2 = 'o' OR t1_1.pk < f4 ); > +drop table t1,t2; > diff --git a/sql/opt_range.cc b/sql/opt_range.cc > index f1d84e5c623..30b7f43ef28 100644 > --- a/sql/opt_range.cc > +++ b/sql/opt_range.cc > @@ -3003,6 +3003,8 @@ int SQL_SELECT::test_quick_select(THD *thd, key_map keys_to_use, > quick=0; > needed_reg.clear_all(); > quick_keys.clear_all(); > + if (head->file->inited != handler::NONE) > + head->file->ha_index_or_rnd_end(); > DBUG_ASSERT(!head->is_filled_at_execution()); > if (keys_to_use.is_clear_all() || head->is_filled_at_execution()) > DBUG_RETURN(0); > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] 7c782988845: MDEV-16240: Assertion `0' failed in row_sel_convert_mysql_key_to_innobase
by Sergei Golubchik 14 Dec '18

14 Dec '18

Hi, Oleksandr! On Dec 13, Oleksandr Byelkin wrote: > revision-id: 7c782988845dec9f0f26a539911f66ed8cecdf83 (mariadb-10.2.19-51-g7c782988845) > parent(s): ad3346dddf419aed3e5d16066471fd5022af1795 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2018-12-13 15:29:52 +0100 > message: > > MDEV-16240: Assertion `0' failed in row_sel_convert_mysql_key_to_innobase > > Set table in row ID position mode before using this function. > > --- > mysql-test/r/multi_update_innodb.result | 40 +++++++++++++++++++++++++++ > mysql-test/t/multi_update_innodb.test | 49 +++++++++++++++++++++++++++++++++ > sql/sql_update.cc | 4 +++ > 3 files changed, 93 insertions(+) > > diff --git a/mysql-test/r/multi_update_innodb.result b/mysql-test/r/multi_update_innodb.result > index 5890fd24f5f..535c5a41d9f 100644 > --- a/mysql-test/r/multi_update_innodb.result > +++ b/mysql-test/r/multi_update_innodb.result > @@ -151,3 +151,43 @@ create table t2 like t1; > insert into t2 select * from t1; > delete t1,t2 from t2,t1 where t1.a<'B' and t2.b=t1.b; > drop table t1,t2; > +# > +# MDEV-16240: Assertion `0' failed in > +# row_sel_convert_mysql_key_to_innobase > +# > +SET @save_sql_mode=@@sql_mode; > +set sql_mode=''; why sql_mode matters here? > +CREATE TABLE `t3` ( > +`col_varchar_nokey` TIMESTAMP NOT NULL DEFAULT '0000-00-00 00:00:00' ON UPDATE current_timestamp(), > +`col_varchar_key` datetime DEFAULT '2000-01-01 00:00:00' ON UPDATE current_timestamp(), > +`col_int_nokey` TIMESTAMP NULL DEFAULT '2000-01-01 00:00:00', > +`pk` datetime NOT NULL DEFAULT '0000-00-00 00:00:00', > +`col_int_key` datetime DEFAULT current_timestamp(), > +PRIMARY KEY (`pk`), > +UNIQUE KEY `col_varchar_key` (`col_varchar_key`), > +KEY `col_int_key` (`col_int_key`) > +) ENGINE=InnoDB; > +INSERT INTO `t3` VALUES ('2018-05-18 15:08:07','2018-05-18 17:08:07','0000-00-00 00:00:00','0000-00-00 00:00:00','0000-00-00 00:00:00'),('0000-00-00 00:00:00','0000-00-00 00:00:00','1999-12-31 23:00:00','2002-07-03 23:04:40','0000-00-00 00:00:00'); > +CREATE VIEW `v1` AS > +SELECT `t3`.`pk` AS `pk`, > +`t3`.`col_int_nokey` AS `col_int_nokey`, > +`t3`.`col_int_key` AS `col_int_key`, > +`t3`.`col_varchar_key` AS `col_varchar_key`, > +`t3`.`col_varchar_nokey` AS `col_varchar_nokey` > +FROM `t3`; > +CREATE TABLE `t4` ( > +`col_varchar_nokey` datetime DEFAULT current_timestamp() ON UPDATE current_timestamp(), > +`col_int_nokey` timestamp NULL DEFAULT NULL, > +`col_varchar_key` timestamp NULL DEFAULT '1999-12-31 23:00:00' ON UPDATE current_timestamp(), > +`pk` int(11) NOT NULL, > +`col_int_key` timestamp NOT NULL DEFAULT current_timestamp() ON UPDATE current_timestamp(), > +PRIMARY KEY (`pk`) > +) ENGINE=InnoDB; > +INSERT INTO `t4` VALUES ('2018-05-18 17:08:06','0000-00-00 00:00:00',NULL,1,'2018-05-18 15:08:06'),('2018-05-18 17:08:06','0000-00-00 00:00:00',NULL,2,'2018-05-18 15:08:06'),('2018-05-18 17:08:06','0000-00-00 00:00:00',NULL,3,'2018-05-18 15:08:06'),('0000-00-00 00:00:00','0000-00-00 00:00:00',NULL,1976,'0000-00-00 00:00:00'),('2018-05-18 17:08:06','0000-00-00 00:00:00',NULL,2000,'2018-05-18 15:08:06'),('2018-05-18 17:08:06','0000-00-00 00:00:00',NULL,2001,'2018-05-18 15:08:06'),('2018-05-18 17:08:06','0000-00-00 00:00:00',NULL,2002,'2018-05-18 15:08:06'),('2018-05-18 17:08:06','0000-00-00 00:00:00',NULL,2003,'2018-05-18 15:08:06'),('2018-05-18 17:08:06','0000-00-00 00:00:00',NULL,2004,'2018-05-18 15:08:06'),('2018-05-18 17:08:06','0000-00-00 00:00:00','2018-05-18 15:08:06',2005,'2018-05-18 15:08:06'),('2018-05-18 17:08:06','0000-00-00 00:00:00','2018-05-18 15:08:06',2018,'2018-05-18 15:08:06'),('2018-05-18 17:08:06','0000-00-00 00:00:00','2018-05-18 15:08:06',2019,'2018-05-1 > 8 15:08:06'),('2018-05-18 17:08:06','0000-00-00 00:00:00','2018-05-18 15:08:06',2024,'2018-05-18 15:08:06'),('2018-05-18 17:08:06','0000-00-00 00:00:00','1999-12-31 23:00:00',2025,'2018-05-18 15:08:06'),('0000-00-00 00:00:00',NULL,'2018-05-18 15:08:06',2026,'2018-05-18 15:08:06'),('2018-05-18 17:08:07','0000-00-00 00:00:00','0000-00-00 00:00:00',2027,'0000-00-00 00:00:00'); > +UPDATE `v1` t1, `t4` t2 > +SET t1.`col_varchar_key` = 6452736 WHERE t1.`col_int_key` = 6272000; > +ERROR 23000: Duplicate entry '0000-00-00 00:00:00' for key 'col_varchar_key' please, rename all columns in the test to have unique names. What col_varchar_key is meant in the error message? > +DROP VIEW v1; > +DROP TABLE t3,t4; > +SET @@sql_mode=@save_sql_mode; > +# End of 10.2 tests > diff --git a/sql/sql_update.cc b/sql/sql_update.cc > index 11ffa684216..6d4c11d494a 100644 > --- a/sql/sql_update.cc > +++ b/sql/sql_update.cc > @@ -211,6 +211,10 @@ static void prepare_record_for_error_message(int error, TABLE *table) > bitmap_union(table->read_set, &unique_map); > /* Tell the engine about the new set. */ > table->file->column_bitmaps_signal(); > + /* Prepare table for random positioning (importent for innodb) */ This definitely needs a more detailed comment. I suspect (remembering your questions on slack) that rnd_pos() in InnoDB ends up using whatever active_index was at the moment, not the primary key. I'm not sure, though. And you have a typo in the word "important". By the way, why using rnd_pos() after index_init() didn't trigger an assert?.. Okay, I see why :) please fix it too. Hmm, with the assert you won't need a detailed comment in sql_update.cc anymore, assert will be a sufficient explanation. > + if (table->file->ha_index_or_rnd_end() || > + table->file->ha_rnd_init(0)) > + DBUG_VOID_RETURN; > /* Read record that is identified by table->file->ref. */ > (void) table->file->ha_rnd_pos(table->record[1], table->file->ref); > /* Copy the newly read columns into the new record. */ Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 1

Re: [Maria-developers] c16ad313bc4: MDEV-16849 Extending indexed VARCHAR column should be instantaneous
by Sergei Golubchik 13 Dec '18

13 Dec '18

Hi, Thirunarayanan! On Dec 13, Thirunarayanan Balathandayuthapani wrote: > revision-id: c16ad313bc4 (mariadb-10.2.19-4-gc16ad313bc4) > parent(s): 7e756437789 > author: Thirunarayanan Balathandayuthapani <thiru(a)mariadb.com> > committer: Thirunarayanan Balathandayuthapani <thiru(a)mariadb.com> > timestamp: 2018-11-15 15:14:02 +0530 > message: > > MDEV-16849 Extending indexed VARCHAR column should be instantaneous ... > diff --git a/mysql-test/suite/innodb/r/alter_varchar_change.result b/mysql-test/suite/innodb/r/alter_varchar_change.result > new file mode 100644 > index 00000000000..5d2c6403aab > --- /dev/null > +++ b/mysql-test/suite/innodb/r/alter_varchar_change.result > @@ -0,0 +1,335 @@ ... > +CREATE TABLE t1(f1 INT NOT NULL, > +f2 VARCHAR(100), > +INDEX idx(f2(10)))ENGINE=InnoDB; > +CALL get_table_id("test/t1", @tbl_id); > +CALL get_index_id(@tbl_id, "idx", @idx_id); > +ALTER TABLE t1 MODIFY f2 VARCHAR(200), DROP INDEX idx, ADD INDEX idx(f2(50)); > +CALL get_table_id("test/t1", @tbl1_id); > +CALL get_index_id(@tbl1_id, "idx", @idx1_id); > +SELECT @tbl1_id = @tbl_id; > +@tbl1_id = @tbl_id > +1 > +SELECT @idx1_id = @idx_id; > +@idx1_id = @idx_id > +1 Is that right? Old index would sort values "aaaaaaaaaa1", "aaaaaaaaaa2", "aaaaaaaaaa3" as equal, so they could be in the index in any order. But in the new index they aren't equal. > +CREATE TABLE t1(f1 INT NOT NULL, > +f2 VARCHAR(100), FULLTEXT idx(f2))ENGINE=InnoDB; > +CALL get_table_id("test/t1", @tbl_id); > +CALL get_index_id(@tbl_id, "idx", @idx_id); > +ALTER TABLE t1 MODIFY f2 VARCHAR(200); please, add a case where there's a FULLTEXT index, as above, but ALTER TABLE makes a column shorter. The index must be rebuilt. > +CREATE TABLE t1(f1 INT NOT NULL, > +f2 VARCHAR(100), > +INDEX idx(f2(10)), > +INDEX idx1(f1))ENGINE=InnoDB; > +CALL get_table_id("test/t1", @tbl_id); > +CALL get_index_id(@tbl_id, "idx", @idx_id); > +ALTER TABLE t1 MODIFY f2 VARCHAR(200), DROP INDEX idx1; Please also add a test case where VARCHAR is made shorter, but still longer than a prefix: ALTER TABLE t1 MODIFY f2 VARCHAR(50) The index should not be rebuilt. And shorter than a prefix: ALTER TABLE t1 MODIFY f2 VARCHAR(5) The index must be rebuilt. Also add a test when ALTER TABLE changes column's charset, or column type (CHAR/VARCHAR/TEXT). And a test where column's length is changed over 255. E.g. from 200 to 300. > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -6687,18 +6687,28 @@ static bool fill_alter_inplace_info(THD *thd, > key_part < end; > key_part++, new_part++) > { > + new_field= get_field_by_index(alter_info, new_part->fieldnr); > + old_field= table->field[key_part->fieldnr - 1]; > /* > + If there is a change in index length due to column expansion > + like varchar(X) changed to varchar(X + N) and has a compatible > + packed data representation, we mark it for fast/INPLACE change > + in index definition. InnoDB supports INPLACE for this cases > + > Key definition has changed if we are using a different field or > - if the used key part length is different. It makes sense to > - check lengths first as in case when fields differ it is likely > - that lengths differ too and checking fields is more expensive > - in general case. > + if the user key part length is different. > */ > - if (key_part->length != new_part->length) > + if (key_part->length <= new_part->length && > + old_field->pack_length() < new_field->pack_length && I still don't understand this condition. Why do you care whether a field length has changed, while you're should interested to know whether an index length have changed? Field length changes are completely irrelevant here. again, the only possible ALTER_COLUMN_INDEX_LENGTH case: * key part length was the same as column length, and it was not decreased. So, your condition should, probably, be something like if (key_part->length == old_field->pack_length() && key_part->length < new_part->length) > + (key_part->field->is_equal((Create_field*) new_field) > + == IS_EQUAL_PACK_LENGTH)) > + { > + ha_alter_info->handler_flags |= > + Alter_inplace_info::ALTER_COLUMN_INDEX_LENGTH; > + } > + else if (key_part->length != new_part->length) > goto index_changed; > > - new_field= get_field_by_index(alter_info, new_part->fieldnr); > - > /* > For prefix keys KEY_PART_INFO::field points to cloned Field > object with adjusted length. So below we have to check field Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] 667d178: MDEV-14576 Include full name of object in message about incorrect value for column.
by Sergei Golubchik 12 Dec '18

12 Dec '18

Hi, Alexey! Looks ok. I didn't expect make_truncated_value_warning to be used that much :( Jusr one comment below: On Nov 13, Alexey Botchkov wrote: > revision-id: 667d1786a96a40ee5074c9f1b493c7cf05a85cca (mariadb-10.2.18-80-g667d178) > parent(s): 59b87e75d04f0ed75256724c78f7fd7af2f96e9b > committer: Alexey Botchkov > timestamp: 2018-11-13 00:11:32 +0400 > message: > > MDEV-14576 Include full name of object in message about incorrect value for column. > > The error message modified. > Then the TABLE_SHARE::error_table_name() implementation taken from 10.3, > to be used as a name of the table in this message. > > diff --git a/sql/field.cc b/sql/field.cc > index caa84dc..4cb6eb2 100644 > --- a/sql/field.cc > +++ b/sql/field.cc > @@ -2079,7 +2079,7 @@ bool Field_num::get_date(MYSQL_TIME *ltime,ulonglong fuzzydate) > longlong nr= val_int(); > bool neg= !(flags & UNSIGNED_FLAG) && nr < 0; > return int_to_datetime_with_warn(neg, neg ? -nr : nr, ltime, fuzzydate, > - field_name); > + table->s, field_name); couldn't you just pass this instead of this->table->s and this->field_name ? > } > > > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index 71d5b80..8f8607d 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -3009,6 +3009,10 @@ int select_export::send_data(List<Item> &items) > error_pos= copier.most_important_error_pos(); > if (error_pos) > { > + /* > + TODO: > + add new error message that will show user this printable_buff > + > char printable_buff[32]; > convert_to_printable(printable_buff, sizeof(printable_buff), > error_pos, res->ptr() + res->length() - error_pos, > @@ -3018,6 +3022,11 @@ int select_export::send_data(List<Item> &items) > ER_THD(thd, ER_TRUNCATED_WRONG_VALUE_FOR_FIELD), > "string", printable_buff, > item->name, static_cast<long>(row_count)); > + */ > + push_warning_printf(thd, Sql_condition::WARN_LEVEL_WARN, > + ER_TRUNCATED_WRONG_VALUE_FOR_FIELD, > + ER_THD(thd, WARN_DATA_TRUNCATED), > + item->name, static_cast<long>(row_count)); Why not ER_TRUNCATED_WRONG_VALUE "Truncated incorrect %-.32s value: '%-.128s'" ? Ah, it has no row number... > } > else if (copier.source_end_pos() < res->ptr() + res->length()) > { Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] Extending storage engine API for random-row extraction for histogram collection (and others)
by Vicențiu Ciorbaru 11 Dec '18

11 Dec '18

Hi! Here is my proposal on extending the storage engine API to provide a functionality for retrieving random rows from tables (those that have indexes). The storage engines for which I plan to implement this are: MyISAM, Aria, Innodb. Possibly RocksDB, TokuDB. --- a/sql/handler.h +++ b/sql/handler.h @@ -2927,7 +2927,7 @@ class handler :public Sql_alloc /** Length of ref (1-8 or the clustered key length) */ uint ref_length; FT_INFO *ft_handler; - enum init_stat { NONE=0, INDEX, RND }; + enum init_stat { NONE=0, INDEX, RND, RANDOM }; init_stat inited, pre_inited; ........ + virtual int ha_random_sample_init() __attribute__((warn_unused_result)) + { + DBUG_ENTER("ha_random_sample_init"); + inited= RANDOM; + DBUG_RETURN(random_sample_init()); + } + virtual int ha_random_sample(uint inx, + key_range *min_key, + key_range *max_key) + __attribute__((warn_unused_result)) + { + DBUG_ENTER("ha_random_sample"); + DBUG_ASSERT(inited == RANDOM); + DBUG_RETURN(random_sample(inx, min_key, max_key)); + } + virtual int ha_random_sample_end() __attribute__((warn_unused_result)) + { + DBUG_ENTER("ha_random_sample_end"); + inited= NONE; + DBUG_RETURN(random_sample_end()); + } + This is the default implementation for a storage engine which does not support it: + virtual int random_sample_init() { return 0; } ; + virtual int random_sample(uint idx, key_range *min_key, key_range *max_key) + { + return HA_ERR_WRONG_COMMAND; + } + virtual int random_sample_end() { return 0; }; Alternative ideas: random_sample_init() takes the idx as a parameter and random_sample just fetches a row from the range using the index previously specified. The range can be left unspecified with nulls to provide a fetch from the full table range. I don't know enough about storage engine internals to know if an index declaration within the init function instead of within the "sample" function is better. Maybe I am complicating it too much and a simple random_sample() function is sufficient, kind of how ha_records_in_range does it. Thoughts? Vicențiu

3 6

Re: [Maria-developers] [Commits] 07f5d03: MDEV-5313 Improving merge audit api.
by Sergei Golubchik 10 Dec '18

10 Dec '18

Hi, Alexey! On Dec 03, Alexey Botchkov wrote: > revision-id: 07f5d036fbb5bdcb011a84f5c882a062d9e609e7 (mariadb-10.4.0-54-g07f5d03) > parent(s): 3afae13b548c903d86a55530d59fbf7d8666281f > committer: Alexey Botchkov > timestamp: 2018-12-03 02:35:52 +0400 > message: > > MDEV-5313 Improving merge audit api. "improving audit api" ? we don't merge it anymore, right? > service_json and service_cfg_table interfaces added. > > diff --git a/include/mysql/service_cfg_table.h b/include/mysql/service_cfg_table.h > new file mode 100644 > index 0000000..36660ff > --- /dev/null > +++ b/include/mysql/service_cfg_table.h > @@ -0,0 +1,109 @@ > +/* Copyright (C) 2018 MariaDB Corporation > + > + This program is free software; you can redistribute it and/or modify > + it under the terms of the GNU General Public License as published by > + the Free Software Foundation; version 2 of the License. > + > + This program is distributed in the hope that it will be useful, > + but WITHOUT ANY WARRANTY; without even the implied warranty of > + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + GNU General Public License for more details. > + > + You should have received a copy of the GNU General Public License > + along with this program; if not, write to the Free Software > + Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02111-1301 USA */ > + > +#ifndef MYSQL_SERVICE_CFG_TABLE > +#define MYSQL_SERVICE_CFG_TABLE > + > +/** > + @file > + cfg table service > + > + Reading plugin settings from the server table. > + > + This service lets the plugin to read it's settings > + from the server table. > + This 'cfg' table has two VARCHAR fields per row > + which are > + 'id' - the name of the setting > + 'value' - the value of the respective 'id' setting > + The structure 'cfg_table_row' reflects this format for > + the data interchange with the service. > + Fuctions of the service: > + cfg_table_check_exists > + checks if the table exists. > + cfg_table_create - creates the table > + parameers: > + table_name - the name of the table > + int id_len, int value_len - the length of the 'id' and 'value' > + fields. > + const struct cfg_table_row *defaults - table will be filled > + with these default rows. > + The list of rows is ended by {0,0} row > + > + cfg_table_set - replaces (inserts if not exists) the setting > + parameters: > + table_name - the name of the table > + const char *id, const char *value - the 'id' and the new value > + > + cfg_table_get - reads the cft tale content > + It return the number of rows in the table or -1 if error. > + parameters: > + table_name, > + struct cfg_table_row *data_buf - buffer to store the records > + int n_buf_row - the size of the buffer > +*/ This is, of course, a very much overdue functionality. I'm not sure it should be limited to key/value pairs. On the other hand, if it's plugin config only, loaded once on startup - then it should be fine. Just need to make clear it's not for run-time data. In the implementation, make sure that when a table is opened, there are no other opened/locked tables. This will avoid deadlocks and prevent plugins from using it from inside another statement. Also, * I'd put the table name into pluginname namespace. Not quite sure how to do it, though. * Why do you need cfg_table_check_exists and cfg_table_create? A plugin only needs set and get methods. A table is automatically created on the first access, a plugin can always assume it exists. * may be even just one table, plugin_config? with two columns plugin_name and json_config? plugin gets the json on startup and that's all. No way for a plugin to access the table directly. > diff --git a/include/mysql/service_json.h b/include/mysql/service_json.h > new file mode 100644 > index 0000000..5d0e260 > --- /dev/null > +++ b/include/mysql/service_json.h ... > +int js_type(const char *js, const char *js_end, > + enum js_value_types *vt, > + const char **v, int *vlen); > +int js_get_array_item(const char *js, const char *js_end, int n_item, > + enum js_value_types *vt, > + const char **v, int *vlen); > +int js_get_object_key(const char *js, const char *js_end, const char *key, > + enum js_value_types *vt, > + const char **v, int *vlen); > +int js_get_object_nkey(const char *js,const char *js_end, int nkey, > + const char **keyname, const char **keyname_end, > + enum js_value_types *vt, > + const char **v, int *vlen); I wouldn't introduce ten different apis to access json. Just json_get(json, path) and json_set(json, path, value). Same path syntax everywhere, like on SQL level. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] MDEV-17658 review.
by Sergei Golubchik 09 Dec '18

09 Dec '18

Hi, Alexey! It's important for USER_TABLE to be last. And there can be only one last value in the enum, if I add TABLES_MAX, someone will eventually put new table just before TABLES_MAX (tests will catch it, but it might take hours to track down to the actual reason). So, it's safer to have just one last element in the enum, USER_TABLE. On Dec 09, Alexey Botchkov wrote: > Hello, Sergei! > > I haven't found anything serious to improve there in your changes. > sql_acl.cc gets nicer with this patch. So ok to push. > > Only question left - > - TABLES_MAX // <== always the last > + USER_TABLE // <== always the last > > Why not preserve the TABLES_MAX? Seems more self-explaining. > +static LEX_CSTRING MYSQL_TABLE_NAME[USER_TABLE+1]= > could be > +static LEX_CSTRING MYSQL_TABLE_NAME[TABLES_MAX]= > and a couple of similar places. > > Best regards. > HF Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] 8bfb140d5dc: Move deletion of old GTID rows to slave background thread
by andrei.elkin＠pp.inet.fi 07 Dec '18

07 Dec '18

Kristian, hello. The patch is great and instructive in many ways. Thanks! There is something to improve in the test organization, like to base two tests of > storage/rocksdb/mysql-test/rocksdb_rpl/t/mdev12179.test > storage/tokudb /mysql-test/tokudb_rpl /t/mdev12179.test on a common parent. I thought for a second to place it in mysql-test/include/ but again the parent file is so specific that I had to stop it. This apparently can wait until a third engine shows up and require the same coverage. Cheers, Andrei > revision-id: 8bfb140d5dc247c183787b8a0a1799cf375845bd (mariadb-10.3.10-25-g8bfb140d5dc) > parent(s): 74387028a06c557f36a0fd1bbde347f1551c8fb7 > author: Kristian Nielsen > committer: Kristian Nielsen > timestamp: 2018-11-25 19:38:33 +0100 > message: > > Move deletion of old GTID rows to slave background thread > > This patch changes how old rows in mysql.gtid_slave_pos* tables are deleted. > Instead of doing it as part of every replicated transaction in > record_gtid(), it is done periodically (every @@gtid_cleanup_batch_size > transaction) in the slave background thread. > > This removes the deletion step from the replication process in SQL or worker > threads, which could speed up replication with many small transactions. It > also decreases contention on the global mutex LOCK_slave_state. And it > simplifies the logic, eg. when a replicated transaction fails after having > deleted old rows. > > With this patch, the deletion of old GTID rows happens asynchroneously and > slightly non-deterministic. Thus the number of old rows in > mysql.gtid_slave_pos can temporarily exceed @@gtid_cleanup_batch_size. But > all old rows will be deleted eventually after sufficiently many new GTIDs > have been replicated. > > --- > mysql-test/main/mysqld--help.result | 10 + > mysql-test/suite/rpl/r/rpl_gtid_mdev4484.result | 40 +- > mysql-test/suite/rpl/r/rpl_gtid_stop_start.result | 8 +- > .../suite/rpl/r/rpl_parallel_optimistic.result | 14 +- > mysql-test/suite/rpl/t/rpl_gtid_mdev4484.test | 68 +++- > .../suite/rpl/t/rpl_parallel_optimistic.test | 42 ++- > .../sys_vars/r/sysvars_server_notembedded.result | 14 + > sql/log_event.cc | 6 +- > sql/mysqld.cc | 1 + > sql/mysqld.h | 1 + > sql/rpl_gtid.cc | 413 +++++++++++++-------- > sql/rpl_gtid.h | 12 +- > sql/rpl_rli.cc | 87 +---- > sql/rpl_rli.h | 11 - > sql/slave.cc | 35 +- > sql/slave.h | 1 + > sql/sys_vars.cc | 13 + > .../mysql-test/rocksdb_rpl/r/mdev12179.result | 18 + > .../mysql-test/rocksdb_rpl/t/mdev12179.test | 85 +++++ > .../mysql-test/tokudb_rpl/r/mdev12179.result | 18 + > .../tokudb/mysql-test/tokudb_rpl/t/mdev12179.test | 85 +++++ > 21 files changed, 675 insertions(+), 307 deletions(-) > > diff --git a/mysql-test/main/mysqld--help.result b/mysql-test/main/mysqld--help.result > index 5a7153f32d3..4f801ec5275 100644 > --- a/mysql-test/main/mysqld--help.result > +++ b/mysql-test/main/mysqld--help.result > @@ -294,6 +294,15 @@ The following specify which files/extra groups are read (specified before remain > --group-concat-max-len=# > The maximum length of the result of function > GROUP_CONCAT() > + --gtid-cleanup-batch-size=# > + Normally does not need tuning. How many old rows must > + accumulate in the mysql.gtid_slave_pos table before a > + background job will be run to delete them. Can be > + increased to reduce number of commits if using many > + different engines with --gtid_pos_auto_engines, or to > + reduce CPU overhead if using a huge number of different > + gtid_domain_ids. Can be decreased to reduce number of old > + rows in the table. > --gtid-domain-id=# Used with global transaction ID to identify logically > independent replication streams. When events can > propagate through multiple parallel paths (for example > @@ -1425,6 +1434,7 @@ gdb FALSE > general-log FALSE > getopt-prefix-matching FALSE > group-concat-max-len 1048576 > +gtid-cleanup-batch-size 64 > gtid-domain-id 0 > gtid-ignore-duplicates FALSE > gtid-pos-auto-engines > diff --git a/mysql-test/suite/rpl/r/rpl_gtid_mdev4484.result b/mysql-test/suite/rpl/r/rpl_gtid_mdev4484.result > index aaeb0c8f119..55d2831dcf4 100644 > --- a/mysql-test/suite/rpl/r/rpl_gtid_mdev4484.result > +++ b/mysql-test/suite/rpl/r/rpl_gtid_mdev4484.result > @@ -16,36 +16,32 @@ INSERT INTO t1 VALUES (1); > connection slave; > connection slave; > include/stop_slave.inc > +SET @old_gtid_cleanup_batch_size= @@GLOBAL.gtid_cleanup_batch_size; > +SET GLOBAL gtid_cleanup_batch_size= 2; > SET @old_dbug= @@GLOBAL.debug_dbug; > SET GLOBAL debug_dbug="+d,gtid_slave_pos_simulate_failed_delete"; > SET sql_log_bin= 0; > -CALL mtr.add_suppression("Can't find file"); > +CALL mtr.add_suppression("<DEBUG> Error deleting old GTID row"); > SET sql_log_bin= 1; > include/start_slave.inc > connection master; > -INSERT INTO t1 VALUES (2); > -connection slave; > -include/wait_for_slave_sql_error.inc [errno=1942] > -STOP SLAVE IO_THREAD; > -SELECT domain_id, server_id, seq_no FROM mysql.gtid_slave_pos > -ORDER BY domain_id, sub_id DESC LIMIT 1; > -domain_id server_id seq_no > -0 1 3 > +connection slave; > +SELECT COUNT(*), MAX(seq_no) INTO @pre_count, @pre_max_seq_no > +FROM mysql.gtid_slave_pos; > +SELECT IF(@pre_count >= 20, "OK", CONCAT("Error: too few rows seen while errors injected: ", @pre_count)); > +IF(@pre_count >= 20, "OK", CONCAT("Error: too few rows seen while errors injected: ", @pre_count)) > +OK > SET GLOBAL debug_dbug= @old_dbug; > -include/start_slave.inc > connection master; > -INSERT INTO t1 VALUES (3); > -connection slave; > -connection slave; > -SELECT domain_id, server_id, seq_no FROM mysql.gtid_slave_pos > -ORDER BY domain_id, sub_id DESC LIMIT 1; > -domain_id server_id seq_no > -0 1 4 > -SELECT * FROM t1 ORDER BY i; > -i > -1 > -2 > -3 > +connection slave; > +connection slave; > +SELECT IF(COUNT(*) >= 1, "OK", CONCAT("Error: too few rows seen after errors no longer injected: ", COUNT(*))) > +FROM mysql.gtid_slave_pos > +WHERE seq_no <= @pre_max_seq_no; > +IF(COUNT(*) >= 1, "OK", CONCAT("Error: too few rows seen after errors no longer injected: ", COUNT(*))) > +OK > connection master; > DROP TABLE t1; > +connection slave; > +SET GLOBAL gtid_cleanup_batch_size= @old_gtid_cleanup_batch_size; > include/rpl_end.inc > diff --git a/mysql-test/suite/rpl/r/rpl_gtid_stop_start.result b/mysql-test/suite/rpl/r/rpl_gtid_stop_start.result > index ff845794c22..b27ffed9f94 100644 > --- a/mysql-test/suite/rpl/r/rpl_gtid_stop_start.result > +++ b/mysql-test/suite/rpl/r/rpl_gtid_stop_start.result > @@ -171,7 +171,7 @@ include/start_slave.inc > *** MDEV-4692: mysql.gtid_slave_pos accumulates values for a domain *** > SELECT domain_id, COUNT(*) FROM mysql.gtid_slave_pos GROUP BY domain_id; > domain_id COUNT(*) > -0 2 > +0 3 > 1 2 > connection server_1; > INSERT INTO t1 VALUES (11); > @@ -179,7 +179,7 @@ connection server_2; > FLUSH NO_WRITE_TO_BINLOG TABLES; > SELECT domain_id, COUNT(*) FROM mysql.gtid_slave_pos GROUP BY domain_id; > domain_id COUNT(*) > -0 2 > +0 4 > 1 2 > include/start_slave.inc > connection server_1; > @@ -189,8 +189,8 @@ connection server_2; > FLUSH NO_WRITE_TO_BINLOG TABLES; > SELECT domain_id, COUNT(*) FROM mysql.gtid_slave_pos GROUP BY domain_id; > domain_id COUNT(*) > -0 2 > -1 2 > +0 3 > +1 1 > *** MDEV-4650: show variables; ERROR 1946 (HY000): Failed to load replication slave GTID position *** > connection server_2; > SET sql_log_bin=0; > diff --git a/mysql-test/suite/rpl/r/rpl_parallel_optimistic.result b/mysql-test/suite/rpl/r/rpl_parallel_optimistic.result > index ca202a66b0e..83343e52cab 100644 > --- a/mysql-test/suite/rpl/r/rpl_parallel_optimistic.result > +++ b/mysql-test/suite/rpl/r/rpl_parallel_optimistic.result > @@ -12,6 +12,8 @@ SET GLOBAL slave_parallel_threads=10; > CHANGE MASTER TO master_use_gtid=slave_pos; > SET @old_parallel_mode=@@GLOBAL.slave_parallel_mode; > SET GLOBAL slave_parallel_mode='optimistic'; > +SET @old_gtid_cleanup_batch_size= @@GLOBAL.gtid_cleanup_batch_size; > +SET GLOBAL gtid_cleanup_batch_size= 1000000; > connection server_1; > INSERT INTO t1 VALUES(1,1); > BEGIN; > @@ -131,6 +133,11 @@ c > 204 > 205 > 206 > +SELECT IF(COUNT(*) >= 30, "OK", CONCAT("Error: too few old rows found: ", COUNT(*))) > +FROM mysql.gtid_slave_pos; > +IF(COUNT(*) >= 30, "OK", CONCAT("Error: too few old rows found: ", COUNT(*))) > +OK > +SET GLOBAL gtid_cleanup_batch_size=1; > *** Test @@skip_parallel_replication. *** > connection server_2; > include/stop_slave.inc > @@ -651,9 +658,10 @@ DROP TABLE t1, t2, t3; > include/save_master_gtid.inc > connection server_2; > include/sync_with_master_gtid.inc > -Check that no more than the expected last four GTIDs are in mysql.gtid_slave_pos > -select count(4) <= 4 from mysql.gtid_slave_pos order by domain_id, sub_id; > -count(4) <= 4 > +SELECT COUNT(*) <= 5*@@GLOBAL.gtid_cleanup_batch_size > +FROM mysql.gtid_slave_pos; > +COUNT(*) <= 5*@@GLOBAL.gtid_cleanup_batch_size > 1 > +SET GLOBAL gtid_cleanup_batch_size= @old_gtid_cleanup_batch_size; > connection server_1; > include/rpl_end.inc > diff --git a/mysql-test/suite/rpl/t/rpl_gtid_mdev4484.test b/mysql-test/suite/rpl/t/rpl_gtid_mdev4484.test > index e1f5696f5a1..a28bff3d27a 100644 > --- a/mysql-test/suite/rpl/t/rpl_gtid_mdev4484.test > +++ b/mysql-test/suite/rpl/t/rpl_gtid_mdev4484.test > @@ -28,37 +28,79 @@ INSERT INTO t1 VALUES (1); > # Inject an artificial error deleting entries, and check that the error handling code works. > --connection slave > --source include/stop_slave.inc > +SET @old_gtid_cleanup_batch_size= @@GLOBAL.gtid_cleanup_batch_size; > +SET GLOBAL gtid_cleanup_batch_size= 2; > SET @old_dbug= @@GLOBAL.debug_dbug; > SET GLOBAL debug_dbug="+d,gtid_slave_pos_simulate_failed_delete"; > SET sql_log_bin= 0; > -CALL mtr.add_suppression("Can't find file"); > +CALL mtr.add_suppression("<DEBUG> Error deleting old GTID row"); > SET sql_log_bin= 1; > --source include/start_slave.inc > > --connection master > -INSERT INTO t1 VALUES (2); > +--disable_query_log > +let $i = 20; > +while ($i) { > + eval INSERT INTO t1 VALUES ($i+10); > + dec $i; > +} > +--enable_query_log > +--save_master_pos > > --connection slave > ---let $slave_sql_errno= 1942 > ---source include/wait_for_slave_sql_error.inc > -STOP SLAVE IO_THREAD; > -SELECT domain_id, server_id, seq_no FROM mysql.gtid_slave_pos > - ORDER BY domain_id, sub_id DESC LIMIT 1; > +--sync_with_master > + > +# Now wait for the slave background thread to try to delete old rows and > +# hit the error injection. > +--let _TEST_MYSQLD_ERROR_LOG=$MYSQLTEST_VARDIR/log/mysqld.2.err > +--perl > + open F, '<', $ENV{'_TEST_MYSQLD_ERROR_LOG'} or die; > + outer: while (1) { > + inner: while (<F>) { > + last outer if /<DEBUG> Error deleting old GTID row/; > + } > + # Easy way to do sub-second sleep without extra modules. > + select(undef, undef, undef, 0.1); > + } > +EOF > + > +# Since we injected error in the cleanup code, the rows should remain in > +# mysql.gtid_slave_pos. Check that we have at least 20 (more robust against > +# non-deterministic cleanup and future changes than checking for exact number). > +SELECT COUNT(*), MAX(seq_no) INTO @pre_count, @pre_max_seq_no > + FROM mysql.gtid_slave_pos; > +SELECT IF(@pre_count >= 20, "OK", CONCAT("Error: too few rows seen while errors injected: ", @pre_count)); > SET GLOBAL debug_dbug= @old_dbug; > ---source include/start_slave.inc > > --connection master > -INSERT INTO t1 VALUES (3); > +--disable_query_log > +let $i = 20; > +while ($i) { > + eval INSERT INTO t1 VALUES ($i+40); > + dec $i; > +} > +--enable_query_log > --sync_slave_with_master > > --connection slave > -SELECT domain_id, server_id, seq_no FROM mysql.gtid_slave_pos > - ORDER BY domain_id, sub_id DESC LIMIT 1; > -SELECT * FROM t1 ORDER BY i; > - > +# Now check that 1) rows are being deleted again after removing error > +# injection, and 2) old rows are left that failed their delete while errors > +# where injected (again compensating for non-deterministic deletion). > +# Deletion is async and slightly non-deterministic, so we wait for at > +# least 10 of the 20 new rows to be deleted. > +let $wait_condition= > + SELECT COUNT(*) <= 20-10 > + FROM mysql.gtid_slave_pos > + WHERE seq_no > @pre_max_seq_no; > +--source include/wait_condition.inc > +SELECT IF(COUNT(*) >= 1, "OK", CONCAT("Error: too few rows seen after errors no longer injected: ", COUNT(*))) > + FROM mysql.gtid_slave_pos > + WHERE seq_no <= @pre_max_seq_no; > > # Clean up > --connection master > DROP TABLE t1; > +--connection slave > +SET GLOBAL gtid_cleanup_batch_size= @old_gtid_cleanup_batch_size; > > --source include/rpl_end.inc > diff --git a/mysql-test/suite/rpl/t/rpl_parallel_optimistic.test b/mysql-test/suite/rpl/t/rpl_parallel_optimistic.test > index e08472d5f51..0060cf4416c 100644 > --- a/mysql-test/suite/rpl/t/rpl_parallel_optimistic.test > +++ b/mysql-test/suite/rpl/t/rpl_parallel_optimistic.test > @@ -21,6 +21,10 @@ SET GLOBAL slave_parallel_threads=10; > CHANGE MASTER TO master_use_gtid=slave_pos; > SET @old_parallel_mode=@@GLOBAL.slave_parallel_mode; > SET GLOBAL slave_parallel_mode='optimistic'; > +# Run the first part of the test with high batch size and see that > +# old rows remain in the table. > +SET @old_gtid_cleanup_batch_size= @@GLOBAL.gtid_cleanup_batch_size; > +SET GLOBAL gtid_cleanup_batch_size= 1000000; > > > --connection server_1 > @@ -108,7 +112,12 @@ SELECT * FROM t3 ORDER BY c; > SELECT * FROM t1 ORDER BY a; > SELECT * FROM t2 ORDER BY a; > SELECT * FROM t3 ORDER BY c; > -#SHOW STATUS LIKE 'Slave_retried_transactions'; > +# Check that we have a bunch of old rows left-over - they were not deleted > +# due to high @@gtid_cleanup_batch_size. Then set a low > +# @@gtid_cleanup_batch_size so we can test that rows start being deleted. > +SELECT IF(COUNT(*) >= 30, "OK", CONCAT("Error: too few old rows found: ", COUNT(*))) > + FROM mysql.gtid_slave_pos; > +SET GLOBAL gtid_cleanup_batch_size=1; > > > --echo *** Test @@skip_parallel_replication. *** > @@ -557,25 +566,18 @@ DROP TABLE t1, t2, t3; > > --connection server_2 > --source include/sync_with_master_gtid.inc > -# Check for left-over rows in table mysql.gtid_slave_pos (MDEV-12147). > -# > -# There was a bug when a transaction got a conflict and was rolled back. It > -# might have also handled deletion of some old rows, and these deletions would > -# then also be rolled back. And since the deletes were never re-tried, old no > -# longer needed rows would accumulate in the table without limit. > -# > -# The earlier part of this test file have plenty of transactions being rolled > -# back. But the last DROP TABLE statement runs on its own and should never > -# conflict, thus at this point the mysql.gtid_slave_pos table should be clean. > -# > -# To support @@gtid_pos_auto_engines, when a row is inserted in the table, it > -# is associated with the engine of the table at insertion time, and it will > -# only be deleted during record_gtid from a table of the same engine. Since we > -# alter the table from MyISAM to InnoDB at the start of this test, we should > -# end up with 4 rows: two left-over from when the table was MyISAM, and two > -# left-over from the InnoDB part. > ---echo Check that no more than the expected last four GTIDs are in mysql.gtid_slave_pos > -select count(4) <= 4 from mysql.gtid_slave_pos order by domain_id, sub_id; > +# Check that old rows are deleted from mysql.gtid_slave_pos. > +# Deletion is asynchronous, so use wait_condition.inc. > +# Also, there is a small amount of non-determinism in the deletion of old > +# rows, so it is not guaranteed that there can never be more than > +# @@gtid_cleanup_batch_size rows in the table; so allow a bit of slack > +# here. > +let $wait_condition= > + SELECT COUNT(*) <= 5*@@GLOBAL.gtid_cleanup_batch_size > + FROM mysql.gtid_slave_pos; > +--source include/wait_condition.inc > +eval $wait_condition; > +SET GLOBAL gtid_cleanup_batch_size= @old_gtid_cleanup_batch_size; > > --connection server_1 > --source include/rpl_end.inc > diff --git a/mysql-test/suite/sys_vars/r/sysvars_server_notembedded.result b/mysql-test/suite/sys_vars/r/sysvars_server_notembedded.result > index e8e4d671eb9..5c5ca8b66b2 100644 > --- a/mysql-test/suite/sys_vars/r/sysvars_server_notembedded.result > +++ b/mysql-test/suite/sys_vars/r/sysvars_server_notembedded.result > @@ -1202,6 +1202,20 @@ NUMERIC_BLOCK_SIZE NULL > ENUM_VALUE_LIST NULL > READ_ONLY NO > COMMAND_LINE_ARGUMENT NULL > +VARIABLE_NAME GTID_CLEANUP_BATCH_SIZE > +SESSION_VALUE NULL > +GLOBAL_VALUE 64 > +GLOBAL_VALUE_ORIGIN COMPILE-TIME > +DEFAULT_VALUE 64 > +VARIABLE_SCOPE GLOBAL > +VARIABLE_TYPE INT UNSIGNED > +VARIABLE_COMMENT Normally does not need tuning. How many old rows must accumulate in the mysql.gtid_slave_pos table before a background job will be run to delete them. Can be increased to reduce number of commits if using many different engines with --gtid_pos_auto_engines, or to reduce CPU overhead if using a huge number of different gtid_domain_ids. Can be decreased to reduce number of old rows in the table. > +NUMERIC_MIN_VALUE 0 > +NUMERIC_MAX_VALUE 2147483647 > +NUMERIC_BLOCK_SIZE 1 > +ENUM_VALUE_LIST NULL > +READ_ONLY NO > +COMMAND_LINE_ARGUMENT REQUIRED > VARIABLE_NAME GTID_CURRENT_POS > SESSION_VALUE NULL > GLOBAL_VALUE > diff --git a/sql/log_event.cc b/sql/log_event.cc > index 8813d20578e..e10480fb015 100644 > --- a/sql/log_event.cc > +++ b/sql/log_event.cc > @@ -5565,7 +5565,7 @@ int Query_log_event::do_apply_event(rpl_group_info *rgi, > gtid= rgi->current_gtid; > if (unlikely(rpl_global_gtid_slave_state->record_gtid(thd, &gtid, > sub_id, > - rgi, false, > + true, false, > &hton))) > { > int errcode= thd->get_stmt_da()->sql_errno(); > @@ -8362,7 +8362,7 @@ Gtid_list_log_event::do_apply_event(rpl_group_info *rgi) > { > if ((ret= rpl_global_gtid_slave_state->record_gtid(thd, &list[i], > sub_id_list[i], > - NULL, false, &hton))) > + false, false, &hton))) > return ret; > rpl_global_gtid_slave_state->update_state_hash(sub_id_list[i], &list[i], > hton, NULL); > @@ -8899,7 +8899,7 @@ int Xid_log_event::do_apply_event(rpl_group_info *rgi) > rgi->gtid_pending= false; > > gtid= rgi->current_gtid; > - err= rpl_global_gtid_slave_state->record_gtid(thd, &gtid, sub_id, rgi, > + err= rpl_global_gtid_slave_state->record_gtid(thd, &gtid, sub_id, true, > false, &hton); > if (unlikely(err)) > { > diff --git a/sql/mysqld.cc b/sql/mysqld.cc > index afef4a5f52c..07bdd66f74c 100644 > --- a/sql/mysqld.cc > +++ b/sql/mysqld.cc > @@ -580,6 +580,7 @@ ulong opt_binlog_commit_wait_count= 0; > ulong opt_binlog_commit_wait_usec= 0; > ulong opt_slave_parallel_max_queued= 131072; > my_bool opt_gtid_ignore_duplicates= FALSE; > +uint opt_gtid_cleanup_batch_size= 64; > > const double log_10[] = { > 1e000, 1e001, 1e002, 1e003, 1e004, 1e005, 1e006, 1e007, 1e008, 1e009, > diff --git a/sql/mysqld.h b/sql/mysqld.h > index d5cabd790b2..261748372f9 100644 > --- a/sql/mysqld.h > +++ b/sql/mysqld.h > @@ -258,6 +258,7 @@ extern ulong opt_slave_parallel_mode; > extern ulong opt_binlog_commit_wait_count; > extern ulong opt_binlog_commit_wait_usec; > extern my_bool opt_gtid_ignore_duplicates; > +extern uint opt_gtid_cleanup_batch_size; > extern ulong back_log; > extern ulong executed_events; > extern char language[FN_REFLEN]; > diff --git a/sql/rpl_gtid.cc b/sql/rpl_gtid.cc > index fabd09adaa7..196c2fe3d16 100644 > --- a/sql/rpl_gtid.cc > +++ b/sql/rpl_gtid.cc > @@ -79,7 +79,7 @@ rpl_slave_state::record_and_update_gtid(THD *thd, rpl_group_info *rgi) > rgi->gtid_pending= false; > if (rgi->gtid_ignore_duplicate_state!=rpl_group_info::GTID_DUPLICATE_IGNORE) > { > - if (record_gtid(thd, &rgi->current_gtid, sub_id, NULL, false, &hton)) > + if (record_gtid(thd, &rgi->current_gtid, sub_id, false, false, &hton)) > DBUG_RETURN(1); > update_state_hash(sub_id, &rgi->current_gtid, hton, rgi); > } > @@ -244,7 +244,7 @@ rpl_slave_state_free_element(void *arg) > > > rpl_slave_state::rpl_slave_state() > - : last_sub_id(0), gtid_pos_tables(0), loaded(false) > + : pending_gtid_count(0), last_sub_id(0), gtid_pos_tables(0), loaded(false) > { > mysql_mutex_init(key_LOCK_slave_state, &LOCK_slave_state, > MY_MUTEX_INIT_SLOW); > @@ -331,14 +331,11 @@ rpl_slave_state::update(uint32 domain_id, uint32 server_id, uint64 sub_id, > } > } > rgi->gtid_ignore_duplicate_state= rpl_group_info::GTID_DUPLICATE_NULL; > - > -#ifdef HAVE_REPLICATION > - rgi->pending_gtid_deletes_clear(); > -#endif > } > > if (!(list_elem= (list_element *)my_malloc(sizeof(*list_elem), MYF(MY_WME)))) > return 1; > + list_elem->domain_id= domain_id; > list_elem->server_id= server_id; > list_elem->sub_id= sub_id; > list_elem->seq_no= seq_no; > @@ -348,6 +345,15 @@ rpl_slave_state::update(uint32 domain_id, uint32 server_id, uint64 sub_id, > if (last_sub_id < sub_id) > last_sub_id= sub_id; > > +#ifdef HAVE_REPLICATION > + ++pending_gtid_count; > + if (pending_gtid_count >= opt_gtid_cleanup_batch_size) > + { > + pending_gtid_count = 0; > + slave_background_gtid_pending_delete_request(); > + } > +#endif > + > return 0; > } > > @@ -382,20 +388,22 @@ rpl_slave_state::get_element(uint32 domain_id) > > > int > -rpl_slave_state::put_back_list(uint32 domain_id, list_element *list) > +rpl_slave_state::put_back_list(list_element *list) > { > - element *e; > + element *e= NULL; > int err= 0; > > mysql_mutex_lock(&LOCK_slave_state); > - if (!(e= (element *)my_hash_search(&hash, (const uchar *)&domain_id, 0))) > - { > - err= 1; > - goto end; > - } > while (list) > { > list_element *next= list->next; > + > + if ((!e || e->domain_id != list->domain_id) && > + !(e= (element *)my_hash_search(&hash, (const uchar *)&list->domain_id, 0))) > + { > + err= 1; > + goto end; > + } > e->add(list); > list= next; > } > @@ -572,12 +580,12 @@ rpl_slave_state::select_gtid_pos_table(THD *thd, LEX_CSTRING *out_tablename) > /* > Write a gtid to the replication slave state table. > > + Do it as part of the transaction, to get slave crash safety, or as a separate > + transaction if !in_transaction (eg. MyISAM or DDL). > + > gtid The global transaction id for this event group. > sub_id Value allocated within the sub_id when the event group was > read (sub_id must be consistent with commit order in master binlog). > - rgi rpl_group_info context, if we are recording the gtid transactionally > - as part of replicating a transactional event. NULL if called from > - outside of a replicated transaction. > > Note that caller must later ensure that the new gtid and sub_id is inserted > into the appropriate HASH element with rpl_slave_state.add(), so that it can > @@ -585,16 +593,13 @@ rpl_slave_state::select_gtid_pos_table(THD *thd, LEX_CSTRING *out_tablename) > */ > int > rpl_slave_state::record_gtid(THD *thd, const rpl_gtid *gtid, uint64 sub_id, > - rpl_group_info *rgi, bool in_statement, > + bool in_transaction, bool in_statement, > void **out_hton) > { > TABLE_LIST tlist; > int err= 0, not_sql_thread; > bool table_opened= false; > TABLE *table; > - list_element *delete_list= 0, *next, *cur, **next_ptr_ptr, **best_ptr_ptr; > - uint64 best_sub_id; > - element *elem; > ulonglong thd_saved_option= thd->variables.option_bits; > Query_tables_list lex_backup; > wait_for_commit* suspended_wfc; > @@ -684,7 +689,7 @@ rpl_slave_state::record_gtid(THD *thd, const rpl_gtid *gtid, uint64 sub_id, > thd->wsrep_ignore_table= true; > #endif > > - if (!rgi) > + if (!in_transaction) > { > DBUG_PRINT("info", ("resetting OPTION_BEGIN")); > thd->variables.option_bits&= > @@ -716,168 +721,280 @@ rpl_slave_state::record_gtid(THD *thd, const rpl_gtid *gtid, uint64 sub_id, > my_error(ER_OUT_OF_RESOURCES, MYF(0)); > goto end; > } > +end: > > - mysql_mutex_lock(&LOCK_slave_state); > - if ((elem= get_element(gtid->domain_id)) == NULL) > +#ifdef WITH_WSREP > + thd->wsrep_ignore_table= false; > +#endif > + > + if (table_opened) > { > - mysql_mutex_unlock(&LOCK_slave_state); > - my_error(ER_OUT_OF_RESOURCES, MYF(0)); > - err= 1; > - goto end; > + if (err || (err= ha_commit_trans(thd, FALSE))) > + ha_rollback_trans(thd, FALSE); > + > + close_thread_tables(thd); > + if (in_transaction) > + thd->mdl_context.release_statement_locks(); > + else > + thd->mdl_context.release_transactional_locks(); > } > + thd->lex->restore_backup_query_tables_list(&lex_backup); > + thd->variables.option_bits= thd_saved_option; > + thd->resume_subsequent_commits(suspended_wfc); > + DBUG_EXECUTE_IF("inject_record_gtid_serverid_100_sleep", > + { > + if (gtid->server_id == 100) > + my_sleep(500000); > + }); > + DBUG_RETURN(err); > +} > > - /* Now pull out all GTIDs that were recorded in this engine. */ > - delete_list = NULL; > - next_ptr_ptr= &elem->list; > - cur= elem->list; > - best_sub_id= 0; > - best_ptr_ptr= NULL; > - while (cur) > + > +/* > + Return a list of all old GTIDs in any mysql.gtid_slave_pos* table that are > + no longer needed and can be deleted from the table. > + > + Within each domain, we need to keep around the latest GTID (the one with the > + highest sub_id), but any others in that domain can be deleted. > +*/ > +rpl_slave_state::list_element * > +rpl_slave_state::gtid_grab_pending_delete_list() > +{ > + uint32 i; > + list_element *full_list; > + > + mysql_mutex_lock(&LOCK_slave_state); > + full_list= NULL; > + for (i= 0; i < hash.records; ++i) > { > - list_element *next= cur->next; > - if (cur->hton == hton) > - { > - /* Belongs to same engine, so move it to the delete list. */ > - cur->next= delete_list; > - delete_list= cur; > - if (cur->sub_id > best_sub_id) > + element *elem= (element *)my_hash_element(&hash, i); > + list_element *elist= elem->list; > + list_element *last_elem, **best_ptr_ptr, *cur, *next; > + uint64 best_sub_id; > + > + if (!elist) > + continue; /* Nothing here */ > + > + /* Delete any old stuff, but keep around the most recent one. */ > + cur= elist; > + best_sub_id= cur->sub_id; > + best_ptr_ptr= &elist; > + last_elem= cur; > + while ((next= cur->next)) { > + last_elem= next; > + if (next->sub_id > best_sub_id) > { > - best_sub_id= cur->sub_id; > - best_ptr_ptr= &delete_list; > - } > - else if (best_ptr_ptr == &delete_list) > + best_sub_id= next->sub_id; > best_ptr_ptr= &cur->next; > - } > - else > - { > - /* Another engine, leave it in the list. */ > - if (cur->sub_id > best_sub_id) > - { > - best_sub_id= cur->sub_id; > - /* Current best is not on the delete list. */ > - best_ptr_ptr= NULL; > } > - *next_ptr_ptr= cur; > - next_ptr_ptr= &cur->next; > + cur= next; > } > - cur= next; > - } > - *next_ptr_ptr= NULL; > - /* > - If the highest sub_id element is on the delete list, put it back on the > - original list, to preserve the highest sub_id element in the table for > - GTID position recovery. > - */ > - if (best_ptr_ptr) > - { > + /* > + Append the new elements to the full list. Note the order is important; > + we do it here so that we do not break the list if best_sub_id is the > + last of the new elements. > + */ > + last_elem->next= full_list; > + /* > + Delete the highest sub_id element from the old list, and put it back as > + the single-element new list. > + */ > cur= *best_ptr_ptr; > *best_ptr_ptr= cur->next; > - cur->next= elem->list; > + cur->next= NULL; > elem->list= cur; > + > + /* > + Collect the full list so far here. Note that elist may have moved if we > + deleted the first element, so order is again important. > + */ > + full_list= elist; > } > mysql_mutex_unlock(&LOCK_slave_state); > > - if (!delete_list) > - goto end; > + return full_list; > +} > + > > - /* Now delete any already committed GTIDs. */ > - bitmap_set_bit(table->read_set, table->field[0]->field_index); > - bitmap_set_bit(table->read_set, table->field[1]->field_index); > +/* Find the mysql.gtid_slave_posXXX table associated with a given hton. */ > +LEX_CSTRING * > +rpl_slave_state::select_gtid_pos_table(void *hton) > +{ > + struct gtid_pos_table *table_entry; > > - if ((err= table->file->ha_index_init(0, 0))) > + /* > + See comments on rpl_slave_state::gtid_pos_tables for rules around proper > + access to the list. > + */ > + table_entry= (struct gtid_pos_table *) > + my_atomic_loadptr_explicit(&gtid_pos_tables, MY_MEMORY_ORDER_ACQUIRE); > + > + while (table_entry) > { > - table->file->print_error(err, MYF(0)); > - goto end; > + if (table_entry->table_hton == hton) > + { > + if (likely(table_entry->state == GTID_POS_AVAILABLE)) > + return &table_entry->table_name; > + } > + table_entry= table_entry->next; > } > - cur = delete_list; > - while (cur) > - { > - uchar key_buffer[4+8]; > > - DBUG_EXECUTE_IF("gtid_slave_pos_simulate_failed_delete", > - { err= ENOENT; > - table->file->print_error(err, MYF(0)); > - /* `break' does not work inside DBUG_EXECUTE_IF */ > - goto dbug_break; }); > + table_entry= (struct gtid_pos_table *) > + my_atomic_loadptr_explicit(&default_gtid_pos_table, MY_MEMORY_ORDER_ACQUIRE); > + return &table_entry->table_name; > +} > > - next= cur->next; > > - table->field[1]->store(cur->sub_id, true); > - /* domain_id is already set in table->record[0] from write_row() above. */ > - key_copy(key_buffer, table->record[0], &table->key_info[0], 0, false); > - if (table->file->ha_index_read_map(table->record[1], key_buffer, > - HA_WHOLE_KEY, HA_READ_KEY_EXACT)) > - /* We cannot find the row, assume it is already deleted. */ > - ; > - else if ((err= table->file->ha_delete_row(table->record[1]))) > - table->file->print_error(err, MYF(0)); > - /* > - In case of error, we still discard the element from the list. We do > - not want to endlessly error on the same element in case of table > - corruption or such. > - */ > - cur= next; > - if (err) > - break; > - } > -IF_DBUG(dbug_break:, ) > - table->file->ha_index_end(); > +void > +rpl_slave_state::gtid_delete_pending(THD *thd, > + rpl_slave_state::list_element **list_ptr) > +{ > + int err= 0; > + ulonglong thd_saved_option; > > -end: > + if (unlikely(!loaded)) > + return; > > #ifdef WITH_WSREP > - thd->wsrep_ignore_table= false; > + /* > + Updates in slave state table should not be appended to galera transaction > + writeset. > + */ > + thd->wsrep_ignore_table= true; > #endif > > - if (table_opened) > + thd_saved_option= thd->variables.option_bits; > + thd->variables.option_bits&= > + ~(ulonglong)(OPTION_NOT_AUTOCOMMIT |OPTION_BEGIN |OPTION_BIN_LOG | > + OPTION_GTID_BEGIN); > + > + while (*list_ptr) > { > - if (err || (err= ha_commit_trans(thd, FALSE))) > - { > - /* > - If error, we need to put any remaining delete_list back into the HASH > - so we can do another delete attempt later. > - */ > - if (delete_list) > - { > - put_back_list(gtid->domain_id, delete_list); > - delete_list = 0; > - } > + LEX_CSTRING *gtid_pos_table_name, *tmp_table_name; > + Query_tables_list lex_backup; > + TABLE_LIST tlist; > + TABLE *table; > + handler::Table_flags direct_pos; > + list_element *cur, **cur_ptr_ptr; > + bool table_opened= false; > + void *hton= (*list_ptr)->hton; > > - ha_rollback_trans(thd, FALSE); > + thd->reset_for_next_command(); > + > + /* > + Only the SQL thread can call select_gtid_pos_table without a mutex > + Other threads needs to use a mutex and take into account that the > + result may change during execution, so we have to make a copy. > + */ > + mysql_mutex_lock(&LOCK_slave_state); > + tmp_table_name= select_gtid_pos_table(hton); > + gtid_pos_table_name= thd->make_clex_string(tmp_table_name->str, > + tmp_table_name->length); > + mysql_mutex_unlock(&LOCK_slave_state); > + if (!gtid_pos_table_name) > + { > + /* Out of memory - we can try again later. */ > + break; > } > - close_thread_tables(thd); > - if (rgi) > + > + thd->lex->reset_n_backup_query_tables_list(&lex_backup); > + tlist.init_one_table(&MYSQL_SCHEMA_NAME, gtid_pos_table_name, NULL, TL_WRITE); > + if ((err= open_and_lock_tables(thd, &tlist, FALSE, 0))) > + goto end; > + table_opened= true; > + table= tlist.table; > + > + if ((err= gtid_check_rpl_slave_state_table(table))) > + goto end; > + > + direct_pos= table->file->ha_table_flags() & HA_PRIMARY_KEY_REQUIRED_FOR_POSITION; > + bitmap_set_all(table->write_set); > + table->rpl_write_set= table->write_set; > + > + /* Now delete any already committed GTIDs. */ > + bitmap_set_bit(table->read_set, table->field[0]->field_index); > + bitmap_set_bit(table->read_set, table->field[1]->field_index); > + > + if (!direct_pos && (err= table->file->ha_index_init(0, 0))) > { > - thd->mdl_context.release_statement_locks(); > - /* > - Save the list of old gtid entries we deleted. If this transaction > - fails later for some reason and is rolled back, the deletion of those > - entries will be rolled back as well, and we will need to put them back > - on the to-be-deleted list so we can re-do the deletion. Otherwise > - redundant rows in mysql.gtid_slave_pos may accumulate if transactions > - are rolled back and retried after record_gtid(). > - */ > -#ifdef HAVE_REPLICATION > - rgi->pending_gtid_deletes_save(gtid->domain_id, delete_list); > -#endif > + table->file->print_error(err, MYF(0)); > + goto end; > } > - else > + > + cur = *list_ptr; > + cur_ptr_ptr = list_ptr; > + do > { > - thd->mdl_context.release_transactional_locks(); > -#ifdef HAVE_REPLICATION > - rpl_group_info::pending_gtid_deletes_free(delete_list); > -#endif > + uchar key_buffer[4+8]; > + list_element *next= cur->next; > + > + if (cur->hton == hton) > + { > + int res; > + > + table->field[0]->store((ulonglong)cur->domain_id, true); > + table->field[1]->store(cur->sub_id, true); > + if (direct_pos) > + { > + res= table->file->ha_rnd_pos_by_record(table->record[0]); > + } > + else > + { > + key_copy(key_buffer, table->record[0], &table->key_info[0], 0, false); > + res= table->file->ha_index_read_map(table->record[0], key_buffer, > + HA_WHOLE_KEY, HA_READ_KEY_EXACT); > + } > + DBUG_EXECUTE_IF("gtid_slave_pos_simulate_failed_delete", > + { res= 1; > + err= ENOENT; > + sql_print_error("<DEBUG> Error deleting old GTID row"); > + }); > + if (res) > + /* We cannot find the row, assume it is already deleted. */ > + ; > + else if ((err= table->file->ha_delete_row(table->record[0]))) > + { > + sql_print_error("Error deleting old GTID row: %s", > + thd->get_stmt_da()->message()); > + /* > + In case of error, we still discard the element from the list. We do > + not want to endlessly error on the same element in case of table > + corruption or such. > + */ > + } > + *cur_ptr_ptr= next; > + my_free(cur); > + } > + else > + { > + /* Leave this one in the list until we get to the table for its hton. */ > + cur_ptr_ptr= &cur->next; > + } > + cur= next; > + if (err) > + break; > + } while (cur); > +end: > + if (table_opened) > + { > + if (!direct_pos) > + table->file->ha_index_end(); > + > + if (err || (err= ha_commit_trans(thd, FALSE))) > + ha_rollback_trans(thd, FALSE); > } > + close_thread_tables(thd); > + thd->mdl_context.release_transactional_locks(); > + thd->lex->restore_backup_query_tables_list(&lex_backup); > + > + if (err) > + break; > } > - thd->lex->restore_backup_query_tables_list(&lex_backup); > thd->variables.option_bits= thd_saved_option; > - thd->resume_subsequent_commits(suspended_wfc); > - DBUG_EXECUTE_IF("inject_record_gtid_serverid_100_sleep", > - { > - if (gtid->server_id == 100) > - my_sleep(500000); > - }); > - DBUG_RETURN(err); > + > +#ifdef WITH_WSREP > + thd->wsrep_ignore_table= false; > +#endif > } > > > @@ -1251,7 +1368,7 @@ rpl_slave_state::load(THD *thd, const char *state_from_master, size_t len, > > if (gtid_parser_helper(&state_from_master, end, &gtid) || > !(sub_id= next_sub_id(gtid.domain_id)) || > - record_gtid(thd, &gtid, sub_id, NULL, in_statement, &hton) || > + record_gtid(thd, &gtid, sub_id, false, in_statement, &hton) || > update(gtid.domain_id, gtid.server_id, sub_id, gtid.seq_no, hton, NULL)) > return 1; > if (state_from_master == end) > diff --git a/sql/rpl_gtid.h b/sql/rpl_gtid.h > index 0fc92d5e33c..60d822f7b0d 100644 > --- a/sql/rpl_gtid.h > +++ b/sql/rpl_gtid.h > @@ -118,8 +118,9 @@ struct rpl_slave_state > { > struct list_element *next; > uint64 sub_id; > - uint64 seq_no; > + uint32 domain_id; > uint32 server_id; > + uint64 seq_no; > /* > hton of mysql.gtid_slave_pos* table used to record this GTID. > Can be NULL if the gtid table failed to load (eg. missing > @@ -191,6 +192,8 @@ struct rpl_slave_state > > /* Mapping from domain_id to its element. */ > HASH hash; > + /* GTIDs added since last purge of old mysql.gtid_slave_pos rows. */ > + uint32 pending_gtid_count; > /* Mutex protecting access to the state. */ > mysql_mutex_t LOCK_slave_state; > /* Auxiliary buffer to sort gtid list. */ > @@ -233,7 +236,10 @@ struct rpl_slave_state > int truncate_state_table(THD *thd); > void select_gtid_pos_table(THD *thd, LEX_CSTRING *out_tablename); > int record_gtid(THD *thd, const rpl_gtid *gtid, uint64 sub_id, > - rpl_group_info *rgi, bool in_statement, void **out_hton); > + bool in_transaction, bool in_statement, void **out_hton); > + list_element *gtid_grab_pending_delete_list(); > + LEX_CSTRING *select_gtid_pos_table(void *hton); > + void gtid_delete_pending(THD *thd, rpl_slave_state::list_element **list_ptr); > uint64 next_sub_id(uint32 domain_id); > int iterate(int (*cb)(rpl_gtid *, void *), void *data, > rpl_gtid *extra_gtids, uint32 num_extra, > @@ -245,7 +251,7 @@ struct rpl_slave_state > bool is_empty(); > > element *get_element(uint32 domain_id); > - int put_back_list(uint32 domain_id, list_element *list); > + int put_back_list(list_element *list); > > void update_state_hash(uint64 sub_id, rpl_gtid *gtid, void *hton, > rpl_group_info *rgi); > diff --git a/sql/rpl_rli.cc b/sql/rpl_rli.cc > index b275ad884bd..2d91620c898 100644 > --- a/sql/rpl_rli.cc > +++ b/sql/rpl_rli.cc > @@ -1820,6 +1820,7 @@ rpl_load_gtid_slave_state(THD *thd) > int err= 0; > uint32 i; > load_gtid_state_cb_data cb_data; > + rpl_slave_state::list_element *old_gtids_list; > DBUG_ENTER("rpl_load_gtid_slave_state"); > > mysql_mutex_lock(&rpl_global_gtid_slave_state->LOCK_slave_state); > @@ -1905,6 +1906,13 @@ rpl_load_gtid_slave_state(THD *thd) > rpl_global_gtid_slave_state->loaded= true; > mysql_mutex_unlock(&rpl_global_gtid_slave_state->LOCK_slave_state); > > + /* Clear out no longer needed elements now. */ > + old_gtids_list= > + rpl_global_gtid_slave_state->gtid_grab_pending_delete_list(); > + rpl_global_gtid_slave_state->gtid_delete_pending(thd, &old_gtids_list); > + if (old_gtids_list) > + rpl_global_gtid_slave_state->put_back_list(old_gtids_list); > + > end: > if (array_inited) > delete_dynamic(&array); > @@ -2086,7 +2094,6 @@ rpl_group_info::reinit(Relay_log_info *rli) > long_find_row_note_printed= false; > did_mark_start_commit= false; > gtid_ev_flags2= 0; > - pending_gtid_delete_list= NULL; > last_master_timestamp = 0; > gtid_ignore_duplicate_state= GTID_DUPLICATE_NULL; > speculation= SPECULATE_NO; > @@ -2217,12 +2224,6 @@ void rpl_group_info::cleanup_context(THD *thd, bool error) > erroneously update the GTID position. > */ > gtid_pending= false; > - > - /* > - Rollback will have undone any deletions of old rows we might have made > - in mysql.gtid_slave_pos. Put those rows back on the list to be deleted. > - */ > - pending_gtid_deletes_put_back(); > } > m_table_map.clear_tables(); > slave_close_thread_tables(thd); > @@ -2448,78 +2449,6 @@ rpl_group_info::unmark_start_commit() > } > > > -/* > - When record_gtid() has deleted any old rows from the table > - mysql.gtid_slave_pos as part of a replicated transaction, save the list of > - rows deleted here. > - > - If later the transaction fails (eg. optimistic parallel replication), the > - deletes will be undone when the transaction is rolled back. Then we can > - put back the list of rows into the rpl_global_gtid_slave_state, so that > - we can re-do the deletes and avoid accumulating old rows in the table. > -*/ > -void > -rpl_group_info::pending_gtid_deletes_save(uint32 domain_id, > - rpl_slave_state::list_element *list) > -{ > - /* > - We should never get to a state where we try to save a new pending list of > - gtid deletes while we still have an old one. But make sure we handle it > - anyway just in case, so we avoid leaving stray entries in the > - mysql.gtid_slave_pos table. > - */ > - DBUG_ASSERT(!pending_gtid_delete_list); > - if (unlikely(pending_gtid_delete_list)) > - pending_gtid_deletes_put_back(); > - > - pending_gtid_delete_list= list; > - pending_gtid_delete_list_domain= domain_id; > -} > - > - > -/* > - Take the list recorded by pending_gtid_deletes_save() and put it back into > - rpl_global_gtid_slave_state. This is needed if deletion of the rows was > - rolled back due to transaction failure. > -*/ > -void > -rpl_group_info::pending_gtid_deletes_put_back() > -{ > - if (pending_gtid_delete_list) > - { > - rpl_global_gtid_slave_state->put_back_list(pending_gtid_delete_list_domain, > - pending_gtid_delete_list); > - pending_gtid_delete_list= NULL; > - } > -} > - > - > -/* > - Free the list recorded by pending_gtid_deletes_save(). Done when the deletes > - in the list have been permanently committed. > -*/ > -void > -rpl_group_info::pending_gtid_deletes_clear() > -{ > - pending_gtid_deletes_free(pending_gtid_delete_list); > - pending_gtid_delete_list= NULL; > -} > - > - > -void > -rpl_group_info::pending_gtid_deletes_free(rpl_slave_state::list_element *list) > -{ > - rpl_slave_state::list_element *next; > - > - while (list) > - { > - next= list->next; > - my_free(list); > - list= next; > - } > -} > - > - > rpl_sql_thread_info::rpl_sql_thread_info(Rpl_filter *filter) > : rpl_filter(filter) > { > diff --git a/sql/rpl_rli.h b/sql/rpl_rli.h > index d9f0e0e5d3b..b8b153c34be 100644 > --- a/sql/rpl_rli.h > +++ b/sql/rpl_rli.h > @@ -757,11 +757,6 @@ struct rpl_group_info > /* Needs room for "Gtid D-S-N\x00". */ > char gtid_info_buf[5+10+1+10+1+20+1]; > > - /* List of not yet committed deletions in mysql.gtid_slave_pos. */ > - rpl_slave_state::list_element *pending_gtid_delete_list; > - /* Domain associated with pending_gtid_delete_list. */ > - uint32 pending_gtid_delete_list_domain; > - > /* > The timestamp, from the master, of the commit event. > Used to do delayed update of rli->last_master_timestamp, for getting > @@ -903,12 +898,6 @@ struct rpl_group_info > char *gtid_info(); > void unmark_start_commit(); > > - static void pending_gtid_deletes_free(rpl_slave_state::list_element *list); > - void pending_gtid_deletes_save(uint32 domain_id, > - rpl_slave_state::list_element *list); > - void pending_gtid_deletes_put_back(); > - void pending_gtid_deletes_clear(); > - > longlong get_row_stmt_start_timestamp() > { > return row_stmt_start_timestamp; > diff --git a/sql/slave.cc b/sql/slave.cc > index bb1300d36e6..f8499513dd6 100644 > --- a/sql/slave.cc > +++ b/sql/slave.cc > @@ -465,6 +465,8 @@ static struct slave_background_gtid_pos_create_t { > void *hton; > } *slave_background_gtid_pos_create_list; > > +static volatile bool slave_background_gtid_pending_delete_flag; > + > > pthread_handler_t > handle_slave_background(void *arg __attribute__((unused))) > @@ -499,6 +501,7 @@ handle_slave_background(void *arg __attribute__((unused))) > { > slave_background_kill_t *kill_list; > slave_background_gtid_pos_create_t *create_list; > + bool pending_deletes; > > thd->ENTER_COND(&COND_slave_background, &LOCK_slave_background, > &stage_slave_background_wait_request, > @@ -508,13 +511,15 @@ handle_slave_background(void *arg __attribute__((unused))) > stop= abort_loop || thd->killed || slave_background_thread_stop; > kill_list= slave_background_kill_list; > create_list= slave_background_gtid_pos_create_list; > - if (stop || kill_list || create_list) > + pending_deletes= slave_background_gtid_pending_delete_flag; > + if (stop || kill_list || create_list || pending_deletes) > break; > mysql_cond_wait(&COND_slave_background, &LOCK_slave_background); > } > > slave_background_kill_list= NULL; > slave_background_gtid_pos_create_list= NULL; > + slave_background_gtid_pending_delete_flag= false; > thd->EXIT_COND(&old_stage); > > while (kill_list) > @@ -541,6 +546,17 @@ handle_slave_background(void *arg __attribute__((unused))) > create_list= next; > } > > + if (pending_deletes) > + { > + rpl_slave_state::list_element *list; > + > + slave_background_gtid_pending_delete_flag= false; > + list= rpl_global_gtid_slave_state->gtid_grab_pending_delete_list(); > + rpl_global_gtid_slave_state->gtid_delete_pending(thd, &list); > + if (list) > + rpl_global_gtid_slave_state->put_back_list(list); > + } > + > mysql_mutex_lock(&LOCK_slave_background); > } while (!stop); > > @@ -615,6 +631,23 @@ slave_background_gtid_pos_create_request( > > > /* > + Request the slave background thread to delete no longer used rows from the > + mysql.gtid_slave_pos* tables. > + > + This is called from time-critical rpl_slave_state::update(), so we avoid > + taking any locks here. This means we may race with the background thread > + to occasionally lose a signal. This is not a problem; any pending rows to > + be deleted will just be deleted a bit later as part of the next batch. > +*/ > +void > +slave_background_gtid_pending_delete_request(void) > +{ > + slave_background_gtid_pending_delete_flag= true; > + mysql_cond_signal(&COND_slave_background); > +} > + > + > +/* > Start the slave background thread. > > This thread is currently used for two purposes: > diff --git a/sql/slave.h b/sql/slave.h > index 649d55b45b9..12d569b0333 100644 > --- a/sql/slave.h > +++ b/sql/slave.h > @@ -276,6 +276,7 @@ bool net_request_file(NET* net, const char* fname); > void slave_background_kill_request(THD *to_kill); > void slave_background_gtid_pos_create_request > (rpl_slave_state::gtid_pos_table *table_entry); > +void slave_background_gtid_pending_delete_request(void); > > extern bool volatile abort_loop; > extern Master_info *active_mi; /* active_mi for multi-master */ > diff --git a/sql/sys_vars.cc b/sql/sys_vars.cc > index 6d4c135683a..9348f4e5c98 100644 > --- a/sql/sys_vars.cc > +++ b/sql/sys_vars.cc > @@ -1942,6 +1942,19 @@ Sys_var_last_gtid::session_value_ptr(THD *thd, const LEX_CSTRING *base) > } > > > +static Sys_var_uint Sys_gtid_cleanup_batch_size( > + "gtid_cleanup_batch_size", > + "Normally does not need tuning. How many old rows must accumulate in " > + "the mysql.gtid_slave_pos table before a background job will be run to " > + "delete them. Can be increased to reduce number of commits if " > + "using many different engines with --gtid_pos_auto_engines, or to " > + "reduce CPU overhead if using a huge number of different " > + "gtid_domain_ids. Can be decreased to reduce number of old rows in the " > + "table.", > + GLOBAL_VAR(opt_gtid_cleanup_batch_size), CMD_LINE(REQUIRED_ARG), > + VALID_RANGE(0,2147483647), DEFAULT(64), BLOCK_SIZE(1)); > + > + > static bool > check_slave_parallel_threads(sys_var *self, THD *thd, set_var *var) > { > diff --git a/storage/rocksdb/mysql-test/rocksdb_rpl/r/mdev12179.result b/storage/rocksdb/mysql-test/rocksdb_rpl/r/mdev12179.result > index 9c20fea97ae..a1e501f78f4 100644 > --- a/storage/rocksdb/mysql-test/rocksdb_rpl/r/mdev12179.result > +++ b/storage/rocksdb/mysql-test/rocksdb_rpl/r/mdev12179.result > @@ -2,6 +2,7 @@ include/master-slave.inc > [connection master] > connection server_2; > include/stop_slave.inc > +SET GLOBAL gtid_cleanup_batch_size = 999999999; > CHANGE MASTER TO master_use_gtid=slave_pos; > SET sql_log_bin=0; > CREATE TABLE mysql.gtid_slave_pos_innodb LIKE mysql.gtid_slave_pos; > @@ -41,6 +42,8 @@ a > 1 > SELECT * FROM mysql.gtid_slave_pos ORDER BY sub_id; > domain_id sub_id server_id seq_no > +0 1 1 1 > +0 2 1 2 > 0 3 1 3 > 0 4 1 4 > SELECT * FROM ( SELECT * FROM mysql.gtid_slave_pos_innodb > @@ -121,6 +124,21 @@ Transactions_multi_engine 6 > DELETE FROM t1 WHERE a >= 100; > DELETE FROM t2 WHERE a >= 100; > DELETE FROM t3 WHERE a >= 100; > +connection server_1; > +include/save_master_gtid.inc > +connection server_2; > +include/sync_with_master_gtid.inc > +SELECT COUNT(*)>=10 FROM mysql.gtid_slave_pos; > +COUNT(*)>=10 > +1 > +SELECT COUNT(*)>=10 FROM ( SELECT * FROM mysql.gtid_slave_pos_innodb > +UNION ALL SELECT * FROM mysql.gtid_slave_pos_innodb_redundant) inner_select; > +COUNT(*)>=10 > +1 > +SELECT COUNT(*)>=10 FROM mysql.gtid_slave_pos_rocksdb; > +COUNT(*)>=10 > +1 > +SET GLOBAL gtid_cleanup_batch_size = 3; > connection server_2; > include/stop_slave.inc > SET sql_log_bin=0; > diff --git a/storage/rocksdb/mysql-test/rocksdb_rpl/t/mdev12179.test b/storage/rocksdb/mysql-test/rocksdb_rpl/t/mdev12179.test > index e0d16e7f242..631d9ca533f 100644 > --- a/storage/rocksdb/mysql-test/rocksdb_rpl/t/mdev12179.test > +++ b/storage/rocksdb/mysql-test/rocksdb_rpl/t/mdev12179.test > @@ -4,6 +4,12 @@ > > --connection server_2 > --source include/stop_slave.inc > + > +# Set GTID cleanup limit high enough that cleanup will not run and we > +# can rely on consistent table output in .result. > +--let $old_gtid_cleanup_batch_size=`SELECT @@GLOBAL.gtid_cleanup_batch_size` > +SET GLOBAL gtid_cleanup_batch_size = 999999999; > + > CHANGE MASTER TO master_use_gtid=slave_pos; > SET sql_log_bin=0; > CREATE TABLE mysql.gtid_slave_pos_innodb LIKE mysql.gtid_slave_pos; > @@ -89,6 +95,82 @@ DELETE FROM t2 WHERE a >= 100; > DELETE FROM t3 WHERE a >= 100; > > > +# Create a bunch more GTIDs in mysql.gtid_slave_pos* tables to test with. > +--connection server_1 > +--disable_query_log > +let $i=10; > +while ($i) { > + eval INSERT INTO t1 VALUES (300+$i); > + eval INSERT INTO t2 VALUES (300+$i); > + eval INSERT INTO t3 VALUES (300+$i); > + dec $i; > +} > +--enable_query_log > +--source include/save_master_gtid.inc > + > +--connection server_2 > +--source include/sync_with_master_gtid.inc > + > +# Check that we have many rows in mysql.gtid_slave_pos now (since > +# @@gtid_cleanup_batch_size was set to a huge value). No need to check > +# for an exact number, since that will require changing .result if > +# anything changes prior to this point, and we just need to know that > +# we have still have some data in the tables to make the following > +# test effective. > +SELECT COUNT(*)>=10 FROM mysql.gtid_slave_pos; > +SELECT COUNT(*)>=10 FROM ( SELECT * FROM mysql.gtid_slave_pos_innodb > + UNION ALL SELECT * FROM mysql.gtid_slave_pos_innodb_redundant) inner_select; > +SELECT COUNT(*)>=10 FROM mysql.gtid_slave_pos_rocksdb; > + > +# Check that old GTID rows will be deleted when batch delete size is > +# set reasonably. Old row deletion is not 100% deterministic (by design), so > +# we must wait for it to occur, but it should occur eventually. > +SET GLOBAL gtid_cleanup_batch_size = 3; > +let $i=40; > +--disable_query_log > +--let $keep_include_silent=1 > +while ($i) { > + let N=`SELECT 1+($i MOD 3)`; > + --connection server_1 > + eval UPDATE t$N SET a=a+1 WHERE a=(SELECT MAX(a) FROM t$N); > + --source include/save_master_gtid.inc > + --connection server_2 > + --source include/sync_with_master_gtid.inc > + let $j=50; > + while ($j) { > + let $is_done=`SELECT SUM(a)=1 FROM ( > + SELECT COUNT(*) AS a FROM mysql.gtid_slave_pos > + UNION ALL > + SELECT COUNT(*) AS a FROM ( SELECT * FROM mysql.gtid_slave_pos_innodb > + UNION ALL SELECT * FROM mysql.gtid_slave_pos_innodb_redundant) inner_select > + UNION ALL > + SELECT COUNT(*) AS a FROM mysql.gtid_slave_pos_rocksdb) outer_select`; > + if ($is_done) { > + let $j=0; > + } > + if (!$is_done) { > + real_sleep 0.1; > + dec $j; > + } > + } > + dec $i; > + if ($is_done) { > + let $i=0; > + } > +} > +--enable_query_log > +--let $keep_include_silent=0 > +if (!$is_done) { > + --echo Timed out waiting for mysql.gtid_slave_pos* tables to be cleaned up > +} > + > +--disable_query_log > +DELETE FROM t1 WHERE a >= 100; > +DELETE FROM t2 WHERE a >= 100; > +DELETE FROM t3 WHERE a >= 100; > +--enable_query_log > + > + > # Test status variables Rpl_transactions_multi_engine and Transactions_gtid_foreign_engine. > # Have mysql.gtid_slave_pos* for myisam and innodb but not rocksdb. > --connection server_2 > @@ -223,6 +305,9 @@ SHOW STATUS LIKE "%transactions%engine"; > SET sql_log_bin=0; > DROP TABLE mysql.gtid_slave_pos_innodb; > SET sql_log_bin=1; > +--disable_query_log > +eval SET GLOBAL gtid_cleanup_batch_size = $old_gtid_cleanup_batch_size; > +--enable_query_log > > --connection server_1 > DROP TABLE t1; > diff --git a/storage/tokudb/mysql-test/tokudb_rpl/r/mdev12179.result b/storage/tokudb/mysql-test/tokudb_rpl/r/mdev12179.result > index d4532eec4e2..d79e7e59aa4 100644 > --- a/storage/tokudb/mysql-test/tokudb_rpl/r/mdev12179.result > +++ b/storage/tokudb/mysql-test/tokudb_rpl/r/mdev12179.result > @@ -2,6 +2,7 @@ include/master-slave.inc > [connection master] > connection server_2; > include/stop_slave.inc > +SET GLOBAL gtid_cleanup_batch_size = 999999999; > CHANGE MASTER TO master_use_gtid=slave_pos; > SET sql_log_bin=0; > CREATE TABLE mysql.gtid_slave_pos_innodb LIKE mysql.gtid_slave_pos; > @@ -41,6 +42,8 @@ a > 1 > SELECT * FROM mysql.gtid_slave_pos ORDER BY sub_id; > domain_id sub_id server_id seq_no > +0 1 1 1 > +0 2 1 2 > 0 3 1 3 > 0 4 1 4 > SELECT * FROM ( SELECT * FROM mysql.gtid_slave_pos_innodb > @@ -121,6 +124,21 @@ Transactions_multi_engine 6 > DELETE FROM t1 WHERE a >= 100; > DELETE FROM t2 WHERE a >= 100; > DELETE FROM t3 WHERE a >= 100; > +connection server_1; > +include/save_master_gtid.inc > +connection server_2; > +include/sync_with_master_gtid.inc > +SELECT COUNT(*)>=10 FROM mysql.gtid_slave_pos; > +COUNT(*)>=10 > +1 > +SELECT COUNT(*)>=10 FROM ( SELECT * FROM mysql.gtid_slave_pos_innodb > +UNION ALL SELECT * FROM mysql.gtid_slave_pos_innodb_redundant) inner_select; > +COUNT(*)>=10 > +1 > +SELECT COUNT(*)>=10 FROM mysql.gtid_slave_pos_tokudb; > +COUNT(*)>=10 > +1 > +SET GLOBAL gtid_cleanup_batch_size = 3; > connection server_2; > include/stop_slave.inc > SET sql_log_bin=0; > diff --git a/storage/tokudb/mysql-test/tokudb_rpl/t/mdev12179.test b/storage/tokudb/mysql-test/tokudb_rpl/t/mdev12179.test > index ceb119cd0dc..1d19a25889e 100644 > --- a/storage/tokudb/mysql-test/tokudb_rpl/t/mdev12179.test > +++ b/storage/tokudb/mysql-test/tokudb_rpl/t/mdev12179.test > @@ -4,6 +4,12 @@ > > --connection server_2 > --source include/stop_slave.inc > + > +# Set GTID cleanup limit high enough that cleanup will not run and we > +# can rely on consistent table output in .result. > +--let $old_gtid_cleanup_batch_size=`SELECT @@GLOBAL.gtid_cleanup_batch_size` > +SET GLOBAL gtid_cleanup_batch_size = 999999999; > + > CHANGE MASTER TO master_use_gtid=slave_pos; > SET sql_log_bin=0; > CREATE TABLE mysql.gtid_slave_pos_innodb LIKE mysql.gtid_slave_pos; > @@ -89,6 +95,82 @@ DELETE FROM t2 WHERE a >= 100; > DELETE FROM t3 WHERE a >= 100; > > > +# Create a bunch more GTIDs in mysql.gtid_slave_pos* tables to test with. > +--connection server_1 > +--disable_query_log > +let $i=10; > +while ($i) { > + eval INSERT INTO t1 VALUES (300+$i); > + eval INSERT INTO t2 VALUES (300+$i); > + eval INSERT INTO t3 VALUES (300+$i); > + dec $i; > +} > +--enable_query_log > +--source include/save_master_gtid.inc > + > +--connection server_2 > +--source include/sync_with_master_gtid.inc > + > +# Check that we have many rows in mysql.gtid_slave_pos now (since > +# @@gtid_cleanup_batch_size was set to a huge value). No need to check > +# for an exact number, since that will require changing .result if > +# anything changes prior to this point, and we just need to know that > +# we have still have some data in the tables to make the following > +# test effective. > +SELECT COUNT(*)>=10 FROM mysql.gtid_slave_pos; > +SELECT COUNT(*)>=10 FROM ( SELECT * FROM mysql.gtid_slave_pos_innodb > + UNION ALL SELECT * FROM mysql.gtid_slave_pos_innodb_redundant) inner_select; > +SELECT COUNT(*)>=10 FROM mysql.gtid_slave_pos_tokudb; > + > +# Check that old GTID rows will be deleted when batch delete size is > +# set reasonably. Old row deletion is not 100% deterministic (by design), so > +# we must wait for it to occur, but it should occur eventually. > +SET GLOBAL gtid_cleanup_batch_size = 3; > +let $i=40; > +--disable_query_log > +--let $keep_include_silent=1 > +while ($i) { > + let N=`SELECT 1+($i MOD 3)`; > + --connection server_1 > + eval UPDATE t$N SET a=a+1 WHERE a=(SELECT MAX(a) FROM t$N); > + --source include/save_master_gtid.inc > + --connection server_2 > + --source include/sync_with_master_gtid.inc > + let $j=50; > + while ($j) { > + let $is_done=`SELECT SUM(a)=1 FROM ( > + SELECT COUNT(*) AS a FROM mysql.gtid_slave_pos > + UNION ALL > + SELECT COUNT(*) AS a FROM ( SELECT * FROM mysql.gtid_slave_pos_innodb > + UNION ALL SELECT * FROM mysql.gtid_slave_pos_innodb_redundant) inner_select > + UNION ALL > + SELECT COUNT(*) AS a FROM mysql.gtid_slave_pos_tokudb) outer_select`; > + if ($is_done) { > + let $j=0; > + } > + if (!$is_done) { > + real_sleep 0.1; > + dec $j; > + } > + } > + dec $i; > + if ($is_done) { > + let $i=0; > + } > +} > +--enable_query_log > +--let $keep_include_silent=0 > +if (!$is_done) { > + --echo Timed out waiting for mysql.gtid_slave_pos* tables to be cleaned up > +} > + > +--disable_query_log > +DELETE FROM t1 WHERE a >= 100; > +DELETE FROM t2 WHERE a >= 100; > +DELETE FROM t3 WHERE a >= 100; > +--enable_query_log > + > + > # Test status variables Rpl_transactions_multi_engine and Transactions_gtid_foreign_engine. > # Have mysql.gtid_slave_pos* for myisam and innodb but not tokudb. > --connection server_2 > @@ -223,6 +305,9 @@ SHOW STATUS LIKE "%transactions%engine"; > SET sql_log_bin=0; > DROP TABLE mysql.gtid_slave_pos_innodb; > SET sql_log_bin=1; > +--disable_query_log > +eval SET GLOBAL gtid_cleanup_batch_size = $old_gtid_cleanup_batch_size; > +--enable_query_log > > --connection server_1 > DROP TABLE t1; > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

2 3

Re: [Maria-developers] [Commits] 2da93752430: MDEV-17255: New optimizer defaults and ANALYZE TABLE
by Sergey Petrunia 06 Dec '18

06 Dec '18

Hi Varun, Nice to see that most of the input has been addressed. However, the issue of missing test coverage is still not resolved: On Sat, Nov 24, 2018 at 09:47:57PM +0300, Sergey Petrunia wrote: >> Please also add test coverage for this particular MDEV. This should be a >> testcase that shows that >> >> - with PREFERABLY, ANALYZE will collect EITS stats, SELECT will use them. >> >> - with PREFERABLY_FOR_READS : >> -- ANALYZE will not collect EITS stats >> -- ANALYZE .. PERSISTENT FOR ... will update the EITS stats. >> -- SELECT will use them. and now it is still not addressed. I made this change: @@ -104,6 +104,8 @@ Use_stat_tables_mode get_use_stat_tables_mode(THD *thd) inline bool check_eits_collection_allowed(THD *thd) { + if (get_use_stat_tables_mode(thd) == PREFERABLY_FOR_QUERIES) + DBUG_ASSERT(0); return (get_use_stat_tables_mode(thd) == COMPLEMENTARY || get_use_stat_tables_mode(thd) == PREFERABLY); } and the testsuite still passes! On Wed, Dec 05, 2018 at 07:57:06PM +0530, Varun wrote: > revision-id: 2da93752430d0dd07344c9d62f2468bba8962b77 (mariadb-10.3.6-266-g2da93752430) > parent(s): e0739064450f2c2be6d5de1d799582121747dd39 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2018-12-05 19:56:23 +0530 > message: > > MDEV-17255: New optimizer defaults and ANALYZE TABLE > > Added to new values to the server variable use_stat_tables. > The values are COMPLEMENTARY_FOR_QUERIES and PREFERABLY_FOR_QUERIES. > Both these values don't allow to collect EITS for queries like > analyze table t1; > To collect EITS we would need to use the syntax with persistent like > analyze table t1 persistent for columns (col1,col2...) index (idx1, idx2...) / ALL > > Changing the default value from NEVER to PREFERABLY_FOR_QUERIES. > > --- > mysql-test/include/default_mysqld.cnf | 1 + > mysql-test/main/stat_tables.result | 59 +++++++++++++++++++++++++++++++ > mysql-test/main/stat_tables.test | 43 ++++++++++++++++++++++ > mysql-test/main/stat_tables_innodb.result | 59 +++++++++++++++++++++++++++++++ > sql/sql_admin.cc | 2 +- > sql/sql_statistics.cc | 5 ++- > sql/sql_statistics.h | 27 ++++++++++++++ > sql/sys_vars.cc | 5 +-- > 8 files changed, 195 insertions(+), 6 deletions(-) > > diff --git a/mysql-test/include/default_mysqld.cnf b/mysql-test/include/default_mysqld.cnf > index 69a2b58288b..5ba3bdeb92c 100644 > --- a/mysql-test/include/default_mysqld.cnf > +++ b/mysql-test/include/default_mysqld.cnf > @@ -107,6 +107,7 @@ loose-performance-schema-consumer-thread-instrumentation=ON > binlog-direct-non-transactional-updates > > default-storage-engine=myisam > +use-stat-tables=preferably_for_queries > > loose-ssl-ca=(a)ENV.MYSQL_TEST_DIR/std_data/cacert.pem > loose-ssl-cert=(a)ENV.MYSQL_TEST_DIR/std_data/server-cert.pem > diff --git a/mysql-test/main/stat_tables.result b/mysql-test/main/stat_tables.result > index 308529ece47..67bf3f9237d 100644 > --- a/mysql-test/main/stat_tables.result > +++ b/mysql-test/main/stat_tables.result > @@ -605,4 +605,63 @@ SELECT MAX(pk) FROM t1; > MAX(pk) > NULL > DROP TABLE t1; > +# > +# MDEV-17255: New optimizer defaults and ANALYZE TABLE > +# > +set @save_use_stat_tables= @@use_stat_tables; > +create table t1 (a int, b int); > +insert into t1(a,b) values (1,2),(1,3),(1,4),(1,5),(2,6),(2,7),(3,8),(3,9),(3,9),(4,10); > +# > +# with use_stat_tables= PREFERABLY_FOR_QUERIES > +# analyze table t1 will not collect statistics > +# > +analyze table t1; > +Table Op Msg_type Msg_text > +test.t1 analyze status Engine-independent statistics collected > +test.t1 analyze status OK > +select * from mysql.column_stats; > +db_name table_name column_name min_value max_value nulls_ratio avg_length avg_frequency hist_size hist_type histogram > +test t1 a 1 4 0.0000 4.0000 2.5000 0 NULL NULL > +test t1 b 2 10 0.0000 4.0000 1.1111 0 NULL NULL > +analyze > +select * from t1 where a = 1 and b=3; > +id select_type table type possible_keys key key_len ref rows r_rows filtered r_filtered Extra > +1 SIMPLE t1 ALL NULL NULL NULL NULL 10 10.00 2.78 10.00 Using where > +# > +# with use_stat_tables= PREFERABLY_FOR_QUERIES > +# analyze table t1 will collect statistics if we use PERSISTENT > +# for columns, indexes or everything > +# > +analyze table t1 persistent for columns (a) indexes (); > +Table Op Msg_type Msg_text > +test.t1 analyze status Engine-independent statistics collected > +test.t1 analyze status Table is already up to date > +select * from mysql.column_stats; > +db_name table_name column_name min_value max_value nulls_ratio avg_length avg_frequency hist_size hist_type histogram > +test t1 a 1 4 0.0000 4.0000 2.5000 0 NULL NULL > +test t1 b 2 10 0.0000 4.0000 1.1111 0 NULL NULL > +# filtered shows that we used the data from stat tables > +analyze > +select * from t1 where a = 1 and b=3; > +id select_type table type possible_keys key key_len ref rows r_rows filtered r_filtered Extra > +1 SIMPLE t1 ALL NULL NULL NULL NULL 10 10.00 2.78 10.00 Using where > +# > +# with use_stat_tables= PREFERABLY > +# analyze table t1 will collect statistics > +# > +set @@use_stat_tables=PREFERABLY; > +analyze table t1; > +Table Op Msg_type Msg_text > +test.t1 analyze status Engine-independent statistics collected > +test.t1 analyze status Table is already up to date > +select * from mysql.column_stats; > +db_name table_name column_name min_value max_value nulls_ratio avg_length avg_frequency hist_size hist_type histogram > +test t1 a 1 4 0.0000 4.0000 2.5000 0 NULL NULL > +test t1 b 2 10 0.0000 4.0000 1.1111 0 NULL NULL > +# filtered shows that we used the data from stat tables > +analyze > +select * from t1 where a=1 and b=3; > +id select_type table type possible_keys key key_len ref rows r_rows filtered r_filtered Extra > +1 SIMPLE t1 ALL NULL NULL NULL NULL 10 10.00 2.78 10.00 Using where > +drop table t1; > set use_stat_tables=@save_use_stat_tables; > diff --git a/mysql-test/main/stat_tables.test b/mysql-test/main/stat_tables.test > index 19bc0fa2f46..8616f6386e1 100644 > --- a/mysql-test/main/stat_tables.test > +++ b/mysql-test/main/stat_tables.test > @@ -384,4 +384,47 @@ SELECT MAX(pk) FROM t1; > > DROP TABLE t1; > > +--echo # > +--echo # MDEV-17255: New optimizer defaults and ANALYZE TABLE > +--echo # > + > +set @save_use_stat_tables= @@use_stat_tables; > +create table t1 (a int, b int); > +insert into t1(a,b) values (1,2),(1,3),(1,4),(1,5),(2,6),(2,7),(3,8),(3,9),(3,9),(4,10); > + > +--echo # > +--echo # with use_stat_tables= PREFERABLY_FOR_QUERIES > +--echo # analyze table t1 will not collect statistics > +--echo # > + > +analyze table t1; > +select * from mysql.column_stats; > +analyze > +select * from t1 where a = 1 and b=3; > + > +--echo # > +--echo # with use_stat_tables= PREFERABLY_FOR_QUERIES > +--echo # analyze table t1 will collect statistics if we use PERSISTENT > +--echo # for columns, indexes or everything > +--echo # > + > +analyze table t1 persistent for columns (a) indexes (); > +select * from mysql.column_stats; > +--echo # filtered shows that we used the data from stat tables > +analyze > +select * from t1 where a = 1 and b=3; > + > +--echo # > +--echo # with use_stat_tables= PREFERABLY > +--echo # analyze table t1 will collect statistics > +--echo # > + > +set @@use_stat_tables=PREFERABLY; > +analyze table t1; > +select * from mysql.column_stats; > +--echo # filtered shows that we used the data from stat tables > +analyze > +select * from t1 where a=1 and b=3; > +drop table t1; > + > set use_stat_tables=@save_use_stat_tables; > diff --git a/mysql-test/main/stat_tables_innodb.result b/mysql-test/main/stat_tables_innodb.result > index 8198e94dc10..4bb8f34712c 100644 > --- a/mysql-test/main/stat_tables_innodb.result > +++ b/mysql-test/main/stat_tables_innodb.result > @@ -632,6 +632,65 @@ SELECT MAX(pk) FROM t1; > MAX(pk) > NULL > DROP TABLE t1; > +# > +# MDEV-17255: New optimizer defaults and ANALYZE TABLE > +# > +set @save_use_stat_tables= @@use_stat_tables; > +create table t1 (a int, b int); > +insert into t1(a,b) values (1,2),(1,3),(1,4),(1,5),(2,6),(2,7),(3,8),(3,9),(3,9),(4,10); > +# > +# with use_stat_tables= PREFERABLY_FOR_QUERIES > +# analyze table t1 will not collect statistics > +# > +analyze table t1; > +Table Op Msg_type Msg_text > +test.t1 analyze status Engine-independent statistics collected > +test.t1 analyze status OK > +select * from mysql.column_stats; > +db_name table_name column_name min_value max_value nulls_ratio avg_length avg_frequency hist_size hist_type histogram > +test t1 a 1 4 0.0000 4.0000 2.5000 0 NULL NULL > +test t1 b 2 10 0.0000 4.0000 1.1111 0 NULL NULL > +analyze > +select * from t1 where a = 1 and b=3; > +id select_type table type possible_keys key key_len ref rows r_rows filtered r_filtered Extra > +1 SIMPLE t1 ALL NULL NULL NULL NULL 10 10.00 2.78 10.00 Using where > +# > +# with use_stat_tables= PREFERABLY_FOR_QUERIES > +# analyze table t1 will collect statistics if we use PERSISTENT > +# for columns, indexes or everything > +# > +analyze table t1 persistent for columns (a) indexes (); > +Table Op Msg_type Msg_text > +test.t1 analyze status Engine-independent statistics collected > +test.t1 analyze status OK > +select * from mysql.column_stats; > +db_name table_name column_name min_value max_value nulls_ratio avg_length avg_frequency hist_size hist_type histogram > +test t1 a 1 4 0.0000 4.0000 2.5000 0 NULL NULL > +test t1 b 2 10 0.0000 4.0000 1.1111 0 NULL NULL > +# filtered shows that we used the data from stat tables > +analyze > +select * from t1 where a = 1 and b=3; > +id select_type table type possible_keys key key_len ref rows r_rows filtered r_filtered Extra > +1 SIMPLE t1 ALL NULL NULL NULL NULL 10 10.00 2.78 10.00 Using where > +# > +# with use_stat_tables= PREFERABLY > +# analyze table t1 will collect statistics > +# > +set @@use_stat_tables=PREFERABLY; > +analyze table t1; > +Table Op Msg_type Msg_text > +test.t1 analyze status Engine-independent statistics collected > +test.t1 analyze status OK > +select * from mysql.column_stats; > +db_name table_name column_name min_value max_value nulls_ratio avg_length avg_frequency hist_size hist_type histogram > +test t1 a 1 4 0.0000 4.0000 2.5000 0 NULL NULL > +test t1 b 2 10 0.0000 4.0000 1.1111 0 NULL NULL > +# filtered shows that we used the data from stat tables > +analyze > +select * from t1 where a=1 and b=3; > +id select_type table type possible_keys key key_len ref rows r_rows filtered r_filtered Extra > +1 SIMPLE t1 ALL NULL NULL NULL NULL 10 10.00 2.78 10.00 Using where > +drop table t1; > set use_stat_tables=@save_use_stat_tables; > set optimizer_switch=@save_optimizer_switch_for_stat_tables_test; > SET SESSION STORAGE_ENGINE=DEFAULT; > diff --git a/sql/sql_admin.cc b/sql/sql_admin.cc > index d0d959de8f9..b39103e382a 100644 > --- a/sql/sql_admin.cc > +++ b/sql/sql_admin.cc > @@ -767,7 +767,7 @@ static bool mysql_admin_table(THD* thd, TABLE_LIST* tables, > } > collect_eis= > (table->table->s->table_category == TABLE_CATEGORY_USER && > - (get_use_stat_tables_mode(thd) > NEVER || > + (check_eits_collection_allowed(thd) || > lex->with_persistent_for_clause)); > > > diff --git a/sql/sql_statistics.cc b/sql/sql_statistics.cc > index 04806f07b3b..8c88f7f927f 100644 > --- a/sql/sql_statistics.cc > +++ b/sql/sql_statistics.cc > @@ -3720,9 +3720,8 @@ void set_statistics_for_table(THD *thd, TABLE *table) > { > TABLE_STATISTICS_CB *stats_cb= &table->s->stats_cb; > Table_statistics *read_stats= stats_cb->table_stats; > - Use_stat_tables_mode use_stat_table_mode= get_use_stat_tables_mode(thd); > table->used_stat_records= > - (use_stat_table_mode <= COMPLEMENTARY || > + (!check_eits_preferred(thd) || > !table->stats_is_read || read_stats->cardinality_is_null) ? > table->file->stats.records : read_stats->cardinality; > KEY *key_info, *key_info_end; > @@ -3730,7 +3729,7 @@ void set_statistics_for_table(THD *thd, TABLE *table) > key_info < key_info_end; key_info++) > { > key_info->is_statistics_from_stat_tables= > - (use_stat_table_mode > COMPLEMENTARY && > + (check_eits_preferred(thd) && > table->stats_is_read && > key_info->read_stats->avg_frequency_is_inited() && > key_info->read_stats->get_avg_frequency(0) > 0.5); > diff --git a/sql/sql_statistics.h b/sql/sql_statistics.h > index 39cddf95188..8439ac8db53 100644 > --- a/sql/sql_statistics.h > +++ b/sql/sql_statistics.h > @@ -16,12 +16,26 @@ > #ifndef SQL_STATISTICS_H > #define SQL_STATISTICS_H > > +/* > + For COMPLEMENTARY_FOR_QUERIES and PREFERABLY_FOR_QUERIES they are > + similar to the COMPLEMENTARY and PREFERABLY respectively except that > + with these values we would not be collecting EITS for queries like > + ANALYZE TABLE t1; > + To collect EITS with these values, we have to use PERSISITENT FOR > + analyze table t1 persistent for > + columns (col1,col2...) index (idx1, idx2...) > + or > + analyze table t1 persistent for all > +*/ > + > typedef > enum enum_use_stat_tables_mode > { > NEVER, > COMPLEMENTARY, > PREFERABLY, > + COMPLEMENTARY_FOR_QUERIES, > + PREFERABLY_FOR_QUERIES > } Use_stat_tables_mode; > > typedef > @@ -87,6 +101,19 @@ Use_stat_tables_mode get_use_stat_tables_mode(THD *thd) > { > return (Use_stat_tables_mode) (thd->variables.use_stat_tables); > } > +inline > +bool check_eits_collection_allowed(THD *thd) > +{ > + return (get_use_stat_tables_mode(thd) == COMPLEMENTARY || > + get_use_stat_tables_mode(thd) == PREFERABLY); > +} > + > +inline > +bool check_eits_preferred(THD *thd) > +{ > + return (get_use_stat_tables_mode(thd) == PREFERABLY || > + get_use_stat_tables_mode(thd) == PREFERABLY_FOR_QUERIES); > +} > > int read_statistics_for_tables_if_needed(THD *thd, TABLE_LIST *tables); > int collect_statistics_for_table(THD *thd, TABLE *table); > diff --git a/sql/sys_vars.cc b/sql/sys_vars.cc > index 420e7feabde..b925de34db4 100644 > --- a/sql/sys_vars.cc > +++ b/sql/sys_vars.cc > @@ -5842,12 +5842,13 @@ static Sys_var_ulong Sys_progress_report_time( > VALID_RANGE(0, UINT_MAX), DEFAULT(5), BLOCK_SIZE(1)); > > const char *use_stat_tables_modes[] = > - {"NEVER", "COMPLEMENTARY", "PREFERABLY", 0}; > + {"NEVER", "COMPLEMENTARY", "PREFERABLY", > + "COMPLEMENTARY_FOR_QUERIES", "PREFERABLY_FOR_QUERIES", 0}; > static Sys_var_enum Sys_optimizer_use_stat_tables( > "use_stat_tables", > "Specifies how to use system statistics tables", > SESSION_VAR(use_stat_tables), CMD_LINE(REQUIRED_ARG), > - use_stat_tables_modes, DEFAULT(2)); > + use_stat_tables_modes, DEFAULT(4)); > > static Sys_var_ulong Sys_histogram_size( > "histogram_size", > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] af6a8ea0f79: MDEV-16266 : New command FLUSH SSL ro reload SSL acceptor context.
by Sergei Golubchik 06 Dec '18

06 Dec '18

Hi, Vladislav! Again, that's quite ok. I don't like a new lock around sslaccept(), but I couldn't find any other solution for it :( See few comments below. On Dec 03, Vladislav Vaintroub wrote: > revision-id: af6a8ea0f79d628384a61986482dec7d3ff751dd (mariadb-10.4.0-55-gaf6a8ea0f79) > parent(s): 757530b83ca1cc550127311dc6edb47152ba0c6a > author: Vladislav Vaintroub <wlad(a)mariadb.com> > committer: Vladislav Vaintroub <wlad(a)mariadb.com> > timestamp: 2018-12-02 01:25:04 +0100 > message: > > MDEV-16266 : New command FLUSH SSL ro reload SSL acceptor context. > > Reloads certificate, key, CA, CRL. > > diff --git a/mysql-test/main/flush_ssl.test b/mysql-test/main/flush_ssl.test > new file mode 100644 > index 00000000000..24dbb4719f3 > --- /dev/null > +++ b/mysql-test/main/flush_ssl.test > @@ -0,0 +1,24 @@ > +source include/have_ssl_communication.inc; > + > +connect ssl_con,localhost,root,,,,,SSL; > +SELECT VARIABLE_VALUE != '0' FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME='SSL_ACCEPTS'; > +FLUSH SSL; > + > +# Check if SSL_ACCEPTS was flushed (SSL_ACCEPTS = 0) > +SELECT VARIABLE_VALUE FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME='SSL_ACCEPTS'; How about the test that a certificate was actually reloaded? That's the main functionality, right? Nobody wanted a new command just to reset SSL_ACCEPTS Note, that if you want to generate a new certificate for mysql-test, don't forget to extend mysql-test/lib/generate-ssl-certs.sh > + > +CREATE USER u; > +connect ssl_con2,localhost,u,,,,,SSL; > + > +# Check SSL_ACCEPTS increased to 1 > +SELECT VARIABLE_VALUE FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME='SSL_ACCEPTS'; > + > +# Check that FLUSH SSL does not work for unprivileged user > +error ER_SPECIFIC_ACCESS_DENIED_ERROR; > +FLUSH SSL; > + > +disconnect ssl_con2; > +disconnect ssl_con; > +connection default; > +DROP USER u; > + > diff --git a/sql/mysqld.cc b/sql/mysqld.cc > index 9ff47dc1ff1..13aad0a0197 100644 > --- a/sql/mysqld.cc > +++ b/sql/mysqld.cc > @@ -4766,6 +4771,56 @@ static void openssl_lock(int mode, openssl_lock_t *lock, const char *file, > } > #endif /* HAVE_OPENSSL10 */ > > + > +struct SSL_ACCEPTOR_STATS > +{ > + long accept; > + long accept_good; > + long cache_size; > + long verify_mode; > + long verify_depth; > + const char *session_cache_mode; > + long zero; > + SSL_ACCEPTOR_STATS() > + { > + session_cache_mode = "NONE"; are you relying on everything else being zero'ed by default because SSL_ACCEPTOR_STATS is instantiated statically? May be, still, add a bzero? > + } > + void init() > + { > + DBUG_ASSERT(ssl_acceptor_fd); > + accept = 0; > + accept_good = 0; > + verify_mode = SSL_CTX_get_verify_mode(ssl_acceptor_fd->ssl_context); > + verify_depth = SSL_CTX_get_verify_depth(ssl_acceptor_fd->ssl_context); > + cache_size = SSL_CTX_sess_get_cache_size(ssl_acceptor_fd->ssl_context); > + switch (SSL_CTX_get_session_cache_mode(ssl_acceptor_fd->ssl_context)) > + { > + case SSL_SESS_CACHE_OFF: > + session_cache_mode = "OFF"; break; > + case SSL_SESS_CACHE_CLIENT: > + session_cache_mode = "CLIENT"; break; > + case SSL_SESS_CACHE_SERVER: > + session_cache_mode = "SERVER"; break; > + case SSL_SESS_CACHE_BOTH: > + session_cache_mode = "BOTH"; break; > + case SSL_SESS_CACHE_NO_AUTO_CLEAR: > + session_cache_mode = "NO_AUTO_CLEAR"; break; > + case SSL_SESS_CACHE_NO_INTERNAL_LOOKUP: > + session_cache_mode = "NO_INTERNAL_LOOKUP"; break; > + default: > + session_cache_mode = "Unknown"; break; > + } > + } > +}; > + > +static SSL_ACCEPTOR_STATS ssl_acceptor_stats; > +void ssl_acceptor_stats_update(int sslaccept_ret) > +{ > + ssl_acceptor_stats.accept++; > + if (!sslaccept_ret) > + ssl_acceptor_stats.accept_good++; use statistic_increment here > +} > + > static void init_ssl() > { > #if defined(HAVE_OPENSSL) && !defined(EMBEDDED_LIBRARY) > @@ -4804,6 +4862,38 @@ static void init_ssl() > #endif /* HAVE_OPENSSL && ! EMBEDDED_LIBRARY */ > } > > +/* Reinitialize SSL (FLUSH SSL) */ > +int reinit_ssl() > +{ > +#if defined(HAVE_OPENSSL) && !defined(EMBEDDED_LIBRARY) > + if (!opt_use_ssl) > + return 0; > +#ifndef DBUG_OFF > + char *old_ssl_cert; it's generally prefered to use __attribute__((unused)) instead of #ifdef > +#endif > + > + DBUG_EXECUTE_IF("simulate_bad_ssl_cert", old_ssl_cert= opt_ssl_cert; opt_ssl_cert=const_cast<char *>("");); > + > + enum enum_ssl_init_error error = SSL_INITERR_NOERROR; > + st_VioSSLFd *new_fd = new_VioSSLAcceptorFd(opt_ssl_key, opt_ssl_cert, > + opt_ssl_ca, opt_ssl_capath, opt_ssl_cipher, &error, opt_ssl_crl, opt_ssl_crlpath); > + > + DBUG_EXECUTE_IF("simulate_bad_ssl_cert", opt_ssl_cert= old_ssl_cert;); > + > + if (!new_fd) > + { > + my_printf_error(ER_UNKNOWN_ERROR, "Failed to refresh SSL, error: %s", MYF(0), > + sslGetErrString(error)); > + return 1; > + } > + mysql_rwlock_wrlock(&LOCK_ssl_refresh); > + free_vio_ssl_acceptor_fd(ssl_acceptor_fd); > + ssl_acceptor_fd= new_fd; > + ssl_acceptor_stats.init(); > + mysql_rwlock_unlock(&LOCK_ssl_refresh); > + return 0; > +#endif > +} > Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] MDEV-16188: Rowid filtering: EXPLAIN FORMAT=JSON support
by Sergey Petrunia 05 Dec '18

05 Dec '18

Hello Igor, I have pushed into 10.4-mdev16188 tree a patch that adds EXPLAIN FORMAT=JSON support. There is no ANALYZE support, yet. main.rowid_filter has an example of EXPLAIN FORMAT=JSON at the bottom. The filter is described like so: + "rowid_filter": { + "range": { + "key": "i_l_shipdate", + "used_key_parts": ["l_shipDATE"] + }, + "rows": 8, + "selectivity_pct": 0.1332 + }, Here "rows" is how many rows we will get from the quick select that is used to build the filter. selectivity_pct is the selectivity of the filter. You will also notice this change from (1%) to (0%): -1 SIMPLE lineitem range|filter PRIMARY,i_l_shipdate,i_l_receiptdate,i_l_orderkey,i_l_orderkey_quantity i_l_receiptdate|i_l_shipdate 4|4 NULL 6 (1%) Using index condition; Using where; Using filter +1 SIMPLE lineitem range|filter PRIMARY,i_l_shipdate,i_l_receiptdate,i_l_orderkey,i_l_orderkey_quantity i_l_receiptdate|i_l_shipdate 4|4 NULL 6 (0%) Using index condition; Using where; Using filter This is because the original code had a piece of logic in JOIN_TAB::save_filter_explain_data: print "1%" if it was "0%". I'm wondering, is there any reason we cannot print the number with greater precision in this case? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] 2397c00086c: MDEV-15649 Speedup search in acl_users and acl_dbs array,
by Sergei Golubchik 03 Dec '18

03 Dec '18

Hi, Vladislav! Okay. Frankly, that's not how I'd done it, but it'll work. Only a few minor comments below, then ok to push. On Dec 03, Vladislav Vaintroub wrote: > revision-id: 2397c00086c4d9df5e2bbad70803e293f5a47984 (mariadb-10.4.0-37-g2397c00086c) > parent(s): 46a411088c5abbacda4e52e89f4dbcf52a451f7a > author: Vladislav Vaintroub <wlad(a)mariadb.com> > committer: Vladislav Vaintroub <wlad(a)mariadb.com> > timestamp: 2018-11-30 18:14:00 +0100 > message: > > MDEV-15649 Speedup search in acl_users and acl_dbs array, > sorting them by usernames first, and then by get_sort() value. > > Search functions now use binary search to find the the first entry with > given name. Then, linear search is done, until the first match. > > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index 052c5ada3a2..932c4189e3f 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -2326,6 +2328,156 @@ static int acl_compare(ACL_ACCESS *a,ACL_ACCESS *b) > + > + > +static inline const char *get_username(const ACL_DB& acl_db) > +{ > + return acl_db.user; > +} > + > + > +static inline const char *get_username(const ACL_USER& acl_user) > +{ > + return acl_user.user.str; > +} I'd do them as methods in ACL_DB and ACL_USER > + > +/* > + Return index of the first entry with given user in the array, > + or UINT_MAX if not found. > + > + Assumes the array is sorted by get_username > +*/ > +template<typename T> uint find_first_user(T* arr, uint len, const char *user) > +{ > + uint low= 0; > + uint high= len - 1; > + uint mid; > + if(!len) > + return UINT_MAX; > + > +#ifndef DBUG_OFF > + for (uint i = 0; i < len - 1; i++) > + DBUG_ASSERT(strcmp(get_username(arr[i]), get_username(arr[i + 1])) <= 0); > +#endif > + while (low <= high) > + { > + mid = low + (high - low) / 2; > + int cmp = strcmp(get_username(arr[mid]), user); > + if (cmp == 0) > + { > + /* > + We are looking for the left-most element in the array with the given value. > + Thus, even if value is found, we still have to check the left neighbor, > + and if it has the same value, continue the search. > + */ > + if (mid > 0 && !strcmp(get_username(arr[mid - 1]), user)) > + cmp = 1; hmm, I thought, normally it's done not by checking the left neighbor, but by `if (cmp <=0 )` and continuing the search until low>high > + } > + > + if (cmp > 0) > + { > + if (mid == 0) > + break; > + high= mid - 1; > + } > + else if (cmp < 0) > + low= mid + 1; > + else > + return mid; > + } > + return UINT_MAX; > +} > + > +static ACL_DB *acl_db_find(const char *db, const char *user, const char *host, const char *ip, my_bool db_is_pattern) > +{ > + uint i; > + ACL_DB *ret = NULL; > + > + // Check databases with matching user name. > + uint start= acl_find_db_by_username(user); > + for (i= start; i < acl_dbs.elements; i++) > + { > + ACL_DB *acl_db= dynamic_element(&acl_dbs, i, ACL_DB*); > + if (i > start && strcmp(user, acl_db->user)) > + break; > + > + if(acl_db_matches(acl_db, db, host, ip,db_is_pattern)) > + { > + ret= acl_db; > + break; > + } > + } > + > + // Look also for anonymous user (at the start of array due to sort by name). > + for (i = 0; i < acl_dbs.elements; i++) > + { > + ACL_DB *acl_db= dynamic_element(&acl_dbs, i, ACL_DB*); > + if (*acl_db->user || (ret && acl_compare(acl_db, ret) >= 0)) > + break; > + if (acl_db_matches(acl_db, db, host, ip, db_is_pattern)) > + { > + ret= acl_db; > + break; > + } > + } > + return ret; > +} that's the same logic as in find_user_or_anon(). May be you could template that out too? Another thought - this is a pretty weird-looking search logic. Please add a comment that its goal is to emulate historical behavior, where entries were found by a linear search in an array sorted by the get_sort() value. > @@ -3437,10 +3579,14 @@ static ACL_USER * find_user_wild(const char *host, const char *user, const char > { > mysql_mutex_assert_owner(&acl_cache->lock); > > - for (uint i=0 ; i < acl_users.elements ; i++) > + uint start = acl_find_user_by_name(user); > + > + for (uint i= start; i < acl_users.elements; i++) > { > ACL_USER *acl_user=dynamic_element(&acl_users,i,ACL_USER*); > - if (acl_user->wild_eq(user, host, ip)) This makes wild_eq method kind of pointless. Remove it? > + if (i > start && strcmp(acl_user->user.str, user)) > + break; > + if (compare_hostname(&acl_user->host, host, ip ? ip : host)) > return acl_user; > } > return 0; Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] Do we support compilers that support C++11 but not thread_local?
by Sergey Petrunia 27 Nov '18

27 Nov '18

Hi, I'm writing this in connection with MyRocks Storage Engine and https://github.com/facebook/mysql-5.6/issues/904 C++ 11 includes "thread_local" specifier : https://en.cppreference.com/w/cpp/keyword/thread_local GNU also has __thread, which was available before C++11. RocksDB (and so MyRocks) uses both __thread and thread_local. thread_local is part of the standard, so they might switch to thread_local. The question: will it cause any issues for us? In other words, do we have compilers/environments that - claim to support C++11 (so that we attempt to compile MyRocks) - but do not support "thread_local" ? For Windows, RocksDB does this in port/win/port_win.h: #ifndef __thread #define __thread __declspec(thread) #endif but this page page about Visual Studio 2015) https://msdn.microsoft.com/en-us/library/6yh4a9k1.aspx says that C++11: The thread_local storage class specifier is the recommended way to specify thread-local storage for objects and class members. I assume this means it's ok to use thread_local on Windows? RocksDB source code mentions one problematic case: // However, thread_local is not supported in all compilers that accept -std=c++11 // (e.g., eg Mac with XCode < 8. XCode 8+ supports thread_local). As far as I understand this is not relevant for us? Anyone aware of other such cases? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

3 2

[Maria-developers] MDEV-17706 Review server changes introduced by Galera 4 wsrep patch
by Alexander Barkov 27 Nov '18

27 Nov '18

Hi Jan, I checked changes to the /sql directory so far. Sorry, I'm completely out of this topic. So I have only general suggestions. 1. General code structure. There are a lot of new #ifdef WITH_WRESP do_something_with_wresp(); #else do_something_without_wresp(); #endif This make the server code harder to follow. I'd prefer a more modular way. Would it be possible to introduce some abstract class with virtual methods? One instantiable class would implement behavior for wresp, and other class would implement behavior for "without wresp". Instead #ifdefs, we would do: behavour->do_something(); WRESP support could be then turned into a real dynamic loadable plugin, the server would be able to load it independently from the server compilation flags. Structures that need extra members, for example, "struct THD_TRANS" could be implemented like this: struct THD_TRANS_WRESP: public THD_TRANS { public: int rw_ha_count; void reset() { THD_TRANS::reset(); rw_ha_count= 0; } }; The virtual interface would do either "new THD_TRANS()" or "new THD_TRANS_WRESP()", depending on the WSREP support availability. 2. Please fix coding style in some places: + bool fix_length_and_dec() + { + max_length = WSREP_GTID_STR_LEN; + maybe_null = true; + return FALSE; + } It should: + bool fix_length_and_dec() + { + max_length= WSREP_GTID_STR_LEN; + maybe_null= true; + return false; + } There are more places that do not follow the coding style. Please check. 3. Why do we need a separate directory /wresp with pure C files? Are they shared by the server and by the clients? If not, why not to put this code together with other wsrep related files? 4. Some code was commented. Why not just remove it? Example: +#ifdef TODO DBUG_SLOW_ASSERT(global_status_var.global_memory_used == 0); +#endif Example: #ifdef WITH_WSREP - if (thd->wsrep_exec_mode == LOCAL_STATE) + // if (thd->wsrep_exec_mode == LOCAL_STATE) WSREP_TO_ISOLATION_BEGIN(WSREP_MYSQL_DB, NULL, NULL); #endif Example: +#include "log_event.h" +// #include "binlog.h" + Example: +int wsrep_write_dummy_event_low(THD *thd, const char *msg) +{ + ::abort(); + return 0; +#if 0 + int ret= 0; .... a lot of dead code + return ret; +#endif +} Example: +int wsrep_write_dummy_event(THD *orig_thd, const char *msg) +{ + return 0; +#if 0 ... a lot of dead code +#endif +} Example: +//#ifdef GTID_SUPPORT +void wsrep_init_sidno(const wsrep::id& uuid) ..... +//#endif /* GTID_SUPPORT */ Example: +static std::string wsrep_server_id() +{ +#ifdef WSREP_TODO + std::stringstream ss; + ss << server_id; + return(ss.str()); +#endif + /* us Example: + // enum wsrep::provider::status ret = wsrep_sync_wait_upto(thd, NULL, -1); + if (thd->wsrep_cs().sync_wait(-1)) Example: +#if UNUSED /* 323f269d4099 (Jan Lindström 2018-07-19) */ static const char* wsrep_get_query_or_msg(const THD* thd) { sw ... a lot of unused code } +#endif //UNUSED Example: +#if 0 + /* todo */ + wsrep_xid_init(&thd->wsrep_xid, + thd->wsrep_trx_meta.gtid.uuid, + thd->wsrep_trx_meta.gtid.seqno); + if (tc_log) tc_log->commit(thd, true); +#endif /* 0 */ Example: +void* start_wsrep_THD(void *arg) +{ + THD *thd; + //wsrep_thd_processor_fun processor= (wsrep_thd_processor_fun)arg; Example: +//wsrep_t *get_wsrep() +//{ +// return wsrep; +//} Example: +//int wsrep_show_status(THD *thd, SHOW_VAR *var, char *buff, +// enum enum_var_type scope); There are many other pieces of the dead code. 5. Why this change? --- a/sql/sql_basic_types.h +++ b/sql/sql_basic_types.h @@ -20,6 +20,8 @@ #ifndef SQL_TYPES_INCLUDED #define SQL_TYPES_INCLUDED +#include <my_global.h> + 6. sql/wsrep_mysqld.cc: something is wrong with indentation: case 1: case 2: case 3: + case 4: 7. According to our coding style, multi-line comments like this: + // Note: We can't call THD destructor without crashing + // if plugins have not been initialized. However, in most of the + // cases this means that pre SE initialization SST failed and + // we are going to exit anyway. should be: /* Note: We can't call THD destructor without crashing if plugins have not been initialized. However, in most of the cases this means that pre SE initialization SST failed and */ we are going to exit anyway. 8. This looks strange: +const char wsrep_defaults_group_suffix[256] = {0}; Why do you need an array of 256 zero bytes in the const static area? 9. You remove a lot of code in one place and add new code in other places. It seems you move some old code to new locations. For example, this code was removed from sql/wsrep_thd.cc.diff: -void wsrep_replay_transaction(THD *thd) I can see its pieces in wsrep_high_priority_service.h, in the class Wsrep_replayer_service. Can refactoring changes like this be done in a separate patch? It would make reviewing easier. 10. Please don't use redundant current_thd calls when a THD pointer is already available in a local variable or a function parameter. Example: @@ -2457,7 +2460,7 @@ void Item_func_rand::seed_random(Item *arg) THD *thd= current_thd; if (WSREP(thd)) { - if (thd->wsrep_exec_mode==REPL_RECV) + if (wsrep_thd_is_applying(current_thd)) 11. When does current_thd return NULL? + if (current_thd) + { + /* Undocking the thread specific data. */ + my_pthread_setspecific_ptr(THR_THD, NULL); + / 12. Perhaps ErrConv family classes could be used in some cases: This: + std::ostringstream os; + os << view.state_id().id(); + wsrep_update_cluster_state_uuid(os.str().c_str()); can probably be shortened to something like this: wsrep_update_cluster_state_uuid(ErrConvInteger(view.state_id().id()).ptr()); 13. There's something wrong with semicolons here: + /* DTRACE begin */ + MYSQL_QUERY_START(rawbuf, thd->thread_id, + (char *)(thd->db.str ? : thd->db.str : "unknown"), + &thd->security_ctx->priv_user[0], + (char *) thd->security_ctx->host_or_ip); Notice, "?" immediately followed by ":". This should not compile. Is it an optionally compiled code? Please make sure to compile in all possible build option combinations, to check all optional code.

2 1

[Maria-developers] Fwd: Re: [Commits] 68b7e231cb3: MDEV-17255: New optimizer defaults and ANALYZE TABLE
by Sergey Petrunia 27 Nov '18

27 Nov '18

----- Forwarded message from Igor Babaev <igor(a)mariadb.com> ----- Date: Sun, 25 Nov 2018 11:45:34 -0800 From: Igor Babaev <igor(a)mariadb.com> To: Sergey Petrunia <sergey(a)mariadb.com> Subject: Fwd: Re: [Commits] 68b7e231cb3: MDEV-17255: New optimizer defaults and ANALYZE TABLE Hi Sergey, PREFERABLY_FOR_READS sounds confusing, because the stat tables still are used for UPDATE/DELETE. Maybe PREFERABLY_FOR_QUERIES? Besides, don't forget that you need COMPLEMENTARY_FOR_READS[QUERIES] as well. Regards, Igor. -------- Forwarded Message -------- Subject: Re: [Commits] 68b7e231cb3: MDEV-17255: New optimizer defaults and ANALYZE TABLE Date: Sat, 24 Nov 2018 21:47:57 +0300 From: Sergey Petrunia <sergey(a)mariadb.com> Reply-To: maria-developers(a)lists.launchpad.net To: Varun <varunraiko1803(a)gmail.com>, maria-developers(a)lists.launchpad.net CC: commits(a)mariadb.org Hi Varun, On Fri, Nov 16, 2018 at 11:48:59PM +0530, Varun wrote: > revision-id: 68b7e231cb3a1cdfbd968c2cbf72687b29b6d2da (mariadb-10.3.6-209-g68b7e231cb3) > parent(s): b9a9055793ab8b9a50200e2f55602463627dedd3 Does the above say the patch is going into 10.3? The MDEV has fixVersion=10.4 and the parent revision is also in 10.4. So I assume the target version is 10.4? > author: Varun Gupta > committer: Varun Gupta > timestamp: 2018-11-16 23:48:11 +0530 > message: > > MDEV-17255: New optimizer defaults and ANALYZE TABLE > > Added another value for the use_stat_tables system variable Please mention in the commit comment: - the name of the new value - the fact that the new value is now the default. > --- > sql/opt_range.cc | 2 +- > sql/sql_admin.cc | 3 ++- > sql/sql_statistics.h | 1 + > sql/sys_vars.cc | 4 ++-- > 4 files changed, 6 insertions(+), 4 deletions(-) Please run the testsuite. I did the 'main' testsuite and: - main.mysqld--help, main.stat_tables_disabled - these need updates - a few other tests need to be updated as now opening any table will now cause the server to attept to read EITS stats for it. Please also add test coverage for this particular MDEV. This should be a testcase that shows that - with PREFERABLY, ANALYZE will collect EITS stats, SELECT will use them. - with PREFERABLY_FOR_READS : -- ANALYZE will not collect EITS stats -- ANALYZE .. PERSISTENT FOR ... will update the EITS stats. -- SELECT will use them. . > > diff --git a/sql/opt_range.cc b/sql/opt_range.cc > index d6db365a8a2..f501a5ae085 100644 > --- a/sql/opt_range.cc > +++ b/sql/opt_range.cc > @@ -3158,7 +3158,7 @@ bool calculate_cond_selectivity_for_table(THD *thd, TABLE *table, Item **cond) > > if (thd->variables.optimizer_use_condition_selectivity > 2 && > !bitmap_is_clear_all(used_fields) && > - thd->variables.use_stat_tables > 0) > + get_use_stat_tables_mode(thd) > NEVER) > { > PARAM param; > MEM_ROOT alloc; > diff --git a/sql/sql_admin.cc b/sql/sql_admin.cc > index d0d959de8f9..b5c732737b7 100644 > --- a/sql/sql_admin.cc > +++ b/sql/sql_admin.cc > @@ -767,7 +767,8 @@ static bool mysql_admin_table(THD* thd, TABLE_LIST* tables, > } > collect_eis= > (table->table->s->table_category == TABLE_CATEGORY_USER && > - (get_use_stat_tables_mode(thd) > NEVER || > + ((get_use_stat_tables_mode(thd) > NEVER && > + get_use_stat_tables_mode(thd) < PREFERABLY_FOR_READS) || > lex->with_persistent_for_clause)); > > > diff --git a/sql/sql_statistics.h b/sql/sql_statistics.h > index 39cddf95188..a942c05be09 100644 > --- a/sql/sql_statistics.h > +++ b/sql/sql_statistics.h > @@ -22,6 +22,7 @@ enum enum_use_stat_tables_mode > NEVER, > COMPLEMENTARY, > PREFERABLY, Please add a comment describing the new value. > + PREFERABLY_FOR_READS, > } Use_stat_tables_mode; > > typedef > diff --git a/sql/sys_vars.cc b/sql/sys_vars.cc > index 92c7d329bb9..3652d728fc1 100644 > --- a/sql/sys_vars.cc > +++ b/sql/sys_vars.cc > @@ -5841,12 +5841,12 @@ static Sys_var_ulong Sys_progress_report_time( > VALID_RANGE(0, UINT_MAX), DEFAULT(5), BLOCK_SIZE(1)); > > const char *use_stat_tables_modes[] = > - {"NEVER", "COMPLEMENTARY", "PREFERABLY", 0}; > + {"NEVER", "COMPLEMENTARY", "PREFERABLY", "PREFERABLY_FOR_READS", 0}; > static Sys_var_enum Sys_optimizer_use_stat_tables( > "use_stat_tables", > "Specifies how to use system statistics tables", > SESSION_VAR(use_stat_tables), CMD_LINE(REQUIRED_ARG), > - use_stat_tables_modes, DEFAULT(0)); > + use_stat_tables_modes, DEFAULT(3)); > > static Sys_var_ulong Sys_histogram_size( > "histogram_size", BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog _______________________________________________ commits mailing list commits(a)mariadb.org https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits ----- End forwarded message ----- -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] [Maria-discuss] Pipeline-stype slave parallel applier
by Kristian Nielsen 26 Nov '18

26 Nov '18

andrei.elkin(a)pp.inet.fi writes: > Your comments, thoughts and critical notes are most welcome! Thanks for the interesting idea and detailed description, Andrei! I have written some initial comments inline, below: > entering the binlog group (ordered) commit module. They can end up > into binlog as A,B or B,A, and regardless of that (Gtid actually) order > also be acknowledged to the user in either way. > > The latter fact infers that slave does not have to commit the group > sub-transaction/branches in the GTID order. It could be done in any > order including no order at all like to do so as one epoch commit. However, MariaDB GTID requires that transactions be binlogged in the same order on the slave as on the master. Otherwise it is not possible to move a lower-level GTID slave from one master to the other, as the slave position only includes a single GTID in MariaDB. > conservative scheduler offers a decent parallelization method still suffering from > the aforementioned uneven branch's sizes and inter-thread > communications caused by ordering. The optimistic scheduler remedies to some level > but not fully (an optimistically executed group may be unbalanced too) > and even conceding some risk of performance degradation (rollback is costly). Jean François Gagné's comprehensive benchmarking of parallel replication actually has some very interesting data on this. He ran a real Booking.com workload with literally thousands of worker threads and had a huge rollback ratio (I remember it as up around 40%), and still saw speedup and good scalabitily. Apparently rollback isn't necessarily so costly; maybe because replication servers often have spare cores idling anyway, and because the aborted transaction acts as a pre-fetcher, improving I/O parallelism. But the fact that thousands of threads still improved speed suggests exactly that there was a problem of uneven transaction size - a few large transactions spread far apart, so lots of transaction lookahead was needed. I do not know if those transactions were single large queries or contained many individual statements, but in the latter case this is exactly what this idea could help with! > There is an idea reminiscent to pipelining to handle the epoch (the > master binlog group) as a single transaction but in parallel. That is > to execute its statements by multiple workers, this time scheduling > them fairly, and "atomically" commit their works. So this is a very interesting idea. Once the transactions are recorded in the binlog, we have a serial stream of statements that will reproduce the original data on the master. And they will do this regardless of if we remove some of the COMMITs/BEGINs - or if we add some. (I wonder if there are any exceptions to this, I can't think of any off the top of my head). So instead of trying to parallelise transactions, just do all the individual statements in parallel, disregarding transaction boundaries (with some restrictions to ensure consistency, as you describe below). I wonder how much this will benefit real-life workloads. There seems to be a lot of potential. For some workloads it will not help - if most transactions are single-statement, or if the multi-statement transactions have dependencies between the statements. But that doesn't seem likely to be the typical scenario. But why tie this to the master binlog group? It seems to fit perfectly to the optimistic parallel replication mode, where different transactions are run in parallel speculatively, and any conflicts are detected and resolved. In fact, it seems the exact same mechanism will work to detect conflicts between the more fine-grained per-statement parallelisation. The conservative parallle replication I consider obsolete. It causes a lot of hassle for the user to try to get good-sized group commits on the master to ensure parallel replication opportunities on the slave. And it doesn't even ensure consistency; there are corner cases (in InnoDB and elsewhere) that can cause conflicts to appear anyway on the slave, so the rollback+retry mechanism is needed anyway for conservative, just like for optimistic. > To the fairness part, workers get assigned by a statement at a time > from the epoch transaction(sub-transaction BEGIN, COMMIT removed). > Say 'E' an epoch consists of 's' of sub-transaction > > E := { T_1, T_2, ... T_s } > > For each m from 1 to s sub-transactions T_m gets scheduled to some worker > > W_1 W_2 ... W_k > > | | | > V V V > > T_m := { e_1, e_2, ... e_l } > > Hereinafter I use the TeX math notation, '^' - a superscript attached to > the event to designate its parent transaction sequence number (Gtid > seq_no), '_' - a subscript to enumerate an object within its > compound object. > > e_i stands for T-ransation's statements, W_t:s are workers. > Effectively the epoch breaks into modified branches executed by > workers on one to one basic: > > E := { T'_1, T'_2, ... T'_k } > > here 'k' is the number of workers engaged, T' indicates the modified > transaction. > The branches are executed until they are ready to prepare which event > is triggered by scheduling of the last statement of the last original > sub-transaction. > > It's clear that sizes of the modified branches are even this time. Well... each worker has exactly one statement at a time, but each statement can be very different in execution cost. But more even than transaction-based scheduling, certainly. > The last statement worker coordinates 2pc. > > Thinking more technically workers can consume/pull from > the current epoch presented as a queue which is pushed into > by statement producer. > > Here is a possible state of the queue when m:th statement of T_n is > about to be pushed: > > e^n_k -> > > > [ e^n_m-1, ..., e^n_2, e^n_1; ...; e^1_l1, ... e^1_2, e^1_1 ] > > ...----- T_n ------------| T_n-1,...T_2 |---------- T_1 ---------| > > A pull by consumer at this point would return e^1_1. How do you imagine integrating this in the current parallel replication scheduling algorithm? In the current algorithm, the producer (the SQL thread) assigns work to worker threads in a round-robin way. There is a lot of care taken to minimise contention on a single global queue-lock or similar, and I think that is an important reason why Jean François' tests were able to scale so well to many thousand worker threads. I wonder if the existing scheduling could not be used directly for this idea also - but it needs to be extended of course to schedule individual statements rather than whole transactions, with some way to let worker threads coordinate transaction start/end between them. What are your thoughts on this? > Very likely that by the epoch prepare event *all* the branches have been already > ready for prepare(). Hm, any particular reason you think so? I would imagine the opposite, that individual statements will be scheduled quickly, and it will be quite random in which order they happen to become ready in? > This pipelining parallelization method can work for the single "large" > transaction in the binlog group and also could drag into the > parallelizable input transactions from later groups if we additionally > create master side dependency tracking (think of mysql logical > timestamp method). > Also notice that the optimisitic scheduler is orthogonal to this > method so the two can be combined. > > Consistency concern > ------------------- > > The epoch parallel execution (scheduling) must respect > intra-transaction statement dependencies (e.g FIFO execution order of > operations over the same object [record, table]). There is no > inter-transaction dependencies, at least in theory. In fact, there do exist such inter-transaction dependencies in a number of corner cases, at least if we want to enforce the same commit order on slave as on master, which is required for MariaDB GTID. But that is already handled in current code, by detecting the dependency inside the storage engine (the thd_rpl_deadlock_check() mechanism). I wonder if the exact same thd_rpl_deadlock_check() mechanism would not work as well for this per-statement scheduling idea? It would be interesting to see a proof-of-concept based on this... > The notion of a leader, the two-phase-commits leader, remains roughly > the same as in the existing conservative scheduler. This time more > than initiating commits of the epoch branches it also takes care to > advance the slave gtid execution status accordingly. > It could be implemented as the very last statement of its own T' branch. Hm, right, interesting, agree that this seems a good fit. In fact, if the existing mechanism can be slightly extended, maybe it can already do much of what is needed to ensure consistency. Eg. suppose we have 3 original transactions each with two statements: e^1_1 e^1_2 e^2_1 e^2_2 e^3_1 e^3_2 Suppose the first 3 of these are ready to commit at some point: e^1_1 e^1_2 e^2_1 The leader can check that it doesn't try to commit only part of an original transaction. In this case it sees that e^2_1 is missing e_2_2, and it can commit only the first part and leave the partial transaction for a following group commit: e^2_1 This way, we already ensure that a crash will not leave us with an inconsistent storage engine state on the slave. Because we never commit part of an original master transaction - all parts of it are always group-committed together. Seems promising, though that code is also heavily optimised for scalability, so will require some case. Hm, one case needs to be handled though - when there are more statements in one original master transaction, than there are worker threads - otherwise we end up with a deadlock. > It's clear that causal dependencies of an user write and a read (to > find the effect of that write) can be tracked by WAIT_FOR_GTID() as > currently. > > A plain two-phase commit does admit possibility to catch on slave combinations > of the old (pre-epoch) and new version (post-epoch) or some objects of > the same transaction if the user hits with non-locking SELECT in a > "tiny" time window between commit():s of two branches > like in the following example. > > Let t1,t2 some tables and the binlog group consists of just one > transaction > > T_1 := { update(t1), update(t2) } > > Let on slave it's modified into two parallel ones: > > T'_1 := { update(t1) } || T'2 := { update(t2) } > > After both respond OK to 2pc prepare() and then the leader initiates > commit() where could be 4 possible result sets to the contemporary SELECT > > SET @@session.tx_isolation='REPEATABLE-READ'; > SELECT * from t1 as t_1, t2 as t_2; > => ... Well, due to the enforced commit order, it will not be possible to see the case where update(t2) is committed but update(t1) is not. In fact, if the group commit is extended as suggested above (to only group-commit the statements of an original transaction together), then I think it is possible for the user to avoid this problem entirely using START TRANSACTION WITH CONSISTENT SNAPSHOT In MariaDB, this ensures a consistent read view against 2pc transactions (reader will always see all of the 2pc commit or none of it). And IIRC, it works by holding the same lock (LOCK_commit_ordered) that is used by the group commit leader. So if the user has some special case where this minor read-inconsistency is sensitive, the START TRANSACTION WITH CONSISTENT SNAPSHOT mechanism can be used to avoid it completely. > While this is perhaps more than imperfect, I believe it may be > relevant only to remote corner cases, but more importantly, we still can > refine that doing some *adjustments* in the Engine studied briefly > myself of what would be a sub-transaction in Innodb, > "ideologically" endorsed and guided by Marko Makela. > The idea is to hold the branch's locks even after the branch commits, > and delegate their release to the leader. > > Recoverability > -------------- > > When slave is configured to log in the binary log > we must make sure to rollback any prepared > epoch branches at the server recovery unless the last branch (of the > leader) is also there, prepared. > > When it does not binlog the problem is that the modified branch's binlog > images can't be just flushed one after another especially > if the user specifies to log on the slave uniformly with the master > that is to preserve the original sub-transaction structures and their > boundaries. "if the user specified" - but this is always a requirement in MariaDB, right? Otherwise GTID order is broken? > In such case there are several multiple option, here are some: > > - use relay-logged image of original sub-transactions (may be limited > to the master and slave version combination though;) I did not follow what this means - can you elaborate? > - reconstruct original sub-transaction's binlog images. That is > binlog events "belonging" to an original sub-transaction will be > cached cooperatively by workers that took share of the > original sub-transaction execution. > > Optionally at least the caching could be set to allow out > of order while the statements are parallelizable, like in the above > T1(t1,t2) example. Otherwise the ordering will be preserved through > implementing the shared cache as as a queue with an interface > to insert an item into the middle. Isn't it possible to do the binlog caching locally, each worker caching just the binlogging for its own statement? If we make sure they group-commit together, they will be written in sequence into the binlog, so they can just write each their part one after the other? > - exotics like to create the slave side "merge" GTID which would > embed the original "commit". (The quotes here to allude to the Git's > way). > > While it's a serious subject I am leaving out mariabackup for this moment. > I believe it will be feasible to adjust its logics to > account as prepared only such transactions that that > are prepared in all their branches. > > More about pipeline style optimization > -------------------------------------- > > As mentioned workers can be made self-serving to assign events for > execution at once when become available. That must turn to think of > producer-consumer model, potentially lock-free queues, dynamical # of > workers to stay in balance with the receiver's pace etc. So are you planning to implement a completely new parallel replication scheduler, rather than build on the existing one? > I think *eventually* we will need to assign the producer role to > the IO (receiver) thread, and convert the current SQL (driver) thread > into (relay-) logger (we still may have to have such logger for > the semisync replication, the relay logger can be opted out > otherwise). > Here is a picture > > IO_thread.push() > | +---- worker.pull() > V V > e^s_i -> [ e^s_i-1, ... e^s_1 }, T_s-1, ... ] > ^ > | > SQL.pull() > > where on the right hand side the two consumers handle > a transaction and when both of them have done with it > (a worker has read its last event and the logger written > it to the relay-log) the query element becomes > garbage-collectable. So overall, this was an interesting idea to see! I never thought before of doing parallelisation beyond the original transaction boundaries from the master. But now that you have described it, it actually seems quite a natural extension of the optimistic parallel replication, and it fits very well with the fundamental principle in MariaDB replication of preserving strict GTID ordering, which might work to handle much of the consistency requirements. It would be quite interesting to see this in practice on real workloads and with benchmarks - I could imagine very large improvements for some workloads. Though I suppose you shouldn't underestimate the work required to complete something like this in production quality (the devil is in the detail, once you get to nasty things like temporary tables and non-transactional engines and other horrors that are part of the MySQL/MariaDB ecosystem). Hope this helps, - Kristian.

2 7

[Maria-developers] Documenting is needed: MDEV-16991 Rounding vs truncation for TIME, DATETIME, TIMESTAMP
by Alexander Barkov 26 Nov '18

26 Nov '18

Hi Kenneth, all, I've pushed MDEV-16991. Can you please document this new feature. Thanks!

2 3

Re: [Maria-developers] MDEV-16991 review.
by Alexander Barkov 26 Nov '18

26 Nov '18

Hi Alexey, Thanks for your review. On 11/26/2018 12:29 AM, Alexey Botchkov wrote: > Ok, fine with leaving sql_base_types.h and the MYSQL_TIME_STATUS as it is. > > The last comment then: > Item_func_trt_id::val_int() > Item_interval_DDhhmmssff_typecast::val_str() > there you get the 'current_thd' which is considered to be slow. > Code looks like > THD *thd= current_thd; > Datetime::Options opt(TIME_CONV_NONE, thd); > if (args[0]->get_date(thd, &commit_ts, opt)) > I think it makes sence to add the Item_func_trt_id::m_opt and fill it in > ::fix_fields(). Unfortunately this does not help much in these two places exactly. We need to pass thd to get_date() anyway. Methods like val_int(), val_str(), etc, should be fixed eventually to get an extra THD* parameter. But I checked the entiner patch again and fixed a few other places where some current_thd calls were redundant. Thanks! > > Ok with me to push if you fix it like that. > > One extra idea though. We can get rid of current_thd > in set_date_length() too - > add the THD * parameter there and call in in ::fix_fields() instead of > ::fix_length_and_dec(). > Since it doesn't seem critical, i'm fine if you leave it as it is :) Alas, fix_length_and_dec() does not get THD* as an argument. Only fix_fields() does. fix_length_and_dec() should also be fixed to get THD*. Btw, perhaps 10.4 is a good timing for this. > > Best regards. > HF > > > > On Sun, Nov 25, 2018 at 9:42 AM Alexander Barkov <bar(a)mariadb.com > <mailto:bar@mariadb.com>> wrote: > > Hi Alexey, > > > On 11/25/2018 09:05 AM, Alexander Barkov wrote: > >> > >> What do you think if we do this: > >> - add THD* memver to the MYSQL_TIME_STATUS structure, > >> so we can get rid of the THD * parameter to many functions? > >> - send one MYSQL_TIME_STATUS* to the add_nanoseconds() > >> instead of three {thd, &status->warnings, and > status->nanoseconds} ? > > > > Changing all functions that accept both &warn and nsec to > > accept a pointer to MYSQL_TIME_STATUS instead, like this: > > > > < xxxx(thd, &status->warnings, and status->nanoseconds); > >> xxxx(thd, &status); > > > > is a very good idea. > > > > > > I'd avoid mix "thd" to MYSQL_TIME_STATUS though. > > It's a pure C structure. > > > > Does it sound OK? > > > > I tried this, but this does not save anything. > See the attached patch. > > git diff --stat > sql/sql_time.cc | 6 +++--- > sql/sql_type.cc | 2 +- > sql/sql_type.h | 8 ++++++++ > 3 files changed, 12 insertions(+), 4 deletions(-) > > > In all other places "*warn" and "nsec" come from different > places (not from the same MYSQL_TIME_STATUS). > > > Btw, MYSQL_TIME_STATUS is actually an interface to low level > pure C conversion functions like str_to_xxx() and number_to_xxx(). > Let's not propagate it all around the code. > > Can we leave the relevant code as is? > > Thanks. >

1 0

Re: [Maria-developers] [Commits] 8bfb140d5dc: Move deletion of old GTID rows to slave background thread
by Kristian Nielsen 25 Nov '18

25 Nov '18

Hi Andrei! Here is the patch to move the cleanup of old GTID rows into the slave background thread. It is mostly identical to the patch I wrote a month ago, but now I got test cases added/updated and I ran it through buildbot. Want to review it? The patch should be complete now. The patch is also available in git here: https://github.com/knielsen/server/tree/gtid_delete_in_background_thread This patch could go into 10.4. With the patch, a request is sent to the background thread to delete old rows every @@gtid_cleanup_batch_size replicated transactions. Some considerations on the default of this value: - It should be reasonably large so that the background thread only needs to wake up occasionally, and a number of row deletions can share the cost of the part that is not inherently per-row. - A large enough value can group a number of row deletions in a single transaction which is probably beneficial. There is also a fixed overhead per existing gtid_domain_id to scan the GTID state, so the value should not be much smaller than the number of domains in use. - It should not be too large either, since that could cause the mysql.gtid_slave_pos table to grow unnecessarily. Ideally the table should fit in one InnoDB data page. I settled on a default of 64. Probably a wide range will work fine in >99% of cases, eg. 10 or 100 should be mostly the same. 64 should be large enough that the per-batch overhead becomes small relative to the per-row, even if hundreds of domains are used. And small enough that the table is likely to be able to fit in a single InnoDB 16k page, even considering that deletion is asynchroneous and more rows can accumulate temporarily due to parallel replication etc. I think this is a nice improvement. Moving the deletion logic away from the main replication SQL/worker threads simplifies the logic, eg. for when a transaction fails after a deletion has already been done. And it also has the potential to be more efficient; the contention on global mutex LOCK_slave_state is reduced, processing is moved out of the main replication threads, and batching row deletions could potentially be cheaper than doing them one-by-one. - Kristian. Kristian Nielsen <knielsen(a)knielsen-hq.org> writes: > revision-id: 8bfb140d5dc247c183787b8a0a1799cf375845bd (mariadb-10.3.10-25-g8bfb140d5dc) > parent(s): 74387028a06c557f36a0fd1bbde347f1551c8fb7 > author: Kristian Nielsen > committer: Kristian Nielsen > timestamp: 2018-11-25 19:38:33 +0100 > message: > > Move deletion of old GTID rows to slave background thread > > This patch changes how old rows in mysql.gtid_slave_pos* tables are deleted. > Instead of doing it as part of every replicated transaction in > record_gtid(), it is done periodically (every @@gtid_cleanup_batch_size > transaction) in the slave background thread. > > This removes the deletion step from the replication process in SQL or worker > threads, which could speed up replication with many small transactions. It > also decreases contention on the global mutex LOCK_slave_state. And it > simplifies the logic, eg. when a replicated transaction fails after having > deleted old rows. > > With this patch, the deletion of old GTID rows happens asynchroneously and > slightly non-deterministic. Thus the number of old rows in > mysql.gtid_slave_pos can temporarily exceed @@gtid_cleanup_batch_size. But > all old rows will be deleted eventually after sufficiently many new GTIDs > have been replicated. > > --- > mysql-test/main/mysqld--help.result | 10 + > mysql-test/suite/rpl/r/rpl_gtid_mdev4484.result | 40 +- > mysql-test/suite/rpl/r/rpl_gtid_stop_start.result | 8 +- > .../suite/rpl/r/rpl_parallel_optimistic.result | 14 +- > mysql-test/suite/rpl/t/rpl_gtid_mdev4484.test | 68 +++- > .../suite/rpl/t/rpl_parallel_optimistic.test | 42 ++- > .../sys_vars/r/sysvars_server_notembedded.result | 14 + > sql/log_event.cc | 6 +- > sql/mysqld.cc | 1 + > sql/mysqld.h | 1 + > sql/rpl_gtid.cc | 413 +++++++++++++-------- > sql/rpl_gtid.h | 12 +- > sql/rpl_rli.cc | 87 +---- > sql/rpl_rli.h | 11 - > sql/slave.cc | 35 +- > sql/slave.h | 1 + > sql/sys_vars.cc | 13 + > .../mysql-test/rocksdb_rpl/r/mdev12179.result | 18 + > .../mysql-test/rocksdb_rpl/t/mdev12179.test | 85 +++++ > .../mysql-test/tokudb_rpl/r/mdev12179.result | 18 + > .../tokudb/mysql-test/tokudb_rpl/t/mdev12179.test | 85 +++++ > 21 files changed, 675 insertions(+), 307 deletions(-) > > diff --git a/mysql-test/main/mysqld--help.result b/mysql-test/main/mysqld--help.result > index 5a7153f32d3..4f801ec5275 100644 > --- a/mysql-test/main/mysqld--help.result > +++ b/mysql-test/main/mysqld--help.result > @@ -294,6 +294,15 @@ The following specify which files/extra groups are read (specified before remain > --group-concat-max-len=# > The maximum length of the result of function > GROUP_CONCAT() > + --gtid-cleanup-batch-size=# > + Normally does not need tuning. How many old rows must > + accumulate in the mysql.gtid_slave_pos table before a > + background job will be run to delete them. Can be > + increased to reduce number of commits if using many > + different engines with --gtid_pos_auto_engines, or to > + reduce CPU overhead if using a huge number of different > + gtid_domain_ids. Can be decreased to reduce number of old > + rows in the table. > --gtid-domain-id=# Used with global transaction ID to identify logically > independent replication streams. When events can > propagate through multiple parallel paths (for example > @@ -1425,6 +1434,7 @@ gdb FALSE > general-log FALSE > getopt-prefix-matching FALSE > group-concat-max-len 1048576 > +gtid-cleanup-batch-size 64 > gtid-domain-id 0 > gtid-ignore-duplicates FALSE > gtid-pos-auto-engines > diff --git a/mysql-test/suite/rpl/r/rpl_gtid_mdev4484.result b/mysql-test/suite/rpl/r/rpl_gtid_mdev4484.result > index aaeb0c8f119..55d2831dcf4 100644 > --- a/mysql-test/suite/rpl/r/rpl_gtid_mdev4484.result > +++ b/mysql-test/suite/rpl/r/rpl_gtid_mdev4484.result > @@ -16,36 +16,32 @@ INSERT INTO t1 VALUES (1); > connection slave; > connection slave; > include/stop_slave.inc > +SET @old_gtid_cleanup_batch_size= @@GLOBAL.gtid_cleanup_batch_size; > +SET GLOBAL gtid_cleanup_batch_size= 2; > SET @old_dbug= @@GLOBAL.debug_dbug; > SET GLOBAL debug_dbug="+d,gtid_slave_pos_simulate_failed_delete"; > SET sql_log_bin= 0; > -CALL mtr.add_suppression("Can't find file"); > +CALL mtr.add_suppression("<DEBUG> Error deleting old GTID row"); > SET sql_log_bin= 1; > include/start_slave.inc > connection master; > -INSERT INTO t1 VALUES (2); > -connection slave; > -include/wait_for_slave_sql_error.inc [errno=1942] > -STOP SLAVE IO_THREAD; > -SELECT domain_id, server_id, seq_no FROM mysql.gtid_slave_pos > -ORDER BY domain_id, sub_id DESC LIMIT 1; > -domain_id server_id seq_no > -0 1 3 > +connection slave; > +SELECT COUNT(*), MAX(seq_no) INTO @pre_count, @pre_max_seq_no > +FROM mysql.gtid_slave_pos; > +SELECT IF(@pre_count >= 20, "OK", CONCAT("Error: too few rows seen while errors injected: ", @pre_count)); > +IF(@pre_count >= 20, "OK", CONCAT("Error: too few rows seen while errors injected: ", @pre_count)) > +OK > SET GLOBAL debug_dbug= @old_dbug; > -include/start_slave.inc > connection master; > -INSERT INTO t1 VALUES (3); > -connection slave; > -connection slave; > -SELECT domain_id, server_id, seq_no FROM mysql.gtid_slave_pos > -ORDER BY domain_id, sub_id DESC LIMIT 1; > -domain_id server_id seq_no > -0 1 4 > -SELECT * FROM t1 ORDER BY i; > -i > -1 > -2 > -3 > +connection slave; > +connection slave; > +SELECT IF(COUNT(*) >= 1, "OK", CONCAT("Error: too few rows seen after errors no longer injected: ", COUNT(*))) > +FROM mysql.gtid_slave_pos > +WHERE seq_no <= @pre_max_seq_no; > +IF(COUNT(*) >= 1, "OK", CONCAT("Error: too few rows seen after errors no longer injected: ", COUNT(*))) > +OK > connection master; > DROP TABLE t1; > +connection slave; > +SET GLOBAL gtid_cleanup_batch_size= @old_gtid_cleanup_batch_size; > include/rpl_end.inc > diff --git a/mysql-test/suite/rpl/r/rpl_gtid_stop_start.result b/mysql-test/suite/rpl/r/rpl_gtid_stop_start.result > index ff845794c22..b27ffed9f94 100644 > --- a/mysql-test/suite/rpl/r/rpl_gtid_stop_start.result > +++ b/mysql-test/suite/rpl/r/rpl_gtid_stop_start.result > @@ -171,7 +171,7 @@ include/start_slave.inc > *** MDEV-4692: mysql.gtid_slave_pos accumulates values for a domain *** > SELECT domain_id, COUNT(*) FROM mysql.gtid_slave_pos GROUP BY domain_id; > domain_id COUNT(*) > -0 2 > +0 3 > 1 2 > connection server_1; > INSERT INTO t1 VALUES (11); > @@ -179,7 +179,7 @@ connection server_2; > FLUSH NO_WRITE_TO_BINLOG TABLES; > SELECT domain_id, COUNT(*) FROM mysql.gtid_slave_pos GROUP BY domain_id; > domain_id COUNT(*) > -0 2 > +0 4 > 1 2 > include/start_slave.inc > connection server_1; > @@ -189,8 +189,8 @@ connection server_2; > FLUSH NO_WRITE_TO_BINLOG TABLES; > SELECT domain_id, COUNT(*) FROM mysql.gtid_slave_pos GROUP BY domain_id; > domain_id COUNT(*) > -0 2 > -1 2 > +0 3 > +1 1 > *** MDEV-4650: show variables; ERROR 1946 (HY000): Failed to load replication slave GTID position *** > connection server_2; > SET sql_log_bin=0; > diff --git a/mysql-test/suite/rpl/r/rpl_parallel_optimistic.result b/mysql-test/suite/rpl/r/rpl_parallel_optimistic.result > index ca202a66b0e..83343e52cab 100644 > --- a/mysql-test/suite/rpl/r/rpl_parallel_optimistic.result > +++ b/mysql-test/suite/rpl/r/rpl_parallel_optimistic.result > @@ -12,6 +12,8 @@ SET GLOBAL slave_parallel_threads=10; > CHANGE MASTER TO master_use_gtid=slave_pos; > SET @old_parallel_mode=@@GLOBAL.slave_parallel_mode; > SET GLOBAL slave_parallel_mode='optimistic'; > +SET @old_gtid_cleanup_batch_size= @@GLOBAL.gtid_cleanup_batch_size; > +SET GLOBAL gtid_cleanup_batch_size= 1000000; > connection server_1; > INSERT INTO t1 VALUES(1,1); > BEGIN; > @@ -131,6 +133,11 @@ c > 204 > 205 > 206 > +SELECT IF(COUNT(*) >= 30, "OK", CONCAT("Error: too few old rows found: ", COUNT(*))) > +FROM mysql.gtid_slave_pos; > +IF(COUNT(*) >= 30, "OK", CONCAT("Error: too few old rows found: ", COUNT(*))) > +OK > +SET GLOBAL gtid_cleanup_batch_size=1; > *** Test @@skip_parallel_replication. *** > connection server_2; > include/stop_slave.inc > @@ -651,9 +658,10 @@ DROP TABLE t1, t2, t3; > include/save_master_gtid.inc > connection server_2; > include/sync_with_master_gtid.inc > -Check that no more than the expected last four GTIDs are in mysql.gtid_slave_pos > -select count(4) <= 4 from mysql.gtid_slave_pos order by domain_id, sub_id; > -count(4) <= 4 > +SELECT COUNT(*) <= 5*@@GLOBAL.gtid_cleanup_batch_size > +FROM mysql.gtid_slave_pos; > +COUNT(*) <= 5*@@GLOBAL.gtid_cleanup_batch_size > 1 > +SET GLOBAL gtid_cleanup_batch_size= @old_gtid_cleanup_batch_size; > connection server_1; > include/rpl_end.inc > diff --git a/mysql-test/suite/rpl/t/rpl_gtid_mdev4484.test b/mysql-test/suite/rpl/t/rpl_gtid_mdev4484.test > index e1f5696f5a1..a28bff3d27a 100644 > --- a/mysql-test/suite/rpl/t/rpl_gtid_mdev4484.test > +++ b/mysql-test/suite/rpl/t/rpl_gtid_mdev4484.test > @@ -28,37 +28,79 @@ INSERT INTO t1 VALUES (1); > # Inject an artificial error deleting entries, and check that the error handling code works. > --connection slave > --source include/stop_slave.inc > +SET @old_gtid_cleanup_batch_size= @@GLOBAL.gtid_cleanup_batch_size; > +SET GLOBAL gtid_cleanup_batch_size= 2; > SET @old_dbug= @@GLOBAL.debug_dbug; > SET GLOBAL debug_dbug="+d,gtid_slave_pos_simulate_failed_delete"; > SET sql_log_bin= 0; > -CALL mtr.add_suppression("Can't find file"); > +CALL mtr.add_suppression("<DEBUG> Error deleting old GTID row"); > SET sql_log_bin= 1; > --source include/start_slave.inc > > --connection master > -INSERT INTO t1 VALUES (2); > +--disable_query_log > +let $i = 20; > +while ($i) { > + eval INSERT INTO t1 VALUES ($i+10); > + dec $i; > +} > +--enable_query_log > +--save_master_pos > > --connection slave > ---let $slave_sql_errno= 1942 > ---source include/wait_for_slave_sql_error.inc > -STOP SLAVE IO_THREAD; > -SELECT domain_id, server_id, seq_no FROM mysql.gtid_slave_pos > - ORDER BY domain_id, sub_id DESC LIMIT 1; > +--sync_with_master > + > +# Now wait for the slave background thread to try to delete old rows and > +# hit the error injection. > +--let _TEST_MYSQLD_ERROR_LOG=$MYSQLTEST_VARDIR/log/mysqld.2.err > +--perl > + open F, '<', $ENV{'_TEST_MYSQLD_ERROR_LOG'} or die; > + outer: while (1) { > + inner: while (<F>) { > + last outer if /<DEBUG> Error deleting old GTID row/; > + } > + # Easy way to do sub-second sleep without extra modules. > + select(undef, undef, undef, 0.1); > + } > +EOF > + > +# Since we injected error in the cleanup code, the rows should remain in > +# mysql.gtid_slave_pos. Check that we have at least 20 (more robust against > +# non-deterministic cleanup and future changes than checking for exact number). > +SELECT COUNT(*), MAX(seq_no) INTO @pre_count, @pre_max_seq_no > + FROM mysql.gtid_slave_pos; > +SELECT IF(@pre_count >= 20, "OK", CONCAT("Error: too few rows seen while errors injected: ", @pre_count)); > SET GLOBAL debug_dbug= @old_dbug; > ---source include/start_slave.inc > > --connection master > -INSERT INTO t1 VALUES (3); > +--disable_query_log > +let $i = 20; > +while ($i) { > + eval INSERT INTO t1 VALUES ($i+40); > + dec $i; > +} > +--enable_query_log > --sync_slave_with_master > > --connection slave > -SELECT domain_id, server_id, seq_no FROM mysql.gtid_slave_pos > - ORDER BY domain_id, sub_id DESC LIMIT 1; > -SELECT * FROM t1 ORDER BY i; > - > +# Now check that 1) rows are being deleted again after removing error > +# injection, and 2) old rows are left that failed their delete while errors > +# where injected (again compensating for non-deterministic deletion). > +# Deletion is async and slightly non-deterministic, so we wait for at > +# least 10 of the 20 new rows to be deleted. > +let $wait_condition= > + SELECT COUNT(*) <= 20-10 > + FROM mysql.gtid_slave_pos > + WHERE seq_no > @pre_max_seq_no; > +--source include/wait_condition.inc > +SELECT IF(COUNT(*) >= 1, "OK", CONCAT("Error: too few rows seen after errors no longer injected: ", COUNT(*))) > + FROM mysql.gtid_slave_pos > + WHERE seq_no <= @pre_max_seq_no; > > # Clean up > --connection master > DROP TABLE t1; > +--connection slave > +SET GLOBAL gtid_cleanup_batch_size= @old_gtid_cleanup_batch_size; > > --source include/rpl_end.inc > diff --git a/mysql-test/suite/rpl/t/rpl_parallel_optimistic.test b/mysql-test/suite/rpl/t/rpl_parallel_optimistic.test > index e08472d5f51..0060cf4416c 100644 > --- a/mysql-test/suite/rpl/t/rpl_parallel_optimistic.test > +++ b/mysql-test/suite/rpl/t/rpl_parallel_optimistic.test > @@ -21,6 +21,10 @@ SET GLOBAL slave_parallel_threads=10; > CHANGE MASTER TO master_use_gtid=slave_pos; > SET @old_parallel_mode=@@GLOBAL.slave_parallel_mode; > SET GLOBAL slave_parallel_mode='optimistic'; > +# Run the first part of the test with high batch size and see that > +# old rows remain in the table. > +SET @old_gtid_cleanup_batch_size= @@GLOBAL.gtid_cleanup_batch_size; > +SET GLOBAL gtid_cleanup_batch_size= 1000000; > > > --connection server_1 > @@ -108,7 +112,12 @@ SELECT * FROM t3 ORDER BY c; > SELECT * FROM t1 ORDER BY a; > SELECT * FROM t2 ORDER BY a; > SELECT * FROM t3 ORDER BY c; > -#SHOW STATUS LIKE 'Slave_retried_transactions'; > +# Check that we have a bunch of old rows left-over - they were not deleted > +# due to high @@gtid_cleanup_batch_size. Then set a low > +# @@gtid_cleanup_batch_size so we can test that rows start being deleted. > +SELECT IF(COUNT(*) >= 30, "OK", CONCAT("Error: too few old rows found: ", COUNT(*))) > + FROM mysql.gtid_slave_pos; > +SET GLOBAL gtid_cleanup_batch_size=1; > > > --echo *** Test @@skip_parallel_replication. *** > @@ -557,25 +566,18 @@ DROP TABLE t1, t2, t3; > > --connection server_2 > --source include/sync_with_master_gtid.inc > -# Check for left-over rows in table mysql.gtid_slave_pos (MDEV-12147). > -# > -# There was a bug when a transaction got a conflict and was rolled back. It > -# might have also handled deletion of some old rows, and these deletions would > -# then also be rolled back. And since the deletes were never re-tried, old no > -# longer needed rows would accumulate in the table without limit. > -# > -# The earlier part of this test file have plenty of transactions being rolled > -# back. But the last DROP TABLE statement runs on its own and should never > -# conflict, thus at this point the mysql.gtid_slave_pos table should be clean. > -# > -# To support @@gtid_pos_auto_engines, when a row is inserted in the table, it > -# is associated with the engine of the table at insertion time, and it will > -# only be deleted during record_gtid from a table of the same engine. Since we > -# alter the table from MyISAM to InnoDB at the start of this test, we should > -# end up with 4 rows: two left-over from when the table was MyISAM, and two > -# left-over from the InnoDB part. > ---echo Check that no more than the expected last four GTIDs are in mysql.gtid_slave_pos > -select count(4) <= 4 from mysql.gtid_slave_pos order by domain_id, sub_id; > +# Check that old rows are deleted from mysql.gtid_slave_pos. > +# Deletion is asynchronous, so use wait_condition.inc. > +# Also, there is a small amount of non-determinism in the deletion of old > +# rows, so it is not guaranteed that there can never be more than > +# @@gtid_cleanup_batch_size rows in the table; so allow a bit of slack > +# here. > +let $wait_condition= > + SELECT COUNT(*) <= 5*@@GLOBAL.gtid_cleanup_batch_size > + FROM mysql.gtid_slave_pos; > +--source include/wait_condition.inc > +eval $wait_condition; > +SET GLOBAL gtid_cleanup_batch_size= @old_gtid_cleanup_batch_size; > > --connection server_1 > --source include/rpl_end.inc > diff --git a/mysql-test/suite/sys_vars/r/sysvars_server_notembedded.result b/mysql-test/suite/sys_vars/r/sysvars_server_notembedded.result > index e8e4d671eb9..5c5ca8b66b2 100644 > --- a/mysql-test/suite/sys_vars/r/sysvars_server_notembedded.result > +++ b/mysql-test/suite/sys_vars/r/sysvars_server_notembedded.result > @@ -1202,6 +1202,20 @@ NUMERIC_BLOCK_SIZE NULL > ENUM_VALUE_LIST NULL > READ_ONLY NO > COMMAND_LINE_ARGUMENT NULL > +VARIABLE_NAME GTID_CLEANUP_BATCH_SIZE > +SESSION_VALUE NULL > +GLOBAL_VALUE 64 > +GLOBAL_VALUE_ORIGIN COMPILE-TIME > +DEFAULT_VALUE 64 > +VARIABLE_SCOPE GLOBAL > +VARIABLE_TYPE INT UNSIGNED > +VARIABLE_COMMENT Normally does not need tuning. How many old rows must accumulate in the mysql.gtid_slave_pos table before a background job will be run to delete them. Can be increased to reduce number of commits if using many different engines with --gtid_pos_auto_engines, or to reduce CPU overhead if using a huge number of different gtid_domain_ids. Can be decreased to reduce number of old rows in the table. > +NUMERIC_MIN_VALUE 0 > +NUMERIC_MAX_VALUE 2147483647 > +NUMERIC_BLOCK_SIZE 1 > +ENUM_VALUE_LIST NULL > +READ_ONLY NO > +COMMAND_LINE_ARGUMENT REQUIRED > VARIABLE_NAME GTID_CURRENT_POS > SESSION_VALUE NULL > GLOBAL_VALUE > diff --git a/sql/log_event.cc b/sql/log_event.cc > index 8813d20578e..e10480fb015 100644 > --- a/sql/log_event.cc > +++ b/sql/log_event.cc > @@ -5565,7 +5565,7 @@ int Query_log_event::do_apply_event(rpl_group_info *rgi, > gtid= rgi->current_gtid; > if (unlikely(rpl_global_gtid_slave_state->record_gtid(thd, &gtid, > sub_id, > - rgi, false, > + true, false, > &hton))) > { > int errcode= thd->get_stmt_da()->sql_errno(); > @@ -8362,7 +8362,7 @@ Gtid_list_log_event::do_apply_event(rpl_group_info *rgi) > { > if ((ret= rpl_global_gtid_slave_state->record_gtid(thd, &list[i], > sub_id_list[i], > - NULL, false, &hton))) > + false, false, &hton))) > return ret; > rpl_global_gtid_slave_state->update_state_hash(sub_id_list[i], &list[i], > hton, NULL); > @@ -8899,7 +8899,7 @@ int Xid_log_event::do_apply_event(rpl_group_info *rgi) > rgi->gtid_pending= false; > > gtid= rgi->current_gtid; > - err= rpl_global_gtid_slave_state->record_gtid(thd, &gtid, sub_id, rgi, > + err= rpl_global_gtid_slave_state->record_gtid(thd, &gtid, sub_id, true, > false, &hton); > if (unlikely(err)) > { > diff --git a/sql/mysqld.cc b/sql/mysqld.cc > index afef4a5f52c..07bdd66f74c 100644 > --- a/sql/mysqld.cc > +++ b/sql/mysqld.cc > @@ -580,6 +580,7 @@ ulong opt_binlog_commit_wait_count= 0; > ulong opt_binlog_commit_wait_usec= 0; > ulong opt_slave_parallel_max_queued= 131072; > my_bool opt_gtid_ignore_duplicates= FALSE; > +uint opt_gtid_cleanup_batch_size= 64; > > const double log_10[] = { > 1e000, 1e001, 1e002, 1e003, 1e004, 1e005, 1e006, 1e007, 1e008, 1e009, > diff --git a/sql/mysqld.h b/sql/mysqld.h > index d5cabd790b2..261748372f9 100644 > --- a/sql/mysqld.h > +++ b/sql/mysqld.h > @@ -258,6 +258,7 @@ extern ulong opt_slave_parallel_mode; > extern ulong opt_binlog_commit_wait_count; > extern ulong opt_binlog_commit_wait_usec; > extern my_bool opt_gtid_ignore_duplicates; > +extern uint opt_gtid_cleanup_batch_size; > extern ulong back_log; > extern ulong executed_events; > extern char language[FN_REFLEN]; > diff --git a/sql/rpl_gtid.cc b/sql/rpl_gtid.cc > index fabd09adaa7..196c2fe3d16 100644 > --- a/sql/rpl_gtid.cc > +++ b/sql/rpl_gtid.cc > @@ -79,7 +79,7 @@ rpl_slave_state::record_and_update_gtid(THD *thd, rpl_group_info *rgi) > rgi->gtid_pending= false; > if (rgi->gtid_ignore_duplicate_state!=rpl_group_info::GTID_DUPLICATE_IGNORE) > { > - if (record_gtid(thd, &rgi->current_gtid, sub_id, NULL, false, &hton)) > + if (record_gtid(thd, &rgi->current_gtid, sub_id, false, false, &hton)) > DBUG_RETURN(1); > update_state_hash(sub_id, &rgi->current_gtid, hton, rgi); > } > @@ -244,7 +244,7 @@ rpl_slave_state_free_element(void *arg) > > > rpl_slave_state::rpl_slave_state() > - : last_sub_id(0), gtid_pos_tables(0), loaded(false) > + : pending_gtid_count(0), last_sub_id(0), gtid_pos_tables(0), loaded(false) > { > mysql_mutex_init(key_LOCK_slave_state, &LOCK_slave_state, > MY_MUTEX_INIT_SLOW); > @@ -331,14 +331,11 @@ rpl_slave_state::update(uint32 domain_id, uint32 server_id, uint64 sub_id, > } > } > rgi->gtid_ignore_duplicate_state= rpl_group_info::GTID_DUPLICATE_NULL; > - > -#ifdef HAVE_REPLICATION > - rgi->pending_gtid_deletes_clear(); > -#endif > } > > if (!(list_elem= (list_element *)my_malloc(sizeof(*list_elem), MYF(MY_WME)))) > return 1; > + list_elem->domain_id= domain_id; > list_elem->server_id= server_id; > list_elem->sub_id= sub_id; > list_elem->seq_no= seq_no; > @@ -348,6 +345,15 @@ rpl_slave_state::update(uint32 domain_id, uint32 server_id, uint64 sub_id, > if (last_sub_id < sub_id) > last_sub_id= sub_id; > > +#ifdef HAVE_REPLICATION > + ++pending_gtid_count; > + if (pending_gtid_count >= opt_gtid_cleanup_batch_size) > + { > + pending_gtid_count = 0; > + slave_background_gtid_pending_delete_request(); > + } > +#endif > + > return 0; > } > > @@ -382,20 +388,22 @@ rpl_slave_state::get_element(uint32 domain_id) > > > int > -rpl_slave_state::put_back_list(uint32 domain_id, list_element *list) > +rpl_slave_state::put_back_list(list_element *list) > { > - element *e; > + element *e= NULL; > int err= 0; > > mysql_mutex_lock(&LOCK_slave_state); > - if (!(e= (element *)my_hash_search(&hash, (const uchar *)&domain_id, 0))) > - { > - err= 1; > - goto end; > - } > while (list) > { > list_element *next= list->next; > + > + if ((!e || e->domain_id != list->domain_id) && > + !(e= (element *)my_hash_search(&hash, (const uchar *)&list->domain_id, 0))) > + { > + err= 1; > + goto end; > + } > e->add(list); > list= next; > } > @@ -572,12 +580,12 @@ rpl_slave_state::select_gtid_pos_table(THD *thd, LEX_CSTRING *out_tablename) > /* > Write a gtid to the replication slave state table. > > + Do it as part of the transaction, to get slave crash safety, or as a separate > + transaction if !in_transaction (eg. MyISAM or DDL). > + > gtid The global transaction id for this event group. > sub_id Value allocated within the sub_id when the event group was > read (sub_id must be consistent with commit order in master binlog). > - rgi rpl_group_info context, if we are recording the gtid transactionally > - as part of replicating a transactional event. NULL if called from > - outside of a replicated transaction. > > Note that caller must later ensure that the new gtid and sub_id is inserted > into the appropriate HASH element with rpl_slave_state.add(), so that it can > @@ -585,16 +593,13 @@ rpl_slave_state::select_gtid_pos_table(THD *thd, LEX_CSTRING *out_tablename) > */ > int > rpl_slave_state::record_gtid(THD *thd, const rpl_gtid *gtid, uint64 sub_id, > - rpl_group_info *rgi, bool in_statement, > + bool in_transaction, bool in_statement, > void **out_hton) > { > TABLE_LIST tlist; > int err= 0, not_sql_thread; > bool table_opened= false; > TABLE *table; > - list_element *delete_list= 0, *next, *cur, **next_ptr_ptr, **best_ptr_ptr; > - uint64 best_sub_id; > - element *elem; > ulonglong thd_saved_option= thd->variables.option_bits; > Query_tables_list lex_backup; > wait_for_commit* suspended_wfc; > @@ -684,7 +689,7 @@ rpl_slave_state::record_gtid(THD *thd, const rpl_gtid *gtid, uint64 sub_id, > thd->wsrep_ignore_table= true; > #endif > > - if (!rgi) > + if (!in_transaction) > { > DBUG_PRINT("info", ("resetting OPTION_BEGIN")); > thd->variables.option_bits&= > @@ -716,168 +721,280 @@ rpl_slave_state::record_gtid(THD *thd, const rpl_gtid *gtid, uint64 sub_id, > my_error(ER_OUT_OF_RESOURCES, MYF(0)); > goto end; > } > +end: > > - mysql_mutex_lock(&LOCK_slave_state); > - if ((elem= get_element(gtid->domain_id)) == NULL) > +#ifdef WITH_WSREP > + thd->wsrep_ignore_table= false; > +#endif > + > + if (table_opened) > { > - mysql_mutex_unlock(&LOCK_slave_state); > - my_error(ER_OUT_OF_RESOURCES, MYF(0)); > - err= 1; > - goto end; > + if (err || (err= ha_commit_trans(thd, FALSE))) > + ha_rollback_trans(thd, FALSE); > + > + close_thread_tables(thd); > + if (in_transaction) > + thd->mdl_context.release_statement_locks(); > + else > + thd->mdl_context.release_transactional_locks(); > } > + thd->lex->restore_backup_query_tables_list(&lex_backup); > + thd->variables.option_bits= thd_saved_option; > + thd->resume_subsequent_commits(suspended_wfc); > + DBUG_EXECUTE_IF("inject_record_gtid_serverid_100_sleep", > + { > + if (gtid->server_id == 100) > + my_sleep(500000); > + }); > + DBUG_RETURN(err); > +} > > - /* Now pull out all GTIDs that were recorded in this engine. */ > - delete_list = NULL; > - next_ptr_ptr= &elem->list; > - cur= elem->list; > - best_sub_id= 0; > - best_ptr_ptr= NULL; > - while (cur) > + > +/* > + Return a list of all old GTIDs in any mysql.gtid_slave_pos* table that are > + no longer needed and can be deleted from the table. > + > + Within each domain, we need to keep around the latest GTID (the one with the > + highest sub_id), but any others in that domain can be deleted. > +*/ > +rpl_slave_state::list_element * > +rpl_slave_state::gtid_grab_pending_delete_list() > +{ > + uint32 i; > + list_element *full_list; > + > + mysql_mutex_lock(&LOCK_slave_state); > + full_list= NULL; > + for (i= 0; i < hash.records; ++i) > { > - list_element *next= cur->next; > - if (cur->hton == hton) > - { > - /* Belongs to same engine, so move it to the delete list. */ > - cur->next= delete_list; > - delete_list= cur; > - if (cur->sub_id > best_sub_id) > + element *elem= (element *)my_hash_element(&hash, i); > + list_element *elist= elem->list; > + list_element *last_elem, **best_ptr_ptr, *cur, *next; > + uint64 best_sub_id; > + > + if (!elist) > + continue; /* Nothing here */ > + > + /* Delete any old stuff, but keep around the most recent one. */ > + cur= elist; > + best_sub_id= cur->sub_id; > + best_ptr_ptr= &elist; > + last_elem= cur; > + while ((next= cur->next)) { > + last_elem= next; > + if (next->sub_id > best_sub_id) > { > - best_sub_id= cur->sub_id; > - best_ptr_ptr= &delete_list; > - } > - else if (best_ptr_ptr == &delete_list) > + best_sub_id= next->sub_id; > best_ptr_ptr= &cur->next; > - } > - else > - { > - /* Another engine, leave it in the list. */ > - if (cur->sub_id > best_sub_id) > - { > - best_sub_id= cur->sub_id; > - /* Current best is not on the delete list. */ > - best_ptr_ptr= NULL; > } > - *next_ptr_ptr= cur; > - next_ptr_ptr= &cur->next; > + cur= next; > } > - cur= next; > - } > - *next_ptr_ptr= NULL; > - /* > - If the highest sub_id element is on the delete list, put it back on the > - original list, to preserve the highest sub_id element in the table for > - GTID position recovery. > - */ > - if (best_ptr_ptr) > - { > + /* > + Append the new elements to the full list. Note the order is important; > + we do it here so that we do not break the list if best_sub_id is the > + last of the new elements. > + */ > + last_elem->next= full_list; > + /* > + Delete the highest sub_id element from the old list, and put it back as > + the single-element new list. > + */ > cur= *best_ptr_ptr; > *best_ptr_ptr= cur->next; > - cur->next= elem->list; > + cur->next= NULL; > elem->list= cur; > + > + /* > + Collect the full list so far here. Note that elist may have moved if we > + deleted the first element, so order is again important. > + */ > + full_list= elist; > } > mysql_mutex_unlock(&LOCK_slave_state); > > - if (!delete_list) > - goto end; > + return full_list; > +} > + > > - /* Now delete any already committed GTIDs. */ > - bitmap_set_bit(table->read_set, table->field[0]->field_index); > - bitmap_set_bit(table->read_set, table->field[1]->field_index); > +/* Find the mysql.gtid_slave_posXXX table associated with a given hton. */ > +LEX_CSTRING * > +rpl_slave_state::select_gtid_pos_table(void *hton) > +{ > + struct gtid_pos_table *table_entry; > > - if ((err= table->file->ha_index_init(0, 0))) > + /* > + See comments on rpl_slave_state::gtid_pos_tables for rules around proper > + access to the list. > + */ > + table_entry= (struct gtid_pos_table *) > + my_atomic_loadptr_explicit(&gtid_pos_tables, MY_MEMORY_ORDER_ACQUIRE); > + > + while (table_entry) > { > - table->file->print_error(err, MYF(0)); > - goto end; > + if (table_entry->table_hton == hton) > + { > + if (likely(table_entry->state == GTID_POS_AVAILABLE)) > + return &table_entry->table_name; > + } > + table_entry= table_entry->next; > } > - cur = delete_list; > - while (cur) > - { > - uchar key_buffer[4+8]; > > - DBUG_EXECUTE_IF("gtid_slave_pos_simulate_failed_delete", > - { err= ENOENT; > - table->file->print_error(err, MYF(0)); > - /* `break' does not work inside DBUG_EXECUTE_IF */ > - goto dbug_break; }); > + table_entry= (struct gtid_pos_table *) > + my_atomic_loadptr_explicit(&default_gtid_pos_table, MY_MEMORY_ORDER_ACQUIRE); > + return &table_entry->table_name; > +} > > - next= cur->next; > > - table->field[1]->store(cur->sub_id, true); > - /* domain_id is already set in table->record[0] from write_row() above. */ > - key_copy(key_buffer, table->record[0], &table->key_info[0], 0, false); > - if (table->file->ha_index_read_map(table->record[1], key_buffer, > - HA_WHOLE_KEY, HA_READ_KEY_EXACT)) > - /* We cannot find the row, assume it is already deleted. */ > - ; > - else if ((err= table->file->ha_delete_row(table->record[1]))) > - table->file->print_error(err, MYF(0)); > - /* > - In case of error, we still discard the element from the list. We do > - not want to endlessly error on the same element in case of table > - corruption or such. > - */ > - cur= next; > - if (err) > - break; > - } > -IF_DBUG(dbug_break:, ) > - table->file->ha_index_end(); > +void > +rpl_slave_state::gtid_delete_pending(THD *thd, > + rpl_slave_state::list_element **list_ptr) > +{ > + int err= 0; > + ulonglong thd_saved_option; > > -end: > + if (unlikely(!loaded)) > + return; > > #ifdef WITH_WSREP > - thd->wsrep_ignore_table= false; > + /* > + Updates in slave state table should not be appended to galera transaction > + writeset. > + */ > + thd->wsrep_ignore_table= true; > #endif > > - if (table_opened) > + thd_saved_option= thd->variables.option_bits; > + thd->variables.option_bits&= > + ~(ulonglong)(OPTION_NOT_AUTOCOMMIT |OPTION_BEGIN |OPTION_BIN_LOG | > + OPTION_GTID_BEGIN); > + > + while (*list_ptr) > { > - if (err || (err= ha_commit_trans(thd, FALSE))) > - { > - /* > - If error, we need to put any remaining delete_list back into the HASH > - so we can do another delete attempt later. > - */ > - if (delete_list) > - { > - put_back_list(gtid->domain_id, delete_list); > - delete_list = 0; > - } > + LEX_CSTRING *gtid_pos_table_name, *tmp_table_name; > + Query_tables_list lex_backup; > + TABLE_LIST tlist; > + TABLE *table; > + handler::Table_flags direct_pos; > + list_element *cur, **cur_ptr_ptr; > + bool table_opened= false; > + void *hton= (*list_ptr)->hton; > > - ha_rollback_trans(thd, FALSE); > + thd->reset_for_next_command(); > + > + /* > + Only the SQL thread can call select_gtid_pos_table without a mutex > + Other threads needs to use a mutex and take into account that the > + result may change during execution, so we have to make a copy. > + */ > + mysql_mutex_lock(&LOCK_slave_state); > + tmp_table_name= select_gtid_pos_table(hton); > + gtid_pos_table_name= thd->make_clex_string(tmp_table_name->str, > + tmp_table_name->length); > + mysql_mutex_unlock(&LOCK_slave_state); > + if (!gtid_pos_table_name) > + { > + /* Out of memory - we can try again later. */ > + break; > } > - close_thread_tables(thd); > - if (rgi) > + > + thd->lex->reset_n_backup_query_tables_list(&lex_backup); > + tlist.init_one_table(&MYSQL_SCHEMA_NAME, gtid_pos_table_name, NULL, TL_WRITE); > + if ((err= open_and_lock_tables(thd, &tlist, FALSE, 0))) > + goto end; > + table_opened= true; > + table= tlist.table; > + > + if ((err= gtid_check_rpl_slave_state_table(table))) > + goto end; > + > + direct_pos= table->file->ha_table_flags() & HA_PRIMARY_KEY_REQUIRED_FOR_POSITION; > + bitmap_set_all(table->write_set); > + table->rpl_write_set= table->write_set; > + > + /* Now delete any already committed GTIDs. */ > + bitmap_set_bit(table->read_set, table->field[0]->field_index); > + bitmap_set_bit(table->read_set, table->field[1]->field_index); > + > + if (!direct_pos && (err= table->file->ha_index_init(0, 0))) > { > - thd->mdl_context.release_statement_locks(); > - /* > - Save the list of old gtid entries we deleted. If this transaction > - fails later for some reason and is rolled back, the deletion of those > - entries will be rolled back as well, and we will need to put them back > - on the to-be-deleted list so we can re-do the deletion. Otherwise > - redundant rows in mysql.gtid_slave_pos may accumulate if transactions > - are rolled back and retried after record_gtid(). > - */ > -#ifdef HAVE_REPLICATION > - rgi->pending_gtid_deletes_save(gtid->domain_id, delete_list); > -#endif > + table->file->print_error(err, MYF(0)); > + goto end; > } > - else > + > + cur = *list_ptr; > + cur_ptr_ptr = list_ptr; > + do > { > - thd->mdl_context.release_transactional_locks(); > -#ifdef HAVE_REPLICATION > - rpl_group_info::pending_gtid_deletes_free(delete_list); > -#endif > + uchar key_buffer[4+8]; > + list_element *next= cur->next; > + > + if (cur->hton == hton) > + { > + int res; > + > + table->field[0]->store((ulonglong)cur->domain_id, true); > + table->field[1]->store(cur->sub_id, true); > + if (direct_pos) > + { > + res= table->file->ha_rnd_pos_by_record(table->record[0]); > + } > + else > + { > + key_copy(key_buffer, table->record[0], &table->key_info[0], 0, false); > + res= table->file->ha_index_read_map(table->record[0], key_buffer, > + HA_WHOLE_KEY, HA_READ_KEY_EXACT); > + } > + DBUG_EXECUTE_IF("gtid_slave_pos_simulate_failed_delete", > + { res= 1; > + err= ENOENT; > + sql_print_error("<DEBUG> Error deleting old GTID row"); > + }); > + if (res) > + /* We cannot find the row, assume it is already deleted. */ > + ; > + else if ((err= table->file->ha_delete_row(table->record[0]))) > + { > + sql_print_error("Error deleting old GTID row: %s", > + thd->get_stmt_da()->message()); > + /* > + In case of error, we still discard the element from the list. We do > + not want to endlessly error on the same element in case of table > + corruption or such. > + */ > + } > + *cur_ptr_ptr= next; > + my_free(cur); > + } > + else > + { > + /* Leave this one in the list until we get to the table for its hton. */ > + cur_ptr_ptr= &cur->next; > + } > + cur= next; > + if (err) > + break; > + } while (cur); > +end: > + if (table_opened) > + { > + if (!direct_pos) > + table->file->ha_index_end(); > + > + if (err || (err= ha_commit_trans(thd, FALSE))) > + ha_rollback_trans(thd, FALSE); > } > + close_thread_tables(thd); > + thd->mdl_context.release_transactional_locks(); > + thd->lex->restore_backup_query_tables_list(&lex_backup); > + > + if (err) > + break; > } > - thd->lex->restore_backup_query_tables_list(&lex_backup); > thd->variables.option_bits= thd_saved_option; > - thd->resume_subsequent_commits(suspended_wfc); > - DBUG_EXECUTE_IF("inject_record_gtid_serverid_100_sleep", > - { > - if (gtid->server_id == 100) > - my_sleep(500000); > - }); > - DBUG_RETURN(err); > + > +#ifdef WITH_WSREP > + thd->wsrep_ignore_table= false; > +#endif > } > > > @@ -1251,7 +1368,7 @@ rpl_slave_state::load(THD *thd, const char *state_from_master, size_t len, > > if (gtid_parser_helper(&state_from_master, end, &gtid) || > !(sub_id= next_sub_id(gtid.domain_id)) || > - record_gtid(thd, &gtid, sub_id, NULL, in_statement, &hton) || > + record_gtid(thd, &gtid, sub_id, false, in_statement, &hton) || > update(gtid.domain_id, gtid.server_id, sub_id, gtid.seq_no, hton, NULL)) > return 1; > if (state_from_master == end) > diff --git a/sql/rpl_gtid.h b/sql/rpl_gtid.h > index 0fc92d5e33c..60d822f7b0d 100644 > --- a/sql/rpl_gtid.h > +++ b/sql/rpl_gtid.h > @@ -118,8 +118,9 @@ struct rpl_slave_state > { > struct list_element *next; > uint64 sub_id; > - uint64 seq_no; > + uint32 domain_id; > uint32 server_id; > + uint64 seq_no; > /* > hton of mysql.gtid_slave_pos* table used to record this GTID. > Can be NULL if the gtid table failed to load (eg. missing > @@ -191,6 +192,8 @@ struct rpl_slave_state > > /* Mapping from domain_id to its element. */ > HASH hash; > + /* GTIDs added since last purge of old mysql.gtid_slave_pos rows. */ > + uint32 pending_gtid_count; > /* Mutex protecting access to the state. */ > mysql_mutex_t LOCK_slave_state; > /* Auxiliary buffer to sort gtid list. */ > @@ -233,7 +236,10 @@ struct rpl_slave_state > int truncate_state_table(THD *thd); > void select_gtid_pos_table(THD *thd, LEX_CSTRING *out_tablename); > int record_gtid(THD *thd, const rpl_gtid *gtid, uint64 sub_id, > - rpl_group_info *rgi, bool in_statement, void **out_hton); > + bool in_transaction, bool in_statement, void **out_hton); > + list_element *gtid_grab_pending_delete_list(); > + LEX_CSTRING *select_gtid_pos_table(void *hton); > + void gtid_delete_pending(THD *thd, rpl_slave_state::list_element **list_ptr); > uint64 next_sub_id(uint32 domain_id); > int iterate(int (*cb)(rpl_gtid *, void *), void *data, > rpl_gtid *extra_gtids, uint32 num_extra, > @@ -245,7 +251,7 @@ struct rpl_slave_state > bool is_empty(); > > element *get_element(uint32 domain_id); > - int put_back_list(uint32 domain_id, list_element *list); > + int put_back_list(list_element *list); > > void update_state_hash(uint64 sub_id, rpl_gtid *gtid, void *hton, > rpl_group_info *rgi); > diff --git a/sql/rpl_rli.cc b/sql/rpl_rli.cc > index b275ad884bd..2d91620c898 100644 > --- a/sql/rpl_rli.cc > +++ b/sql/rpl_rli.cc > @@ -1820,6 +1820,7 @@ rpl_load_gtid_slave_state(THD *thd) > int err= 0; > uint32 i; > load_gtid_state_cb_data cb_data; > + rpl_slave_state::list_element *old_gtids_list; > DBUG_ENTER("rpl_load_gtid_slave_state"); > > mysql_mutex_lock(&rpl_global_gtid_slave_state->LOCK_slave_state); > @@ -1905,6 +1906,13 @@ rpl_load_gtid_slave_state(THD *thd) > rpl_global_gtid_slave_state->loaded= true; > mysql_mutex_unlock(&rpl_global_gtid_slave_state->LOCK_slave_state); > > + /* Clear out no longer needed elements now. */ > + old_gtids_list= > + rpl_global_gtid_slave_state->gtid_grab_pending_delete_list(); > + rpl_global_gtid_slave_state->gtid_delete_pending(thd, &old_gtids_list); > + if (old_gtids_list) > + rpl_global_gtid_slave_state->put_back_list(old_gtids_list); > + > end: > if (array_inited) > delete_dynamic(&array); > @@ -2086,7 +2094,6 @@ rpl_group_info::reinit(Relay_log_info *rli) > long_find_row_note_printed= false; > did_mark_start_commit= false; > gtid_ev_flags2= 0; > - pending_gtid_delete_list= NULL; > last_master_timestamp = 0; > gtid_ignore_duplicate_state= GTID_DUPLICATE_NULL; > speculation= SPECULATE_NO; > @@ -2217,12 +2224,6 @@ void rpl_group_info::cleanup_context(THD *thd, bool error) > erroneously update the GTID position. > */ > gtid_pending= false; > - > - /* > - Rollback will have undone any deletions of old rows we might have made > - in mysql.gtid_slave_pos. Put those rows back on the list to be deleted. > - */ > - pending_gtid_deletes_put_back(); > } > m_table_map.clear_tables(); > slave_close_thread_tables(thd); > @@ -2448,78 +2449,6 @@ rpl_group_info::unmark_start_commit() > } > > > -/* > - When record_gtid() has deleted any old rows from the table > - mysql.gtid_slave_pos as part of a replicated transaction, save the list of > - rows deleted here. > - > - If later the transaction fails (eg. optimistic parallel replication), the > - deletes will be undone when the transaction is rolled back. Then we can > - put back the list of rows into the rpl_global_gtid_slave_state, so that > - we can re-do the deletes and avoid accumulating old rows in the table. > -*/ > -void > -rpl_group_info::pending_gtid_deletes_save(uint32 domain_id, > - rpl_slave_state::list_element *list) > -{ > - /* > - We should never get to a state where we try to save a new pending list of > - gtid deletes while we still have an old one. But make sure we handle it > - anyway just in case, so we avoid leaving stray entries in the > - mysql.gtid_slave_pos table. > - */ > - DBUG_ASSERT(!pending_gtid_delete_list); > - if (unlikely(pending_gtid_delete_list)) > - pending_gtid_deletes_put_back(); > - > - pending_gtid_delete_list= list; > - pending_gtid_delete_list_domain= domain_id; > -} > - > - > -/* > - Take the list recorded by pending_gtid_deletes_save() and put it back into > - rpl_global_gtid_slave_state. This is needed if deletion of the rows was > - rolled back due to transaction failure. > -*/ > -void > -rpl_group_info::pending_gtid_deletes_put_back() > -{ > - if (pending_gtid_delete_list) > - { > - rpl_global_gtid_slave_state->put_back_list(pending_gtid_delete_list_domain, > - pending_gtid_delete_list); > - pending_gtid_delete_list= NULL; > - } > -} > - > - > -/* > - Free the list recorded by pending_gtid_deletes_save(). Done when the deletes > - in the list have been permanently committed. > -*/ > -void > -rpl_group_info::pending_gtid_deletes_clear() > -{ > - pending_gtid_deletes_free(pending_gtid_delete_list); > - pending_gtid_delete_list= NULL; > -} > - > - > -void > -rpl_group_info::pending_gtid_deletes_free(rpl_slave_state::list_element *list) > -{ > - rpl_slave_state::list_element *next; > - > - while (list) > - { > - next= list->next; > - my_free(list); > - list= next; > - } > -} > - > - > rpl_sql_thread_info::rpl_sql_thread_info(Rpl_filter *filter) > : rpl_filter(filter) > { > diff --git a/sql/rpl_rli.h b/sql/rpl_rli.h > index d9f0e0e5d3b..b8b153c34be 100644 > --- a/sql/rpl_rli.h > +++ b/sql/rpl_rli.h > @@ -757,11 +757,6 @@ struct rpl_group_info > /* Needs room for "Gtid D-S-N\x00". */ > char gtid_info_buf[5+10+1+10+1+20+1]; > > - /* List of not yet committed deletions in mysql.gtid_slave_pos. */ > - rpl_slave_state::list_element *pending_gtid_delete_list; > - /* Domain associated with pending_gtid_delete_list. */ > - uint32 pending_gtid_delete_list_domain; > - > /* > The timestamp, from the master, of the commit event. > Used to do delayed update of rli->last_master_timestamp, for getting > @@ -903,12 +898,6 @@ struct rpl_group_info > char *gtid_info(); > void unmark_start_commit(); > > - static void pending_gtid_deletes_free(rpl_slave_state::list_element *list); > - void pending_gtid_deletes_save(uint32 domain_id, > - rpl_slave_state::list_element *list); > - void pending_gtid_deletes_put_back(); > - void pending_gtid_deletes_clear(); > - > longlong get_row_stmt_start_timestamp() > { > return row_stmt_start_timestamp; > diff --git a/sql/slave.cc b/sql/slave.cc > index bb1300d36e6..f8499513dd6 100644 > --- a/sql/slave.cc > +++ b/sql/slave.cc > @@ -465,6 +465,8 @@ static struct slave_background_gtid_pos_create_t { > void *hton; > } *slave_background_gtid_pos_create_list; > > +static volatile bool slave_background_gtid_pending_delete_flag; > + > > pthread_handler_t > handle_slave_background(void *arg __attribute__((unused))) > @@ -499,6 +501,7 @@ handle_slave_background(void *arg __attribute__((unused))) > { > slave_background_kill_t *kill_list; > slave_background_gtid_pos_create_t *create_list; > + bool pending_deletes; > > thd->ENTER_COND(&COND_slave_background, &LOCK_slave_background, > &stage_slave_background_wait_request, > @@ -508,13 +511,15 @@ handle_slave_background(void *arg __attribute__((unused))) > stop= abort_loop || thd->killed || slave_background_thread_stop; > kill_list= slave_background_kill_list; > create_list= slave_background_gtid_pos_create_list; > - if (stop || kill_list || create_list) > + pending_deletes= slave_background_gtid_pending_delete_flag; > + if (stop || kill_list || create_list || pending_deletes) > break; > mysql_cond_wait(&COND_slave_background, &LOCK_slave_background); > } > > slave_background_kill_list= NULL; > slave_background_gtid_pos_create_list= NULL; > + slave_background_gtid_pending_delete_flag= false; > thd->EXIT_COND(&old_stage); > > while (kill_list) > @@ -541,6 +546,17 @@ handle_slave_background(void *arg __attribute__((unused))) > create_list= next; > } > > + if (pending_deletes) > + { > + rpl_slave_state::list_element *list; > + > + slave_background_gtid_pending_delete_flag= false; > + list= rpl_global_gtid_slave_state->gtid_grab_pending_delete_list(); > + rpl_global_gtid_slave_state->gtid_delete_pending(thd, &list); > + if (list) > + rpl_global_gtid_slave_state->put_back_list(list); > + } > + > mysql_mutex_lock(&LOCK_slave_background); > } while (!stop); > > @@ -615,6 +631,23 @@ slave_background_gtid_pos_create_request( > > > /* > + Request the slave background thread to delete no longer used rows from the > + mysql.gtid_slave_pos* tables. > + > + This is called from time-critical rpl_slave_state::update(), so we avoid > + taking any locks here. This means we may race with the background thread > + to occasionally lose a signal. This is not a problem; any pending rows to > + be deleted will just be deleted a bit later as part of the next batch. > +*/ > +void > +slave_background_gtid_pending_delete_request(void) > +{ > + slave_background_gtid_pending_delete_flag= true; > + mysql_cond_signal(&COND_slave_background); > +} > + > + > +/* > Start the slave background thread. > > This thread is currently used for two purposes: > diff --git a/sql/slave.h b/sql/slave.h > index 649d55b45b9..12d569b0333 100644 > --- a/sql/slave.h > +++ b/sql/slave.h > @@ -276,6 +276,7 @@ bool net_request_file(NET* net, const char* fname); > void slave_background_kill_request(THD *to_kill); > void slave_background_gtid_pos_create_request > (rpl_slave_state::gtid_pos_table *table_entry); > +void slave_background_gtid_pending_delete_request(void); > > extern bool volatile abort_loop; > extern Master_info *active_mi; /* active_mi for multi-master */ > diff --git a/sql/sys_vars.cc b/sql/sys_vars.cc > index 6d4c135683a..9348f4e5c98 100644 > --- a/sql/sys_vars.cc > +++ b/sql/sys_vars.cc > @@ -1942,6 +1942,19 @@ Sys_var_last_gtid::session_value_ptr(THD *thd, const LEX_CSTRING *base) > } > > > +static Sys_var_uint Sys_gtid_cleanup_batch_size( > + "gtid_cleanup_batch_size", > + "Normally does not need tuning. How many old rows must accumulate in " > + "the mysql.gtid_slave_pos table before a background job will be run to " > + "delete them. Can be increased to reduce number of commits if " > + "using many different engines with --gtid_pos_auto_engines, or to " > + "reduce CPU overhead if using a huge number of different " > + "gtid_domain_ids. Can be decreased to reduce number of old rows in the " > + "table.", > + GLOBAL_VAR(opt_gtid_cleanup_batch_size), CMD_LINE(REQUIRED_ARG), > + VALID_RANGE(0,2147483647), DEFAULT(64), BLOCK_SIZE(1)); > + > + > static bool > check_slave_parallel_threads(sys_var *self, THD *thd, set_var *var) > { > diff --git a/storage/rocksdb/mysql-test/rocksdb_rpl/r/mdev12179.result b/storage/rocksdb/mysql-test/rocksdb_rpl/r/mdev12179.result > index 9c20fea97ae..a1e501f78f4 100644 > --- a/storage/rocksdb/mysql-test/rocksdb_rpl/r/mdev12179.result > +++ b/storage/rocksdb/mysql-test/rocksdb_rpl/r/mdev12179.result > @@ -2,6 +2,7 @@ include/master-slave.inc > [connection master] > connection server_2; > include/stop_slave.inc > +SET GLOBAL gtid_cleanup_batch_size = 999999999; > CHANGE MASTER TO master_use_gtid=slave_pos; > SET sql_log_bin=0; > CREATE TABLE mysql.gtid_slave_pos_innodb LIKE mysql.gtid_slave_pos; > @@ -41,6 +42,8 @@ a > 1 > SELECT * FROM mysql.gtid_slave_pos ORDER BY sub_id; > domain_id sub_id server_id seq_no > +0 1 1 1 > +0 2 1 2 > 0 3 1 3 > 0 4 1 4 > SELECT * FROM ( SELECT * FROM mysql.gtid_slave_pos_innodb > @@ -121,6 +124,21 @@ Transactions_multi_engine 6 > DELETE FROM t1 WHERE a >= 100; > DELETE FROM t2 WHERE a >= 100; > DELETE FROM t3 WHERE a >= 100; > +connection server_1; > +include/save_master_gtid.inc > +connection server_2; > +include/sync_with_master_gtid.inc > +SELECT COUNT(*)>=10 FROM mysql.gtid_slave_pos; > +COUNT(*)>=10 > +1 > +SELECT COUNT(*)>=10 FROM ( SELECT * FROM mysql.gtid_slave_pos_innodb > +UNION ALL SELECT * FROM mysql.gtid_slave_pos_innodb_redundant) inner_select; > +COUNT(*)>=10 > +1 > +SELECT COUNT(*)>=10 FROM mysql.gtid_slave_pos_rocksdb; > +COUNT(*)>=10 > +1 > +SET GLOBAL gtid_cleanup_batch_size = 3; > connection server_2; > include/stop_slave.inc > SET sql_log_bin=0; > diff --git a/storage/rocksdb/mysql-test/rocksdb_rpl/t/mdev12179.test b/storage/rocksdb/mysql-test/rocksdb_rpl/t/mdev12179.test > index e0d16e7f242..631d9ca533f 100644 > --- a/storage/rocksdb/mysql-test/rocksdb_rpl/t/mdev12179.test > +++ b/storage/rocksdb/mysql-test/rocksdb_rpl/t/mdev12179.test > @@ -4,6 +4,12 @@ > > --connection server_2 > --source include/stop_slave.inc > + > +# Set GTID cleanup limit high enough that cleanup will not run and we > +# can rely on consistent table output in .result. > +--let $old_gtid_cleanup_batch_size=`SELECT @@GLOBAL.gtid_cleanup_batch_size` > +SET GLOBAL gtid_cleanup_batch_size = 999999999; > + > CHANGE MASTER TO master_use_gtid=slave_pos; > SET sql_log_bin=0; > CREATE TABLE mysql.gtid_slave_pos_innodb LIKE mysql.gtid_slave_pos; > @@ -89,6 +95,82 @@ DELETE FROM t2 WHERE a >= 100; > DELETE FROM t3 WHERE a >= 100; > > > +# Create a bunch more GTIDs in mysql.gtid_slave_pos* tables to test with. > +--connection server_1 > +--disable_query_log > +let $i=10; > +while ($i) { > + eval INSERT INTO t1 VALUES (300+$i); > + eval INSERT INTO t2 VALUES (300+$i); > + eval INSERT INTO t3 VALUES (300+$i); > + dec $i; > +} > +--enable_query_log > +--source include/save_master_gtid.inc > + > +--connection server_2 > +--source include/sync_with_master_gtid.inc > + > +# Check that we have many rows in mysql.gtid_slave_pos now (since > +# @@gtid_cleanup_batch_size was set to a huge value). No need to check > +# for an exact number, since that will require changing .result if > +# anything changes prior to this point, and we just need to know that > +# we have still have some data in the tables to make the following > +# test effective. > +SELECT COUNT(*)>=10 FROM mysql.gtid_slave_pos; > +SELECT COUNT(*)>=10 FROM ( SELECT * FROM mysql.gtid_slave_pos_innodb > + UNION ALL SELECT * FROM mysql.gtid_slave_pos_innodb_redundant) inner_select; > +SELECT COUNT(*)>=10 FROM mysql.gtid_slave_pos_rocksdb; > + > +# Check that old GTID rows will be deleted when batch delete size is > +# set reasonably. Old row deletion is not 100% deterministic (by design), so > +# we must wait for it to occur, but it should occur eventually. > +SET GLOBAL gtid_cleanup_batch_size = 3; > +let $i=40; > +--disable_query_log > +--let $keep_include_silent=1 > +while ($i) { > + let N=`SELECT 1+($i MOD 3)`; > + --connection server_1 > + eval UPDATE t$N SET a=a+1 WHERE a=(SELECT MAX(a) FROM t$N); > + --source include/save_master_gtid.inc > + --connection server_2 > + --source include/sync_with_master_gtid.inc > + let $j=50; > + while ($j) { > + let $is_done=`SELECT SUM(a)=1 FROM ( > + SELECT COUNT(*) AS a FROM mysql.gtid_slave_pos > + UNION ALL > + SELECT COUNT(*) AS a FROM ( SELECT * FROM mysql.gtid_slave_pos_innodb > + UNION ALL SELECT * FROM mysql.gtid_slave_pos_innodb_redundant) inner_select > + UNION ALL > + SELECT COUNT(*) AS a FROM mysql.gtid_slave_pos_rocksdb) outer_select`; > + if ($is_done) { > + let $j=0; > + } > + if (!$is_done) { > + real_sleep 0.1; > + dec $j; > + } > + } > + dec $i; > + if ($is_done) { > + let $i=0; > + } > +} > +--enable_query_log > +--let $keep_include_silent=0 > +if (!$is_done) { > + --echo Timed out waiting for mysql.gtid_slave_pos* tables to be cleaned up > +} > + > +--disable_query_log > +DELETE FROM t1 WHERE a >= 100; > +DELETE FROM t2 WHERE a >= 100; > +DELETE FROM t3 WHERE a >= 100; > +--enable_query_log > + > + > # Test status variables Rpl_transactions_multi_engine and Transactions_gtid_foreign_engine. > # Have mysql.gtid_slave_pos* for myisam and innodb but not rocksdb. > --connection server_2 > @@ -223,6 +305,9 @@ SHOW STATUS LIKE "%transactions%engine"; > SET sql_log_bin=0; > DROP TABLE mysql.gtid_slave_pos_innodb; > SET sql_log_bin=1; > +--disable_query_log > +eval SET GLOBAL gtid_cleanup_batch_size = $old_gtid_cleanup_batch_size; > +--enable_query_log > > --connection server_1 > DROP TABLE t1; > diff --git a/storage/tokudb/mysql-test/tokudb_rpl/r/mdev12179.result b/storage/tokudb/mysql-test/tokudb_rpl/r/mdev12179.result > index d4532eec4e2..d79e7e59aa4 100644 > --- a/storage/tokudb/mysql-test/tokudb_rpl/r/mdev12179.result > +++ b/storage/tokudb/mysql-test/tokudb_rpl/r/mdev12179.result > @@ -2,6 +2,7 @@ include/master-slave.inc > [connection master] > connection server_2; > include/stop_slave.inc > +SET GLOBAL gtid_cleanup_batch_size = 999999999; > CHANGE MASTER TO master_use_gtid=slave_pos; > SET sql_log_bin=0; > CREATE TABLE mysql.gtid_slave_pos_innodb LIKE mysql.gtid_slave_pos; > @@ -41,6 +42,8 @@ a > 1 > SELECT * FROM mysql.gtid_slave_pos ORDER BY sub_id; > domain_id sub_id server_id seq_no > +0 1 1 1 > +0 2 1 2 > 0 3 1 3 > 0 4 1 4 > SELECT * FROM ( SELECT * FROM mysql.gtid_slave_pos_innodb > @@ -121,6 +124,21 @@ Transactions_multi_engine 6 > DELETE FROM t1 WHERE a >= 100; > DELETE FROM t2 WHERE a >= 100; > DELETE FROM t3 WHERE a >= 100; > +connection server_1; > +include/save_master_gtid.inc > +connection server_2; > +include/sync_with_master_gtid.inc > +SELECT COUNT(*)>=10 FROM mysql.gtid_slave_pos; > +COUNT(*)>=10 > +1 > +SELECT COUNT(*)>=10 FROM ( SELECT * FROM mysql.gtid_slave_pos_innodb > +UNION ALL SELECT * FROM mysql.gtid_slave_pos_innodb_redundant) inner_select; > +COUNT(*)>=10 > +1 > +SELECT COUNT(*)>=10 FROM mysql.gtid_slave_pos_tokudb; > +COUNT(*)>=10 > +1 > +SET GLOBAL gtid_cleanup_batch_size = 3; > connection server_2; > include/stop_slave.inc > SET sql_log_bin=0; > diff --git a/storage/tokudb/mysql-test/tokudb_rpl/t/mdev12179.test b/storage/tokudb/mysql-test/tokudb_rpl/t/mdev12179.test > index ceb119cd0dc..1d19a25889e 100644 > --- a/storage/tokudb/mysql-test/tokudb_rpl/t/mdev12179.test > +++ b/storage/tokudb/mysql-test/tokudb_rpl/t/mdev12179.test > @@ -4,6 +4,12 @@ > > --connection server_2 > --source include/stop_slave.inc > + > +# Set GTID cleanup limit high enough that cleanup will not run and we > +# can rely on consistent table output in .result. > +--let $old_gtid_cleanup_batch_size=`SELECT @@GLOBAL.gtid_cleanup_batch_size` > +SET GLOBAL gtid_cleanup_batch_size = 999999999; > + > CHANGE MASTER TO master_use_gtid=slave_pos; > SET sql_log_bin=0; > CREATE TABLE mysql.gtid_slave_pos_innodb LIKE mysql.gtid_slave_pos; > @@ -89,6 +95,82 @@ DELETE FROM t2 WHERE a >= 100; > DELETE FROM t3 WHERE a >= 100; > > > +# Create a bunch more GTIDs in mysql.gtid_slave_pos* tables to test with. > +--connection server_1 > +--disable_query_log > +let $i=10; > +while ($i) { > + eval INSERT INTO t1 VALUES (300+$i); > + eval INSERT INTO t2 VALUES (300+$i); > + eval INSERT INTO t3 VALUES (300+$i); > + dec $i; > +} > +--enable_query_log > +--source include/save_master_gtid.inc > + > +--connection server_2 > +--source include/sync_with_master_gtid.inc > + > +# Check that we have many rows in mysql.gtid_slave_pos now (since > +# @@gtid_cleanup_batch_size was set to a huge value). No need to check > +# for an exact number, since that will require changing .result if > +# anything changes prior to this point, and we just need to know that > +# we have still have some data in the tables to make the following > +# test effective. > +SELECT COUNT(*)>=10 FROM mysql.gtid_slave_pos; > +SELECT COUNT(*)>=10 FROM ( SELECT * FROM mysql.gtid_slave_pos_innodb > + UNION ALL SELECT * FROM mysql.gtid_slave_pos_innodb_redundant) inner_select; > +SELECT COUNT(*)>=10 FROM mysql.gtid_slave_pos_tokudb; > + > +# Check that old GTID rows will be deleted when batch delete size is > +# set reasonably. Old row deletion is not 100% deterministic (by design), so > +# we must wait for it to occur, but it should occur eventually. > +SET GLOBAL gtid_cleanup_batch_size = 3; > +let $i=40; > +--disable_query_log > +--let $keep_include_silent=1 > +while ($i) { > + let N=`SELECT 1+($i MOD 3)`; > + --connection server_1 > + eval UPDATE t$N SET a=a+1 WHERE a=(SELECT MAX(a) FROM t$N); > + --source include/save_master_gtid.inc > + --connection server_2 > + --source include/sync_with_master_gtid.inc > + let $j=50; > + while ($j) { > + let $is_done=`SELECT SUM(a)=1 FROM ( > + SELECT COUNT(*) AS a FROM mysql.gtid_slave_pos > + UNION ALL > + SELECT COUNT(*) AS a FROM ( SELECT * FROM mysql.gtid_slave_pos_innodb > + UNION ALL SELECT * FROM mysql.gtid_slave_pos_innodb_redundant) inner_select > + UNION ALL > + SELECT COUNT(*) AS a FROM mysql.gtid_slave_pos_tokudb) outer_select`; > + if ($is_done) { > + let $j=0; > + } > + if (!$is_done) { > + real_sleep 0.1; > + dec $j; > + } > + } > + dec $i; > + if ($is_done) { > + let $i=0; > + } > +} > +--enable_query_log > +--let $keep_include_silent=0 > +if (!$is_done) { > + --echo Timed out waiting for mysql.gtid_slave_pos* tables to be cleaned up > +} > + > +--disable_query_log > +DELETE FROM t1 WHERE a >= 100; > +DELETE FROM t2 WHERE a >= 100; > +DELETE FROM t3 WHERE a >= 100; > +--enable_query_log > + > + > # Test status variables Rpl_transactions_multi_engine and Transactions_gtid_foreign_engine. > # Have mysql.gtid_slave_pos* for myisam and innodb but not tokudb. > --connection server_2 > @@ -223,6 +305,9 @@ SHOW STATUS LIKE "%transactions%engine"; > SET sql_log_bin=0; > DROP TABLE mysql.gtid_slave_pos_innodb; > SET sql_log_bin=1; > +--disable_query_log > +eval SET GLOBAL gtid_cleanup_batch_size = $old_gtid_cleanup_batch_size; > +--enable_query_log > > --connection server_1 > DROP TABLE t1;

1 0

Re: [Maria-developers] [Commits] 1ed2d8b98ad: MDEV-17032: Estimates are higher for partitions of a table with @@use_stat_tables= PREFERABLY
by Sergey Petrunia 25 Nov '18

25 Nov '18

Hi Varun, On Fri, Nov 16, 2018 at 07:41:07PM +0530, Varun wrote: > revision-id: 1ed2d8b98ade099fe23b7d5c00d23364388e15aa (mariadb-10.0.36-80-g1ed2d8b98ad) > parent(s): a84d87fde8c0bc325c8e00f06ea02bcd84a75d55 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2018-11-16 19:40:47 +0530 > message: > > MDEV-17032: Estimates are higher for partitions of a table with @@use_stat_tables= PREFERABLY > > The problem here is EITS statistics does not calculate statistics for the partitions of the table. > So a temporary solution would be to not read EITS statistics for partitioned tables. > > Also disabling reading of EITS for columns that participate in the partition list of a table. > > --- > mysql-test/r/partition.result | 88 +++++++++++++++++++++++++++++++++++++++++++ > mysql-test/t/partition.test | 58 ++++++++++++++++++++++++++++ > sql/opt_range.cc | 12 +++++- > sql/partition_info.cc | 26 +++++++++++++ > sql/partition_info.h | 1 + > sql/sql_statistics.cc | 16 ++++++++ > 6 files changed, 199 insertions(+), 2 deletions(-) > > diff --git a/mysql-test/r/partition.result b/mysql-test/r/partition.result > index c6669176b3d..aedf9f89f0e 100644 > --- a/mysql-test/r/partition.result > +++ b/mysql-test/r/partition.result > @@ -2645,3 +2645,91 @@ Warnings: > Note 1517 Duplicate partition name p2 > DEALLOCATE PREPARE stmt; > DROP TABLE t1; > +# > +# MDEV-17032: Estimates are higher for partitions of a table with @@use_stat_tables= PREFERABLY > +# > +create table t0(a int); > +insert into t0 values (0),(1),(2),(3),(4),(5),(6),(7),(8),(9); > +create table t1(a int); > +insert into t1 select A.a + B.a* 10 + C.a * 100 from t0 A, t0 B, t0 C; > +create table t2 ( > +part_key int, > +a int, > +b int > +) partition by list(part_key) ( > +partition p0 values in (0), > +partition p1 values in (1), > +partition p2 values in (2), > +partition p3 values in (3), > +partition p4 values in (4) > +); > +insert into t2 > +select mod(a,5), a/100, a from t1; > +set @save_use_stat_tables= @@use_stat_tables; > +set @save_optimizer_use_condition_selectivity=@@optimizer_use_condition_selectivity; > +# > +# Tests using stats provided by the storage engine > +# > +explain extended select * from t2 where part_key=1; > +id select_type table type possible_keys key key_len ref rows filtered Extra > +1 SIMPLE t2 ALL NULL NULL NULL NULL 200 100.00 Using where > +Warnings: > +Note 1003 select `test`.`t2`.`part_key` AS `part_key`,`test`.`t2`.`a` AS `a`,`test`.`t2`.`b` AS `b` from `test`.`t2` where (`test`.`t2`.`part_key` = 1) > +explain partitions select * from t2 where part_key=1; > +id select_type table partitions type possible_keys key key_len ref rows Extra > +1 SIMPLE t2 p1 ALL NULL NULL NULL NULL 200 Using where > +explain extended select * from t2 where part_key in (1,2); > +id select_type table type possible_keys key key_len ref rows filtered Extra > +1 SIMPLE t2 ALL NULL NULL NULL NULL 400 100.00 Using where > +Warnings: > +Note 1003 select `test`.`t2`.`part_key` AS `part_key`,`test`.`t2`.`a` AS `a`,`test`.`t2`.`b` AS `b` from `test`.`t2` where (`test`.`t2`.`part_key` in (1,2)) > +explain partitions select * from t2 where part_key=1; > +id select_type table partitions type possible_keys key key_len ref rows Extra > +1 SIMPLE t2 p1 ALL NULL NULL NULL NULL 200 Using where > +explain extended select * from t2 where part_key in (1,2); > +id select_type table type possible_keys key key_len ref rows filtered Extra > +1 SIMPLE t2 ALL NULL NULL NULL NULL 400 100.00 Using where > +Warnings: > +Note 1003 select `test`.`t2`.`part_key` AS `part_key`,`test`.`t2`.`a` AS `a`,`test`.`t2`.`b` AS `b` from `test`.`t2` where (`test`.`t2`.`part_key` in (1,2)) > +explain partitions select * from t2 where part_key in (1,2); > +id select_type table partitions type possible_keys key key_len ref rows Extra > +1 SIMPLE t2 p1,p2 ALL NULL NULL NULL NULL 400 Using where > +set @save_histogram_size=@@histogram_size; > +set @@histogram_size=100; > +set @@use_stat_tables= PREFERABLY; > +set @@optimizer_use_condition_selectivity=4; > +analyze table t2; > +Table Op Msg_type Msg_text > +test.t2 analyze status Engine-independent statistics collected > +test.t2 analyze status OK > +# > +# Tests using EITS > +# > +explain extended select * from t2 where part_key=1; > +id select_type table type possible_keys key key_len ref rows filtered Extra > +1 SIMPLE t2 ALL NULL NULL NULL NULL 200 100.00 Using where > +Warnings: > +Note 1003 select `test`.`t2`.`part_key` AS `part_key`,`test`.`t2`.`a` AS `a`,`test`.`t2`.`b` AS `b` from `test`.`t2` where (`test`.`t2`.`part_key` = 1) > +explain partitions select * from t2 where part_key=1; > +id select_type table partitions type possible_keys key key_len ref rows Extra > +1 SIMPLE t2 p1 ALL NULL NULL NULL NULL 200 Using where > +explain extended select * from t2 where part_key in (1,2); > +id select_type table type possible_keys key key_len ref rows filtered Extra > +1 SIMPLE t2 ALL NULL NULL NULL NULL 400 100.00 Using where > +Warnings: > +Note 1003 select `test`.`t2`.`part_key` AS `part_key`,`test`.`t2`.`a` AS `a`,`test`.`t2`.`b` AS `b` from `test`.`t2` where (`test`.`t2`.`part_key` in (1,2)) > +explain partitions select * from t2 where part_key=1; > +id select_type table partitions type possible_keys key key_len ref rows Extra > +1 SIMPLE t2 p1 ALL NULL NULL NULL NULL 200 Using where > +explain extended select * from t2 where part_key in (1,2); > +id select_type table type possible_keys key key_len ref rows filtered Extra > +1 SIMPLE t2 ALL NULL NULL NULL NULL 400 100.00 Using where > +Warnings: > +Note 1003 select `test`.`t2`.`part_key` AS `part_key`,`test`.`t2`.`a` AS `a`,`test`.`t2`.`b` AS `b` from `test`.`t2` where (`test`.`t2`.`part_key` in (1,2)) > +explain partitions select * from t2 where part_key in (1,2); > +id select_type table partitions type possible_keys key key_len ref rows Extra > +1 SIMPLE t2 p1,p2 ALL NULL NULL NULL NULL 400 Using where > +set @@use_stat_tables= @save_use_stat_tables; > +set @@optimizer_use_condition_selectivity= @save_optimizer_use_condition_selectivity; > +set @@histogram_size= @save_histogram_size; > +drop table t0,t1,t2; > diff --git a/mysql-test/t/partition.test b/mysql-test/t/partition.test > index 1c8cd0375d6..00c6f1ce77c 100644 > --- a/mysql-test/t/partition.test > +++ b/mysql-test/t/partition.test > @@ -2897,3 +2897,61 @@ EXECUTE stmt; > DEALLOCATE PREPARE stmt; > DROP TABLE t1; > > +--echo # > +--echo # MDEV-17032: Estimates are higher for partitions of a table with @@use_stat_tables= PREFERABLY > +--echo # > + > +create table t0(a int); > +insert into t0 values (0),(1),(2),(3),(4),(5),(6),(7),(8),(9); > + > +create table t1(a int); > +insert into t1 select A.a + B.a* 10 + C.a * 100 from t0 A, t0 B, t0 C; > + > + > +create table t2 ( > + part_key int, > + a int, > + b int > +) partition by list(part_key) ( > + partition p0 values in (0), > + partition p1 values in (1), > + partition p2 values in (2), > + partition p3 values in (3), > + partition p4 values in (4) > +); > +insert into t2 > +select mod(a,5), a/100, a from t1; > + > +set @save_use_stat_tables= @@use_stat_tables; > +set @save_optimizer_use_condition_selectivity=@@optimizer_use_condition_selectivity; > +--echo # > +--echo # Tests using stats provided by the storage engine > +--echo # > +explain extended select * from t2 where part_key=1; > +explain partitions select * from t2 where part_key=1; > +explain extended select * from t2 where part_key in (1,2); > +explain partitions select * from t2 where part_key=1; ^^ Isn't this query the same like one two lines above? > +explain extended select * from t2 where part_key in (1,2); > +explain partitions select * from t2 where part_key in (1,2); > + > +set @save_histogram_size=@@histogram_size; > +set @@histogram_size=100; > +set @@use_stat_tables= PREFERABLY; > +set @@optimizer_use_condition_selectivity=4; > +analyze table t2; > +--echo # > +--echo # Tests using EITS > +--echo # > +# filtered should be 100 Please change to use '--echo #' so that the above is visible in the .result file as well. > +explain extended select * from t2 where part_key=1; > +explain partitions select * from t2 where part_key=1; > +# filtered should be 100 > +explain extended select * from t2 where part_key in (1,2); > +explain partitions select * from t2 where part_key=1; > +# filtered should be 100 > +explain extended select * from t2 where part_key in (1,2); > +explain partitions select * from t2 where part_key in (1,2); Please add a test that shows that condition selectivity is taken into account for non-partitioned columns. > +set @@use_stat_tables= @save_use_stat_tables; > +set @@optimizer_use_condition_selectivity= @save_optimizer_use_condition_selectivity; > +set @@histogram_size= @save_histogram_size; > +drop table t0,t1,t2; > diff --git a/sql/opt_range.cc b/sql/opt_range.cc > index 3bcaa72e32f..59aa2f3f280 100644 > --- a/sql/opt_range.cc > +++ b/sql/opt_range.cc > @@ -3322,6 +3322,10 @@ bool create_key_parts_for_pseudo_indexes(RANGE_OPT_PARAM *param, > { > Field **field_ptr; > TABLE *table= param->table; > + partition_info *part_info= NULL; > + #ifdef WITH_PARTITION_STORAGE_ENGINE > + part_info= table->part_info; > + #endif > uint parts= 0; > > for (field_ptr= table->field; *field_ptr; field_ptr++) > @@ -3329,7 +3333,9 @@ bool create_key_parts_for_pseudo_indexes(RANGE_OPT_PARAM *param, > Column_statistics* col_stats= (*field_ptr)->read_stats; > if (bitmap_is_set(used_fields, (*field_ptr)->field_index) > && col_stats && !col_stats->no_stat_values_provided() > - && !((*field_ptr)->type() == MYSQL_TYPE_GEOMETRY)) > + && !((*field_ptr)->type() == MYSQL_TYPE_GEOMETRY) > + && (!part_info || > + part_info->field_in_partition_expr(*field_ptr))) > parts++; > } > > @@ -3350,7 +3356,9 @@ bool create_key_parts_for_pseudo_indexes(RANGE_OPT_PARAM *param, > if (bitmap_is_set(used_fields, (*field_ptr)->field_index)) > { > Field *field= *field_ptr; > - if (field->type() == MYSQL_TYPE_GEOMETRY) > + if (field->type() == MYSQL_TYPE_GEOMETRY > + || (!part_info || > + part_info->field_in_partition_expr(field))) > continue; This condition is wrong. (field->type() == MYSQL_TYPE_GEOMETRY || (!part_info || part_info->field_in_partition_expr(field))) is the same as (field->type() == MYSQL_TYPE_GEOMETRY || !part_info || ... which means if there is no partitioning, we won't count the selectivity. Indeed, if I apply this patch, all EITS tests start to fail, selectivity is no longer counted: --- /home/psergey/dev-git/10.1/mysql-test/r/selectivity_innodb.result 2018-11-16 14:06:27.231202907 +0300 +++ /home/psergey/dev-git/10.1/mysql-test/r/selectivity_innodb.reject 2018-11-25 01:03:14.661647893 +0300 @@ -28,13 +28,13 @@ explain extended select * from t1 where a is null; id select_type table type possible_keys key key_len ref rows filtered Extra -1 SIMPLE t1 ALL NULL NULL NULL NULL 10 40.00 Using where +1 SIMPLE t1 ALL NULL NULL NULL NULL 10 100.00 Using where Warnings: Note 1003 select `test`.`t1`.`a` AS `a` from `test`.`t1` where isnull(`test`.`t1`.`a`) explain extended select * from t1 where a is not null; id select_type table type possible_keys key key_len ref rows filtered Extra -1 SIMPLE t1 ALL NULL NULL NULL NULL 10 60.00 Using where +1 SIMPLE t1 ALL NULL NULL NULL NULL 10 100.00 Using where Warnings: See also the comment below about the return value of field_in_partition_expr. > > uint16 store_length; > diff --git a/sql/partition_info.cc b/sql/partition_info.cc > index 52bda560c1c..0111fc1451d 100644 > --- a/sql/partition_info.cc > +++ b/sql/partition_info.cc > @@ -3164,6 +3164,32 @@ void partition_info::print_debug(const char *str, uint *value) > DBUG_PRINT("info", ("parser: %s", str)); > DBUG_VOID_RETURN; > } > + > +/* > + > + Disabling reading EITS statistics for columns involved in the > + partition list of a table. > + We assume the selecticivity for such columns would be handled > + during partition pruning. > + > +*/ > + This comment here doesn't make any sense anymore. Please move it to create_key_parts_for_pseudo_indexes. Also, the name and return value of the function is confusing now. If one calls the function for the partitioning column: field_in_partition_expr(partition_column) == false. Please change the return value to be true in this case, and false otherwise. > +bool partition_info::field_in_partition_expr(Field *field) const > +{ > + uint i; > + for (i= 0; i < num_part_fields; i++) > + { > + if (field->eq(part_field_array[i])) > + return FALSE; > + } > + for (i= 0; i < num_subpart_fields; i++) > + { > + if (field->eq(subpart_field_array[i])) > + return FALSE; > + } > + return TRUE; > +} > + > #else /* WITH_PARTITION_STORAGE_ENGINE */ > /* > For builds without partitioning we need to define these functions > diff --git a/sql/partition_info.h b/sql/partition_info.h > index f250c5496bf..10b8954ace7 100644 > --- a/sql/partition_info.h > +++ b/sql/partition_info.h > @@ -384,6 +384,7 @@ class partition_info : public Sql_alloc > bool is_full_part_expr_in_fields(List<Item> &fields); > public: > bool has_unique_name(partition_element *element); > + bool field_in_partition_expr(Field *field) const; > }; > > uint32 get_next_partition_id_range(struct st_partition_iter* part_iter); > diff --git a/sql/sql_statistics.cc b/sql/sql_statistics.cc > index cb75a5c2176..02dd4970c99 100644 > --- a/sql/sql_statistics.cc > +++ b/sql/sql_statistics.cc > @@ -3589,6 +3589,22 @@ void set_statistics_for_table(THD *thd, TABLE *table) > (use_stat_table_mode <= COMPLEMENTARY || > !table->stats_is_read || read_stats->cardinality_is_null) ? > table->file->stats.records : read_stats->cardinality; > + > + /* > + For partitioned table, EITS statistics is based on data from all partitions. > + > + On the other hand, Partition Pruning figures which partitions will be > + accessed and then computes the estimate of rows in used_partitions. > + > + Use the estimate from Partition Pruning as it is typically more precise. > + Ideally, EITS should provide per-partition statistics but this is not > + implemented currently. > + */ Good. > + #ifdef WITH_PARTITION_STORAGE_ENGINE > + if (table->part_info) > + table->used_stat_records= table->file->stats.records; > + #endif > + > KEY *key_info, *key_info_end; > for (key_info= table->key_info, key_info_end= key_info+table->s->keys; > key_info < key_info_end; key_info++) -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] MDEV-16991 review.
by Alexander Barkov 25 Nov '18

25 Nov '18

Hi Alexey, On 11/25/2018 03:00 AM, Alexey Botchkov wrote: > Hi, Alexander. > > First question about the sql/sql_basic_types.h. > To me it looks overcomplicated. Do we really need classes over > the simple enums? > I attached my version of this file here. In my opinion it's shorter itself, > and makes other code look nicer. This is intentional, for strict data type control. Simple enums have some problems. For example, you can pass enums to a function that expects an integer data type. I want any mistakes like this to cause a compilation error. I would avoid using the traditional enums. Can you please leave it as is? I just found that C++11 introduced enum classes. They don't convert to int implicitly, and don't export their values to the public name space. https://www.codeguru.com/cpp/cpp/article.php/c19083/C-2011-Stronglytyped-En… We have enabled C++11 in 10.4 So now this could be changed to: class enum date_conv_mode_t { CONV_NONE= 0U, FUZZY_DATES= 1U, TIME_ONLY= 4U, INTERVAL_hhmmssff= 8U, INTERVAL_DAY= 16U, RANGE0_LAST= INTERVAL_DAY, NO_ZERO_IN_DATE= (1UL << 23), // MODE_NO_ZERO_IN_DATE NO_ZERO_DATE= (1UL << 24), // MODE_NO_ZERO_DATE INVALID_DATES= (1UL << 25) // MODE_INVALID_DATES }; We could try this. > > Other than that > > lines like this can be shortened > -TIME_FUZZY_DATES (date_mode_t::value_t::FUZZY_DATES), > +TIME_FUZZY_DATES (date_mode_t::FUZZY_DATES), Thanks. Fixed. > > Three variables all defined differently. Looks funny :) > const date_conv_mode_t > ... > static const date_conv_mode_t > ... > static time_round_mode_t > Why is that? Thanks for noticing this. I've changed them all to "static const". > > > What do you think if we do this: > - add THD* memver to the MYSQL_TIME_STATUS structure, > so we can get rid of the THD * parameter to many functions? > - send one MYSQL_TIME_STATUS* to the add_nanoseconds() > instead of three {thd, &status->warnings, and status->nanoseconds} ? Changing all functions that accept both &warn and nsec to accept a pointer to MYSQL_TIME_STATUS instead, like this: < xxxx(thd, &status->warnings, and status->nanoseconds); > xxxx(thd, &status); is a very good idea. I'd avoid mix "thd" to MYSQL_TIME_STATUS though. It's a pure C structure. Does it sound OK? > > You do like this: > Datetime::Options opt(TIME_CONV_NONE, thd); > Datetime dt(thd, item->arguments()[0], opt); > > Why not just > Datetime dt(thd, item->arguments()[0], Datetime::Options > opt(TIME_CONV_NONE, thd))? > I'm not sure about this very case, but ofter code like yours translated > with extra > constructor call. When the line fits the entire call, I do like this: a. Datetime dt(thd, item, Datetime::Options opt(mode, thd)); When the line does not fit, I tried both ways: b. Datetime dt(thd, item->arguments()[0], Datetime::Options opt(TIME_CONV_NONE, thd)); and c. Datetime::Options opt(TIME_CONV_NONE, thd); Datetime dt(thd, item->arguments()[0], opt); and "c" looked easier to read for me than "b". But this causes a constructor call indeed. Although, in case of date_mode_t this is very cheap (as cheap as copying of a 4 byte m_mode value), it's still a good idea to avoid this. I'll change like you suggest. Moreover, it's very likely that date_mode_t can change to something mode complex in the future. So an extra constructor call can get more expensive. > > Best regards. > HF >

1 1

Re: [Maria-developers] [Commits] 68b7e231cb3: MDEV-17255: New optimizer defaults and ANALYZE TABLE
by Sergey Petrunia 24 Nov '18

24 Nov '18

Hi Varun, On Fri, Nov 16, 2018 at 11:48:59PM +0530, Varun wrote: > revision-id: 68b7e231cb3a1cdfbd968c2cbf72687b29b6d2da (mariadb-10.3.6-209-g68b7e231cb3) > parent(s): b9a9055793ab8b9a50200e2f55602463627dedd3 Does the above say the patch is going into 10.3? The MDEV has fixVersion=10.4 and the parent revision is also in 10.4. So I assume the target version is 10.4? > author: Varun Gupta > committer: Varun Gupta > timestamp: 2018-11-16 23:48:11 +0530 > message: > > MDEV-17255: New optimizer defaults and ANALYZE TABLE > > Added another value for the use_stat_tables system variable Please mention in the commit comment: - the name of the new value - the fact that the new value is now the default. > --- > sql/opt_range.cc | 2 +- > sql/sql_admin.cc | 3 ++- > sql/sql_statistics.h | 1 + > sql/sys_vars.cc | 4 ++-- > 4 files changed, 6 insertions(+), 4 deletions(-) Please run the testsuite. I did the 'main' testsuite and: - main.mysqld--help, main.stat_tables_disabled - these need updates - a few other tests need to be updated as now opening any table will now cause the server to attept to read EITS stats for it. Please also add test coverage for this particular MDEV. This should be a testcase that shows that - with PREFERABLY, ANALYZE will collect EITS stats, SELECT will use them. - with PREFERABLY_FOR_READS : -- ANALYZE will not collect EITS stats -- ANALYZE .. PERSISTENT FOR ... will update the EITS stats. -- SELECT will use them. . > > diff --git a/sql/opt_range.cc b/sql/opt_range.cc > index d6db365a8a2..f501a5ae085 100644 > --- a/sql/opt_range.cc > +++ b/sql/opt_range.cc > @@ -3158,7 +3158,7 @@ bool calculate_cond_selectivity_for_table(THD *thd, TABLE *table, Item **cond) > > if (thd->variables.optimizer_use_condition_selectivity > 2 && > !bitmap_is_clear_all(used_fields) && > - thd->variables.use_stat_tables > 0) > + get_use_stat_tables_mode(thd) > NEVER) > { > PARAM param; > MEM_ROOT alloc; > diff --git a/sql/sql_admin.cc b/sql/sql_admin.cc > index d0d959de8f9..b5c732737b7 100644 > --- a/sql/sql_admin.cc > +++ b/sql/sql_admin.cc > @@ -767,7 +767,8 @@ static bool mysql_admin_table(THD* thd, TABLE_LIST* tables, > } > collect_eis= > (table->table->s->table_category == TABLE_CATEGORY_USER && > - (get_use_stat_tables_mode(thd) > NEVER || > + ((get_use_stat_tables_mode(thd) > NEVER && > + get_use_stat_tables_mode(thd) < PREFERABLY_FOR_READS) || > lex->with_persistent_for_clause)); > > > diff --git a/sql/sql_statistics.h b/sql/sql_statistics.h > index 39cddf95188..a942c05be09 100644 > --- a/sql/sql_statistics.h > +++ b/sql/sql_statistics.h > @@ -22,6 +22,7 @@ enum enum_use_stat_tables_mode > NEVER, > COMPLEMENTARY, > PREFERABLY, Please add a comment describing the new value. > + PREFERABLY_FOR_READS, > } Use_stat_tables_mode; > > typedef > diff --git a/sql/sys_vars.cc b/sql/sys_vars.cc > index 92c7d329bb9..3652d728fc1 100644 > --- a/sql/sys_vars.cc > +++ b/sql/sys_vars.cc > @@ -5841,12 +5841,12 @@ static Sys_var_ulong Sys_progress_report_time( > VALID_RANGE(0, UINT_MAX), DEFAULT(5), BLOCK_SIZE(1)); > > const char *use_stat_tables_modes[] = > - {"NEVER", "COMPLEMENTARY", "PREFERABLY", 0}; > + {"NEVER", "COMPLEMENTARY", "PREFERABLY", "PREFERABLY_FOR_READS", 0}; > static Sys_var_enum Sys_optimizer_use_stat_tables( > "use_stat_tables", > "Specifies how to use system statistics tables", > SESSION_VAR(use_stat_tables), CMD_LINE(REQUIRED_ARG), > - use_stat_tables_modes, DEFAULT(0)); > + use_stat_tables_modes, DEFAULT(3)); > > static Sys_var_ulong Sys_histogram_size( > "histogram_size", BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] afedc29f773: MDEV-14024 PCRE2.
by Sergei Golubchik 23 Nov '18

23 Nov '18

Hi, Alexey! Thanks! Few minor comments, see below On Nov 23, Alexey Botchkov wrote: > revision-id: afedc29f773362f342756e9cdd9bbd9ed0f1abe2 (versioning-1.0.3-68-gafedc29f773) > parent(s): 4ec8598c1dcb63499bce998142b8e5b8b09b2d30 > author: Alexey Botchkov <holyfoot(a)askmonty.org> > committer: Alexey Botchkov <holyfoot(a)askmonty.org> > timestamp: 2018-06-26 14:10:46 +0400 > message: > > MDEV-14024 PCRE2. > > Related changes in the server code. > > diff --git a/cmake/pcre.cmake b/cmake/pcre.cmake > index 4c113929866..264f93a60a2 100644 > --- a/cmake/pcre.cmake > +++ b/cmake/pcre.cmake > @@ -5,24 +5,17 @@ SET(WITH_PCRE "auto" CACHE STRING > > MACRO (CHECK_PCRE) > IF(WITH_PCRE STREQUAL "system" OR WITH_PCRE STREQUAL "auto") > - CHECK_LIBRARY_EXISTS(pcre pcre_stack_guard "" HAVE_PCRE_STACK_GUARD) > + CHECK_LIBRARY_EXISTS(pcre2-8 pcre2_match "" HAVE_PCRE2) > - IF(NOT CMAKE_CROSSCOMPILING) > - SET(CMAKE_REQUIRED_LIBRARIES "pcre") > - CHECK_C_SOURCE_RUNS(" > - #include <pcre.h> > - int main() { > - return -pcre_exec(NULL, NULL, NULL, -999, -999, 0, NULL, 0) < 256; > - }" PCRE_STACK_SIZE_OK) > - SET(CMAKE_REQUIRED_LIBRARIES) > - ENDIF() > ENDIF() > - IF(NOT HAVE_PCRE_STACK_GUARD OR NOT PCRE_STACK_SIZE_OK OR > - WITH_PCRE STREQUAL "bundled") > + IF(NOT HAVE_PCRE2 OR WITH_PCRE STREQUAL "bundled") > IF (WITH_PCRE STREQUAL "system") > - MESSAGE(FATAL_ERROR "system pcre is not found or unusable") > + MESSAGE(FATAL_ERROR "system pcre2-8 library is not found or unusable") > ENDIF() > - SET(PCRE_INCLUDES ${CMAKE_BINARY_DIR}/pcre ${CMAKE_SOURCE_DIR}/pcre) > - ADD_SUBDIRECTORY(pcre) > + SET(PCRE_INCLUDES ${CMAKE_BINARY_DIR}/pcre2 ${CMAKE_SOURCE_DIR}/pcre2 > + ${CMAKE_BINARY_DIR}/pcre2/src ${CMAKE_SOURCE_DIR}/pcre2/src) That's a bit strange. I'd expect only .../pcre2/src paths, not .../pcre2. you need pcre2.h and pcre2posix.h, they're both in src/ right? > + SET(PCRE_BUILD_TESTS OFF CACHE BOOL "Disable tests.") > + SET(PCRE2_BUILD_PCRE2GREP OFF CACHE BOOL "Disable pcre2grep") > + ADD_SUBDIRECTORY(pcre2) > ENDIF() > ENDMACRO() > > diff --git a/extra/mariabackup/CMakeLists.txt b/extra/mariabackup/CMakeLists.txt > index 7df5fa17903..a9d7153b893 100644 > --- a/extra/mariabackup/CMakeLists.txt > +++ b/extra/mariabackup/CMakeLists.txt > @@ -37,7 +37,7 @@ INCLUDE_DIRECTORIES( > ) > > IF(NOT HAVE_SYSTEM_REGEX) > - INCLUDE_DIRECTORIES(${CMAKE_SOURCE_DIR}/pcre) > + INCLUDE_DIRECTORIES(${CMAKE_SOURCE_DIR}/pcre2 ${CMAKE_SOURCE_DIR}/pcre2/src) Hmm, not INCLUDE_DIRECTORIES(${PCRE_INCLUDES}) ? pcre2.h is in the ${CMAKE_BINARY_DIR}, it's generated. > ENDIF() > > IF(WITH_WSREP) > diff --git a/mysql-test/main/func_regexp_pcre.test b/mysql-test/main/func_regexp_pcre.test > index 21600390bb2..30969b3e9ae 100644 > --- a/mysql-test/main/func_regexp_pcre.test > +++ b/mysql-test/main/func_regexp_pcre.test > @@ -382,6 +382,7 @@ SELECT 'AB' RLIKE 'A B'; > SELECT 'AB' RLIKE 'A# this is a comment\nB'; > SET default_regex_flags=DEFAULT; > > +--error ER_REGEXP_ERROR > SELECT 'Aq' RLIKE 'A\\q'; you forgot to update func_regexp_pcre.result > SET default_regex_flags='EXTRA'; > --error ER_REGEXP_ERROR > diff --git a/sql/item_cmpfunc.cc b/sql/item_cmpfunc.cc > index 76f4788c1cf..450a9c54176 100644 > --- a/sql/item_cmpfunc.cc > +++ b/sql/item_cmpfunc.cc > @@ -5485,124 +5486,44 @@ bool Regexp_processor_pcre::compile(Item *item, bool send_error) > */ > void Regexp_processor_pcre::pcre_exec_warn(int rc) const > { > - char buf[64]; > - const char *errmsg= NULL; > + PCRE2_UCHAR8 buf[128]; > THD *thd= current_thd; > > - /* > - Make a descriptive message only for those pcre_exec() error codes > - that can actually happen in MariaDB. > - */ > - switch (rc) > + int errlen= pcre2_get_error_message(rc, buf, sizeof(buf)); > + if (errlen <= 0) > { > - case PCRE_ERROR_NULL: > - errmsg= "pcre_exec: null argument passed"; > - break; > - case PCRE_ERROR_BADOPTION: > - errmsg= "pcre_exec: bad option"; > - break; > - case PCRE_ERROR_BADMAGIC: > - errmsg= "pcre_exec: bad magic - not a compiled regex"; > - break; > - case PCRE_ERROR_UNKNOWN_OPCODE: > - errmsg= "pcre_exec: error in compiled regex"; > - break; > - case PCRE_ERROR_NOMEMORY: > - errmsg= "pcre_exec: Out of memory"; > - break; > - case PCRE_ERROR_NOSUBSTRING: > - errmsg= "pcre_exec: no substring"; > - break; > - case PCRE_ERROR_MATCHLIMIT: > - errmsg= "pcre_exec: match limit exceeded"; > - break; > - case PCRE_ERROR_CALLOUT: > - errmsg= "pcre_exec: callout error"; > - break; > - case PCRE_ERROR_BADUTF8: > - errmsg= "pcre_exec: Invalid utf8 byte sequence in the subject string"; > - break; > - case PCRE_ERROR_BADUTF8_OFFSET: > - errmsg= "pcre_exec: Started at invalid location within utf8 byte sequence"; > - break; > - case PCRE_ERROR_PARTIAL: > - errmsg= "pcre_exec: partial match"; > - break; > - case PCRE_ERROR_INTERNAL: > - errmsg= "pcre_exec: internal error"; > - break; > - case PCRE_ERROR_BADCOUNT: > - errmsg= "pcre_exec: ovesize is negative"; > - break; > - case PCRE_ERROR_RECURSIONLIMIT: > - my_snprintf(buf, sizeof(buf), "pcre_exec: recursion limit of %ld exceeded", > - m_pcre_extra.match_limit_recursion); > - errmsg= buf; > - break; > - case PCRE_ERROR_BADNEWLINE: > - errmsg= "pcre_exec: bad newline options"; > - break; > - case PCRE_ERROR_BADOFFSET: > - errmsg= "pcre_exec: start offset negative or greater than string length"; > - break; > - case PCRE_ERROR_SHORTUTF8: > - errmsg= "pcre_exec: ended in middle of utf8 sequence"; > - break; > - case PCRE_ERROR_JIT_STACKLIMIT: > - errmsg= "pcre_exec: insufficient stack memory for JIT compile"; > - break; > - case PCRE_ERROR_RECURSELOOP: > - errmsg= "pcre_exec: Recursion loop detected"; > - break; > - case PCRE_ERROR_BADMODE: > - errmsg= "pcre_exec: compiled pattern passed to wrong bit library function"; > - break; > - case PCRE_ERROR_BADENDIANNESS: > - errmsg= "pcre_exec: compiled pattern passed to wrong endianness processor"; > - break; > - case PCRE_ERROR_JIT_BADOPTION: > - errmsg= "pcre_exec: bad jit option"; > - break; > - case PCRE_ERROR_BADLENGTH: > - errmsg= "pcre_exec: negative length"; > - break; > - default: > - /* > - As other error codes should normally not happen, > - we just report the error code without textual description > - of the code. > - */ > - my_snprintf(buf, sizeof(buf), "pcre_exec: Internal error (%d)", rc); > - errmsg= buf; > + my_snprintf((char *)buf, sizeof(buf), "pcre_exec: Internal error (%d)", rc); > } > push_warning_printf(thd, Sql_condition::WARN_LEVEL_WARN, > - ER_REGEXP_ERROR, ER_THD(thd, ER_REGEXP_ERROR), errmsg); > + ER_REGEXP_ERROR, ER_THD(thd, ER_REGEXP_ERROR), buf); > } > > > /** > Call pcre_exec() and send a warning if pcre_exec() returned with an error. > */ > -int Regexp_processor_pcre::pcre_exec_with_warn(const pcre *code, > - const pcre_extra *extra, > +int Regexp_processor_pcre::pcre_exec_with_warn(const pcre2_code *code, > + pcre2_match_data *data, > const char *subject, > int length, int startoffset, > - int options, int *ovector, > - int ovecsize) > + uint options) > { > - int rc= pcre_exec(code, extra, subject, length, > - startoffset, options, ovector, ovecsize); > + int rc= pcre2_match(code, (PCRE2_SPTR8) subject, (PCRE2_SIZE) length, > + (PCRE2_SIZE) startoffset, options, data, NULL); > DBUG_EXECUTE_IF("pcre_exec_error_123", rc= -123;); > - if (unlikely(rc < PCRE_ERROR_NOMATCH)) > - pcre_exec_warn(rc); > + if (unlikely(rc < PCRE2_ERROR_NOMATCH)) > + m_SubStrVec= NULL; > + else > + m_SubStrVec= pcre2_get_ovector_pointer(data); > + The function name is still pcre_exec_with_warn, the comment still says "Call pcre_exec() and send a warning". But neither seems to be happening here. where do you do warnings now? > return rc; > } > > > bool Regexp_processor_pcre::exec(const char *str, size_t length, size_t offset) > { > - m_pcre_exec_rc= pcre_exec_with_warn(m_pcre, &m_pcre_extra, str, (int)length, (int)offset, 0, > - m_SubStrVec, array_elements(m_SubStrVec)); > + m_pcre_exec_rc= pcre_exec_with_warn(m_pcre, m_pcre_match_data, > + str, (int)length, (int)offset, 0); > return false; > } > > diff --git a/sql/sys_vars.cc b/sql/sys_vars.cc > index ccb7d3b29ed..e8abae3487d 100644 > --- a/sql/sys_vars.cc > +++ b/sql/sys_vars.cc > @@ -5692,19 +5692,21 @@ static const char *default_regex_flags_names[]= > "DOTALL", // (?s) . matches anything including NL > "DUPNAMES", // (?J) Allow duplicate names for subpatterns > "EXTENDED", // (?x) Ignore white space and # comments > - "EXTRA", // (?X) extra features (e.g. error on unknown escape character) > + "EXTENDED_MORE",//(?xx) Ignore white space and # comments inside cheracter perhaps, update all flag descriptions from man pcre2syntax? Like "EXTENDED", // (?x) extended: ignore white space except in classes "EXTENDED_MORE",//(?xx) as (?x) but also ignore space and tab in classes also, I don't quite like EXTENDED_MORE name. But I don't have a good suggestion. May be just re-purpose EXTENDED to do (?xx) without introducing a new flag? > + "EXTRA", // means nothing since PCRE2 may be a warning when it's used? > "MULTILINE", // (?m) ^ and $ match newlines within data > "UNGREEDY", // (?U) Invert greediness of quantifiers > 0 > }; > static const int default_regex_flags_to_pcre[]= > { > - PCRE_DOTALL, > - PCRE_DUPNAMES, > - PCRE_EXTENDED, > - PCRE_EXTRA, > - PCRE_MULTILINE, > - PCRE_UNGREEDY, > + PCRE2_DOTALL, > + PCRE2_DUPNAMES, > + PCRE2_EXTENDED, > + PCRE2_EXTENDED_MORE, > + 0, /* EXTRA flag not available since PCRE2 */ > + PCRE2_MULTILINE, > + PCRE2_UNGREEDY, > 0 > }; > int default_regex_flags_pcre(const THD *thd) > diff --git a/storage/tokudb/PerconaFT/cmake_modules/TokuFeatureDetection.cmake b/storage/tokudb/PerconaFT/cmake_modules/TokuFeatureDetection.cmake > index 2f04a33558a..51257febc0d 100644 > --- a/storage/tokudb/PerconaFT/cmake_modules/TokuFeatureDetection.cmake > +++ b/storage/tokudb/PerconaFT/cmake_modules/TokuFeatureDetection.cmake > @@ -1,6 +1,8 @@ > ## feature detection > find_package(Threads) > -find_package(ZLIB REQUIRED) > +IF(WITH_EXTERNAL_ZLIB) > + find_package(ZLIB REQUIRED) > +ENDIF() push it in separate commit, please. > > option(USE_VALGRIND "Build to run safely under valgrind (often slower)." ON) > if(USE_VALGRIND) > Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] 49b63dcc048: MDEV-15073: Generic UDAF parser code in server for windows functions
by Sergei Golubchik 23 Nov '18

23 Nov '18

Hi, Oleksandr! On Nov 21, Oleksandr Byelkin wrote: > revision-id: 49b63dcc0482bcb0fc7f642b0c19c8e24740ab27 (mariadb-10.4.0-20-g49b63dcc048) > parent(s): b5ac863f1494920b5e7035c9dfa0ebfdaa50a15d > author: Oleksandr Byelkin <sanja(a)mariadb.com> > committer: Oleksandr Byelkin <sanja(a)mariadb.com> > timestamp: 2018-11-20 10:58:34 +0100 > message: > > MDEV-15073: Generic UDAF parser code in server for windows functions > > Added support for usual agreggate UDF (UDAF) > Added remove() call support for more efficient window function processing > Added example of aggregate UDF with efficient windows function support Looks good, see a couple of minor comments below. I wouldn't extend UDF API, really. It's ancient and it should be removed and replaced with a new plugin type, not extended. But if you still want to do it make sure it's documented https://mariadb.com/kb/en/user-defined-functions/ > diff --git a/mysql-test/main/udf.test b/mysql-test/main/udf.test > index c3a25c6bcce..e72f8b219af 100644 > --- a/mysql-test/main/udf.test > +++ b/mysql-test/main/udf.test > @@ -528,3 +528,62 @@ DROP FUNCTION METAPHON; > #INSERT INTO t1 (a) VALUES ('Hello'); > #SELECT * FROM t1; > DROP TABLE t1; > + > +--echo > +--echo MDEV-15073: Generic UDAF parser code in server for windows functions > +--echo use --echo # so that your echo commands would stand out in the result file. currently they look like an output from some query and it's confusing. The test itself is good. > diff --git a/sql/item_sum.cc b/sql/item_sum.cc > index 0e52b2988a3..d19645b5020 100644 > --- a/sql/item_sum.cc > +++ b/sql/item_sum.cc > @@ -3235,6 +3235,25 @@ bool Item_udf_sum::add() > DBUG_RETURN(0); > } > > + > +bool Item_udf_sum::supports_removal() > +{ > + DBUG_ENTER("Item_udf_sum::supports_remove"); > + DBUG_PRINT("info", ("support: %d", udf.supports_removal())); > + DBUG_RETURN(udf.supports_removal()); > +} > + > + > +void Item_udf_sum::remove() > +{ > + my_bool tmp_null_value; > + DBUG_ENTER("Item_udf_sum::remove"); > + udf.remove(&tmp_null_value); > + null_value= tmp_null_value; > + DBUG_VOID_RETURN; > +} > + > + > void Item_udf_sum::cleanup() > { > /* > diff --git a/sql/item_sum.h b/sql/item_sum.h > index 1a21c257221..663dea1ed7b 100644 > --- a/sql/item_sum.h > +++ b/sql/item_sum.h > @@ -574,7 +574,7 @@ class Item_sum :public Item_func_or_sum > virtual bool add()= 0; > virtual bool setup(THD *thd) { return false; } > > - virtual bool supports_removal() const { return false; } > + virtual bool supports_removal() { return false; } why? > virtual void remove() { DBUG_ASSERT(0); } Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 2

[Maria-developers] MariaDB Developers Unconference
by Ian Gilfillan 19 Nov '18

19 Nov '18

The MariaDB Developer Unconference is taking place in New York 23-24 Feb, just before OpenWorks: https://mariadb.org/2019-developers-unconference-new-york/ Please sign up on the Meetup page: https://www.meetup.com/MariaDB-Developers/events/256385430/ - security is very tight at the building, so everyone needs to be signed up in advance to be given access. There's a cap on the attendance, so don't delay too long. If you want to suggest a session, the spreadsheet is open for proposals.

1 0

[Maria-developers] 7c088b624f4: MDEV-14500 filesort to support engines with slow rnd_pos
by Sergei Golubchik 19 Nov '18

19 Nov '18

revision-id: 7c088b624f4f50e1e9f11891c371eba912228a2d (mariadb-10.4.0-14-g7c088b624f4) parent(s): da8a589a982c45ed161b9a25d61282a6bc1fa037 author: Sergei Golubchik <serg(a)mariadb.org> committer: Sergei Golubchik <serg(a)mariadb.org> timestamp: 2018-11-17 14:13:37 +0100 message: MDEV-14500 filesort to support engines with slow rnd_pos If the engine wants to avoid rnd_pos() - force a temporary table before a filesort. But don't do it if addon_fields are used. --- mysql-test/suite/archive/rnd_pos.result | 56 +++++++++++++++++++++++++++++++++ mysql-test/suite/archive/rnd_pos.test | 27 ++++++++++++++++ sql/filesort.cc | 55 ++++++++++++++++++-------------- sql/filesort.h | 3 ++ sql/handler.h | 7 +++++ sql/sql_select.cc | 15 +++++++++ storage/archive/ha_archive.h | 2 +- storage/csv/ha_tina.h | 2 +- 8 files changed, 142 insertions(+), 25 deletions(-) diff --git a/mysql-test/suite/archive/rnd_pos.result b/mysql-test/suite/archive/rnd_pos.result new file mode 100644 index 00000000000..b6b6748d53f --- /dev/null +++ b/mysql-test/suite/archive/rnd_pos.result @@ -0,0 +1,56 @@ +create table t1(c1 int not null, c2 double not null, c3 char(255) not null) engine=archive; +insert t1 select seq, seq+0.7, concat('row with c1 = ', seq) from seq_1_to_10; +explain partitions select c1,c3 from t1 order by c2; +id select_type table partitions type possible_keys key key_len ref rows Extra +1 SIMPLE t1 NULL ALL NULL NULL NULL NULL 10 Using filesort +set max_length_for_sort_data = 4; +explain partitions select c1,c3 from t1 order by c2; +id select_type table partitions type possible_keys key key_len ref rows Extra +1 SIMPLE t1 NULL ALL NULL NULL NULL NULL 10 Using temporary; Using filesort +flush status; +select c1,c3 from t1 order by c2; +c1 c3 +1 row with c1 = 1 +2 row with c1 = 2 +3 row with c1 = 3 +4 row with c1 = 4 +5 row with c1 = 5 +6 row with c1 = 6 +7 row with c1 = 7 +8 row with c1 = 8 +9 row with c1 = 9 +10 row with c1 = 10 +set max_length_for_sort_data = default; +show status where variable_name like '%tmp%' and value != 0; +Variable_name Value +Created_tmp_tables 1 +Handler_tmp_write 10 +Rows_tmp_read 20 +alter table t1 partition by hash (c1) partitions 3; +explain partitions select c1,c3 from t1 order by c2; +id select_type table partitions type possible_keys key key_len ref rows Extra +1 SIMPLE t1 p0,p1,p2 ALL NULL NULL NULL NULL 10 Using filesort +set max_length_for_sort_data = 4; +explain partitions select c1,c3 from t1 order by c2; +id select_type table partitions type possible_keys key key_len ref rows Extra +1 SIMPLE t1 p0,p1,p2 ALL NULL NULL NULL NULL 10 Using temporary; Using filesort +flush status; +select c1,c3 from t1 order by c2; +c1 c3 +1 row with c1 = 1 +2 row with c1 = 2 +3 row with c1 = 3 +4 row with c1 = 4 +5 row with c1 = 5 +6 row with c1 = 6 +7 row with c1 = 7 +8 row with c1 = 8 +9 row with c1 = 9 +10 row with c1 = 10 +set max_length_for_sort_data = default; +show status where variable_name like '%tmp%' and value != 0; +Variable_name Value +Created_tmp_tables 1 +Handler_tmp_write 10 +Rows_tmp_read 20 +drop table t1; diff --git a/mysql-test/suite/archive/rnd_pos.test b/mysql-test/suite/archive/rnd_pos.test new file mode 100644 index 00000000000..7a1d78ea003 --- /dev/null +++ b/mysql-test/suite/archive/rnd_pos.test @@ -0,0 +1,27 @@ +# +# MDEV-14500 Support engines without rnd_pos +# +source include/have_archive.inc; +source include/have_sequence.inc; +source include/have_partition.inc; + +create table t1(c1 int not null, c2 double not null, c3 char(255) not null) engine=archive; +insert t1 select seq, seq+0.7, concat('row with c1 = ', seq) from seq_1_to_10; +explain partitions select c1,c3 from t1 order by c2; +set max_length_for_sort_data = 4; +explain partitions select c1,c3 from t1 order by c2; +flush status; +select c1,c3 from t1 order by c2; +set max_length_for_sort_data = default; +show status where variable_name like '%tmp%' and value != 0; + +alter table t1 partition by hash (c1) partitions 3; +explain partitions select c1,c3 from t1 order by c2; +set max_length_for_sort_data = 4; +explain partitions select c1,c3 from t1 order by c2; +flush status; +select c1,c3 from t1 order by c2; +set max_length_for_sort_data = default; +show status where variable_name like '%tmp%' and value != 0; + +drop table t1; diff --git a/sql/filesort.cc b/sql/filesort.cc index e682f3389da..cbe79967647 100644 --- a/sql/filesort.cc +++ b/sql/filesort.cc @@ -1957,6 +1957,30 @@ sortlength(THD *thd, SORT_FIELD *sortorder, uint s_length, return length; } +bool filesort_use_addons(TABLE *table, uint sortlength, + uint *length, uint *fields, uint *null_fields) +{ + Field **pfield, *field; + *length= *fields= *null_fields= 0; + + for (pfield= table->field; (field= *pfield) ; pfield++) + { + if (!bitmap_is_set(table->read_set, field->field_index)) + continue; + if (field->flags & BLOB_FLAG) + return false; + (*length)+= field->max_packed_col_length(field->pack_length()); + if (field->maybe_null()) + (*null_fields)++; + (*fields)++; + } + if (!*fields) + return false; + (*length)+= (*null_fields+7)/8; + + return *length + sortlength < + table->in_use->variables.max_length_for_sort_data; +} /** Get descriptors of fields appended to sorted fields and @@ -1991,11 +2015,8 @@ get_addon_fields(TABLE *table, uint sortlength, LEX_STRING *addon_buf) Field **pfield; Field *field; SORT_ADDON_FIELD *addonf; - uint length= 0; - uint fields= 0; - uint null_fields= 0; + uint length, fields, null_fields; MY_BITMAP *read_set= table->read_set; - ulong max_sort_len= table->in_use->variables.max_length_for_sort_data; DBUG_ENTER("get_addon_fields"); /* @@ -2011,26 +2032,14 @@ get_addon_fields(TABLE *table, uint sortlength, LEX_STRING *addon_buf) addon_buf->str= 0; addon_buf->length= 0; - for (pfield= table->field; (field= *pfield) ; pfield++) - { - if (!bitmap_is_set(read_set, field->field_index)) - continue; - if (field->flags & BLOB_FLAG) - DBUG_RETURN(0); - length+= field->max_packed_col_length(field->pack_length()); - if (field->maybe_null()) - null_fields++; - fields++; - } - if (!fields) - DBUG_RETURN(0); - length+= (null_fields+7)/8; + // see remove_const() for HA_SLOW_RND_POS explanation + if (table->file->ha_table_flags() & HA_SLOW_RND_POS) + sortlength= 0; - if (length+sortlength > max_sort_len || - !my_multi_malloc(MYF(MY_WME | MY_THREAD_SPECIFIC), - &addonf, sizeof(SORT_ADDON_FIELD) * (fields+1), - &addon_buf->str, length, - NullS)) + if (!filesort_use_addons(table, sortlength, &length, &fields, &null_fields) || + !my_multi_malloc(MYF(MY_WME | MY_THREAD_SPECIFIC), &addonf, + sizeof(SORT_ADDON_FIELD) * (fields+1), + &addon_buf->str, length, NullS)) DBUG_RETURN(0); diff --git a/sql/filesort.h b/sql/filesort.h index bd1d81f91ef..359f44a3907 100644 --- a/sql/filesort.h +++ b/sql/filesort.h @@ -161,6 +161,9 @@ SORT_INFO *filesort(THD *thd, TABLE *table, Filesort *filesort, Filesort_tracker* tracker, JOIN *join=NULL, table_map first_table_bit=0); +bool filesort_use_addons(TABLE *table, uint sortlength, + uint *length, uint *fields, uint *null_fields); + void change_double_for_sort(double nr,uchar *to); #endif /* FILESORT_INCLUDED */ diff --git a/sql/handler.h b/sql/handler.h index b98244fe47e..ea514634b86 100644 --- a/sql/handler.h +++ b/sql/handler.h @@ -308,6 +308,13 @@ enum enum_alter_inplace_result { #define HA_SLOW_CMP_REF (1ULL << 54) #define HA_CMP_REF_IS_EXPENSIVE HA_SLOW_CMP_REF +/** + Some engines are unable to provide an efficient implementation for rnd_pos(). + Server will try to avoid it, if possible + + TODO better to do it with cost estimates, not with an explicit flag +*/ +#define HA_SLOW_RND_POS (1ULL << 55) /* bits in index_flags(index_number) for what you can do with index */ #define HA_READ_NEXT 1 /* TODO really use this flag */ diff --git a/sql/sql_select.cc b/sql/sql_select.cc index 3f5def0838e..c16521068c6 100644 --- a/sql/sql_select.cc +++ b/sql/sql_select.cc @@ -13181,6 +13181,21 @@ remove_const(JOIN *join,ORDER *first_order, COND *cond, JOIN_TAB *head= join->join_tab + join->const_tables; *simple_order= head->on_expr_ref[0] == NULL; + if (*simple_order && head->table->file->ha_table_flags() & HA_SLOW_RND_POS) + { + uint u1, u2, u3; + /* + normally the condition is (see filesort_use_addons()) + + length + sortlength <= max_length_for_sort_data + + but for HA_SLOW_RND_POS tables we relax it a bit, as the alternative + is to use a temporary table, which is rather expensive. + + TODO proper cost estimations + */ + *simple_order= filesort_use_addons(head->table, 0, &u1, &u2, &u3); + } } else { diff --git a/storage/archive/ha_archive.h b/storage/archive/ha_archive.h index 56ff566db8c..1f25fba4eed 100644 --- a/storage/archive/ha_archive.h +++ b/storage/archive/ha_archive.h @@ -108,7 +108,7 @@ class ha_archive: public handler return (HA_NO_TRANSACTIONS | HA_REC_NOT_IN_SEQ | HA_CAN_BIT_FIELD | HA_BINLOG_ROW_CAPABLE | HA_BINLOG_STMT_CAPABLE | HA_STATS_RECORDS_IS_EXACT | HA_CAN_EXPORT | - HA_HAS_RECORDS | HA_CAN_REPAIR | + HA_HAS_RECORDS | HA_CAN_REPAIR | HA_SLOW_RND_POS | HA_FILE_BASED | HA_CAN_INSERT_DELAYED | HA_CAN_GEOMETRY); } ulong index_flags(uint idx, uint part, bool all_parts) const diff --git a/storage/csv/ha_tina.h b/storage/csv/ha_tina.h index c75a64faa52..5b389d984d6 100644 --- a/storage/csv/ha_tina.h +++ b/storage/csv/ha_tina.h @@ -107,7 +107,7 @@ class ha_tina: public handler { return (HA_NO_TRANSACTIONS | HA_REC_NOT_IN_SEQ | HA_NO_AUTO_INCREMENT | HA_BINLOG_ROW_CAPABLE | HA_BINLOG_STMT_CAPABLE | HA_CAN_EXPORT | - HA_CAN_REPAIR); + HA_CAN_REPAIR | HA_SLOW_RND_POS); } ulong index_flags(uint idx, uint part, bool all_parts) const {

1 0

Re: [Maria-developers] [Commits] 3f4dab80912: MDEV-17032: Estimates are higher for partitions of a table with @@use_stat_tables= PREFERABLY
by Sergey Petrunia 14 Nov '18

14 Nov '18

Hi Varun, On Mon, Nov 12, 2018 at 02:41:51PM +0530, Varun wrote: > revision-id: 3f4dab80912682325b61b288178febe9ee55de49 (mariadb-10.0.36-78-g3f4dab80912) > parent(s): 6cecb10a2f8b6536bed78ab6d3791d8befc9d732 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2018-11-12 14:40:07 +0530 > message: > > MDEV-17032: Estimates are higher for partitions of a table with @@use_stat_tables= PREFERABLY > > The problem here is EITS statistics does not calculate statistics for the partitions of the table. > So a temporary solution would be to not read EITS statistics for partitioned tables. > > Also disabling reading of EITS for columns that participate in the partition list of a table. > > --- > mysql-test/r/partition.result | 37 +++++++++++++++++++++++++++++++++++++ > mysql-test/t/partition.test | 35 +++++++++++++++++++++++++++++++++++ > sql/opt_range.cc | 12 ++++++++++-- > sql/partition_info.cc | 21 +++++++++++++++++++++ > sql/partition_info.h | 1 + > sql/sql_statistics.cc | 6 ++++++ > 6 files changed, 110 insertions(+), 2 deletions(-) > > diff --git a/mysql-test/r/partition.result b/mysql-test/r/partition.result > index c6669176b3d..abfbc21dccb 100644 > --- a/mysql-test/r/partition.result > +++ b/mysql-test/r/partition.result > @@ -2645,3 +2645,40 @@ Warnings: > Note 1517 Duplicate partition name p2 > DEALLOCATE PREPARE stmt; > DROP TABLE t1; > +# > +# MDEV-17032: Estimates are higher for partitions of a table with @@use_stat_tables= PREFERABLY > +# > +create table ten(a int); > +insert into ten values (0),(1),(2),(3),(4),(5),(6),(7),(8),(9); > +create table one_k(a int); > +insert into one_k select A.a + B.a* 10 + C.a * 100 from ten A, ten B, ten C; Please rename the tables to follow the t0,t1,t2... convention. > +create table t1 ( > +part_key int, > +a int, > +b int > +) partition by list(part_key) ( > +partition p0 values in (0), > +partition p1 values in (1), > +partition p2 values in (2), > +partition p3 values in (3), > +partition p4 values in (4) > +); > +insert into t1 > +select mod(a,5), a/100, a from one_k; > +set @save_use_stat_tables= @@use_stat_tables; > +set @save_optimizer_use_condition_selectivity=@@optimizer_use_condition_selectivity; > +set @@use_stat_tables= PREFERABLY; > +set @@optimizer_use_condition_selectivity=4; > +analyze table t1; > +Table Op Msg_type Msg_text > +test.t1 analyze status Engine-independent statistics collected > +test.t1 analyze status OK After some time passes, it gets hard to understand what exactly is being checked. Then, there's some change in the EXPLAIN (typically for an unrelated reason) and one is left wondering, whether the new EXPLAIN output is ok or not. * please use EXPLAIN PARTITIONS. * please add something like "--echo # rows should be X, not Y" before the query. > +explain select * from t1 where part_key in (1,2); > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 ALL NULL NULL NULL NULL 400 Using where > +explain select * from t1 where part_key > 1; > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 ALL NULL NULL NULL NULL 600 Using where This does not provide sufficient coverage. I get this result if I just apply the change in sql_statistics.cc. mysql> explain select * from t1 where part_key in (1,2); +------+-------------+-------+------+---------------+------+---------+------+------+-------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +------+-------------+-------+------+---------------+------+---------+------+------+-------------+ | 1 | SIMPLE | t1 | ALL | NULL | NULL | NULL | NULL | 400 | Using where | +------+-------------+-------+------+---------------+------+---------+------+------+-------------+ 1 row in set (0.01 sec) mysql> explain select * from t1 where part_key > 1; +------+-------------+-------+------+---------------+------+---------+------+------+-------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +------+-------------+-------+------+---------------+------+---------+------+------+-------------+ | 1 | SIMPLE | t1 | ALL | NULL | NULL | NULL | NULL | 600 | Using where | +------+-------------+-------+------+---------------+------+---------+------+------+-------------+ 1 row in set (0.00 sec) Please add testcases for other changes. > +set @@optimizer_use_condition_selectivity=@save_optimizer_use_condition_selectivity; > +set @@use_stat_tables= @save_use_stat_tables; > +drop table t1, one_k, ten; > diff --git a/sql/opt_range.cc b/sql/opt_range.cc > index 3bcaa72e32f..993b7d57e0a 100644 > --- a/sql/opt_range.cc > +++ b/sql/opt_range.cc > @@ -3322,6 +3322,10 @@ bool create_key_parts_for_pseudo_indexes(RANGE_OPT_PARAM *param, > { > Field **field_ptr; > TABLE *table= param->table; > + partition_info *part_info= NULL; > + #ifdef WITH_PARTITION_STORAGE_ENGINE > + part_info= table->part_info; > + #endif > uint parts= 0; > > for (field_ptr= table->field; *field_ptr; field_ptr++) > @@ -3329,7 +3333,9 @@ bool create_key_parts_for_pseudo_indexes(RANGE_OPT_PARAM *param, > Column_statistics* col_stats= (*field_ptr)->read_stats; > if (bitmap_is_set(used_fields, (*field_ptr)->field_index) > && col_stats && !col_stats->no_stat_values_provided() > - && !((*field_ptr)->type() == MYSQL_TYPE_GEOMETRY)) > + && !((*field_ptr)->type() == MYSQL_TYPE_GEOMETRY) > + && (!part_info || > + part_info->disable_eits_for_partitioning_columns(*field_ptr))) > parts++; > } > > @@ -3350,7 +3356,9 @@ bool create_key_parts_for_pseudo_indexes(RANGE_OPT_PARAM *param, > if (bitmap_is_set(used_fields, (*field_ptr)->field_index)) > { > Field *field= *field_ptr; > - if (field->type() == MYSQL_TYPE_GEOMETRY) > + if (field->type() == MYSQL_TYPE_GEOMETRY > + && (!part_info || > + part_info->disable_eits_for_partitioning_columns(*field_ptr))) > continue; Why use 'field_ptr' if you have 'field' ? The above condition seems to be wrong. Suppose that the following holds: field->type()==MYSQL_TYPE_GEOMETRY, part_info!=NULL part_info->disable_eits_for_partitioning_columns(field) == FALSE // makes // sense as one typically does not do partitioning on geometry columns. if-condition evaluates to: (TRUE && (FALSE || FALSE)) -> (TRUE && FALSE) -> FALSE. So we will not execute 'continue;' for a geometry column. This seems incorrect. > > uint16 store_length; > diff --git a/sql/partition_info.cc b/sql/partition_info.cc > index 52bda560c1c..91d74b28d0e 100644 > --- a/sql/partition_info.cc > +++ b/sql/partition_info.cc > @@ -3164,6 +3164,27 @@ void partition_info::print_debug(const char *str, uint *value) > DBUG_PRINT("info", ("parser: %s", str)); > DBUG_VOID_RETURN; > } > + > +/* > + > + Disabling reading EITS statistics for columns involved in the > + partition list of a table. > + We assume the selecticivity for such columns would be handled > + during partition pruning. > + > +*/ > + > +bool partition_info::disable_eits_for_partitioning_columns(Field *field) The function name is misleading. If a function is named disable_XYZ() I would have expected the code inside to actually take the steps to prevent XYZ from happening. I think it would be more straightforward if it was called bool partition_info::field_is_used_in_partition_expr() const or something like that. Please also declare the function as 'const' and maybe its parameter too. (As discussed on Slack, the function should also search for the field in the list of sub-partitioning columns) > +{ > + uint i; > + for (i= 0; i < num_part_fields; i++) > + { > + if (field->eq(part_field_array[i])) > + return FALSE; > + } > + return TRUE; > +} > + > #else /* WITH_PARTITION_STORAGE_ENGINE */ > /* > For builds without partitioning we need to define these functions > diff --git a/sql/sql_statistics.cc b/sql/sql_statistics.cc > index cb75a5c2176..33a12588434 100644 > --- a/sql/sql_statistics.cc > +++ b/sql/sql_statistics.cc > @@ -3589,6 +3589,12 @@ void set_statistics_for_table(THD *thd, TABLE *table) > (use_stat_table_mode <= COMPLEMENTARY || > !table->stats_is_read || read_stats->cardinality_is_null) ? > table->file->stats.records : read_stats->cardinality; > + > + #ifdef WITH_PARTITION_STORAGE_ENGINE > + if (table->part_info) > + table->used_stat_records= table->file->stats.records; > + #endif This looks cryptic. Please add a comment, something like: /* For partitioned table, EITS statistics is based on data from all partitions. On the other hand, Partition Pruning figures which partitions will be accessed and then computes the estimate of rows in used_partitions. Use the estimate from Partition Pruning as it is typically more precise. Ideally, EITS should provide per-partition statistics but this is not implemented currently. */ > KEY *key_info, *key_info_end; > for (key_info= table->key_info, key_info_end= key_info+table->s->keys; > key_info < key_info_end; key_info++) BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] [Commits] a9c1420233f: MDEV-17589: Stack-buffer-overflow with indexed varchar (utf8) field
by Sergey Petrunia 14 Nov '18

14 Nov '18

Hi Varun, The patch is ok to push. One last question: why is it made against 10.2? I see the MDEV issue has AffectsVersion to start from 10.2, but to my knowledge the code and constants involved were not changed in 10.2, so the issue should be present in the earlier versions as well? On Mon, Nov 12, 2018 at 04:27:08PM +0530, Varun wrote: > revision-id: a9c1420233f3f65aee00a8e7bdd1c8481b0bbc37 (mariadb-10.2.18-74-ga9c1420233f) > parent(s): b290ef8c76e2d7dfbae7a85766694a6fd4648eac > author: Varun Gupta > committer: Varun Gupta > timestamp: 2018-11-12 16:25:25 +0530 > message: > > MDEV-17589: Stack-buffer-overflow with indexed varchar (utf8) field > > Create a new constant MAX_DATA_LENGTH_FOR_KEY. > Replace the value of MAX_KEY_LENGTH to also include the LENGTH and NULL BYTES. > > --- > mysql-test/r/func_group_innodb.result | 23 +++++++++++++++++++++++ > mysql-test/t/func_group_innodb.test | 18 ++++++++++++++++++ > sql/handler.h | 12 +++++++++--- > sql/partition_info.cc | 4 ++-- > sql/sql_const.h | 13 ++++++++++++- > sql/table.cc | 2 +- > 6 files changed, 65 insertions(+), 7 deletions(-) > > diff --git a/mysql-test/r/func_group_innodb.result b/mysql-test/r/func_group_innodb.result > index 52d5922df95..c2bfe9bf81c 100644 > --- a/mysql-test/r/func_group_innodb.result > +++ b/mysql-test/r/func_group_innodb.result > @@ -246,4 +246,27 @@ EXPLAIN SELECT MIN(c) FROM t1 GROUP BY b; > id select_type table type possible_keys key key_len ref rows Extra > 1 SIMPLE t1 range NULL b 263 NULL 3 Using index for group-by > DROP TABLE t1; > +# > +# MDEV-17589: Stack-buffer-overflow with indexed varchar (utf8) field > +# > +CREATE TABLE t1 (v1 varchar(1020), v2 varchar(2), v3 varchar(2), KEY k1 (v3,v2,v1)) ENGINE=InnoDB CHARACTER SET=utf8; > +INSERT INTO t1 VALUES ('king', 'qu','qu'), ('bad','go','go'); > +explain > +SELECT MIN(t1.v1) FROM t1 where t1.v2='qu' and t1.v3='qu'; > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE NULL NULL NULL NULL NULL NULL NULL Select tables optimized away > +SELECT MIN(t1.v1) FROM t1 where t1.v2='qu' and t1.v3='qu'; > +MIN(t1.v1) > +king > +drop table t1; > +CREATE TABLE t1 (v1 varchar(1024) CHARACTER SET utf8, KEY v1 (v1)) ENGINE=InnoDB; > +INSERT INTO t1 VALUES ('king'), ('bad'); > +explain > +SELECT MIN(x.v1) FROM (SELECT t1.* FROM t1 WHERE t1.v1 >= 'p') x; > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE NULL NULL NULL NULL NULL NULL NULL No matching min/max row > +SELECT MIN(x.v1) FROM (SELECT t1.* FROM t1 WHERE t1.v1 >= 'p') x; > +MIN(x.v1) > +NULL > +drop table t1; > End of 5.5 tests > diff --git a/mysql-test/t/func_group_innodb.test b/mysql-test/t/func_group_innodb.test > index c62d3d08496..0d24f37363b 100644 > --- a/mysql-test/t/func_group_innodb.test > +++ b/mysql-test/t/func_group_innodb.test > @@ -192,4 +192,22 @@ EXPLAIN SELECT MIN(c) FROM t1 GROUP BY b; > > DROP TABLE t1; > > +--echo # > +--echo # MDEV-17589: Stack-buffer-overflow with indexed varchar (utf8) field > +--echo # > + > +CREATE TABLE t1 (v1 varchar(1020), v2 varchar(2), v3 varchar(2), KEY k1 (v3,v2,v1)) ENGINE=InnoDB CHARACTER SET=utf8; > +INSERT INTO t1 VALUES ('king', 'qu','qu'), ('bad','go','go'); > +explain > +SELECT MIN(t1.v1) FROM t1 where t1.v2='qu' and t1.v3='qu'; > +SELECT MIN(t1.v1) FROM t1 where t1.v2='qu' and t1.v3='qu'; > +drop table t1; > + > +CREATE TABLE t1 (v1 varchar(1024) CHARACTER SET utf8, KEY v1 (v1)) ENGINE=InnoDB; > +INSERT INTO t1 VALUES ('king'), ('bad'); > +explain > +SELECT MIN(x.v1) FROM (SELECT t1.* FROM t1 WHERE t1.v1 >= 'p') x; > +SELECT MIN(x.v1) FROM (SELECT t1.* FROM t1 WHERE t1.v1 >= 'p') x; > +drop table t1; > + > --echo End of 5.5 tests > diff --git a/sql/handler.h b/sql/handler.h > index ed2ef822c88..6745312a17d 100644 > --- a/sql/handler.h > +++ b/sql/handler.h > @@ -378,6 +378,12 @@ enum enum_alter_inplace_result { > #define HA_KEY_NULL_LENGTH 1 > #define HA_KEY_BLOB_LENGTH 2 > > +/* Maximum length of any index lookup key, in bytes */ > + > +#define MAX_KEY_LENGTH (MAX_DATA_LENGTH_FOR_KEY \ > + +(MAX_REF_PARTS \ > + *(HA_KEY_NULL_LENGTH + HA_KEY_BLOB_LENGTH))) > + > #define HA_LEX_CREATE_TMP_TABLE 1U > #define HA_CREATE_TMP_ALTER 8U > > @@ -3421,14 +3427,14 @@ class handler :public Sql_alloc > uint max_key_parts() const > { return MY_MIN(MAX_REF_PARTS, max_supported_key_parts()); } > uint max_key_length() const > - { return MY_MIN(MAX_KEY_LENGTH, max_supported_key_length()); } > + { return MY_MIN(MAX_DATA_LENGTH_FOR_KEY, max_supported_key_length()); } > uint max_key_part_length() const > - { return MY_MIN(MAX_KEY_LENGTH, max_supported_key_part_length()); } > + { return MY_MIN(MAX_DATA_LENGTH_FOR_KEY, max_supported_key_part_length()); } > > virtual uint max_supported_record_length() const { return HA_MAX_REC_LENGTH; } > virtual uint max_supported_keys() const { return 0; } > virtual uint max_supported_key_parts() const { return MAX_REF_PARTS; } > - virtual uint max_supported_key_length() const { return MAX_KEY_LENGTH; } > + virtual uint max_supported_key_length() const { return MAX_DATA_LENGTH_FOR_KEY; } > virtual uint max_supported_key_part_length() const { return 255; } > virtual uint min_record_length(uint options) const { return 1; } > > diff --git a/sql/partition_info.cc b/sql/partition_info.cc > index f96882552fa..8be3f878f8d 100644 > --- a/sql/partition_info.cc > +++ b/sql/partition_info.cc > @@ -1713,12 +1713,12 @@ bool partition_info::check_partition_field_length() > > for (i= 0; i < num_part_fields; i++) > store_length+= get_partition_field_store_length(part_field_array[i]); > - if (store_length > MAX_KEY_LENGTH) > + if (store_length > MAX_DATA_LENGTH_FOR_KEY) > DBUG_RETURN(TRUE); > store_length= 0; > for (i= 0; i < num_subpart_fields; i++) > store_length+= get_partition_field_store_length(subpart_field_array[i]); > - if (store_length > MAX_KEY_LENGTH) > + if (store_length > MAX_DATA_LENGTH_FOR_KEY) > DBUG_RETURN(TRUE); > DBUG_RETURN(FALSE); > } > diff --git a/sql/sql_const.h b/sql/sql_const.h > index 7395ae3c08a..0c781855302 100644 > --- a/sql/sql_const.h > +++ b/sql/sql_const.h > @@ -33,7 +33,18 @@ > #define MAX_SYS_VAR_LENGTH 32 > #define MAX_KEY MAX_INDEXES /* Max used keys */ > #define MAX_REF_PARTS 32 /* Max parts used as ref */ > -#define MAX_KEY_LENGTH 3072 /* max possible key */ > + > +/* > + Maximum length of the data part of an index lookup key. > + > + The "data part" is defined as the value itself, not including the > + NULL-indicator bytes or varchar length bytes ("the Extras"). We need this > + value because there was a bug where length of the Extras were not counted. > + > + You probably need MAX_KEY_LENGTH, not this constant. > +*/ > + > +#define MAX_DATA_LENGTH_FOR_KEY 3072 > #if SIZEOF_OFF_T > 4 > #define MAX_REFLENGTH 8 /* Max length for record ref */ > #else > diff --git a/sql/table.cc b/sql/table.cc > index 2b278288784..5b163d501d1 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -2169,7 +2169,7 @@ int TABLE_SHARE::init_from_binary_frm_image(THD *thd, bool write, > uint pk_part_length= key_first_info->key_part[i].store_length; > if (keyinfo->ext_key_part_map & 1<<i) > { > - if (ext_key_length + pk_part_length > MAX_KEY_LENGTH) > + if (ext_key_length + pk_part_length > MAX_DATA_LENGTH_FOR_KEY) > { > add_keyparts_for_this_key= i; > break; > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] a04c60e6c99: MDEV-16849 Extending indexed VARCHAR column should be instantaneous
by Sergei Golubchik 13 Nov '18

13 Nov '18

Hi, Thirunarayanan! On Nov 13, Thirunarayanan Balathandayuthapani wrote: > revision-id: a04c60e6c99097c9b52f582b2af891f48b8f662d (mariadb-10.2.18-40-ga04c60e6c99) > parent(s): ab1ce2204e959bea596817494e932754ab5cbe88 > author: Thirunarayanan Balathandayuthapani <thiru(a)mariadb.com> > committer: Thirunarayanan Balathandayuthapani <thiru(a)mariadb.com> > timestamp: 2018-11-12 16:30:34 +0530 > message: > > MDEV-16849 Extending indexed VARCHAR column should be instantaneous > > Analysis: > ======== > Increasing the length of the indexed varchar column is not an instant operation for > innodb. > > diff --git a/mysql-test/suite/innodb/r/alter_varchar_change.result b/mysql-test/suite/innodb/r/alter_varchar_change.result > new file mode 100644 > index 00000000000..b60bd7847db > --- /dev/null > +++ b/mysql-test/suite/innodb/r/alter_varchar_change.result > @@ -0,0 +1,269 @@ ... > +CALL get_table_id("test/t1", @tbl1_id); > +SELECT @tbl1_id - @tbl_id; > +@tbl1_id - @tbl_id > +1 That's a bit fragile, you test that table id was changed exactly by 1. I don't think the exact difference matters here, so it'd be safer to test for `SELECT @tbl1_id = @tbl_id` (here and everywhere). > +SHOW CREATE TABLE t1; > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index 02780e7d1e0..b9668ad0979 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -6687,18 +6687,29 @@ static bool fill_alter_inplace_info(THD *thd, > key_part < end; > key_part++, new_part++) > { > + new_field= get_field_by_index(alter_info, new_part->fieldnr); > + old_field= table->field[key_part->fieldnr - 1]; > /* > + If there is a change in index length due to column expansion > + like varchar(X) changed to varchar(X + N) and has a compatible > + packed data representation, we mark it for fast/INPLACE change > + in index definition. InnoDB supports INPLACE for this cases no tabs in the server code, please. If you're using vim, I can show you my config that has different formatting style for innodb and the rest of the server :) > + > Key definition has changed if we are using a different field or > - if the used key part length is different. It makes sense to > - check lengths first as in case when fields differ it is likely > - that lengths differ too and checking fields is more expensive > - in general case. > + if the user key part length is different. > */ > - if (key_part->length != new_part->length) > + if (key_part->length != new_part->length && > + new_field->field != NULL && can new_field->field be NULL here? when? > + old_field->pack_length() < new_field->pack_length && Do you need it? I believe IS_EQUAL_PACK_LENGTH includes this check. > + (key_part->field->is_equal((Create_field*) new_field) > + == IS_EQUAL_PACK_LENGTH)) > + { > + ha_alter_info->handler_flags |= > + Alter_inplace_info::ALTER_COLUMN_INDEX_LENGTH; > + } > + else if (key_part->length != new_part->length) > goto index_changed; > > - new_field= get_field_by_index(alter_info, new_part->fieldnr); > - Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] 031efde365c: MDEV-16217: Assertion `!table || (!table->read_set || bitmap_is_set(table->read_set, field_index))' failed in Field_num::get_date
by Sergei Golubchik 13 Nov '18

13 Nov '18

Hi, Oleksandr! On Nov 07, Oleksandr Byelkin wrote: > revision-id: 031efde365c674dbdbaada95aa6d42a4274db438 (mariadb-10.2.18-65-g031efde365c) > parent(s): 89f948c766721a26e110bc9da0ca5ebc20f65112 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2018-11-07 14:29:47 +0100 > message: > > MDEV-16217: Assertion `!table || (!table->read_set || bitmap_is_set(table->read_set, field_index))' failed in Field_num::get_date > > - clean up DEFAULT() to work only with default value and correctly print > itself. > - fix of DBUG_ASSERT about fields read/write > - fix of field marking for write based really on the thd->mark_used_columns flag > diff --git a/sql/field.cc b/sql/field.cc > index caa84dc9932..6cd8940a893 100644 > --- a/sql/field.cc > +++ b/sql/field.cc > @@ -70,8 +70,21 @@ const char field_separator=','; > #define BLOB_PACK_LENGTH_TO_MAX_LENGH(arg) \ > ((ulong) ((1LL << MY_MIN(arg, 4) * 8) - 1)) > > -#define ASSERT_COLUMN_MARKED_FOR_READ DBUG_ASSERT(!table || (!table->read_set || bitmap_is_set(table->read_set, field_index))) > -#define ASSERT_COLUMN_MARKED_FOR_WRITE_OR_COMPUTED DBUG_ASSERT(is_stat_field || !table || (!table->write_set || bitmap_is_set(table->write_set, field_index) || (table->vcol_set && bitmap_is_set(table->vcol_set, field_index)))) > +// Column marked for read or the field set to read out or record[0] or [1] > +#define ASSERT_COLUMN_MARKED_FOR_READ \ > + DBUG_ASSERT(!table || \ > + (!table->read_set || \ > + bitmap_is_set(table->read_set, field_index) || \ > + (!(ptr >= table->record[0] && \ > + ptr < table->record[0] + table->s->reclength)))) > + > +#define ASSERT_COLUMN_MARKED_FOR_WRITE_OR_COMPUTED \ > + DBUG_ASSERT(is_stat_field || !table || \ > + (!table->write_set || \ > + bitmap_is_set(table->write_set, field_index) || \ > + (!(ptr >= table->record[0] && \ > + ptr < table->record[0] + table->s->reclength))) || \ > + (table->vcol_set && bitmap_is_set(table->vcol_set, field_index))) Do you need this ptr check in ASSERT_COLUMN_MARKED_FOR_WRITE_OR_COMPUTED ? I'd expect you only needing it in ASSERT_COLUMN_MARKED_FOR_READ. > > #define FLAGSTR(S,F) ((S) & (F) ? #F " " : "") > > diff --git a/sql/item.cc b/sql/item.cc > index 2adec33491b..6828a74f9ff 100644 > --- a/sql/item.cc > +++ b/sql/item.cc > @@ -8813,15 +8824,19 @@ bool Item_default_value::fix_fields(THD *thd, Item **items) > goto error; > memcpy((void *)def_field, (void *)field_arg->field, > field_arg->field->size_of()); > - IF_DBUG(def_field->is_stat_field=1,); // a hack to fool ASSERT_COLUMN_MARKED_FOR_WRITE_OR_COMPUTED > + // If non-constant default value expression > if (def_field->default_value && def_field->default_value->flags) > { > uchar *newptr= (uchar*) thd->alloc(1+def_field->pack_length()); > if (!newptr) > goto error; > + /* > + Even if DEFAULT() do not read tables fields, the default value > + expression can do it. > + */ > fix_session_vcol_expr_for_read(thd, def_field, def_field->default_value); > if (thd->mark_used_columns != MARK_COLUMNS_NONE) > - def_field->default_value->expr->walk(&Item::register_field_in_read_map, 1, 0); > + def_field->default_value->expr->update_used_tables(); what's the difference? > def_field->move_field(newptr+1, def_field->maybe_null() ? newptr : 0, 1); > } > else > diff --git a/sql/item.h b/sql/item.h > index 8d02d981d38..5866f328f38 100644 > --- a/sql/item.h > +++ b/sql/item.h > @@ -5229,6 +5229,11 @@ class Item_default_value : public Item_field > return false; > } > table_map used_tables() const; > + virtual void update_used_tables() > + { > + if (field && field->default_value) > + field->default_value->expr->walk(&Item::register_field_in_read_map, 1, 0); why not field->default_value->expr->update_used_tables() ? > + } > Field *get_tmp_table_field() { return 0; } > Item *get_tmp_table_item(THD *thd) { return this; } > Item_field *field_for_view_update() { return 0; } > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > index c282db42fdd..0deb5ec1362 100644 > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -5737,7 +5737,7 @@ find_field_in_table_ref(THD *thd, TABLE_LIST *table_list, > TABLE *table= field_to_set->table; > if (thd->mark_used_columns == MARK_COLUMNS_READ) > bitmap_set_bit(table->read_set, field_to_set->field_index); > - else > + else if (thd->mark_used_columns == MARK_COLUMNS_WRITE) > bitmap_set_bit(table->write_set, field_to_set->field_index); what does it affect? > } > } Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 3

Re: [Maria-developers] 031efde365c: MDEV-16217: Assertion `!table || (!table->read_set || bitmap_is_set(table->read_set, field_index))' failed in Field_num::get_date
by Sergei Golubchik 12 Nov '18

12 Nov '18

Hi, Oleksandr! On Nov 12, Oleksandr Byelkin wrote: > > On Nov 07, Oleksandr Byelkin wrote: > > > revision-id: 031efde365c674dbdbaada95aa6d42a4274db438 (mariadb-10.2.18-65-g031efde365c) > > > parent(s): 89f948c766721a26e110bc9da0ca5ebc20f65112 > > > author: Oleksandr Byelkin > > > committer: Oleksandr Byelkin > > > timestamp: 2018-11-07 14:29:47 +0100 > > > message: > > > > > > MDEV-16217: Assertion `!table || (!table->read_set || > > bitmap_is_set(table->read_set, field_index))' failed in Field_num::get_date > > > > > > - clean up DEFAULT() to work only with default value and correctly print > > > itself. > > > - fix of DBUG_ASSERT about fields read/write > > > - fix of field marking for write based really on the thd->mark_used_columns flag > > > > > diff --git a/sql/field.cc b/sql/field.cc > > > index caa84dc9932..6cd8940a893 100644 > > > --- a/sql/field.cc > > > +++ b/sql/field.cc > > > @@ -70,8 +70,21 @@ const char field_separator=','; > > > #define BLOB_PACK_LENGTH_TO_MAX_LENGH(arg) \ > > > ((ulong) ((1LL << MY_MIN(arg, 4) * 8) - 1)) > > > > > > -#define ASSERT_COLUMN_MARKED_FOR_READ DBUG_ASSERT(!table || > > (!table->read_set || bitmap_is_set(table->read_set, field_index))) > > > -#define ASSERT_COLUMN_MARKED_FOR_WRITE_OR_COMPUTED > > DBUG_ASSERT(is_stat_field || !table || (!table->write_set || > > bitmap_is_set(table->write_set, field_index) || (table->vcol_set && > > bitmap_is_set(table->vcol_set, field_index)))) > > > +// Column marked for read or the field set to read out or record[0] or > > [1] > > > +#define ASSERT_COLUMN_MARKED_FOR_READ \ > > > + DBUG_ASSERT(!table || \ > > > + (!table->read_set || \ > > > + bitmap_is_set(table->read_set, field_index) || \ > > > + (!(ptr >= table->record[0] && \ > > > + ptr < table->record[0] + table->s->reclength)))) > > > + > > > +#define ASSERT_COLUMN_MARKED_FOR_WRITE_OR_COMPUTED \ > > > + DBUG_ASSERT(is_stat_field || !table || \ > > > + (!table->write_set || \ > > > + bitmap_is_set(table->write_set, field_index) || \ > > > + (!(ptr >= table->record[0] && \ > > > + ptr < table->record[0] + table->s->reclength))) || \ > > > + (table->vcol_set && bitmap_is_set(table->vcol_set, > > field_index))) > > > > Do you need this ptr check in ASSERT_COLUMN_MARKED_FOR_WRITE_OR_COMPUTED ? > > I'd expect you only needing it in ASSERT_COLUMN_MARKED_FOR_READ. > > I'd prefer to have them symmetric Right, but these macros already have non-symmetric names and non-symmetric definition - the second checks for vcol_set. The way you did it only makes the reader to pause and think in what case one can get `ptr` outside of table->record[0] for writes. Because your assert says "yes, it's ok to write into the column if ptr is outside of table->record[0]". If ptr cannot possibly be outside of record[0] for write, then it shouldn't be in the assert. > > > #define FLAGSTR(S,F) ((S) & (F) ? #F " " : "") > > > > > > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > > > index c282db42fdd..0deb5ec1362 100644 > > > --- a/sql/sql_base.cc > > > +++ b/sql/sql_base.cc > > > @@ -5737,7 +5737,7 @@ find_field_in_table_ref(THD *thd, TABLE_LIST > > *table_list, > > > TABLE *table= field_to_set->table; > > > if (thd->mark_used_columns == MARK_COLUMNS_READ) > > > bitmap_set_bit(table->read_set, field_to_set->field_index); > > > - else > > > + else if (thd->mark_used_columns == MARK_COLUMNS_WRITE) > > > bitmap_set_bit(table->write_set, field_to_set->field_index); > > > > what does it affect? > > > > enum enum_mark_columns > { MARK_COLUMNS_NONE, MARK_COLUMNS_READ, MARK_COLUMNS_WRITE}; > > MARK_COLUMNS_NONE really used in the code, I remember that something was > marked incorrectly (it is how I found it) but what it is too late to ask > after many month as fix is done. this code is inside of if (thd->mark_used_columns != MARK_COLUMNS_NONE) so I don't see what your change could possibly do. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] bc7d600a27f: MDEV-15951 system versioning by trx id doesn't work with partitioning
by Sergei Golubchik 12 Nov '18

12 Nov '18

Hi, Nikita! On Nov 12, Nikita Malyavin wrote: > revision-id: bc7d600a27f4c6c14deea5f1d674c621f85b23ba (versioning-1.0.5-141-gbc7d600a27f) > parent(s): 90b292ce31170be33796b07723ffcef56bf8a679 > author: Nikita Malyavin <nikitamalyavin(a)gmail.com> > committer: Nikita Malyavin <nikitamalyavin(a)gmail.com> > timestamp: 2018-09-21 22:42:03 +1000 > message: > > MDEV-15951 system versioning by trx id doesn't work with partitioning > > Fix partitioning for trx_id-versioned tables. > `partition by hash`, `range` and others now work. > `partition by system_time` is forbidden. > Currently we cannot use row_start and row_end in `partition by`, because > insertion of versioned field is done by engine's handler, as well as > row_start/row_end's value set up, which is a transaction id -- so it's > also forbidden. > > The drawback is that it's now impossible to use `partition by key()` > without parameters for such tables, because it references row_start and > row_end implicitly. > > * add handler::vers_can_native() > * drop Table_scope_and_contents_source_st::vers_native() > * drop partition_element::find_engine_flag as unused > * forbid versioning partitioning for trx_id as not supported > * adopt vers tests for trx_id partitioning > * forbid any row_end referencing in `partition by` clauses, > including implicit `by key()` > > --- > mysql-test/suite/versioning/r/partition.result | 79 ++++++++++++++++-- > .../suite/versioning/t/partition.combinations | 3 + > mysql-test/suite/versioning/t/partition.test | 97 ++++++++++++++++++++-- > sql/ha_partition.h | 10 +++ > sql/handler.cc | 34 ++------ > sql/handler.h | 8 +- > sql/partition_element.h | 15 ---- > sql/sql_class.h | 4 +- > sql/sql_partition.cc | 9 ++ > sql/sql_table.cc | 6 +- > sql/table.cc | 4 +- > sql/temporary_tables.cc | 10 ++- > 12 files changed, 214 insertions(+), 65 deletions(-) > > diff --git a/mysql-test/suite/versioning/r/partition.result b/mysql-test/suite/versioning/r/partition.result > index bfec0ce2d4b..cf4203b7949 100644 > --- a/mysql-test/suite/versioning/r/partition.result > +++ b/mysql-test/suite/versioning/r/partition.result > @@ -1,6 +1,6 @@ > set system_versioning_alter_history=keep; > # Check conventional partitioning on temporal tables > -create table t1 (x int) > +create or replace table t1 (x int, ROW_START, ROW_END, period for system_time(row_start, row_end)) ouch. that looks very weird. please, only replace the type, not the whole column definition. Like everywhere else: --replace_result $sys_datatype_expl SYS_DATATYPE > with system versioning > partition by range columns (x) ( > partition p0 values less than (100), > @@ -501,6 +504,72 @@ set timestamp=1523466002.799571; > insert into t1 values (11),(12); > set timestamp=1523466004.169435; > delete from t1 where pk in (11, 12); > +# MDEV-15951 system versioning by trx id doesn't work with partitioning > +# currently trx_id does not support partitioning by system_time > +create or replace table t1( > +i int, > +row_start bigint unsigned generated always as row start, > +row_end bigint unsigned generated always as row end, > +period for system_time(row_start, row_end) > +) engine=InnoDB with system versioning partition by system_time ( if you have tests with innodb and trx_id, please, put them in a separate test file, say, partition_innodb.test. partition.test is for common tests that are repeated three times, with myisam, innodb, timestamps, and trx_id. There is no need to repeat your test three times. > diff --git a/mysql-test/suite/versioning/t/partition.combinations b/mysql-test/suite/versioning/t/partition.combinations > index 4d73ef5a5ea..561c5656929 100644 > --- a/mysql-test/suite/versioning/t/partition.combinations > +++ b/mysql-test/suite/versioning/t/partition.combinations > @@ -1,5 +1,8 @@ > [timestamp] > default-storage-engine=innodb > > +[trx_id] > +default-storage-engine=innodb > + > [myisam] > default-storage-engine=myisam But then you don't need partition.combinations at all, just add --source engines.inc like other tests do. > diff --git a/sql/ha_partition.h b/sql/ha_partition.h > index 8a251016703..ef49011b7f8 100644 > --- a/sql/ha_partition.h > +++ b/sql/ha_partition.h > @@ -413,6 +413,16 @@ class ha_partition :public handler > > virtual void return_record_by_parent(); > > + virtual bool vers_can_native(THD *thd) > + { > + // PARTITION BY SYSTEM_TIME is not supported for now > + bool can= !thd->lex->part_info > + || thd->lex->part_info->part_type != VERSIONING_PARTITION; > + for (uint i= 0; i < m_tot_parts && can; i++) > + can= can && m_file[i]->vers_can_native(thd); > + return can; > + } > + > /* > ------------------------------------------------------------------------- > MODULE create/delete handler object > diff --git a/sql/handler.cc b/sql/handler.cc > index 306b0868d15..a7962039463 100644 > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -4938,6 +4938,7 @@ int ha_create_table(THD *thd, const char *path, > !create_info->tmp_table(); > > share.frm_image= frm; > + share.orig_table_name= table_name; why? if orig_table_name is NULL, error_table_name() will use table_name, so there's never need to make orig_table_name==table_name. and I don't understand all your other manipulations with orig_table_name. Could you please explain that? > > // open an frm image > if (share.init_from_binary_frm_image(thd, write_frm_now, > @@ -6952,29 +6953,6 @@ bool Vers_parse_info::fix_implicit(THD *thd, Alter_info *alter_info) > return false; > } > > -bool Table_scope_and_contents_source_st::vers_native(THD *thd) const > -{ > - if (ha_check_storage_engine_flag(db_type, HTON_NATIVE_SYS_VERSIONING)) > - return true; > - > -#ifdef WITH_PARTITION_STORAGE_ENGINE > - partition_info *info= thd->work_part_info; > - if (info && !(used_fields & HA_CREATE_USED_ENGINE)) > - { > - if (handlerton *hton= info->default_engine_type) > - return ha_check_storage_engine_flag(hton, HTON_NATIVE_SYS_VERSIONING); > - > - List_iterator_fast<partition_element> it(info->partitions); > - while (partition_element *partition_element= it++) > - { > - if (partition_element->find_engine_flag(HTON_NATIVE_SYS_VERSIONING)) > - return true; > - } > - } > -#endif > - return false; > -} what was wrong with that (and all other changed code in this file) ? > bool Table_scope_and_contents_source_st::vers_fix_system_fields( > THD *thd, Alter_info *alter_info, const TABLE_LIST &create_table, > bool create_select) Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] MDEV-17292.
by Sergei Golubchik 12 Nov '18

12 Nov '18

Hi, Alexey! On Nov 09, Alexey Botchkov wrote: > Hi, Sergei. > > I'm not sure what we want to do about this issue > https://jira.mariadb.org/browse/MDEV-17292 > > It's about pam_user_map.so packaging. > So we do like this compiled an put into the 'appropriate' > directory as /lib/security during install. Right? Yes. Not sure whether it should be hard-coded /lib/security or there's some distro-specific way of determining the correct path. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] e60cbbbda31: MDEV-12575: Server crash in AGGR_OP::put_record or in JOIN_CACHE::free or Invalid write in JOIN::make_aggr_tables_info
by Sergey Petrunia 09 Nov '18

09 Nov '18

Hi Varun, On Fri, Nov 09, 2018 at 02:26:04PM +0530, Varun wrote: > revision-id: e60cbbbda31d7a10e87c16313171b2036a4519c4 (mariadb-10.2.18-52-ge60cbbbda31) > parent(s): f8268f3cce4577c28ab62e53293556d05a74fb1a > author: Varun Gupta > committer: Varun Gupta > timestamp: 2018-11-01 14:27:24 +0530 > message: > > MDEV-12575: Server crash in AGGR_OP::put_record or in JOIN_CACHE::free or Invalid write in JOIN::make_aggr_tables_info > > During the optimize state of a query, we come know that the result set > would atmost contain one row, then for such a query we don't need > to compute GROUP BY, ORDER BY and DISTINCT. > ... > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index 1fa80da85a6..a721aed6ce3 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -2245,6 +2245,18 @@ JOIN::optimize_inner() > if (!tables_list || !table_count) > { > choose_tableless_subquery_plan(); > + > + /* The output has atmost one row */ > + if (group_list) > + { > + group_list= 0; > + group_optimized_away= 1; > + rollup.state= ROLLUP::STATE_NONE; > + } > + order=0; > + simple_order=1; > + select_distinct=0; > + Please fix the coding style: - "x= y" (space after the =) - use NULL for pointers - use false for booleans s/atmost/at most/ Ok to push after the above is addressed. > if (select_lex->have_window_funcs()) > { > if (!(join_tab= (JOIN_TAB*) thd->alloc(sizeof(JOIN_TAB)))) > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] [Commits] 0187bb03884: MDEV_-17589: Stack-buffer-overflow with indexed varchar (utf8) field
by Sergey Petrunia 08 Nov '18

08 Nov '18

Hi Varun, On Mon, Nov 05, 2018 at 07:36:17PM +0530, Varun wrote: > revision-id: 0187bb03884a34ca8de96e47f4cb1f3d860e6db9 (mariadb-10.2.16-224-g0187bb03884) > parent(s): 8a346f31b913daa011085afec2b2d38450c73e00 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2018-11-05 19:32:26 +0530 > message: > > MDEV_-17589: Stack-buffer-overflow with indexed varchar (utf8) field > > Create a new constant MAX_DATA_LENGTH_FOR_KEY. > Replace the value of MAX_KEY_LENGTH to also include the LENGTH and NULL BYTES. > ... > diff --git a/mysql-test/r/partition_datatype.result b/mysql-test/r/partition_datatype.result > index 2e518c194f0..66d003c97cb 100644 > --- a/mysql-test/r/partition_datatype.result > +++ b/mysql-test/r/partition_datatype.result > @@ -314,7 +314,7 @@ bbbb > drop table t1; > set sql_mode=''; > create table t1 (a varchar(3070)) partition by key (a); > -ERROR HY000: The total length of the partitioning fields is too large > +drop table t1; I don't think it's a good idea that after this patch we allow the table definitions that we didn't allow before. This raises the question of datadir downgrades, etc. I think we should only remove the buffer overrun. Here's patch how to avoid the above: https://gist.github.com/spetrunia/7e7e3fcc60a81a6cb854e77c12c367d3 > create table t1 (a varchar(65532) not null) partition by key (a); > ERROR HY000: The total length of the partitioning fields is too large > create table t1 (a varchar(65533)) partition by key (a); ... > diff --git a/mysql-test/r/innodb_ext_key.result b/mysql-test/r/innodb_ext_key.result > index c55e8d138f8..c76c8613c57 100644 > --- a/mysql-test/r/innodb_ext_key.result > +++ b/mysql-test/r/innodb_ext_key.result > @@ -1168,8 +1168,8 @@ EXPLAIN > "access_type": "range", > "possible_keys": ["f2"], > "key": "f2", > - "key_length": "3070", > - "used_key_parts": ["f2", "pk1"], > + "key_length": "3074", > + "used_key_parts": ["f2", "pk1", "pk2"], Same reaction as above: This change shows that "Extended keys" feature now allows bigger set of key extensions than before. Can we avoid making this change by changing MAX_KEY_LENGTH to MAX_DATA_LENGTH_FOR_KEY somewhere? > "rows": 1, > "filtered": 100, > "index_condition": "t1.pk1 <= 5 and t1.pk2 <= 5 and t1.f2 = 'abc'", > diff --git a/sql/handler.h b/sql/handler.h > index ed2ef822c88..75f72bdfb53 100644 > --- a/sql/handler.h > +++ b/sql/handler.h > @@ -378,6 +378,10 @@ enum enum_alter_inplace_result { > #define HA_KEY_NULL_LENGTH 1 > #define HA_KEY_BLOB_LENGTH 2 > > +#define MAX_KEY_LENGTH (MAX_DATA_LENGTH_FOR_KEY \ > + +(MAX_REF_PARTS \ > + *(HA_KEY_NULL_LENGTH + HA_KEY_BLOB_LENGTH))) > + I think this requires a comment. Please add something like /* Maximum length of any index lookup key, in bytes */ > #define HA_LEX_CREATE_TMP_TABLE 1U > #define HA_CREATE_TMP_ALTER 8U > > @@ -3421,14 +3425,14 @@ class handler :public Sql_alloc > uint max_key_parts() const > { return MY_MIN(MAX_REF_PARTS, max_supported_key_parts()); } > uint max_key_length() const > - { return MY_MIN(MAX_KEY_LENGTH, max_supported_key_length()); } > + { return MY_MIN(MAX_DATA_LENGTH_FOR_KEY, max_supported_key_length()); } > uint max_key_part_length() const > - { return MY_MIN(MAX_KEY_LENGTH, max_supported_key_part_length()); } > + { return MY_MIN(MAX_DATA_LENGTH_FOR_KEY, max_supported_key_part_length()); } > > virtual uint max_supported_record_length() const { return HA_MAX_REC_LENGTH; } > virtual uint max_supported_keys() const { return 0; } > virtual uint max_supported_key_parts() const { return MAX_REF_PARTS; } > - virtual uint max_supported_key_length() const { return MAX_KEY_LENGTH; } > + virtual uint max_supported_key_length() const { return MAX_DATA_LENGTH_FOR_KEY; } > virtual uint max_supported_key_part_length() const { return 255; } > virtual uint min_record_length(uint options) const { return 1; } > > diff --git a/sql/sql_const.h b/sql/sql_const.h > index 7395ae3c08a..0be63a2a5ad 100644 > --- a/sql/sql_const.h > +++ b/sql/sql_const.h > @@ -33,7 +33,7 @@ > #define MAX_SYS_VAR_LENGTH 32 > #define MAX_KEY MAX_INDEXES /* Max used keys */ > #define MAX_REF_PARTS 32 /* Max parts used as ref */ Please add a comment, something like /* Maximum length of the data part of an index lookup key. The "data part" is defined as the value itself, not including the NULL-indicator bytes or varchar length bytes ("the Extras"). We need this value because there was a bug where length of the Extras were not counted. You probably need MAX_KEY_LENGTH, not this constant. */ > -#define MAX_KEY_LENGTH 3072 /* max possible key */ > +#define MAX_DATA_LENGTH_FOR_KEY 3072 > #if SIZEOF_OFF_T > 4 > #define MAX_REFLENGTH 8 /* Max length for record ref */ > #else BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] b0d21b7: MDEV-14567 Include full name of object in message about incorrect value for column.
by Sergei Golubchik 07 Nov '18

07 Nov '18

Hi, Alexey! On Nov 07, Alexey Botchkov wrote: > > But that's fine to show, right? No NULL, no garbage. > > Looks like this: > Incorrect datetime value: '0000-00-00 00:00:00' for column > 'test.#sql-691c_15d.c1' at row 1 > No NULL, but that #sql-something name doesn't tell anything and is > always different so we have to add lots of --replace-result in tests. Right, but we already have a lot of replace-result in tests. #sql-691c_15d showing in error messages in a different issue (but see below). > My patch shows it as.this: > Incorrect datetime value: '0000-00-00 00:00:00' for column ' {temporary}.c1' at row 1 This isn't much better in my opinion. It only helps tests to avoid replace-result, and I always believed in solving test problems in tests, not in the server. Meaning, exactly, replace-result. For the end user #sql-691c_15d and {temporary} has about the save amount of information. > > By the way, can this error ever happen for internal temporary tables? > Yes. > In our tests it happens pretty often. > Code like this: > CREATE TABLE t3(c1 DATETIME NOT NULL); > INSERT INTO t3 VALUES (0); > > --echo > SET sql_mode = TRADITIONAL; > > --echo > --error ER_TRUNCATED_WRONG_VALUE > ALTER TABLE t3 ADD INDEX(c1); Oh, ALTER TABLE. This is different. In one of the system versioning related fixes, midenok added a TABLE_SHARE::error_table_name() method to be used in error messages in ALTER. See 912769b7dc. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] cd41d376d41: MDEV-16697: Fix difference between 32bit/windows and 64bit systems in allowed select nest level
by Sergei Golubchik 06 Nov '18

06 Nov '18

Hi, Oleksandr! Why did you do that? You wrote "Fix difference between 32bit/windows and 64bit systems", but to fix just that it would've been enough to change nesting_map to ulonglong. Using Bitmap<> you're basically preparing for any arbitrary >64 nesting depth. Is this your goal? What feature do you need it for? On Nov 06, Oleksandr Byelkin wrote: > revision-id: cd41d376d413d066f3d497a67d039515fd385a1c (versioning-1.0.5-36-gcd41d376d41) > parent(s): 1748a31ae8d69e4939336f644f884e9de3039e7f > author: Oleksandr Byelkin <sanja(a)mariadb.com> > committer: Oleksandr Byelkin <sanja(a)mariadb.com> > timestamp: 2018-07-05 17:49:44 +0200 > message: > > MDEV-16697: Fix difference between 32bit/windows and 64bit systems in allowed select nest level > > diff --git a/include/my_global.h b/include/my_global.h > index 11d17ad146a..26078114464 100644 > --- a/include/my_global.h > +++ b/include/my_global.h > @@ -984,7 +984,6 @@ typedef unsigned long my_off_t; > TODO Convert these to use Bitmap class. > */ > typedef ulonglong table_map; /* Used for table bits in join */ > -typedef ulong nesting_map; /* Used for flags of nesting constructs */ > > /* often used type names - opaque declarations */ > typedef const struct charset_info_st CHARSET_INFO; > diff --git a/sql/sql_lex.h b/sql/sql_lex.h > --- a/sql/sql_lex.h > +++ b/sql/sql_lex.h > @@ -33,6 +33,10 @@ > #include "sql_tvc.h" > #include "item.h" > > +/* Used for flags of nesting constructs */ > +#define SELECT_NESTING_MAP_SIZE 64 > +typedef Bitmap<SELECT_NESTING_MAP_SIZE> nesting_map; > + > /* YACC and LEX Definitions */ Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 2

Re: [Maria-developers] fa8879bc800: MDEV-16370 row-based binlog events (updates by primary key) can not be applied multiple times to system versioned tables
by Sergei Golubchik 06 Nov '18

06 Nov '18

Hi, Aleksey! Looking at the patch and the test, I get the impression you're fixing the wrong issue. The problem in MDEV-16370 happened when replicating from non-system-versioned table on the master to a system-versioned table on the slave. You don't have a test case for that. Your fix doesn't change the behavior for this use case. So, I don't understand what you've actually fixed. On Nov 06, Aleksey Midenkov wrote: > revision-id: fa8879bc800d2babf48d082ce8c50ffe520e12cc (versioning-1.0.5-140-gfa8879bc800) > parent(s): 28ae79650d3057ffcd849d2ef28dcd80282c85d2 > author: Aleksey Midenkov <midenok(a)gmail.com> > committer: Aleksey Midenkov <midenok(a)gmail.com> > timestamp: 2018-09-20 10:04:57 +0300 > message: > > MDEV-16370 row-based binlog events (updates by primary key) can not be applied multiple times to system versioned tables > > Rows_log_event::write_row() always overwrite historical row. > > diff --git a/mysql-test/suite/versioning/t/rpl.test b/mysql-test/suite/versioning/t/rpl.test > index a9e3af45af8..1985b17bf6d 100644 > --- a/mysql-test/suite/versioning/t/rpl.test > +++ b/mysql-test/suite/versioning/t/rpl.test > @@ -133,4 +133,24 @@ sync_slave_with_master; > connection master; > drop table t1; > > +--echo # MDEV-16370 row-based binlog events (updates by primary key) > +--echo # can not be applied multiple times to system versioned tables > +create or replace table t1 (pk int primary key, x int) with system versioning; > +insert into t1 values (1, 1); > +--sync_slave_with_master > +flush logs; > +--connection master > +update t1 set x= 2; > +--sync_slave_with_master > +select * from t1; > +--let $datadir= `select @@datadir` > +--let $f2= query_get_value(SHOW SLAVE STATUS, Relay_Log_File, 1) > +--echo # Replaying binlog... > +--exec $MYSQL_BINLOG $datadir/$f2| $MYSQL -S $SLAVE_MYSOCK test > +select * from t1; > +--connection master > + > +drop database test; > +create database test; > + > --source include/rpl_end.inc > diff --git a/sql/log_event.cc b/sql/log_event.cc > index c550adea26b..c4c04743a30 100644 > --- a/sql/log_event.cc > +++ b/sql/log_event.cc > @@ -13139,13 +13139,26 @@ Rows_log_event::write_row(rpl_group_info *rgi, > } > > // Handle INSERT. > - if (table->versioned(VERS_TIMESTAMP)) > + if (table->versioned()) > { > - ulong sec_part; > - bitmap_set_bit(table->read_set, table->vers_start_field()->field_index); > - // Check whether a row came from unversioned table and fix vers fields. > - if (table->vers_start_field()->get_timestamp(&sec_part) == 0 && sec_part == 0) > - table->vers_update_fields(); > + if (table->versioned(VERS_TIMESTAMP)) > + { > + // Set vers fields when replicating from not system-versioned table. > + ulong sec_part; > + bitmap_set_bit(table->read_set, table->vers_start_field()->field_index); > + // Check whether a row came from unversioned table and fix vers fields. > + if (table->vers_start_field()->get_timestamp(&sec_part) == 0 && sec_part == 0) > + table->vers_update_fields(); > + else > + goto overwrite_historical_row; > + } > + else > + { > +overwrite_historical_row: > + bitmap_set_bit(table->read_set, table->vers_end_field()->field_index); > + if (!overwrite && !table->vers_end_field()->is_max()) > + overwrite= true; > + } > } Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] 503e494ec6e: MDEV-16241 Assertion `inited==RND' failed in handler::ha_rnd_end()
by Sergei Golubchik 06 Nov '18

06 Nov '18

Hi, Aleksey! On Nov 06, Aleksey Midenkov wrote: > revision-id: 503e494ec6e103b23dc55f8fc8ae68ecec8b8316 (versioning-1.0.6-1-g503e494ec6e) > parent(s): 6120ae4acafb23badecc623e33797949e3f6fbbb > author: Aleksey Midenkov <midenok(a)gmail.com> > committer: Aleksey Midenkov <midenok(a)gmail.com> > timestamp: 2018-10-22 11:50:10 +0300 > message: > > MDEV-16241 Assertion `inited==RND' failed in handler::ha_rnd_end() > > Discrepancy in open indexes due to overwritten `read_partitions` upon > `ha_open()` in `ha_partition::clone()`. > > [Fixes tempesta-tech/mariadb#551] > > diff --git a/mysql-test/main/partition_innodb.result b/mysql-test/main/partition_innodb.result > index d27abc07984..c63cbc4601d 100644 > --- a/mysql-test/main/partition_innodb.result > +++ b/mysql-test/main/partition_innodb.result > @@ -925,3 +925,14 @@ test_jfg test_jfg11 > test_jfg test_jfg12#P#p1000 > test_jfg test_jfg12#P#pmax > DROP DATABASE test_jfg; > +# > +# MDEV-16241 Assertion `inited==RND' failed in handler::ha_rnd_end() > +# > +CREATE TABLE t1 (pk INT PRIMARY KEY, x INT, y INT, z INT, KEY (x), KEY (y, z)) > +WITH SYSTEM VERSIONING > +PARTITION BY SYSTEM_TIME (PARTITION p1 HISTORY, PARTITION pn CURRENT); > +INSERT INTO t1 VALUES (1, 7, 8, 9), (2, NULL, NULL, NULL), (3, NULL, NULL, NULL); > +SELECT COUNT(*) FROM t1 WHERE x IS NULL AND y IS NULL AND z IS NULL; > +COUNT(*) > +2 > +DROP TABLE t1; please, add a non-system-versioning test too. It's in one of the comments in the bug report. > diff --git a/sql/ha_partition.cc b/sql/ha_partition.cc > index 40380ffeec5..901f812a0e6 100644 > --- a/sql/ha_partition.cc > +++ b/sql/ha_partition.cc > @@ -3725,6 +3725,9 @@ void ha_partition::rebind_psi() > handler *ha_partition::clone(const char *name, MEM_ROOT *mem_root) > { > ha_partition *new_handler; > + MY_BITMAP read_partitions; > + MY_BITMAP lock_partitions; > + int error; > > DBUG_ENTER("ha_partition::clone"); > new_handler= new (mem_root) ha_partition(ht, table_share, m_part_info, > @@ -3748,9 +3751,21 @@ handler *ha_partition::clone(const char *name, MEM_ROOT *mem_root) > ALIGN_SIZE(m_ref_length)*2))) > goto err; > > - if (new_handler->ha_open(table, name, > + DBUG_ASSERT(m_tot_parts == m_part_info->read_partitions.n_bits); > + DBUG_ASSERT(m_tot_parts == m_part_info->lock_partitions.n_bits); > + my_bitmap_init(&read_partitions, NULL, m_tot_parts, FALSE); > + my_bitmap_init(&lock_partitions, NULL, m_tot_parts, FALSE); > + bitmap_copy(&read_partitions, &m_part_info->read_partitions); > + bitmap_copy(&lock_partitions, &m_part_info->lock_partitions); > + error= new_handler->ha_open(table, name, > table->db_stat, > - HA_OPEN_IGNORE_IF_LOCKED | HA_OPEN_NO_PSI_CALL)) > + HA_OPEN_IGNORE_IF_LOCKED | HA_OPEN_NO_PSI_CALL); > + bitmap_copy(&m_part_info->read_partitions, &read_partitions); > + bitmap_copy(&m_part_info->lock_partitions, &lock_partitions); > + my_bitmap_free(&read_partitions); > + my_bitmap_free(&lock_partitions); > + > + if (error) I don't think this is needed. ha_partition::open() is aware of clones, it has `if (m_is_clone_of)` to do some stuff only for clones and some only for fresh opens. Perhaps you just need to move the bitmap-resetting code to be done only for fresh opens? Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] b0d21b7: MDEV-14567 Include full name of object in message about incorrect value for column.
by Sergei Golubchik 06 Nov '18

06 Nov '18

Hi, Alexey! On Nov 06, Alexey Botchkov wrote: > > That's why I asked what temporary tables have in table->s->db and > > in table->s->table_name. What do they have there? > > 'Persistent' temporary tables (those created by CREATE TEMPORARY > TABLE) have their database/name there. > Auxiliary temporary tables created to execute queries like SELECT have > 'normal' db value and soumething like #sql_1234 as a table name. But that's fine to show, right? No NULL, no garbage. No need to treat them specially. By the way, can this error ever happen for internal temporary tables? Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] b0d21b7: MDEV-14567 Include full name of object in message about incorrect value for column.
by Sergei Golubchik 06 Nov '18

06 Nov '18

Hi, Alexey! On Nov 06, Alexey Botchkov wrote: > > >> + long_field_name(buf, sizeof(buf), table->s, field_name), > >Better change the error message to be > > Incorrect decimal value: '%s' for column %`s.%`s.%`s at row %d > > and use like > > - field_name, > > + table->s->db.str, table->s->table_name.str, field_name, > > Don't think it is better. > There are temporary tables and just Item's that launch that error. So > the 'db' and 'table' do not always defined properly. > The long_field_name() i added handles the temporary table case. Yes, I've noticed you had a specia case for temporary tables. That's why I asked what temporary tables have in table->s->db and in table->s->table_name. What do they have there? Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] dc9f113111e: MDEV-17376 Server fails to set ADD_PK_INDEX, DROP_PK_INDEX
by Sergei Golubchik 05 Nov '18

05 Nov '18

Hi, Thirunarayanan! On Nov 05, Thirunarayanan Balathandayuthapani wrote: > revision-id: dc9f113111ee4daf3964db938ef3ddd82c777328 (mariadb-10.0.37-6-gdc9f113111e) > parent(s): 38b3e52c3c40b35339b1d18c09053be416a93420 > author: Thirunarayanan Balathandayuthapani <thiru(a)mariadb.com> > committer: Thirunarayanan Balathandayuthapani <thiru(a)mariadb.com> > timestamp: 2018-11-01 21:24:03 +0530 > message: > > MDEV-17376 Server fails to set ADD_PK_INDEX, DROP_PK_INDEX > if unique index nominated as PK Subject line in git commits is always one line. Try to prefer the format as of MDEV-17376 Server fails to set ADD_PK_INDEX, DROP_PK_INDEX if unique index nominated as PK that is, MDEV-number, one space, MDEV summary verbatim, all on one line. This way it'll look good in `git log --oneline` and one can search for a bug description (and new lines won't cause the search to fail). > - Server sets the flag ADD_PK_INDEX, DROP_PK_INDEX if the unique key > nominated as Primary key. this isn't a very helpful comment, just a rephrased commit title :) Up to you, of course, it doesn't hurt. But generally, if you feel the fix it kind of obvious and you cannot come up with anything to put into commit comment body, just don't put there anything at all. > > diff --git a/mysql-test/suite/innodb/t/alter_candidate_key.test b/mysql-test/suite/innodb/t/alter_candidate_key.test > --- /dev/null > +++ b/mysql-test/suite/innodb/t/alter_candidate_key.test > @@ -0,0 +1,47 @@ > +--source include/have_innodb.inc > +--source include/have_debug.inc > +--source include/have_debug_sync.inc > + > +CREATE TABLE t1 (f1 INT NOT NULL, f2 INT NOT NULL, > + UNIQUE KEY uidx2(f1,f2), > + UNIQUE KEY uidx1(f2) > +) ENGINE=InnoDB; > + > +SET DEBUG_SYNC = 'innodb_inplace_alter_table_enter SIGNAL conc_dml WAIT_FOR go_ahead'; > +INSERT INTO t1 VALUES(1,1); > +--send > +alter table t1 change column f1 f11 int, algorithm=inplace; > +connect (con1,localhost,root,,); > +SET DEBUG_SYNC = 'now WAIT_FOR conc_dml'; > +DELETE FROM t1; > +SET DEBUG_SYNC = 'now SIGNAL go_ahead'; > +connection default; > +reap; > +CHECK TABLE t1; > +DROP TABLE t1; Could you explain the test case, please? What's going on here? I don't understand what you're fixing and what I'm reviewing, sorry :( Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] 5403baabc7e: MDEV-16849 Extending indexed VARCHAR column should be instantaneous
by Sergei Golubchik 05 Nov '18

05 Nov '18

Hi, Thirunarayanan! On Nov 05, Thirunarayanan Balathandayuthapani wrote: > revision-id: 5403baabc7ef87e301964434e3670ded06af0802 (mariadb-10.2.18-40-g5403baabc7e) > parent(s): ab1ce2204e959bea596817494e932754ab5cbe88 > author: Thirunarayanan Balathandayuthapani <thiru(a)mariadb.com> > committer: Thirunarayanan Balathandayuthapani <thiru(a)mariadb.com> > timestamp: 2018-10-26 16:49:08 +0530 > message: > > MDEV-16849 Extending indexed VARCHAR column should be instantaneous > > Analysis: > ======== > Increasing the length of the indexed varchar column is not an instant operation for > innodb. > > Fix: > === > - Introduce the new handler flag 'Alter_inplace_info::ALTER_COLUMN_INDEX_LENGTH' to > indicate the index length differs due to change of column length changes. > > - InnoDB makes the ALTER_COLUMN_INDEX_LENGTH flag as instant operation. > > This is a port of Mysql fix. > > diff --git a/sql/handler.cc b/sql/handler.cc > index da41daf2440..b1fa53fb145 100644 > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -4253,7 +4253,8 @@ handler::check_if_supported_inplace_alter(TABLE *altered_table, > Alter_inplace_info::CHANGE_CREATE_OPTION | > Alter_inplace_info::ALTER_PARTITIONED | > Alter_inplace_info::ALTER_VIRTUAL_GCOL_EXPR | > - Alter_inplace_info::ALTER_RENAME; > + Alter_inplace_info::ALTER_RENAME | > + Alter_inplace_info::ALTER_COLUMN_INDEX_LENGTH; You sure? It means that any engine by default can do ALTER_COLUMN_INDEX_LENGTH inplace, unless some other check in the engine disallows it. I'd rather not enable it for all engine, so better remove it. InnoDB has his own check inside ha_innobase::check_if_supported_inplace_alter() anyway. > /* Is there at least one operation that requires copy algorithm? */ > if (ha_alter_info->handler_flags & ~inplace_offline_operations) > diff --git a/sql/handler.h b/sql/handler.h > index ed2ef822c88..f2250699cfb 100644 > --- a/sql/handler.h > +++ b/sql/handler.h > @@ -1994,6 +1994,11 @@ class Alter_inplace_info > > static const HA_ALTER_FLAGS ALTER_DROP_CHECK_CONSTRAINT= 1ULL << 40; > > + /** > + Change in index length such that it doesn't require index rebuild. > + */ > + static const HA_ALTER_FLAGS ALTER_COLUMN_INDEX_LENGTH= 1ULL << 41; > + > /** > Create options (like MAX_ROWS) for the new version of table. > > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index 02780e7d1e0..94d35c99d0a 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -6687,18 +6687,27 @@ static bool fill_alter_inplace_info(THD *thd, > key_part < end; > key_part++, new_part++) > { > + new_field= get_field_by_index(alter_info, new_part->fieldnr); > /* > + If there is a change in index length due to column expansion > + like varchar(X) changed to varchar(X + N) and has a compatible > + packed data representation, we mark it for fast/INPLACE change > + in index definition. InnoDB supports INPLACE for this cases > + > Key definition has changed if we are using a different field or > - if the used key part length is different. It makes sense to > - check lengths first as in case when fields differ it is likely > - that lengths differ too and checking fields is more expensive > - in general case. > + if the user key part length is different. > */ > - if (key_part->length != new_part->length) > + if (key_part->length != new_part->length && > + (ha_alter_info->alter_info->flags & Alter_info::ALTER_CHANGE_COLUMN) && why is that? Can it happen that ALTER_CHANGE_COLUMN flag is not set? > + (key_part->field->is_equal((Create_field*) new_field) > + == IS_EQUAL_PACK_LENGTH)) > + { > + ha_alter_info->handler_flags |= > + Alter_inplace_info::ALTER_COLUMN_INDEX_LENGTH; > + } > + else if (key_part->length != new_part->length) > goto index_changed; > > - new_field= get_field_by_index(alter_info, new_part->fieldnr); > - > /* > For prefix keys KEY_PART_INFO::field points to cloned Field > object with adjusted length. So below we have to check field Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] b0d21b7: MDEV-14567 Include full name of object in message about incorrect value for column.
by Sergei Golubchik 05 Nov '18

05 Nov '18

Hi, Alexey! On Nov 04, Alexey Botchkov wrote: > revision-id: b0d21b7832cea92522d0f8d280ceb8b13b335cbf (mariadb-10.2.18-56-gb0d21b7) > parent(s): 1c6b982e02eeaa75bb6c2f2a3c2b64491dd6d3c8 > committer: Alexey Botchkov > timestamp: 2018-11-04 09:14:40 +0400 > message: > > MDEV-14567 Include full name of object in message about incorrect value for column. Wrong MDEV number again. It should be MDEV-14576 > > Put the full field name in the ER_TRUNCATED_WRONG_VALUE_FOR_FIELD > error message. > > +static char *long_field_name(char *buf, size_t buf_len, > + const TABLE_SHARE *s, const char *field_name) > +{ > + bool is_temp= !s|| > + (tmp_file_prefix_length < s->table_name.length && > + memcmp(s->table_name.str, > + tmp_file_prefix, tmp_file_prefix_length) == 0); > + > + if (is_temp) > + strxnmov(buf, buf_len, "{temporary}.", field_name, NullS); > + else > + strxnmov(buf, buf_len, > + s->db.str, ".", s->table_name.str, ".", field_name, NullS); > + > + return buf; > +} > + > + > int Field_geom::store(const char *from, uint length, CHARSET_INFO *cs) > { > if (!length) > @@ -8620,10 +8638,11 @@ int Field_geom::store(const char *from, uint length, CHARSET_INFO *cs) > geom_type != Field::GEOM_GEOMETRYCOLLECTION && > (uint32) geom_type != wkb_type) > { > + char buf[193]; > my_error(ER_TRUNCATED_WRONG_VALUE_FOR_FIELD, MYF(0), > Geometry::ci_collection[geom_type]->m_name.str, > Geometry::ci_collection[wkb_type]->m_name.str, > - field_name, > + long_field_name(buf, sizeof(buf), table->s, field_name), > (ulong) table->in_use->get_stmt_da()-> > current_row_for_warning()); > goto err_exit; Better change the error message to be Incorrect decimal value: '%s' for column %`s.%`s.%`s at row %d and use like my_error(ER_TRUNCATED_WRONG_VALUE_FOR_FIELD, MYF(0), Geometry::ci_collection[geom_type]->m_name.str, Geometry::ci_collection[wkb_type]->m_name.str, - field_name, + table->s->db.str, table->s->table_name.str, field_name, (ulong) table->in_use->get_stmt_da()-> current_row_for_warning()); What would that do for temporary tables? Anyway, the correct quoting is still `db`.`table`.`column`, not 'db.table.column'. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0