- developers - lists.mariadb.org

Re: [Maria-developers] 8ffded0: MDEV-9312: storage engine not enforced during galera cluster replication
by Sergei Golubchik 24 Sep '16

24 Sep '16

Hi, Nirbhay! On Sep 24, Nirbhay Choubey wrote: > diff --git a/sql/sql_plugin.cc b/sql/sql_plugin.cc > index 60248f3..99b3311 100644 > --- a/sql/sql_plugin.cc > +++ b/sql/sql_plugin.cc > @@ -3116,7 +3118,12 @@ void plugin_thdvar_init(THD *thd) > thd->variables.dynamic_variables_size= 0; > thd->variables.dynamic_variables_ptr= 0; > > - if (IF_WSREP((!WSREP(thd) || !thd->wsrep_applier),1)) > + /* > + The following initializations are deferred for some wsrep system threads > + created during startup as they could be created even before LOCK_plugin > + and plugins are initialized. > + */ > + if (IF_WSREP((plugins_are_initialized),1)) why do you need that? I think you can safely call plugin_thdvar_init() twice, so you don't really need to skip the first invocation. It's that principle that the one reponsible for the problem should pay for it, others shouldn't suffer :) In ha_maria:implicit_commit, it was Aria's hack, and that if() was added inside Aria. Here is purely wsrep problem, putting an if() on a common code path for all threads looks just wrong. > { > mysql_mutex_lock(&LOCK_plugin); > thd->variables.table_plugin= Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] Unique Blob Index for lower transaction isolation levels
by Sachin Setia 22 Sep '16

22 Sep '16

Hi Sergei, Actually I was thinking a different way of implementing this. in function ha_write_row , after it calls storage engine write, we can check for inserted values and see if it is inserted more than once. Of course, it can be I/O intensive (or it may not be , because data should be in buffer ). We have to do this only when there is multiple clients doing inserts in same table. But I am not sure if there is any way to find this. What do you think ? Regards sachin

1 0

Re: [Maria-developers] [Commits] 949e2ae: MDEV-10315 - Online ALTER TABLE may get stuck in tdc_remove_table
by Sergei Golubchik 21 Sep '16

21 Sep '16

Hi, Sergey! On Jul 01, Sergey Vojtovich wrote: > revision-id: 949e2aede909686053ab83ebf2741901be416a07 (mariadb-10.0.26-4-g949e2ae) > parent(s): 0fdb17e6c3f50ae22eb97b6363bcbd8b0cd9e040 > committer: Sergey Vojtovich > timestamp: 2016-07-01 13:57:18 +0400 > message: > > MDEV-10315 - Online ALTER TABLE may get stuck in tdc_remove_table > > There was race condition between online ALTER TABLE and statements performing > TABLE_SHARE release without marking it flushed (e.g. in case of table cache > overflow, SET @@global.table_open_cache, manager thread purgine table cache). > > The reason was missing mysql_cond_broadcast(). ok to push Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] MDEV-10425 Assertion `collation.derivation == DERIVATION_IMPLICIT' failed in Item_func_conv_charset::fix_length_and_dec()
by Alexander Barkov 21 Sep '16

21 Sep '16

Hello Sergei, Please review. Thanks!

2 1

Re: [Maria-developers] [Commits] a288cb6: MDEV-8320 Allow index usage for DATE(datetime_column) = const.
by Sergey Petrunia 20 Sep '16

20 Sep '16

Hi Alexey, The patch doesn't have any testcase. Did you forget to add them? On Tue, Sep 20, 2016 at 01:22:19PM +0400, Alexey Botchkov wrote: > revision-id: a288cb698195b1e57abbb426f1cc9a804d65ff45 (mariadb-10.1.8-262-ga288cb6) > parent(s): cb575abf76be82553b9c1c12c9112cbc6f53a547 > committer: Alexey Botchkov > timestamp: 2016-09-20 13:19:08 +0400 > message: > > MDEV-8320 Allow index usage for DATE(datetime_column) = const. > > create_reverse_func() method added so functions can specify how > to unpack field argument out of it. > opt_arguments added to Item_bool_func2 so it can have different > arguments for the optimizer and the calcualtion itself. > > --- > sql/item.h | 8 +++++ > sql/item_cmpfunc.h | 52 ++++++++++++------------------- > sql/item_func.h | 5 +++ > sql/item_timefunc.cc | 87 ++++++++++++++++++++++++++++++++++++++++++++++++++++ > sql/item_timefunc.h | 49 ++++++++++++++++++++++++++++- > sql/opt_range.cc | 48 +++++++++++++++++++++++++++++ > sql/sql_select.cc | 70 ++++++++++++++++++++++++++++++++++++++---- > 7 files changed, 279 insertions(+), 40 deletions(-) > > diff --git a/sql/item.h b/sql/item.h > index 5b82548..200e2e0 100644 > --- a/sql/item.h > +++ b/sql/item.h > @@ -1212,6 +1212,14 @@ class Item: public Value_source, > { > return; > } > + virtual bool add_extra_key_fields(THD *thd, > + JOIN *join, KEY_FIELD **key_fields, > + uint *and_level, > + table_map usable_tables, > + SARGABLE_PARAM **sargables) > + { > + return false; > + } > /* > Make a select tree for all keys in a condition or a condition part > @param param Context > diff --git a/sql/item_cmpfunc.h b/sql/item_cmpfunc.h > index 6d432bd..516bb07 100644 > --- a/sql/item_cmpfunc.h > +++ b/sql/item_cmpfunc.h > @@ -136,6 +136,14 @@ class Item_bool_func :public Item_int_func > { > protected: > /* > + Some functions modify it's arguments for the optimizer. > + So for example the condition 'Func(fieldX) = constY' turned into > + 'fieldX = cnuR(constY)' so that optimizer can use an index on fieldX. > + */ > + Item *opt_args[3]; > + uint opt_arg_count; > + > + /* > Build a SEL_TREE for a simple predicate > @param param PARAM from SQL_SELECT::test_quick_select > @param field field in the predicate > @@ -189,12 +197,12 @@ class Item_bool_func :public Item_int_func > KEY_PART *key_part, > Item_func::Functype type, Item *value); > public: > - Item_bool_func(THD *thd): Item_int_func(thd) {} > - Item_bool_func(THD *thd, Item *a): Item_int_func(thd, a) {} > - Item_bool_func(THD *thd, Item *a, Item *b): Item_int_func(thd, a, b) {} > - Item_bool_func(THD *thd, Item *a, Item *b, Item *c): Item_int_func(thd, a, b, c) {} > - Item_bool_func(THD *thd, List<Item> &list): Item_int_func(thd, list) { } > - Item_bool_func(THD *thd, Item_bool_func *item) :Item_int_func(thd, item) {} > + Item_bool_func(THD *thd): Item_int_func(thd), opt_arg_count(0) {} > + Item_bool_func(THD *thd, Item *a): Item_int_func(thd, a), opt_arg_count(0) {} > + Item_bool_func(THD *thd, Item *a, Item *b): Item_int_func(thd, a, b), opt_arg_count(0) {} > + Item_bool_func(THD *thd, Item *a, Item *b, Item *c): Item_int_func(thd, a, b, c), opt_arg_count(0) {} > + Item_bool_func(THD *thd, List<Item> &list): Item_int_func(thd, list), opt_arg_count(0) { } > + Item_bool_func(THD *thd, Item_bool_func *item) :Item_int_func(thd, item), opt_arg_count(0) {} > bool is_bool_type() { return true; } > virtual CHARSET_INFO *compare_collation() const { return NULL; } > void fix_length_and_dec() { decimals=0; max_length=1; } > @@ -436,33 +444,7 @@ class Item_bool_func2_with_rev :public Item_bool_func2 > Item_bool_func2_with_rev(THD *thd, Item *a, Item *b): > Item_bool_func2(thd, a, b) { } > virtual enum Functype rev_functype() const= 0; > - SEL_TREE *get_mm_tree(RANGE_OPT_PARAM *param, Item **cond_ptr) > - { > - DBUG_ENTER("Item_bool_func2_with_rev::get_mm_tree"); > - DBUG_ASSERT(arg_count == 2); > - SEL_TREE *ftree; > - /* > - Even if get_full_func_mm_tree_for_args(param, args[0], args[1]) will not > - return a range predicate it may still be possible to create one > - by reversing the order of the operands. Note that this only > - applies to predicates where both operands are fields. Example: A > - query of the form > - > - WHERE t1.a OP t2.b > - > - In this case, args[0] == t1.a and args[1] == t2.b. > - When creating range predicates for t2, > - get_full_func_mm_tree_for_args(param, args[0], args[1]) > - will return NULL because 'field' belongs to t1 and only > - predicates that applies to t2 are of interest. In this case a > - call to get_full_func_mm_tree_for_args() with reversed operands > - may succeed. > - */ > - if (!(ftree= get_full_func_mm_tree_for_args(param, args[0], args[1])) && > - !(ftree= get_full_func_mm_tree_for_args(param, args[1], args[0]))) > - ftree= Item_func::get_mm_tree(param, cond_ptr); > - DBUG_RETURN(ftree); > - } > + SEL_TREE *get_mm_tree(RANGE_OPT_PARAM *param, Item **cond_ptr); > }; > > > @@ -504,6 +486,10 @@ class Item_bool_rowready_func2 :public Item_bool_func2_with_rev > Item_bool_func2::cleanup(); > cmp.cleanup(); > } > + bool add_extra_key_fields(THD *thd, > + JOIN *join, KEY_FIELD **key_fields, > + uint *and_level, table_map usable_tables, > + SARGABLE_PARAM **sargables); > void add_key_fields(JOIN *join, KEY_FIELD **key_fields, > uint *and_level, table_map usable_tables, > SARGABLE_PARAM **sargables) > diff --git a/sql/item_func.h b/sql/item_func.h > index ca7c481..1f802db 100644 > --- a/sql/item_func.h > +++ b/sql/item_func.h > @@ -358,6 +358,11 @@ class Item_func :public Item_func_or_sum > - or replaced to an Item_int_with_ref > */ > bool setup_args_and_comparator(THD *thd, Arg_comparator *cmp); > + virtual bool create_reverse_func(enum Functype cmp_type, > + THD *thd, Item *r_arg, uint *a_cnt, Item** a) > + { > + return false; > + } > }; > > > diff --git a/sql/item_timefunc.cc b/sql/item_timefunc.cc > index 41dc967..3124444 100644 > --- a/sql/item_timefunc.cc > +++ b/sql/item_timefunc.cc > @@ -2569,6 +2569,39 @@ bool Item_date_typecast::get_date(MYSQL_TIME *ltime, ulonglong fuzzy_date) > } > > > +bool Item_date_typecast::create_reverse_func(enum Functype cmp_type, > + THD *thd, Item *r_arg, uint *a_cnt, Item** a) > +{ > + switch (cmp_type) > + { > + case GT_FUNC: > + case LE_FUNC: > + (*a_cnt)++; > + if (!(a[0]= new (thd->mem_root) Item_func_day_end(thd, r_arg)) || > + a[0]->fix_fields(thd, a+1)) > + return true; > + break; > + case LT_FUNC: > + case GE_FUNC: > + (*a_cnt)++; > + if (!(a[0]= new (thd->mem_root) Item_func_day_begin(thd, r_arg)) || > + a[0]->fix_fields(thd, a+1)) > + return true; > + break; > + case EQ_FUNC: > + (*a_cnt)+= 2; > + if (!(a[0]= new (thd->mem_root) Item_func_day_begin(thd, r_arg)) || > + a[0]->fix_fields(thd, a+1)) > + return true; > + if (!(a[1]= new (thd->mem_root) Item_func_day_end(thd, r_arg)) || > + a[1]->fix_fields(thd, a+2)) > + return true; > + default:; > + } > + return false; > +} > + > + > bool Item_datetime_typecast::get_date(MYSQL_TIME *ltime, ulonglong fuzzy_date) > { > fuzzy_date |= sql_mode_for_dates(current_thd); > @@ -3240,3 +3273,57 @@ bool Item_func_last_day::get_date(MYSQL_TIME *ltime, ulonglong fuzzy_date) > ltime->time_type= MYSQL_TIMESTAMP_DATE; > return (null_value= 0); > } > + > + > +bool Item_func_day_begin::get_date(MYSQL_TIME *res, ulonglong fuzzy_date) > +{ > + if (get_arg0_date(res, fuzzy_date)) > + return (null_value=1); > + > + res->second_part= res->second= res->minute= res->hour= 0; > + res->time_type= MYSQL_TIMESTAMP_DATETIME; > + > + return null_value= 0; > +} > + > + > +bool Item_func_day_end::get_date(MYSQL_TIME *res, ulonglong fuzzy_date) > +{ > + if (get_arg0_date(res, fuzzy_date)) > + return (null_value=1); > + > + res->hour= 23; > + res->second= res->minute= 59; > + res->second_part= 999999; > + res->time_type= MYSQL_TIMESTAMP_DATETIME; > + return null_value= 0; > +} > + > + > +bool Item_func_year_begin::get_date(MYSQL_TIME *res, ulonglong fuzzy_date) > +{ > + res->year= args[0]->val_int(); > + if ((null_value= args[0]->null_value || res->year >= 9999)) > + return 0; > + > + res->day= res->month= 1; > + res->second_part= res->second= res->minute= res->hour= 0; > + res->time_type= MYSQL_TIMESTAMP_DATETIME; > + return null_value= 0; > +} > + > + > +bool Item_func_year_end::get_date(MYSQL_TIME *res, ulonglong fuzzy_date) > +{ > + res->year= args[0]->val_int(); > + if ((null_value= args[0]->null_value || res->year >= 9999)) > + return 0; > + > + res->month= 12; > + res->day= 31; > + res->hour= 23; > + res->second= res->minute= 59; > + res->second_part= 999999; > + res->time_type= MYSQL_TIMESTAMP_DATETIME; > + return null_value= 0; > +} > diff --git a/sql/item_timefunc.h b/sql/item_timefunc.h > index a853c63..b4f64ef 100644 > --- a/sql/item_timefunc.h > +++ b/sql/item_timefunc.h > @@ -745,7 +745,7 @@ class Item_func_now_local :public Item_func_now > { > public: > Item_func_now_local(THD *thd, uint dec): Item_func_now(thd, dec) {} > - const char *func_name() const { return "now"; } > + const char *func_name() const { return "day_start"; } > virtual void store_now_in_TIME(THD *thd, MYSQL_TIME *now_time); > virtual enum Functype functype() const { return NOW_FUNC; } > Item *get_copy(THD *thd, MEM_ROOT *mem_root) > @@ -1074,6 +1074,8 @@ class Item_date_typecast :public Item_temporal_typecast > bool get_date(MYSQL_TIME *ltime, ulonglong fuzzy_date); > const char *cast_type() const { return "date"; } > enum_field_types field_type() const { return MYSQL_TYPE_DATE; } > + bool create_reverse_func(enum Functype cmp_type, > + THD *thd, Item *r_arg, uint *a_cnt, Item** a); > Item *get_copy(THD *thd, MEM_ROOT *mem_root) > { return get_item_copy<Item_date_typecast>(thd, mem_root, this); } > }; > @@ -1268,4 +1270,49 @@ class Item_func_last_day :public Item_datefunc > { return get_item_copy<Item_func_last_day>(thd, mem_root, this); } > }; > > + > +class Item_func_day_begin :public Item_datetimefunc > +{ > +public: > + Item_func_day_begin(THD *thd, Item *a): Item_datetimefunc(thd, a) {} > + const char *func_name() const { return "day_begin"; } > + bool get_date(MYSQL_TIME *res, ulonglong fuzzy_date); > + Item *get_copy(THD *thd, MEM_ROOT *mem_root) > + { return get_item_copy<Item_func_day_begin>(thd, mem_root, this); } > +}; > + > + > +class Item_func_day_end :public Item_datetimefunc > +{ > +public: > + Item_func_day_end(THD *thd, Item *a): Item_datetimefunc(thd, a) {} > + const char *func_name() const { return "day_end"; } > + bool get_date(MYSQL_TIME *res, ulonglong fuzzy_date); > + Item *get_copy(THD *thd, MEM_ROOT *mem_root) > + { return get_item_copy<Item_func_day_end>(thd, mem_root, this); } > +}; > + > + > +class Item_func_year_begin :public Item_datetimefunc > +{ > +public: > + Item_func_year_begin(THD *thd, Item *a): Item_datetimefunc(thd, a) {} > + const char *func_name() const { return "year_begin"; } > + bool get_date(MYSQL_TIME *res, ulonglong fuzzy_date); > + Item *get_copy(THD *thd, MEM_ROOT *mem_root) > + { return get_item_copy<Item_func_year_begin>(thd, mem_root, this); } > +}; > + > + > +class Item_func_year_end :public Item_datetimefunc > +{ > +public: > + Item_func_year_end(THD *thd, Item *a): Item_datetimefunc(thd, a) {} > + const char *func_name() const { return "year_end"; } > + bool get_date(MYSQL_TIME *res, ulonglong fuzzy_date); > + Item *get_copy(THD *thd, MEM_ROOT *mem_root) > + { return get_item_copy<Item_func_year_end>(thd, mem_root, this); } > +}; > + > + > #endif /* ITEM_TIMEFUNC_INCLUDED */ > diff --git a/sql/opt_range.cc b/sql/opt_range.cc > index 3ea9f4e..e533608 100644 > --- a/sql/opt_range.cc > +++ b/sql/opt_range.cc > @@ -6998,6 +6998,54 @@ SEL_TREE *Item_bool_func::get_ne_mm_tree(RANGE_OPT_PARAM *param, > } > > > +SEL_TREE *Item_bool_func2_with_rev::get_mm_tree(RANGE_OPT_PARAM *param, Item **cond_ptr) > +{ > + DBUG_ENTER("Item_bool_func2_with_rev::get_mm_tree"); > + DBUG_ASSERT(arg_count == 2); > + SEL_TREE *ftree; > + /* > + Even if get_full_func_mm_tree_for_args(param, args[0], args[1]) will not > + return a range predicate it may still be possible to create one > + by reversing the order of the operands. Note that this only > + applies to predicates where both operands are fields. Example: A > + query of the form > + > + WHERE t1.a OP t2.b > + > + In this case, args[0] == t1.a and args[1] == t2.b. > + When creating range predicates for t2, > + get_full_func_mm_tree_for_args(param, args[0], args[1]) > + will return NULL because 'field' belongs to t1 and only > + predicates that applies to t2 are of interest. In this case a > + call to get_full_func_mm_tree_for_args() with reversed operands > + may succeed. > + */ > + if (opt_arg_count) > + { > + if (opt_arg_count == 2) > + { > + ftree= get_full_func_mm_tree_for_args(param, opt_args[0], opt_args[1]); > + } > + else if (opt_arg_count == 3) > + { > + Field *f= ((Item_field *) opt_args[0])->field; > + ftree= get_mm_parts(param, f, Item_func::GE_FUNC, opt_args[1]); > + if (ftree) > + { > + ftree= tree_and(param, ftree, > + get_mm_parts(param, f, > + Item_func::LE_FUNC, opt_args[2])); > + } > + } > + } > + if (!ftree && > + !(ftree= get_full_func_mm_tree_for_args(param, args[0], args[1])) && > + !(ftree= get_full_func_mm_tree_for_args(param, args[1], args[0]))) > + ftree= Item_func::get_mm_tree(param, cond_ptr); > + DBUG_RETURN(ftree); > +}; > + > + > SEL_TREE *Item_func_between::get_func_mm_tree(RANGE_OPT_PARAM *param, > Field *field, Item *value) > { > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index aa08420..51f6204 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -4833,6 +4833,30 @@ is_local_field (Item *field) > } > > > +static Item_field *get_local_field (Item *field) > +{ > + Item *ri= field->real_item(); > + return (ri->type() == Item::FIELD_ITEM > + && !(field->used_tables() & OUTER_REF_TABLE_BIT) > + && !((Item_field *)ri)->get_depended_from()) ? (Item_field *) ri : 0; > +} > + > + > +static Item_field *field_in_sargable_func(Item *fn) > +{ > + fn= fn->real_item(); > + > + if (fn->type() == Item::FUNC_ITEM && > + strcmp(((Item_func *)fn)->func_name(), "cast_as_date") == 0) > + > + { > + Item_date_typecast *dt= (Item_date_typecast *) fn; > + return get_local_field(dt->arguments()[0]); > + } > + return 0; > +} > + > + > /* > In this and other functions, and_level is a number that is ever-growing > and is different for the contents of every AND or OR clause. For example, > @@ -5036,6 +5060,25 @@ Item_func_like::add_key_fields(JOIN *join, KEY_FIELD **key_fields, > } > > > +bool Item_bool_rowready_func2::add_extra_key_fields(THD *thd, > + JOIN *join, KEY_FIELD **key_fields, > + uint *and_level, > + table_map usable_tables, > + SARGABLE_PARAM **sargables) > +{ > + Item_field *f; > + if ((f= field_in_sargable_func(args[0])) && args[1]->const_item()) > + { > + opt_arg_count= 1; > + opt_args[0]= f; > + if (((Item_func *) args[0])->create_reverse_func( > + functype(), thd, args[1], &opt_arg_count, opt_args+1)) > + return true; > + } > + return false; > +} > + > + > void > Item_bool_func2::add_key_fields_optimize_op(JOIN *join, KEY_FIELD **key_fields, > uint *and_level, > @@ -5043,19 +5086,28 @@ Item_bool_func2::add_key_fields_optimize_op(JOIN *join, KEY_FIELD **key_fields, > SARGABLE_PARAM **sargables, > bool equal_func) > { > + Item_field *f; > /* If item is of type 'field op field/constant' add it to key_fields */ > - if (is_local_field(args[0])) > + if ((f= get_local_field(args[0]))) > { > - add_key_equal_fields(join, key_fields, *and_level, this, > - (Item_field*) args[0]->real_item(), equal_func, > + add_key_equal_fields(join, key_fields, *and_level, this, f, equal_func, > args + 1, 1, usable_tables, sargables); > } > - if (is_local_field(args[1])) > + else if ((f= get_local_field(args[1]))) > { > - add_key_equal_fields(join, key_fields, *and_level, this, > - (Item_field*) args[1]->real_item(), equal_func, > + add_key_equal_fields(join, key_fields, *and_level, this, f, equal_func, > args, 1, usable_tables, sargables); > } > + if (opt_arg_count == 2) > + { > + add_key_equal_fields(join, key_fields, *and_level, this, opt_args[0], > + equal_func, opt_args+1, 1, usable_tables, sargables); > + } > + else if (opt_arg_count == 3) > + { > + add_key_equal_fields(join, key_fields, *and_level, this, opt_args[0], > + false, opt_args+1, 2, usable_tables, sargables); > + } > } > > > @@ -5521,8 +5573,14 @@ update_ref_and_keys(THD *thd, DYNAMIC_ARRAY *keyuse,JOIN_TAB *join_tab, > if (cond) > { > KEY_FIELD *saved_field= field; > + > + if (cond->add_extra_key_fields(thd, join_tab->join, &end, &and_level, > + normal_tables, sargables)) > + DBUG_RETURN(TRUE); > + > cond->add_key_fields(join_tab->join, &end, &and_level, normal_tables, > sargables); > + > for (; field != end ; field++) > { > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] 8cd65cd: MDEV-9143 JSON_xxx functions.
by Sergei Golubchik 15 Sep '16

15 Sep '16

Hi, Alexey! On Sep 13, Alexey Botchkov wrote: > revision-id: 8cd65cd3e31aa6129573c6fbba9feb06c714e00c (mariadb-10.1.8-247-g8cd65cd) > parent(s): 76a0ed2e03c6ae1ff791534e742c812d5e83ba63 > committer: Alexey Botchkov > timestamp: 2016-09-13 15:48:03 +0400 > message: > > MDEV-9143 JSON_xxx functions. > > Library added to operate JSON format. > SQL functions to handle JSON data added: > required by SQL standard: > JSON_VALUE > JSON_QUERY > JSON_EXISTS > MySQL functions: > JSON_VALID > JSON_ARRAY > JSON_ARRAY_APPEND > JSON_CONTAINS_PATH > JSON_EXTRACT > JSON_OBJECT > JSON_QUOTE > JSON_MERGE > Some MySQL functions still missing, but no specific > difficulties to implement them too in same manner. > JSON_TABLE of the SQL standard is missing as it's not > clean how and whether we'd like to implement it. Good comment, thanks. > include/CMakeLists.txt | 1 + > include/json_lib.h | 338 ++++++++++ > mysql-test/r/func_json.result | 99 +++ > mysql-test/t/func_json.test | 44 ++ > sql/CMakeLists.txt | 2 +- > sql/item.h | 15 + > sql/item_create.cc | 351 ++++++++++ > sql/item_jsonfunc.cc | 898 ++++++++++++++++++++++++++ > sql/item_jsonfunc.h | 240 +++++++ > sql/item_xmlfunc.cc | 21 +- > sql/item_xmlfunc.h | 5 - > sql/sql_yacc.yy | 4 +- > strings/CMakeLists.txt | 2 +- > strings/ctype-ucs2.c | 19 +- > strings/json_lib.c | 1426 +++++++++++++++++++++++++++++++++++++++++ > 15 files changed, 3433 insertions(+), 32 deletions(-) Is the json library code the same as in the previous commit? Or should I look at it again? And *please* add unit tests for the json library. > diff --git a/include/json_lib.h b/include/json_lib.h > new file mode 100644 > index 0000000..592096f > --- /dev/null > +++ b/include/json_lib.h ... > + do > + { > + // The parser has read next piece of JSON > + // and set fields of j_eng structure accordingly. > + // So let's see what we have: Oops. Didn't notice that earlier, sorry. No C++ comments in C files, please, some older compilers don't like that. > diff --git a/mysql-test/r/func_json.result b/mysql-test/r/func_json.result > new file mode 100644 > index 0000000..1ebe0b2 > --- /dev/null > +++ b/mysql-test/r/func_json.result > @@ -0,0 +1,99 @@ > +select json_value('{"key1":[1,2,3]}', '$.key1'); > +json_value('{"key1":[1,2,3]}', '$.key1') > +NULL I'd expect an error here, but the standard explictly says: 5) If <JSON value error behavior> is not specified, then NULL ON ERROR is implicit. that is, returning NULL on errors is correct. Would be nice to have a comment about it in the code (around Item_func_json_value::val_str), otherwise someone might change this behavior to return an error. > +select json_value('{"key1": [1,2,3], "key1":123}', '$.key1'); > +json_value('{"key1": [1,2,3], "key1":123}', '$.key1') > +123 Eh... Okay, so you show the *last* key, right? SQL standard says it's implementation defined, so I suppose that's fine. > +select json_object("ki", 1, "mi", "ya"); > +json_object("ki", 1, "mi", "ya") > +{"ki": 1, "mi": "ya"} add tests for json constructing functions (json_object, json_array_append, etc) with key/values that need quoting or escaping. I mean, tests to show that json_object (for example) it more complex than concat(). > diff --git a/mysql-test/t/func_json.test b/mysql-test/t/func_json.test > new file mode 100644 > index 0000000..a5b84e0c > --- /dev/null > +++ b/mysql-test/t/func_json.test ... > + > +select json_merge('string', 123); how comes this test is not in the result file? > diff --git a/sql/item_jsonfunc.h b/sql/item_jsonfunc.h > new file mode 100644 > index 0000000..aff19ee > --- /dev/null > +++ b/sql/item_jsonfunc.h > +class json_path_with_flags > +{ > +public: > + json_path_t p; > + bool constant; > + bool parsed; > + json_path_step_t *cur_step; > + void set_constant_flag(bool s_constant) > + { > + constant= s_constant; > + parsed= FALSE; why set_constant_flag() resets 'parsed'? Better to set parsed=false in the constructor. > + } > +}; > + > diff --git a/sql/item_jsonfunc.cc b/sql/item_jsonfunc.cc > new file mode 100644 > index 0000000..5649989 > --- /dev/null > +++ b/sql/item_jsonfunc.cc ... > +/* > + Appends ASCII string to the String object taking it's charset in > + consideration. > +*/ > +static int st_append_ascii(String *s, const char *ascii, uint ascii_len) Why did you do that, instead of using String::append() ? I'd use String::append(), and if that would happen to be too slow, I'd implement String::append_ascii(), not something json-specific. ... > +/* > + Appends arbitrary String to the JSON string taking charsets in > + consideration. > +*/ > +static int st_append_escaped(String *s, const String *a) > +{ > + /* > + In the worst case one character from the 'a' string > + turns into '\uXXXX\uXXXX' which is 12. how comes? add an example, please. Like "for example, character x'1234' in the charset A becomes '\u1234\u5678' if JSON string is in the charset B" > + */ > + int str_len= a->length() * 12 * s->charset()->mbmaxlen / > + a->charset()->mbminlen; > + if (!s->reserve(str_len, 1024) && > + (str_len= > + json_escape(a->charset(), (uchar *) a->ptr(), (uchar *)a->end(), > + s->charset(), > + (uchar *) s->end(), (uchar *)s->end() + str_len)) > 0) > + { > + s->length(s->length() + str_len); > + return 0; > + } > + > + return a->length(); > +} ... > +longlong Item_func_json_exists::val_int() > +{ > + json_engine_t je; > + String *js= args[0]->val_str(&tmp_js); > + > + if (!path.parsed) > + { > + String *s_p= args[1]->val_str(&tmp_path); > + if (s_p && > + json_path_setup(&path.p, s_p->charset(), (const uchar *) s_p->ptr(), > + (const uchar *) s_p->ptr() + s_p->length())) 1. why wouldn't you do it in fix_length_and_dec for constant paths? 2. please add tests when path is not constant > + goto err_return; > + path.parsed= path.constant; > + } > + > + if ((null_value= args[0]->null_value || args[1]->null_value)) > + { > + null_value= 1; > + return 0; > + } > + > + null_value= 0; > + json_scan_start(&je, js->charset(),(const uchar *) js->ptr(), > + (const uchar *) js->ptr() + js->length()); > + > + path.cur_step= path.p.steps; > + if (json_find_value(&je, &path.p, &path.cur_step)) > + { > + if (je.s.error) > + goto err_return; > + return 0; > + } > + > + return 1; > + > +err_return: > + null_value= 1; > + return 0; > +} ... > +longlong Item_func_json_contains_path::val_int() > +{ > + String *js= args[0]->val_str(&tmp_js); > + json_engine_t je; > + uint n_arg; > + longlong result; > + > + if ((null_value= args[0]->null_value)) > + return 0; ... > + result= !mode_one; > + for (n_arg=2; n_arg < arg_count; n_arg++) > + { > + json_path_with_flags *c_path= paths + n_arg - 2; > + if (!c_path->parsed) > + { > + String *s_p= args[n_arg]->val_str(tmp_paths+(n_arg-2)); > + if (s_p && > + json_path_setup(&c_path->p,s_p->charset(),(const uchar *) s_p->ptr(), > + (const uchar *) s_p->ptr() + s_p->length())) > + goto error; > + c_path->parsed= TRUE; eh? not c_path->parsed= c_path->constant ? > + } > + > + json_scan_start(&je, js->charset(),(const uchar *) js->ptr(), > + (const uchar *) js->ptr() + js->length()); > + > + c_path->cur_step= c_path->p.steps; > + if (json_find_value(&je, &c_path->p, &c_path->cur_step)) > + { > + /* Path wasn't found. */ > + if (je.s.error) > + goto error; > + > + if (!mode_one) > + { > + result= 0; > + break; > + } > + } > + else if (mode_one) > + { > + result= 1; > + break; > + } > + } > + > + > + return result; > + > +error: > + null_value= 1; > + return 0; > +} Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] Questions on recent window function code changes
by Sergey Petrunia 14 Sep '16

14 Sep '16

Hi Vicențiu, > item_windowfunc.cc|168| peer_tracker = new Group_bound_tracker(thd, window_spec->order_list); > item_windowfunc.cc|176| peer_tracker = new Group_bound_tracker(thd, window_spec->order_list); > item_windowfunc.cc|218| peer_tracker = new Group_bound_tracker(thd, window_spec->order_list); Coding style violation: Use peer_tracker=. > sql_window.cc: > // TODO check if this can be placed outside the loop. > err= tbl->file->ha_update_row(tbl->record[1], tbl->record[0]); Yes, it should be. Can you try it? (and if it doesn't work that's surprising and we could discuss why..) > grep fech_prev_row *.h *.cc > sql_window.cc|633| bool fetch_prev_row() This function is not used anymore? > static bool is_computed_with_remove(Item_sum::Sumfunctype sum_func) What is the difference between this function and Item_sum::supports_removal ? Does one imply/require the other? > sql_window.h > /* > This handles computation of one window function. > > Currently, we make a spearate filesort() call for each window function. > */ > > class Window_func_runner : public Sql_alloc Both statements in the comment are not true anymore, right? Can you update the comment? > sql_window.cc > /* > Regular frame cursors add or remove values from the sum functions they > manage. By calling this method, they will only perform the required > movement within the table, but no adding/removing will happen. > */ > void set_no_action() > { > perform_no_action= true; > } Is there any difference between perform_no_action=true and Frame_cursor::sum_functions being an empty list? Please clarify. > sql_window.cc > /* > A class that owns cursor objects associated with a specific window function. > */ > class Cursor_manager > But as I understand, your recent changes make window functions share the frame cursors. That is, window frame has its cursors, and all window functions that are sharing the frame, share the cursors. Is Cursor_manager still needed? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] MariaDB update deficiency
by Sachin Setia 14 Sep '16

14 Sep '16

Hello Everyone! This thread is about solving this issue https://jira.mariadb.org/browse/MDEV-10220. I made a google doc https://docs.google.com/document/d/1sor3nVGz2v9Jm4-MDDFfMSn7d7xOip7Qio_T0Aq… In which I am explaining my idea , which i think should solve this problem. Please have a look and tell me what you think ? Regards sachin

1 0

Re: [Maria-developers] [Commits] 49b2502: Fix assertion/hang in read_init_file()
by Kristian Nielsen 12 Sep '16

12 Sep '16

Monty, My latest push uncovered a problem in the 10.2 tree; however the problem was there before, just not triggered by the testsuite. I pushed the below patch to fix the test failures in Buildbot, but I am not sure it is the correct solution, so please check it. It appears the problem was introduced with this patch: commit 3d4a7390c1a94ef6e07b04b52ea94a95878cda1b Author: Monty <monty(a)mariadb.org> Date: Mon Feb 1 12:45:39 2016 +0200 MDEV-6150 Speed up connection speed by moving creation of THD to new thread diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc index 84f0c63..355f62d 100644 --- a/sql/sql_parse.cc +++ b/sql/sql_parse.cc @@ -844,12 +844,13 @@ void do_handle_bootstrap(THD *thd) delete thd; #ifndef EMBEDDED_LIBRARY - thread_safe_decrement32(&thread_count); + DBUG_ASSERT(thread_count == 1); in_bootstrap= FALSE; - - mysql_mutex_lock(&LOCK_thread_count); - mysql_cond_broadcast(&COND_thread_count); - mysql_mutex_unlock(&LOCK_thread_count); + /* + dec_thread_count will signal bootstrap() function that we have ended as + thread_count will become 0. + */ + dec_thread_count(); my_thread_end(); pthread_exit(0); #endif The problem is that do_handle_bootstrap is not only used for bootstrap - it is also used for read_init_file(). In the latter case, it is _not_ guaranteed that thread_count will drop to zero. For example, the binlog background thread might be running. You can see this in 10.2 (before my just-pushed fix) by running ./mtr main.init_file --mysqld=--log-bin=mysql-bin It will assert, or hang if assert is disabled. To get rid of the failures in Buildbot, I pushed the below patch. I think the patch is perfectly safe. However, maybe there is a better way - it seems a bit messy, mixing bootstrap and read_init_file() and having these if (opt_bootstrap) conditions. And maybe there are other problems caused by this as well that I did not notice? As I am not much familiar with how this thread count checking / bootstrap / init file handling works. Thanks, - Kristian. Kristian Nielsen <knielsen(a)knielsen-hq.org> writes: > revision-id: 49b25020ef512866751f192b91d8439670d0430b (mariadb-10.1.8-257-g49b2502) > parent(s): be2b833c426b420073c50564125049e2b4a95e8b > committer: Kristian Nielsen > timestamp: 2016-09-09 18:09:59 +0200 > message: > > Fix assertion/hang in read_init_file() > > If there are other threads running (for example binlog background > thread), then the thread count may not drop to zero at the end of > do_handle_bootstrap(). This caused an assertion and missing wakeup of > the main thread. The missing wakeup is because THD::~THD() only > signals the COND_thread_count mutex when the number of threads drops > to zero. > > Signed-off-by: Kristian Nielsen <knielsen(a)knielsen-hq.org> > > --- > sql/sql_parse.cc | 13 ++++++++++++- > 1 file changed, 12 insertions(+), 1 deletion(-) > > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > index 7263082..6baff31 100644 > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -1082,9 +1082,20 @@ void do_handle_bootstrap(THD *thd) > end: > in_bootstrap= FALSE; > delete thd; > + if (!opt_bootstrap) > + { > + /* > + We need to wake up main thread in case of read_init_file(). > + This is not done by THD::~THD() when there are other threads running > + (binlog background thread, for example). So do it here again. > + */ > + mysql_mutex_lock(&LOCK_thread_count); > + mysql_cond_broadcast(&COND_thread_count); > + mysql_mutex_unlock(&LOCK_thread_count); > + } > > #ifndef EMBEDDED_LIBRARY > - DBUG_ASSERT(thread_count == 0); > + DBUG_ASSERT(!opt_bootstrap || thread_count == 0); > my_thread_end(); > pthread_exit(0); > #endif > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

2 3

Re: [Maria-developers] Sachin weekly report
by Sachin Setia 10 Sep '16

10 Sep '16

Hi Sergei, Weekly Report for 4th week of gsoc 1. Field property is_row_hash , field_visibility successfully saved and retrived from frm , using extra2 space 2. Some tests added. 3. Solved the error when there is another primary key(it used to accept duplicate in this case ). 4. Added hidden in parser. 5. Identified the memory leak 1 is because of malloc db_row_hash str.I did not freed it. second memory leak i am searching for it. Work for this week. 1 First solve the memory leak problem. 2 Work on FULL_HIDDEN_FIELDS. 3 in mysql_prepare_create_table I am using an iterator it would be better if i can add custom field when it says an error. So will not have to use iterator as suggested by you sir. 4 rename the hash field automatically in the case clash. This week On Thu, Jun 16, 2016 at 11:46 PM, Sergei Golubchik <serg(a)mariadb.org> wrote: > Hi, Sachin! > > On Jun 15, Sachin Setia wrote: > > > > But the major problem is:- > > Consider this case > > > > create table tbl(abc int primary key,xyz blob unique); > > > > In this case , second key_info will have one user_defined_key_parts but > two > > ext_key_parts > > second key_part refers to primary key. > > because of this ha_index_read_idx_map always return HA_ERR_KEY_NOT_FOUND > > I am trying to solve this problem. > > I've seen you solved this, but I do not understand the problem (and so I > cannot understand the fix either). Problem was consider this create table tbl(abc int primary key , xyz blob unique); insert into tbl value(1,12); insert into tbl value(2,12); # no error , details in commit comment https://github.com/MariaDB/server/commit/baecc73380084c61b9323a30f3e2597717… > Please, try to add a test case for > the problem you're fixing. In the same commit, preferrably. > Now you can still commit a test case for this problem and your fix, > then, I hope, I'll be able to understand better what the problem was. > > Regards, > Sergei > Chief Architect MariaDB > and security(a)mariadb.org >

4 29

Re: [Maria-developers] [Commits] d6f760d: MDEV-10296 - Multi-instance table cache
by Sergei Golubchik 09 Sep '16

09 Sep '16

Hi, Sergey! On Sep 08, Sergey Vojtovich wrote: > revision-id: d6f760d344f45177936c72cb65ba1ffff596710e (mariadb-10.1.8-230-gd6f760d) > parent(s): 9f7e77b5414b5b2ef9b5f50f131966821fefb9c7 > committer: Sergey Vojtovich > timestamp: 2016-09-08 16:02:49 +0400 > message: > > MDEV-10296 - Multi-instance table cache > > Table cache instances autosizing. Looks fairly simple, good :) Code-wise it's ok, I've added a comment in Jira. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] 9f7e77b: MDEV-10296 - Multi-instance table cache
by Sergei Golubchik 09 Sep '16

09 Sep '16

Hi, Sergey! On Sep 07, Sergey Vojtovich wrote: > revision-id: 9f7e77b5414b5b2ef9b5f50f131966821fefb9c7 (mariadb-10.1.8-229-g9f7e77b) > parent(s): 58634f6e50b40b28533a03a1afcb68f139937351 > committer: Sergey Vojtovich > timestamp: 2016-09-07 12:47:43 +0400 > message: > > MDEV-10296 - Multi-instance table cache > > - simplified access to per-share free tables list > - explain paddings > > @@ -43,7 +52,9 @@ struct TDC_element > for this share. > */ > All_share_tables_list all_tables; > - char pad[CPU_LEVEL1_DCACHE_LINESIZE]; // free_tables follows this immediately > + /** Avoid false sharing between TDC_element and free_tables */ > + char pad[CPU_LEVEL1_DCACHE_LINESIZE]; > + Share_free_tables free_tables[0]; > }; I vaguely remember that zero-length arrays are not supported by some compilers (they're gcc extension, after all). So I generally use [1] in cases like this, not [0]. Otherwise ok. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 1

Re: [Maria-developers] [Commits] 66b4459: MDEV-9143 JSON_xxx functions.
by Sergei Golubchik 09 Sep '16

09 Sep '16

Hi, Alexey! On Sep 05, Alexey Botchkov wrote: > revision-id: 66b4459e38f5df4913ecce1c9e3d71c7afa7860d (mariadb-10.2.1-21-g66b4459) > parent(s): 31a8cf54c8a7913338480a0571feaf32143b5f64 > committer: Alexey Botchkov > timestamp: 2016-09-05 14:03:33 +0400 > message: > > MDEV-9143 JSON_xxx functions. > > Library with JSON-related functions added. I've just briefly looked through the code, assuming it's mostly the same as the in previous commit. Looks much better comment- and naming-wise, thanks. Could please you add some unit tests for it into unittest/strings ? In a separate commit, if you'd like, that doesn't matter. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] 0a86a91: MDEV-10765: Wrong result - query does not retrieve values from default partition on a table partitioned by list columns
by Sergey Petrunia 08 Sep '16

08 Sep '16

Hi Sanja, > revision-id: 0a86a915842d268477c5febd8481263f00d6c792 (mariadb-10.1.8-242-g0a86a91) > parent(s): effb65bc863da0f1115e16ef5f11d11a13cdc7a0 > committer: Oleksandr Byelkin > timestamp: 2016-09-08 19:43:09 +0200 > message: > > MDEV-10765: Wrong result - query does not retrieve values from default partition on a table partitioned by list columns > > Partial matches should be treat as not exact one. > So I'm running this example: create table t14n ( a int not null, b int not null, c int ) partition by list columns(a,b) ( partition p1 values in ((10,10)), partition p2 values in ((10,20)), partition p3 values in ((10,30)), partition p4 values in ((10,40)), partition p5 values in ((10,50)) ); insert into t14n values (10,10,1234), (10,20,1234), (10,30,1234), (10,40,1234), (10,50,1234); explain partitions select * from t14n where a>=10 and (a <=10 and b <=30); and then I get: #1 0x0000555555eb88ef in get_part_iter_for_interval_cols_via_map (part_info=0x7fff90046208, is_subpart=false, store_length_array=0x7ffff436a0a0, min_value=0x7fff90069518 "\n", max_value=0x7fff90069528 "\n", min_len=8, max_len=4, flags=0, part_iter=0x7ffff436a858) at /home/psergey/dev-git/10.2/sql/sql_partition.cc:7751 This means I get all the way to the memcmp(min_value, max_value, min_len) call, where min_len=8, max_len=4, which means we're comparing garbage. Please add a check that min_len==max_len. Ok to push after this is addressed. On Thu, Sep 08, 2016 at 07:43:10PM +0200, Oleksandr Byelkin wrote: > diff --git a/sql/sql_partition.cc b/sql/sql_partition.cc > index 54396b9..24dff23 100644 > --- a/sql/sql_partition.cc > +++ b/sql/sql_partition.cc > @@ -7723,6 +7723,7 @@ int get_part_iter_for_interval_cols_via_map(partition_info *part_info, > bool can_match_multiple_values; > uint32 nparts; > get_col_endpoint_func UNINIT_VAR(get_col_endpoint); > + uint full_length= 0; > DBUG_ENTER("get_part_iter_for_interval_cols_via_map"); > > if (part_info->part_type == RANGE_PARTITION) > @@ -7740,9 +7741,13 @@ int get_part_iter_for_interval_cols_via_map(partition_info *part_info, > else > assert(0); > > + for (uint32 i= 0; i < part_info->num_columns; i++) > + full_length+= store_length_array[i]; > + > can_match_multiple_values= ((flags & > (NO_MIN_RANGE | NO_MAX_RANGE | NEAR_MIN | > NEAR_MAX)) || > + (min_len != full_length) || > memcmp(min_value, max_value, min_len)); > DBUG_ASSERT(can_match_multiple_values || (flags & EQ_RANGE) || flags == 0); > if (can_match_multiple_values && part_info->has_default_partititon()) > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Contents of information_schema.partitions.PARTITION_DESCRIPTION for DEFAULT partitions?
by Sergey Petrunia 08 Sep '16

08 Sep '16

Hi Sanja, I'm looking at I_S description for the DEFAULT partition and I see this: PARTITION_DESCRIPTION: (MAXVALUE,MAXVALUE) Is this normal? The full example below: show create table t10; +-------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | Table | Create Table | +-------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ | t10 | CREATE TABLE `t10` ( `i` varchar(10) DEFAULT NULL, `j` varchar(10) DEFAULT NULL ) ENGINE=InnoDB DEFAULT CHARSET=latin1 /*!50500 PARTITION BY LIST COLUMNS(i,j) (PARTITION p1 VALUES IN (('10','10')) ENGINE = InnoDB, PARTITION p2 DEFAULT ENGINE = InnoDB) */ | +-------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+ 1 row in set (0.01 sec) MariaDB [j4]> select * from information_schema.partitions where table_name='t10'\G *************************** 1. row *************************** TABLE_CATALOG: def TABLE_SCHEMA: j4 TABLE_NAME: t10 PARTITION_NAME: p1 SUBPARTITION_NAME: NULL PARTITION_ORDINAL_POSITION: 1 SUBPARTITION_ORDINAL_POSITION: NULL PARTITION_METHOD: LIST COLUMNS SUBPARTITION_METHOD: NULL PARTITION_EXPRESSION: `i`,`j` SUBPARTITION_EXPRESSION: NULL PARTITION_DESCRIPTION: ('10','10') TABLE_ROWS: 0 AVG_ROW_LENGTH: 0 DATA_LENGTH: 16384 MAX_DATA_LENGTH: NULL INDEX_LENGTH: 0 DATA_FREE: 0 CREATE_TIME: 2016-09-08 21:22:45 UPDATE_TIME: NULL CHECK_TIME: NULL CHECKSUM: NULL PARTITION_COMMENT: NODEGROUP: default TABLESPACE_NAME: NULL *************************** 2. row *************************** TABLE_CATALOG: def TABLE_SCHEMA: j4 TABLE_NAME: t10 PARTITION_NAME: p2 SUBPARTITION_NAME: NULL PARTITION_ORDINAL_POSITION: 2 SUBPARTITION_ORDINAL_POSITION: NULL PARTITION_METHOD: LIST COLUMNS SUBPARTITION_METHOD: NULL PARTITION_EXPRESSION: `i`,`j` SUBPARTITION_EXPRESSION: NULL PARTITION_DESCRIPTION: (MAXVALUE,MAXVALUE) TABLE_ROWS: 0 AVG_ROW_LENGTH: 0 DATA_LENGTH: 16384 MAX_DATA_LENGTH: NULL INDEX_LENGTH: 0 DATA_FREE: 0 CREATE_TIME: 2016-09-08 21:22:45 UPDATE_TIME: NULL CHECK_TIME: NULL CHECKSUM: NULL PARTITION_COMMENT: NODEGROUP: default TABLESPACE_NAME: NULL 2 rows in set (0.01 sec) BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] [Maria-discuss] Known limitation with TokuDB in Read Free Replication & parallel replication ?
by Kristian Nielsen 08 Sep '16

08 Sep '16

Rich Prohaska <prohaska7(a)gmail.com> writes: > The group lock retry algorithm is on the https://github.com/ > prohaska7/tokuft/tree/killwait branch. Its unit tests pass. Needed to add > some test only functions to get reproducible behaviour. > > The group lock retry algorithm is integrated into my mariadb server on the > https://github.com/prohaska7/mariadb-server/tree/toku_opr3 branch. Ran > sysbench oltp on a small 1000 row table successfully. Looks great, thanks! It passes tests for me, as well. > I am going to write up the tokudb lock tree races that were fixed and email > to George Lorch @ Percona so that this code can be integrated into > PerconaFT. Ok, sounds great! I will push the replication part of the patch to MariaDB 10.1, then (the async deadlock kill). >From the git history, it looks like new TokuDB releases (from Percona Server) are regularly merged into MariaDB 10.1, so I'm thinking that we can get your TokuDB/tokuft changes into MariaDB that way, in the next regular TokuDB merges. I will check it and add any missing MariaDB stuff, if it is not part of the changes that go upstream. Does that sounds ok to you? > Removed the lock wait for report from the lock request start method since > it is redundant with the report that will occur when the lock request is > retried in the lock request wait method. The reason I added this reporting originally was for the case where a deadlock is detected. If transaction T1 tries to get a lock with lock_request::start(), but a deadlock is detected (DB_LOCK_DEADLOCK is returned), the lock request wait method will not be called (if I understand the code correctly), so the reporting in lock_request::start() was not redundant. The rationale is that if T1 gets aborted due to a deadlock with T2, and T2 is later in the replication commit order, then when T1 is run again by replication, it will almost certainly conflict with T2 again. So we might as well get T2 killed early (by doing the report already in start()). But on the other hand, things will work correctly without any reporting in start(), and with only a slight delay in case of a conflict. And the assumption in optimistic parallel replication is that conflicts will be relatively rare. So I'm fine without reporting in start(), as you have in your current code. Looks like we are close now to having optimistic parallel replication working with TokuDB. Thanks for all your work on this, Rich! - Kristian.

1 0

[Maria-developers] Missing tests for MDEV-10057 and MDEV-10058
by Alexander Barkov 07 Sep '16

07 Sep '16

Hello Sergey and Igor, You recently closed MDEV-10057 and MDEV-10058, but forgot to add tests. - MDEV-10057 does not have tests at all. - MDEV-10058 has only the parser related test, but does not have this test: EXPLAIN SELECT * FROM (WITH a AS (SELECT * FROM t1) SELECT * FROM t2 NATURAL JOIN t3) AS d1; Can you please add the missing tests? Adding patches without tests makes merging painful. You never know if a merge (especially an automatic one) breaks something. Thanks!

2 2

[Maria-developers] MDEV-8909 union parser cleanup
by Alexander Barkov 07 Sep '16

07 Sep '16

Hi Sergei, Please review a patch for MDEV-8909. It removes some old remainders that are not needed anymore. Thanks.

2 2

[Maria-developers] MDEV-7145
by Phil Sweeney 05 Sep '16

05 Sep '16

Hi all, Re: https://jira.mariadb.org/browse/MDEV-7145 Delayed replication (a feature shipped in MySQL 5.6) has been a MariaDB feature request since 2014, and has been listed as 'major/red' priority in Jira since. It initially targeted 10.1, but missed the cut for some reason, and is now targeted at 10.2. It has the most votes of any open ticket at 26 (and only 1 closed ticket has more). I noticed we've had a couple of alpha releases of 10.2, and this ticket is open not assigned to anyone. Can anyone comment on whether this is still likely to make it into 10.2. Just wanting to know if it needs more lobbying or if it's 100% going to get looked at. This is certainly not a complaint - short of providing a patch (which I'm not capable of), is there anything else needed to increase the chances of this being implemented? Thanks! Phil

2 1

Re: [Maria-developers] [Commits] 97d212a: MDEV-10545: Server crashed in my_copy_fix_mb on querying I_S and P_S tables
by Nirbhay Choubey 02 Sep '16

02 Sep '16

Hi Svoj! Thanks for the review. On Fri, Sep 2, 2016 at 1:41 AM, Sergey Vojtovich <svoj(a)mariadb.org> wrote: > Hi Nirbhay, > > Ok to push, one question below. > > On Thu, Sep 01, 2016 at 12:46:58PM -0400, Nirbhay Choubey wrote: > > revision-id: 97d212a34ff4e0558126bc393bbef97036611d83 > (mariadb-10.1.17-4-g97d212a) > > parent(s): a322651b8aa702e58d473edfae26606f10a089fb > > author: Nirbhay Choubey > > committer: Nirbhay Choubey > > timestamp: 2016-09-01 12:46:56 -0400 > > message: > > > > MDEV-10545: Server crashed in my_copy_fix_mb on querying I_S and P_S > tables > > > > Once THDs have been added to the global "threads" list, > > they must modify query_string only after acquiring per- > > thread LOCK_thd_data mutex. > > > > --- > > sql/log_event.cc | 2 +- > > sql/sql_acl.cc | 8 ++++---- > > sql/sql_parse.cc | 7 +++---- > > 3 files changed, 8 insertions(+), 9 deletions(-) > > > > diff --git a/sql/log_event.cc b/sql/log_event.cc > > index afa58af..66e7c60 100644 > > --- a/sql/log_event.cc > > +++ b/sql/log_event.cc > > @@ -6022,7 +6022,7 @@ int Load_log_event::do_apply_event(NET* net, > rpl_group_info *rgi, > > new_db.str= (char *) rpl_filter->get_rewrite_db(db, &new_db.length); > > thd->set_db(new_db.str, new_db.length); > > DBUG_ASSERT(thd->query() == 0); > > - thd->reset_query_inner(); // Should not be needed > > + if (thd->query()) thd->reset_query(); // Should not be needed > Why if? Original code didn't have it. > Now that we call reset_query(), the check has been added to avoid it and an extra mutex. And from the above assert, I think it will be ever better have this check within unlikely(). Best, Nirbhay > > Regards, > Sergey > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

2 2

Re: [Maria-developers] 811e062: MDEV-10035: DBUG_ASSERT on CREATE VIEW v1 AS SELECT * FROM t1 FOR UPDATE
by Sergei Golubchik 31 Aug '16

31 Aug '16

Hi, Oleksandr! On Jun 02, Oleksandr Byelkin wrote: > revision-id: 811e062f812e450a2d0ae2c3be7f4ad1b90eadc8 (mariadb-10.2.0-53-g811e062) > parent(s): eb86c32225db2a76c6f256130e7bb316900a9408 > committer: Oleksandr Byelkin > timestamp: 2016-06-02 12:12:26 +0200 > message: > > MDEV-10035: DBUG_ASSERT on CREATE VIEW v1 AS SELECT * FROM t1 FOR UPDATE > > Ability to print lock type added. > Restoring correct lock type for CREATE VIEW added. So your fix is to save/print the lock option, but not let it to affect the locking... I suppose it may be ok for views, let's keep it your way for now. Ok to push Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] 48d1ba6: MDEV-10296 - Multi-instance table cache
by Sergei Golubchik 31 Aug '16

31 Aug '16

Hi, Sergey! Ok, see comments below. But why is this supposed to improve scalability? Are you talking about the use case of one table being access by multiple threads? Or this helps in other cases too? And what is the effect of the more expensive tc_remove_all_unused_tables() ? On Jun 29, Sergey Vojtovich wrote: > revision-id: 48d1ba6097939efe5efd28dd3ed9d281cc2bc2f4 (mariadb-10.2.0-93-g48d1ba6) > parent(s): 8bec9746f0d5b363f385713035ca3f2daff34e1c > committer: Sergey Vojtovich > timestamp: 2016-06-29 16:33:08 +0400 > message: > > MDEV-10296 - Multi-instance table cache > > Improve scalability by implementing multi-instance table cache. > > diff --git a/mysql-test/suite/sys_vars/r/sysvars_server_notembedded.result b/mysql-test/suite/sys_vars/r/sysvars_server_notembedded.result > index e075c64..22b12a9 100644 > --- a/mysql-test/suite/sys_vars/r/sysvars_server_notembedded.result > +++ b/mysql-test/suite/sys_vars/r/sysvars_server_notembedded.result > @@ -4559,6 +4559,20 @@ NUMERIC_BLOCK_SIZE 1 > ENUM_VALUE_LIST NULL > READ_ONLY NO > COMMAND_LINE_ARGUMENT REQUIRED > +VARIABLE_NAME TABLE_OPEN_CACHE_INSTANCES > +SESSION_VALUE NULL > +GLOBAL_VALUE 1 > +GLOBAL_VALUE_ORIGIN COMPILE-TIME Default value 1 and it's not auto-tuned on startup? You think multi-instance table cache is better stay unused? :) If not, either use a reasonable default or, may be, choose it on startup based on my_gencpus() > +DEFAULT_VALUE 1 > +VARIABLE_SCOPE GLOBAL > +VARIABLE_TYPE BIGINT UNSIGNED > +VARIABLE_COMMENT The number of table cache instances > +NUMERIC_MIN_VALUE 1 > +NUMERIC_MAX_VALUE 64 > +NUMERIC_BLOCK_SIZE 1 > +ENUM_VALUE_LIST NULL > +READ_ONLY YES > +COMMAND_LINE_ARGUMENT REQUIRED > VARIABLE_NAME THREAD_CACHE_SIZE > SESSION_VALUE NULL > GLOBAL_VALUE 151 > diff --git a/sql/table_cache.cc b/sql/table_cache.cc > index bfe51df..f1eb006 100644 > --- a/sql/table_cache.cc > +++ b/sql/table_cache.cc > @@ -125,67 +114,101 @@ static int fix_thd_pins(THD *thd) > part of table definition cache. > */ > > +struct Share_free_tables > { > + typedef I_P_List <TABLE, TABLE_share> List; > + List list; > + char pad[CPU_LEVEL1_DCACHE_LINESIZE]; a comment, please, explaining the padding. Or see a suggestion at the end. > +}; > > + > +struct Table_cache_instance > +{ > + /** > + Protects free_tables (TABLE::global_free_next and TABLE::global_free_prev), > + records, Share_free_tables::List (TABLE::prev and TABLE::next), > + TABLE::in_use. > + */ > + mysql_mutex_t LOCK_table_cache; > + I_P_List <TABLE, I_P_List_adapter<TABLE, &TABLE::global_free_next, > + &TABLE::global_free_prev>, > + I_P_List_null_counter, I_P_List_fast_push_back<TABLE> > > + free_tables; > + ulong records; > + char pad[CPU_LEVEL1_DCACHE_LINESIZE]; why is it padded? > + > + Table_cache_instance(): records(0) > { > + mysql_mutex_init(key_LOCK_table_cache, &LOCK_table_cache, > + MY_MUTEX_INIT_FAST); > } > + > + ~Table_cache_instance() > + { > + mysql_mutex_destroy(&LOCK_table_cache); > + DBUG_ASSERT(free_tables.is_empty()); > + DBUG_ASSERT(records == 0); > + } > +}; > > > +static Table_cache_instance *tc; > > + > +static inline Share_free_tables *SHARE_FREE_TABLES(TDC_element *element) fairly unconventional letter case choice :) Besides, instead of pointer arithmetics, you could've simply put Share_free_tables free_tables[1]; at the end of TDC_element. > { > + return (Share_free_tables*) (element + 1); > } > > > +static void intern_close_table(TABLE *table) > +{ > + delete table->triggers; > + DBUG_ASSERT(table->file); old intern_close_table had if (table->file) // Not true if placeholder do you mean, these placeholder TABLEs can no longer be in the table cache? > + closefrm(table); > + tdc_release_share(table->s); old intern_close_table() also did table->alias.free(); > + my_free(table); > +} > > + > +/** > + Get number of TABLE objects (used and unused) in table cache. > */ > > +uint tc_records(void) > { > + ulong total= 0; > + for (ulong i= 0; i < tc_instances; i++) > + { > + mysql_mutex_lock(&tc[i].LOCK_table_cache); > + total+= tc[i].records; > + mysql_mutex_unlock(&tc[i].LOCK_table_cache); > + } > + return total; > } > > > /** > Remove TABLE object from table cache. > */ > > static void tc_remove_table(TABLE *table) > { > + TDC_element *element= table->s->tdc; > + > + mysql_mutex_lock(&element->LOCK_table_share); > + /* Wait for MDL deadlock detector to complete traversing tdc.all_tables. */ > + while (element->all_tables_refs) > + mysql_cond_wait(&element->COND_release, &element->LOCK_table_share); > + element->all_tables.remove(table); > + mysql_mutex_unlock(&element->LOCK_table_share); > + > + intern_close_table(table); > } > > > static void tc_remove_all_unused_tables(TDC_element *element, > + Share_free_tables::List *purge_tables, > bool mark_flushed) > { > TABLE *table; > @@ -200,10 +223,18 @@ static void tc_remove_all_unused_tables(TDC_element *element, > */ > if (mark_flushed) > element->flushed= true; > + for (ulong i= 0; i < tc_instances; i++) > { > + mysql_mutex_lock(&tc[i].LOCK_table_cache); > + while ((table= SHARE_FREE_TABLES(element)[i].list.pop_front())) > + { > + tc[i].records--; > + tc[i].free_tables.remove(table); > + DBUG_ASSERT(element->all_tables_refs == 0); > + element->all_tables.remove(table); > + purge_tables->push_front(table); > + } > + mysql_mutex_unlock(&tc[i].LOCK_table_cache); > } > } > > @@ -225,7 +256,7 @@ static void tc_remove_all_unused_tables(TDC_element *element, > > struct tc_purge_arg > { > + Share_free_tables::List purge_tables; > bool mark_flushed; > }; > > @@ -281,79 +298,33 @@ static TABLE *tc_free_tables_back(TDC_element *element) > - free evicted object > */ > > +void tc_add_table(THD *thd, TABLE *table) > { > + ulong i= thd->thread_id % tc_instances; > + TABLE *LRU_table= 0; > + TDC_element *element= table->s->tdc; > > + DBUG_ASSERT(table->in_use == thd); > mysql_mutex_lock(&element->LOCK_table_share); > + /* Wait for MDL deadlock detector to complete traversing tdc.all_tables. */ > + while (element->all_tables_refs) > + mysql_cond_wait(&element->COND_release, &element->LOCK_table_share); > + element->all_tables.push_front(table); > mysql_mutex_unlock(&element->LOCK_table_share); > > + mysql_mutex_lock(&tc[i].LOCK_table_cache); > + if (tc[i].records == tc_size && (LRU_table= tc[i].free_tables.pop_front())) > { > + SHARE_FREE_TABLES(LRU_table->s->tdc)[i].list.remove(LRU_table); > + /* Needed if MDL deadlock detector chimes in before tc_remove_table() */ > + LRU_table->in_use= thd; How can MDL deadlock detector get to the unused table? It traverses the lock dependency graph, and an unused table cannot be locked, can it? > } > + else > + tc[i].records++; > + mysql_mutex_unlock(&tc[i].LOCK_table_cache); > + > + if (LRU_table) > + tc_remove_table(LRU_table); > } > > > @@ -369,10 +340,11 @@ void tc_add_table(THD *thd, TABLE *table) > > static TABLE *tc_acquire_table(THD *thd, TDC_element *element) > { > + ulong i= thd->thread_id % tc_instances; > TABLE *table; > > - mysql_mutex_lock(&element->LOCK_table_share); > - table= element->free_tables.pop_front(); > + mysql_mutex_lock(&tc[i].LOCK_table_cache); > + table= SHARE_FREE_TABLES(element)[i].list.pop_front(); > if (table) > { > DBUG_ASSERT(!table->in_use); > @@ -381,8 +353,9 @@ static TABLE *tc_acquire_table(THD *thd, TDC_element *element) > DBUG_ASSERT(table->db_stat && table->file); > /* The children must be detached from the table. */ > DBUG_ASSERT(!table->file->extra(HA_EXTRA_IS_ATTACHED_CHILDREN)); > + tc[i].free_tables.remove(table); > } > - mysql_mutex_unlock(&element->LOCK_table_share); > + mysql_mutex_unlock(&tc[i].LOCK_table_cache); > return table; > } > > @@ -413,40 +386,27 @@ static TABLE *tc_acquire_table(THD *thd, TDC_element *element) > @retval false object released > */ > > -bool tc_release_table(TABLE *table) > +void tc_release_table(TABLE *table) > { > + ulong i= table->in_use->thread_id % tc_instances; > DBUG_ASSERT(table->in_use); > DBUG_ASSERT(table->file); > > - if (table->needs_reopen() || tc_records() > tc_size) > + mysql_mutex_lock(&tc[i].LOCK_table_cache); > + if (table->needs_reopen() || table->s->tdc->flushed || > + tc[i].records > tc_size) > { > - mysql_mutex_lock(&table->s->tdc->LOCK_table_share); > - goto purge; > + tc[i].records--; > + mysql_mutex_unlock(&tc[i].LOCK_table_cache); > + tc_remove_table(table); Does it mean that if the table cache is full, you free the table, instead of adding it to the cache? That's not very LRU-ish > + } > + else > + { > + table->in_use= 0; > + SHARE_FREE_TABLES(table->s->tdc)[i].list.push_front(table); > + tc[i].free_tables.push_back(table); > + mysql_mutex_unlock(&tc[i].LOCK_table_cache); > } > } > > > @@ -573,23 +535,29 @@ static uchar *tdc_hash_key(const TDC_element *element, size_t *length, > Initialize table definition cache. > */ > > -void tdc_init(void) > +bool tdc_init(void) > { > DBUG_ENTER("tdc_init"); > #ifdef HAVE_PSI_INTERFACE > - init_tc_psi_keys(); > + mysql_mutex_register("sql", all_tc_mutexes, array_elements(all_tc_mutexes)); > + mysql_cond_register("sql", all_tc_conds, array_elements(all_tc_conds)); > #endif > + /* Extra instance is allocated to avoid false sharing */ > + if (!(tc= new Table_cache_instance[tc_instances + 1])) > + DBUG_RETURN(true); > tdc_inited= true; > mysql_mutex_init(key_LOCK_unused_shares, &LOCK_unused_shares, > MY_MUTEX_INIT_FAST); > tdc_version= 1L; /* Increments on each reload */ > - lf_hash_init(&tdc_hash, sizeof(TDC_element), LF_HASH_UNIQUE, 0, 0, > + lf_hash_init(&tdc_hash, sizeof(TDC_element) + > + sizeof(Share_free_tables) * tc_instances, > + LF_HASH_UNIQUE, 0, 0, > (my_hash_get_key) tdc_hash_key, > &my_charset_bin); > tdc_hash.alloc.constructor= lf_alloc_constructor; > tdc_hash.alloc.destructor= lf_alloc_destructor; > tdc_hash.initializer= (lf_hash_initializer) tdc_hash_initializer; > - DBUG_VOID_RETURN; > + DBUG_RETURN(false); > } > > > @@ -631,6 +599,7 @@ void tdc_deinit(void) > tdc_inited= false; > lf_hash_destroy(&tdc_hash); > mysql_mutex_destroy(&LOCK_unused_shares); > + delete [] tc; > } > DBUG_VOID_RETURN; > } > @@ -1038,7 +1007,7 @@ bool tdc_remove_table(THD *thd, enum_tdc_remove_table_type remove_type, > const char *db, const char *table_name, > bool kill_delayed_threads) > { > - TDC_element::TABLE_list purge_tables; > + Share_free_tables::List purge_tables; > TABLE *table; > TDC_element *element; > uint my_refs= 1; > diff --git a/sql/table_cache.h b/sql/table_cache.h > index c05baae..4f449cb 100644 > --- a/sql/table_cache.h > +++ b/sql/table_cache.h > @@ -45,7 +43,7 @@ struct TDC_element > for this share. > */ > All_share_tables_list all_tables; > - TABLE_list free_tables; > + char pad[CPU_LEVEL1_DCACHE_LINESIZE]; // free_tables follows this immediately hmm. may be you'd rather put padding in one place - at the beginning of Share_free_tables? > }; Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 3

Re: [Maria-developers] 9019cfd: MDEV-9593 - Print the real version in the error log
by Sergei Golubchik 31 Aug '16

31 Aug '16

Hi, Sergey! Ok to push. On Aug 26, Sergey Vojtovich wrote: > revision-id: 9019cfd215819dd765501e4b8d63b36396f3e558 (mariadb-10.2.1-22-g9019cfd) > parent(s): b4eed0afa2aef878ad7b9849a2ff2dabfe85dfe0 > committer: Sergey Vojtovich > timestamp: 2016-08-26 16:39:32 +0400 > message: > > MDEV-9593 - Print the real version in the error log Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] db9822d: MDEV-8931: (server part of) session state tracking
by Sergei Golubchik 31 Aug '16

31 Aug '16

Hi, Oleksandr! Looks ok to push, thanks! On Aug 09, Oleksandr Byelkin wrote: > revision-id: db9822df31917c3d0162322a77bcd4433bfb705e (mariadb-10.2.1-15-gdb9822d) > parent(s): b5e0f70e04cec625c99760e5bfb28bb31bc972ed > committer: Oleksandr Byelkin > timestamp: 2016-08-09 15:49:30 +0200 > message: > > MDEV-8931: (server part of) session state tracking > > Postreview fixes. > New MySQL tests fixes. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] 4949925: MDEV-10421 duplicate CHECK CONSTRAINTs.
by Sergei Golubchik 31 Aug '16

31 Aug '16

Hi, Alexey! Looks good, thanks! See few comments below, then ok to push! On Aug 30, Alexey Botchkov wrote: > revision-id: 49499250c04b631bd908945b7c59d6affaeb2703 (mariadb-10.2.1-9-g4949925) > parent(s): 08683a726773f8cdf16a4a3dfb3920e5f7842481 > committer: Alexey Botchkov > timestamp: 2016-08-30 16:43:05 +0400 > message: > > MDEV-10421 duplicate CHECK CONSTRAINTs. > > mysql_prepare_create_table fixed so it doesn't let duplicating > constraint names. Syntax for CONSTRAINT IF NOT EXISTS added > and handled in mysql_alter_table. > > diff --git a/sql/field.h b/sql/field.h > index 13a0e91..7b271b0 100644 > --- a/sql/field.h > +++ b/sql/field.h > @@ -606,6 +606,7 @@ class Virtual_column_info: public Sql_alloc > /* Flag indicating that the field is physically stored in the database */ > bool stored_in_db; > bool utf8; /* Already in utf8 */ > + bool create_if_not_exists; You apparently forgot to remove create_if_not_exists. It doesn't seem to be used anywhere. > /* The expression to compute the value of the virtual column */ > Item *expr_item; > /* Text representation of the defining expression */ > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index e745fe8..c120fbc 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -6130,6 +6146,38 @@ handle_if_exists_options(THD *thd, TABLE *table, Alter_info *alter_info) > } > #endif /*WITH_PARTITION_STORAGE_ENGINE*/ > > + /* ADD CONSTRAINT IF NOT EXISTS. */ > + { > + List_iterator<Virtual_column_info> it(alter_info->check_constraint_list); > + Virtual_column_info *check; > + TABLE_SHARE *share= table->s; > + uint c; > + while ((check=it++)) > + { > + if (!check->flags /* & Alter_info::CHECK_CONSTRAINT_IF_NOT_EXISTS */ && why is that? because CHECK_CONSTRAINT_IF_NOT_EXISTS is the only possible flag now? I'd still prefer it a bit more future-proof, better to check the flag here. > + check->name.length) > + continue; > + check->flags= 0; > + for (c= share->field_check_constraints; > + c < share->table_check_constraints ; c++) > + { > + Virtual_column_info *dup= table->check_constraints[c]; > + if (dup->name.length == check->name.length && > + my_strcasecmp(system_charset_info, > + check->name.str, dup->name.str) == 0) > + { > + push_warning_printf(thd, Sql_condition::WARN_LEVEL_NOTE, > + ER_DUP_CONSTRAINT_NAME, ER_THD(thd, ER_DUP_CONSTRAINT_NAME), > + "CHECK", check->name.str); > + it.remove(); > + if (alter_info->check_constraint_list.elements == 0) > + alter_info->flags&= ~Alter_info::ALTER_ADD_CHECK_CONSTRAINT; > + > + break; > + } > + } > + } > + } > DBUG_VOID_RETURN; > } You seem to forgot to add #define ER_FK_DUP_NAME ER_DUP_CONSTRAINT_NAME to mysql.h. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] MDEV-10702 Crash in SET STATEMENT FOR EXECUTE
by Alexander Barkov 31 Aug '16

31 Aug '16

Hello Sergei, Please review a patch for mdev-10702.diff Thanks!

1 0

Re: [Maria-developers] [Maria-discuss] Known limitation with TokuDB in Read Free Replication & parallel replication ?
by Kristian Nielsen 30 Aug '16

30 Aug '16

Rich Prohaska <prohaska7(a)gmail.com> writes: > I rearranged the tokudb lock request wait function a little bit, and got > the lock tree unit tests to compile again (since the API changed). commit > on this tree: https://github.com/prohaska7/mariadb-server/tree/toku_opr3 Thanks! So I took a look at the complete patch we have so far, thinking what remains to do to get this included in main MariaDB. Do you have any suggestions for how/whether to upstream the TokuDB parts? I assume the new kill_query functionality would make sense to upstream. The callback to report waits could be upstreamed, but would not be used in non-mariadb builds, probably. And the actual tokudb_lock_wait_needed_callback() would probably not be upstreamed (or would go in #ifdef MARIADB_BASE_VERSION perhaps). Any thoughts? I still need to look into the should_retry_lock_requests, currently I get hangs if it is enabled. And I want to see if the wait_needed_callback interface can be cleaned up a bit. Otherwise the patch looks fairly complete to me, do you agree? I think this could go in 10.1, to fix optimistic parallel replication with TokuDB (which does not currently work at all). - Kristian.

2 1

Re: [Maria-developers] [Commits] 2a4ed5f: MDEV-10017: Get unexpected `Empty Set` for correlated subquery with aggregate functions (part 1)
by Sergey Petrunia 29 Aug '16

29 Aug '16

Hi Sanja, Table t2 is the testcase is redundant. Ok to push after this is addressed. On Mon, Aug 29, 2016 at 01:50:09PM +0200, Oleksandr Byelkin wrote: > revision-id: 2a4ed5f7a74860608cb36b761eb139a57cbd4463 (mariadb-10.1.16-15-g2a4ed5f) > parent(s): 76bd9f655a42dc1e21acfa26650a0cb76c70a22e > committer: Oleksandr Byelkin > timestamp: 2016-08-29 13:50:09 +0200 > message: > > MDEV-10017: Get unexpected `Empty Set` for correlated subquery with aggregate functions (part 1) > > Make aggregate function dependency visible. > > --- > mysql-test/r/func_group.result | 19 +++++++++++++++++++ > mysql-test/r/subselect3.result | 1 + > mysql-test/r/subselect3_jcl6.result | 1 + > mysql-test/r/subselect_mat.result | 1 + > mysql-test/r/subselect_sj_mat.result | 1 + > mysql-test/t/func_group.test | 11 +++++++++++ > sql/item_sum.cc | 10 ++++++++++ > sql/share/errmsg-utf8.txt | 3 +++ > 8 files changed, 47 insertions(+) > > diff --git a/mysql-test/r/func_group.result b/mysql-test/r/func_group.result > index 0d9d4ff..4bc5ad1 100644 > --- a/mysql-test/r/func_group.result > +++ b/mysql-test/r/func_group.result > @@ -2340,5 +2340,24 @@ companynr AVG(fld1) avg1 avg2 > 37 9223372036854775805.0000 9223372036854775805 9223372036854775805 > DROP TABLE t1; > # > +# case where aggregate resolved in the local SELECT > +# but outer ones are checked > +# > +create table t10 (a int , b int, c int); > +insert into t10 values (0,0,0),(1,1,1); > +create table t11 as select * from t10; > +create table t12 as select * from t10; > +create table t2 as select * from t10; > +explain extended select a from t10 where c<3 or a in (select c from t12 union select max(t10.b) from t11 group by t11.c); > +id select_type table type possible_keys key key_len ref rows filtered Extra > +1 PRIMARY t10 ALL NULL NULL NULL NULL 2 100.00 Using where > +2 DEPENDENT SUBQUERY t12 ALL NULL NULL NULL NULL 2 100.00 Using where > +3 DEPENDENT UNION t11 ALL NULL NULL NULL NULL 2 100.00 Using temporary > +NULL UNION RESULT <union2,3> ALL NULL NULL NULL NULL NULL NULL > +Warnings: > +Note 1276 Field or reference 'test.t10.b' of SELECT #3 was resolved in SELECT #1 > +Note 1003 select `test`.`t10`.`a` AS `a` from `test`.`t10` where ((`test`.`t10`.`c` < 3) or <expr_cache><`test`.`t10`.`a`,`test`.`t10`.`b`>(<in_optimizer>(`test`.`t10`.`a`,<exists>(select `test`.`t12`.`c` from `test`.`t12` where (<cache>(`test`.`t10`.`a`) = `test`.`t12`.`c`) union select max(`test`.`t10`.`b`) from `test`.`t11` group by `test`.`t11`.`c` having (<cache>(`test`.`t10`.`a`) = <ref_null_helper>(max(`test`.`t10`.`b`))))))) > +drop table t10,t11,t12,t2; > +# > # End of 10.1 tests > # > diff --git a/mysql-test/r/subselect3.result b/mysql-test/r/subselect3.result > index 24d9f0d..052b46d 100644 > --- a/mysql-test/r/subselect3.result > +++ b/mysql-test/r/subselect3.result > @@ -889,6 +889,7 @@ ERROR 42S22: Unknown column 'c' in 'field list' > SHOW WARNINGS; > Level Code Message > Note 1276 Field or reference 'test.t1.a' of SELECT #3 was resolved in SELECT #2 > +Note 1981 Aggregate function 'count()' of SELECT #3 belongs to SELECT #2 > Note 1276 Field or reference 'test.t1.c' of SELECT #3 was resolved in SELECT #2 > Error 1054 Unknown column 'c' in 'field list' > DROP TABLE t1; > diff --git a/mysql-test/r/subselect3_jcl6.result b/mysql-test/r/subselect3_jcl6.result > index 19d3d25..28f4e40 100644 > --- a/mysql-test/r/subselect3_jcl6.result > +++ b/mysql-test/r/subselect3_jcl6.result > @@ -899,6 +899,7 @@ ERROR 42S22: Unknown column 'c' in 'field list' > SHOW WARNINGS; > Level Code Message > Note 1276 Field or reference 'test.t1.a' of SELECT #3 was resolved in SELECT #2 > +Note 1981 Aggregate function 'count()' of SELECT #3 belongs to SELECT #2 > Note 1276 Field or reference 'test.t1.c' of SELECT #3 was resolved in SELECT #2 > Error 1054 Unknown column 'c' in 'field list' > DROP TABLE t1; > diff --git a/mysql-test/r/subselect_mat.result b/mysql-test/r/subselect_mat.result > index d0c5c2e..bfb40b9 100644 > --- a/mysql-test/r/subselect_mat.result > +++ b/mysql-test/r/subselect_mat.result > @@ -1139,6 +1139,7 @@ id select_type table type possible_keys key key_len ref rows filtered Extra > 3 DEPENDENT SUBQUERY t3 ALL NULL NULL NULL NULL 4 100.00 Using where > Warnings: > Note 1276 Field or reference 'test.t1.b' of SELECT #3 was resolved in SELECT #1 > +Note 1981 Aggregate function 'max()' of SELECT #3 belongs to SELECT #1 > Note 1003 select `test`.`t1`.`a` AS `a` from `test`.`t1` group by `test`.`t1`.`a` having <expr_cache><`test`.`t1`.`a`,`test`.`t1`.`b`,max(`test`.`t1`.`b`),max(`test`.`t1`.`b`)>(<in_optimizer>(`test`.`t1`.`a`,<exists>(select `test`.`t2`.`c` from `test`.`t2` where (<nop>(<expr_cache><`test`.`t2`.`d`,`test`.`t1`.`b`,max(`test`.`t1`.`b`),max(`test`.`t1`.`b`)>(<in_optimizer>(`test`.`t2`.`d`,<exists>(select `test`.`t3`.`e` from `test`.`t3` where (max(`test`.`t1`.`b`) = `test`.`t3`.`e`) having (<cache>(`test`.`t2`.`d`) >= <ref_null_helper>(`test`.`t3`.`e`)))))) and (<cache>(`test`.`t1`.`a`) = `test`.`t2`.`c`))))) > select a from t1 group by a > having a in (select c from t2 where d >= some(select e from t3 where max(b)=e)); > diff --git a/mysql-test/r/subselect_sj_mat.result b/mysql-test/r/subselect_sj_mat.result > index e846549..366a03b 100644 > --- a/mysql-test/r/subselect_sj_mat.result > +++ b/mysql-test/r/subselect_sj_mat.result > @@ -1173,6 +1173,7 @@ id select_type table type possible_keys key key_len ref rows filtered Extra > 3 DEPENDENT SUBQUERY t3 ALL NULL NULL NULL NULL 4 100.00 Using where > Warnings: > Note 1276 Field or reference 'test.t1.b' of SELECT #3 was resolved in SELECT #1 > +Note 1981 Aggregate function 'max()' of SELECT #3 belongs to SELECT #1 > Note 1003 select `test`.`t1`.`a` AS `a` from `test`.`t1` group by `test`.`t1`.`a` having <expr_cache><`test`.`t1`.`a`,`test`.`t1`.`b`,max(`test`.`t1`.`b`),max(`test`.`t1`.`b`)>(<in_optimizer>(`test`.`t1`.`a`,<exists>(select `test`.`t2`.`c` from `test`.`t2` where (<nop>(<expr_cache><`test`.`t2`.`d`,`test`.`t1`.`b`,max(`test`.`t1`.`b`),max(`test`.`t1`.`b`)>(<in_optimizer>(`test`.`t2`.`d`,<exists>(select `test`.`t3`.`e` from `test`.`t3` where (max(`test`.`t1`.`b`) = `test`.`t3`.`e`) having (<cache>(`test`.`t2`.`d`) >= <ref_null_helper>(`test`.`t3`.`e`)))))) and (<cache>(`test`.`t1`.`a`) = `test`.`t2`.`c`))))) > select a from t1 group by a > having a in (select c from t2 where d >= some(select e from t3 where max(b)=e)); > diff --git a/mysql-test/t/func_group.test b/mysql-test/t/func_group.test > index 47477c7..4a8b76b 100644 > --- a/mysql-test/t/func_group.test > +++ b/mysql-test/t/func_group.test > @@ -1617,5 +1617,16 @@ SELECT companynr, AVG(fld1), AVG(fld1)<<0 AS avg1, CAST(AVG(fld1) AS UNSIGNED)<< > DROP TABLE t1; > > --echo # > +--echo # case where aggregate resolved in the local SELECT > +--echo # but outer ones are checked > +--echo # > +create table t10 (a int , b int, c int); > +insert into t10 values (0,0,0),(1,1,1); > +create table t11 as select * from t10; > +create table t12 as select * from t10; > +create table t2 as select * from t10; > +explain extended select a from t10 where c<3 or a in (select c from t12 union select max(t10.b) from t11 group by t11.c); > +drop table t10,t11,t12,t2; > +--echo # > --echo # End of 10.1 tests > --echo # > diff --git a/sql/item_sum.cc b/sql/item_sum.cc > index 1cfee1a..2a2b7f9 100644 > --- a/sql/item_sum.cc > +++ b/sql/item_sum.cc > @@ -380,6 +380,16 @@ bool Item_sum::register_sum_func(THD *thd, Item **ref) > sl->master_unit()->item->with_sum_func= 1; > } > thd->lex->current_select->mark_as_dependent(thd, aggr_sel, NULL); > + > + if ((thd->lex->describe & DESCRIBE_EXTENDED) && aggr_sel) > + { > + push_warning_printf(thd, Sql_condition::WARN_LEVEL_NOTE, > + ER_WARN_AGGFUNC_DEPENDENCE, > + ER_THD(thd, ER_WARN_AGGFUNC_DEPENDENCE), > + func_name(), > + thd->lex->current_select->select_number, > + aggr_sel->select_number); > + } > return FALSE; > } > > diff --git a/sql/share/errmsg-utf8.txt b/sql/share/errmsg-utf8.txt > index 709ac55..a6eaf2b 100644 > --- a/sql/share/errmsg-utf8.txt > +++ b/sql/share/errmsg-utf8.txt > @@ -7139,3 +7139,6 @@ ER_KILL_QUERY_DENIED_ERROR > ER_NO_EIS_FOR_FIELD > eng "Engine-independent statistics are not collected for column '%s'" > ukr "Незалежна від типу таблиці статистика не збирається для стовбця '%s'" > +ER_WARN_AGGFUNC_DEPENDENCE > + eng "Aggregate function '%-.192s)' of SELECT #%d belongs to SELECT #%d" > + ukr "Агрегатна функція '%-.192s)' з SELECTу #%d належить до SELECTу #%d" > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] [Commits] c584f16: MDEV-10017: Get unexpected `Empty Set` for correlated subquery with aggregate functions (part 1)
by Sergey Petrunia 29 Aug '16

29 Aug '16

Hi Sanja, On Thu, Jun 23, 2016 at 05:50:08PM +0200, Oleksandr Byelkin wrote: > revision-id: c584f1655ef45dafd2b5d0eb55ca21421de9a266 (mariadb-10.1.14-25-gc584f16) > parent(s): 773ce408762b5f8256d4053b6d0d418d15657b92 > committer: Oleksandr Byelkin > timestamp: 2016-06-23 17:50:07 +0200 > message: > > MDEV-10017: Get unexpected `Empty Set` for correlated subquery with aggregate functions (part 1) > > Make aggregate function dependency visible. > * I was able to get a crash with the patch. The details are below. * The wording of the warning feels odd: > Field or reference 'test.t10.b' of SELECT #3 was resolved in SELECT #1 > ... > Aggregate function 'max()' of SELECT #3 belongs to SELECT #1 + I like how the wording of the new warning is similar to the old one. - It is not immediately clear what does "belong to" mean. I would change it to " Aggregate function 'max()' of SELECT #3 is aggregated in SELECT #1 " This looks much clearer. OTOH, "Aggregate .. is aggregated" doesn't sound very good. We probably need to consult a native speaker (of both English and SQL :). == Details about the crash == create table ten(a int); insert into ten values (0),(1),(2),(3),(4),(5),(6),(7),(8),(9); create table t10 (a int , b int, c int); insert into t10 select a,a,a from ten; create table t11 as select * from t10; create table t2 as select * from t10; explain extended select a from t10 where c<3 or a in (select c from t12 union select max(t10.b) from t11 group by t11.c); Program received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7ffff7f1fb00 (LWP 4805)] 0x0000555555d0dd3f in Item_sum::register_sum_func (this=0x7fff89845680, thd=0x7fff9cbf6070, ref=0x7fff898457d0) at /home/psergey/dev-git/10.1/sql/item_sum.cc:391 (gdb) wher #0 0x0000555555d0dd3f in Item_sum::register_sum_func (this=0x7fff89845680, thd=0x7fff9cbf6070, ref=0x7fff898457d0) at /home/psergey/dev-git/10.1/sql/item_sum.cc:391 #1 0x0000555555d0d6f2 in Item_sum::check_sum_func (this=0x7fff89845680, thd=0x7fff9cbf6070, ref=0x7fff898457d0) at /home/psergey/dev-git/10.1/sql/item_sum.cc:177 #2 0x0000555555d1041e in Item_sum_hybrid::fix_fields (this=0x7fff89845680, thd=0x7fff9cbf6070, ref=0x7fff898457d0) at /home/psergey/dev-git/10.1/sql/item_sum.cc:1222 #3 0x00005555559c3cf2 in setup_fields (thd=0x7fff9cbf6070, ref_pointer_array=0x7fff898480f0, fields=..., mark_used_columns=MARK_COLUMNS_READ, sum_func_list=0x7fff89847ec8, allow_sum_func=true) at /home/psergey/dev-git/10.1/sql/sql_base.cc:7899 #4 0x0000555555a5b886 in JOIN::prepare (this=0x7fff89847b68, rref_pointer_array=0x7fff89845138, tables_init=0x7fff89845830, wild_num=0, conds_init=0x0, og_num=1, order_init=0x0, skip_order_by=true, group_init=0x7fff89845f38, having_init=0x0, proc_param_init=0x0, select_lex_arg=0x7fff89844ec0, unit_arg=0x7fff89844078) at /home/psergey/dev-git/10.1/sql/sql_select.cc:795 #5 0x0000555555af7082 in st_select_lex_unit::prepare (this=0x7fff89844078, thd_arg=0x7fff9cbf6070, sel_result=0x7fff89846118, additional_options=268435456) at /home/psergey/dev-git/10.1/sql/sql_union.cc:454 #6 0x0000555555d04d2c in subselect_union_engine::prepare (this=0x7fff89846140, thd_arg=0x7fff9cbf6070) at /home/psergey/dev-git/10.1/sql/item_subselect.cc:3590 #7 0x0000555555cfabe4 in Item_subselect::fix_fields (this=0x7fff89845f88, thd_param=0x7fff9cbf6070, ref=0x7fff89846270) at /home/psergey/dev-git/10.1/sql/item_subselect.cc:262 #8 0x0000555555d03f3d in Item_in_subselect::fix_fields (this=0x7fff89845f88, thd_arg=0x7fff9cbf6070, ref=0x7fff89846270) at /home/psergey/dev-git/10.1/sql/item_subselect.cc:3253 #9 0x0000555555c999cf in Item_cond::fix_fields (this=0x7fff89846178, thd=0x7fff9cbf6070, ref=0x7fff89846e98) at /home/psergey/dev-git/10.1/sql/item_cmpfunc.cc:4523 #10 0x00005555559c5bf1 in setup_conds (thd=0x7fff9cbf6070, tables=0x7fff89843330, leaves=..., conds=0x7fff89846e98) at /home/psergey/dev-git/10.1/sql/sql_base.cc:8627 #11 0x0000555555aa3eab in setup_without_group (thd=0x7fff9cbf6070, ref_pointer_array=0x7fff89846fd8, tables=0x7fff89843330, leaves=..., fields=..., all_fields=..., conds=0x7fff89846e98, order=0x0, group=0x0, hidden_group_fields=0x7fff89846d78, reserved=0x7fff9cbfa454) at /home/psergey/dev-git/10.1/sql/sql_select.cc:645 #12 0x0000555555a5b94b in JOIN::prepare (this=0x7fff89846a50, rref_pointer_array=0x7fff9cbfa430, tables_init=0x7fff89843330, wild_num=0, conds_init=0x7fff89846178, og_num=0, order_init=0x0, skip_order_by=false, group_init=0x0, having_init=0x0, proc_param_init=0x0, select_lex_arg=0x7fff9cbfa1b8, unit_arg=0x7fff9cbf9ab8) at /home/psergey/dev-git/10.1/sql/sql_select.cc:799 #13 0x0000555555a64f91 in mysql_select (thd=0x7fff9cbf6070, rref_pointer_array=0x7fff9cbfa430, tables=0x7fff89843330, wild_num=0, fields=..., conds=0x7fff89846178, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=2147748612, result=0x7fff89846378, unit=0x7fff9cbf9ab8, select_lex=0x7fff9cbfa1b8) at /home/psergey/dev-git/10.1/sql/sql_select.cc:3415 #14 0x0000555555a9a6f5 in mysql_explain_union (thd=0x7fff9cbf6070, unit=0x7fff9cbf9ab8, result=0x7fff89846378) at /home/psergey/dev-git/10.1/sql/sql_select.cc:24737 #15 0x0000555555a2a9aa in execute_sqlcom_select (thd=0x7fff9cbf6070, all_tables=0x7fff89843330) at /home/psergey/dev-git/10.1/sql/sql_parse.cc:5834 #16 0x0000555555a20c72 in mysql_execute_command (thd=0x7fff9cbf6070) at /home/psergey/dev-git/10.1/sql/sql_parse.cc:2961 #17 0x0000555555a2e4be in mysql_parse (thd=0x7fff9cbf6070, rawbuf=0x7fff89843088 "explain extended select a from t10 where c<3 or a in (select c from t12 union select max(t10.b) from t11 group by t11.c)", length=120, parser_state=0x7ffff7f1e630) at /home/psergey/dev-git/10.1/sql/sql_parse.cc:7319 BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Aggregate Functions
by Varun Gupta 26 Aug '16

26 Aug '16

Hi, As GSOC is nearing its end, I have written a blog which contains how can aggregate functions be written and executed to get the desired results.In the blog I also have added the technical details of this project on aggregate functions. Here are the links: Jira issue: https://jira.mariadb.org/browse/MDEV-7773 Blog Link: http://varunraiko.github.io/aggregate-functions/ Disclaimer: I hereby declare that all my work done so far (and future) by me on the Aggregate Functions’ project, is released under the NEW BSD licence. Regards, Varun

2 1

Re: [Maria-developers] Sachin weekly report
by Sachin Setia 24 Aug '16

24 Aug '16

Hi Sergei! Actually I completed the work on update and delete. Now they will use index for looking up records. But I am thinking I have done a lot of changes in optimizer which may break it , and also there are lots of queries where my code does not work, fixing this might take a long amount of time. I am thinking of a change in my existing code :- Suppose a table t1 create table t1 (a blob, b blob, c blob, unique(a,b,c)); In current code , for query like there will a KEY with only one keypart which points to field DB_ROW_HASH_1. It was okay for normal updates , insert and delete , but in the case of where optimization I have do a lot of stuff , first to match field (like in add_key_part), then see whether all the fields in hash_str are present in where or not, then create keys by calculating hash. I do this by checking the HA_UNIQUE_HASH flag in KEY , but this also makes (I think) optimizer code bad because of too much dependence. Also I need to patch get_mm_parts and get_mm_leaf function , which I think should not be patched. I am thinking of a another approach to this problem at server level instead of having just one keypart we can have 1+3 keypart. Last three keypart will be for field a, b, c and first one for DB_ROW_HASH_1 .These will be only at server level not at storage level. key_info->key_part will point at keypart containing field a , while key_part having field DB_ROW_HASH_1 will -1 index. By this way I do not have to patch more of optimizer code. But there is one problem , what should be the length of key_part? I am thinking of it equal to field->pack_length(), this would not work because while creating keys optimizer calls get_key_image() (which is real data so can exceed pack_lenght() in case of blob), so to get this work I have to patch optimizer where it calls get_key_image() and see if key is HA_UNIQUE_HASH . If yes then instead of get_key_image just use memcpy(key, field->ptr(), field->pack_length()); this wont copy the actual data, but we do not need actual data. I will patch handler methods like ha_index_read, ha_index_idx_read , multi_range_read_info_const basically handler methods which are related to index or range search. In these methods i need to calculate hash , which I can calculate from key_ptr but key_ptr doe not have actual data(in case of blobs etc).So to get the date for hash , I will make a field clone of (a,b,c etc) but there ptr will point in key_ptr. Then field->val_str() method will work simply and i can calculate hash. And also I can compare returned result with actual key in handler method itself. What do you think of this approach ? Regards sachin On Sat, Aug 20, 2016 at 11:16 PM, Sergei Golubchik <serg(a)mariadb.org> wrote: > Hi, Sachin! > > On Aug 19, Sachin Setia wrote: > > On Fri, Aug 19, 2016 at 2:42 PM, Sergei Golubchik <serg(a)mariadb.org> > wrote: > > > > > First. I believe you'll need to do your final evaluation soon, and > > > it will need to have a link to the code. Did you check google > > > guidelines about it? Is everything clear there? Do you need help > > > publishing your work in a format that google requires? > > > > > > They don't accept delays for any reasons, so even if your code is > > > not 100% complete and ready, you'd better still publish it and > > > submit the evaluation, because otherwise google will fail you and > > > that'd be too sad. > > > > > > If you'd like you can publish the google-way only the > > > unique-constraint part without further optimizer work. Or at least > > > please mention that you'd completed the original project and went > > > working on extensions. I mean, it's better than saying "the code is > > > not 100% complete" :) > > > > > Okay I am thinking of writing a blog post with a link to my github > > repository. > > Blog Link <http://sachin1001gsoc.blogspot.in/2016/08/gsoc-2016.html> > > Please check this. > > I think that'll do, yes. > > Regards, > Sergei > Chief Architect MariaDB > and security(a)mariadb.org >

2 3

Re: [Maria-developers] [Maria-discuss] Known limitation with TokuDB in Read Free Replication & parallel replication ?
by Kristian Nielsen 24 Aug '16

24 Aug '16

Rich Prohaska <prohaska7(a)gmail.com> writes: > On Tue, Aug 23, 2016 at 1:45 PM, Kristian Nielsen <knielsen(a)knielsen-hq.org> > wrote: >> In the original parallel replication patch, thd_report_wait_for() did not >> call back directly into tokudb_kill_query(). The kill happened >> asynchronously, in a separate background thread. Then there is no problem >> with TokuDB (or InnoDB) holding locks over the call to >> thd_report_wait_for(). >> >> I am considering re-introducing that orginal code - this might simplify >> TokuDB implementation (and would also simplify InnoDB implementation). >> I was never very happy about the way thd_report_wait_for() works currently. Ok, so I implemented this, available in this branch: https://github.com/knielsen/server/commits/toku_opr3 With this code, I am no longer able to reproduce any hangs with the tests I've been running so far. > BTW, the current_lock_tree_mutex logic is broken. The underlying tokuft > lock manager has one manager object (and its mutex) and many lock tree > objects (each with its own mutex). Since the thd_report_wait_for function > is called when holding the lock tree mutex (not the manager mutex), it can > be called in parallel; thus there is a race on the current_lock_tree_mutex > logic. Yes, and this is the main problem solved by doing the kill asynchronously. And from my tests, it looks like this was actually what was causing the problems - crash, assertion, and hangs. (I haven't determined this conclusively, but it seems at least plausible). It really was always an ugly hack around the locking problem with thd_report_wait_for() (InnoDB had a similar issue), it seems good to get rid of it. I still have the should_retry_lock_requests disabled in retry_all_lock_requests() - otherwise I get hangs. I haven't investigated this deeply yet. I also want to go back over the entire set of patches and see what needs cleaning up, and think about how this could go into 10.2 and possibly 10.1. - Kristian.

1 0

[Maria-developers] Commit emails reply-to maria-developers@lists.launchpad.net
by Sergey Vojtovich 24 Aug '16

24 Aug '16

Hi! Why do commit emails have "Reply-To: maria-developers(a)lists.launchpad.net"? I believe it is wrong at least for the following reasons: - I must be careful if I want to reply privately, e.g. something like "hold on, I'll review it in a few hours" doesn't deserve to hit maria-developers@ - I want a person, I'm replying to, to be primary recepient, but Reply-To substitutes his email with maria-devlopers@. Makes it easy to miss reviews. Sergei mentioned that one of reasons might be so that we want to keep commits(a)mariadb.org. I believe it doesn't work because: - if I do group reply, I end up with email that has CC: commits(a)mariadb.org Thanks, Sergey

1 0

Re: [Maria-developers] [Commits] 294961c: MDEV-10538: MariaDB fails to start without galera_recovery in systemd mode
by Sergey Vojtovich 24 Aug '16

24 Aug '16

Hi Nirbhay, Ok to push. I suggest that we avoid such overcomplicated constructs in our default systemd configuration. Consider integrating recovery into the server (I know it's not trivial) or creating "included" config for galera. Thanks, Sergey On Mon, Aug 22, 2016 at 06:38:07PM -0400, Nirbhay Choubey wrote: > revision-id: 294961cc4d4d55730e807b3f0a7f93a3dd4d4f7b (mariadb-10.1.16-14-g294961c) > parent(s): d0d99dec914d409d7889fcec2efd126edad10bde > author: Nirbhay Choubey > committer: Nirbhay Choubey > timestamp: 2016-08-22 18:38:06 -0400 > message: > > MDEV-10538: MariaDB fails to start without galera_recovery in systemd mode > > Update ExecStartPre scripts to not fail if 'galera_recovery' > script is not available. > > --- > support-files/mariadb.service.in | 6 ++++-- > support-files/mariadb@.service.in | 15 +++++++++------ > 2 files changed, 13 insertions(+), 8 deletions(-) > > diff --git a/support-files/mariadb.service.in b/support-files/mariadb.service.in > index 6b8b2ba..9b97172 100644 > --- a/support-files/mariadb.service.in > +++ b/support-files/mariadb.service.in > @@ -63,9 +63,11 @@ PermissionsStartOnly=true > # galera_recovery simply returns an empty string. In any case, however, > # the script is not expected to return with a non-zero status. > # It is always safe to unset _WSREP_START_POSITION environment variable. > +# Do not panic if galera_recovery script is not available. (MDEV-10538) > ExecStartPre=/bin/sh -c "systemctl unset-environment _WSREP_START_POSITION" > -ExecStartPre=/bin/sh -c "VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ] && \ > - systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1" > +ExecStartPre=/bin/sh -c "[ ! -e /usr/bin/galera_recovery ] && VAR= || \ > + VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ] \ > + && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1" > > # Needed to create system tables etc. > # ExecStartPre=/usr/bin/mysql_install_db -u mysql > diff --git a/support-files/mariadb@.service.in b/support-files/mariadb@.service.in > index 965e852..e941f21 100644 > --- a/support-files/mariadb@.service.in > +++ b/support-files/mariadb@.service.in > @@ -70,16 +70,19 @@ PermissionsStartOnly=true > # galera_recovery simply returns an empty string. In any case, however, > # the script is not expected to return with a non-zero status. > # It is always safe to unset _WSREP_START_POSITION%I environment variable. > +# Do not panic if galera_recovery script is not available. (MDEV-10538) > ExecStartPre=/bin/sh -c "systemctl unset-environment _WSREP_START_POSITION%I" > -ExecStartPre=/bin/sh -c "VAR=`/usr/bin/galera_recovery \ > - --defaults-file=@INSTALL_SYSCONF2DIR@/my%I.cnf`; [ $? -eq 0 ] && \ > - systemctl set-environment _WSREP_START_POSITION%I=$VAR || exit 1" > + > +ExecStartPre=/bin/sh -c "[ ! -e /usr/bin/galera_recovery ] && VAR= || \ > + VAR=`/usr/bin/galera_recovery --defaults-file=@INSTALL_SYSCONF2DIR@/my%I.cnf`; [ $? -eq 0 ] \ > + && systemctl set-environment _WSREP_START_POSITION%I=$VAR || exit 1" > + > # Alternate: (remove ConditionPathExists above) > # use [mysqld.INSTANCENAME] as sections in my.cnf > # > -#ExecStartPre=/bin/sh -c "VAR=`/usr/bin/galera_recovery \ > -# --defaults-group-suffix=%I`; [ $? -eq 0 ] && \ > -# systemctl set-environment _WSREP_START_POSITION%I=$VAR || exit 1" > +#ExecStartPre=/bin/sh -c "[ ! -e /usr/bin/galera_recovery ] && VAR= || \ > +# VAR=`/usr/bin/galera_recovery --defaults-group-suffix=%I`; [ $? -eq 0 ] \ > +# && systemctl set-environment _WSREP_START_POSITION%I=$VAR || exit 1" > > # Needed to create system tables etc. > # ExecStartPre=/usr/bin/mysql_install_db -u mysql > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

[Maria-developers] Please review MDEV-10604 Disable unstable MTR tests by default
by Elena Stepanova 24 Aug '16

24 Aug '16

Hi Sergey (Vojtovich), Please review the patch that we've discussed at length. Serg, FYI. The length of the "unstable" list shouldn't scare anyone. At the end, this task should cause no change in our current procedure, the tests will still be run in buildbot and hopefully on developers' local machines (and every external user who wants to run them can do it easily). The change should mainly help those who run tests as a part of their build process. The lists can be tuned later, now they have been created generously, to solve the most urgent issue in shortest time possible. Please note that it's still not a 100% guarantee that nothing will fail sporadically, I've already got an absolutely new random failure after disabling all known ones, so it will still require some iterations. Regards, Elena -------- Forwarded Message -------- Subject: [Commits] 5ac5050: MDEV-10604 Disable unstable MTR tests by default Date: Mon, 22 Aug 2016 14:44:30 +0300 (EEST) From: elenst(a)montyprogram.com Reply-To: maria-developers(a)lists.launchpad.net To: commits(a)mariadb.org revision-id: 5ac505064a75508213d4dbce388ff5ee62d1ad56 (mariadb-10.0.26-92-g5ac5050) parent(s): 05f61ba46046ca835071a73b9255e787dcce9255 author: Elena Stepanova committer: Elena Stepanova timestamp: 2016-08-22 14:38:27 +0300 message: MDEV-10604 Disable unstable MTR tests by default - add new syntax to disabled.def in MTR to distinguish "unstable" tests from disabled tests; - allow wildcards in test names in disabled.def; - add a new option enable-unstable and env option MTR_ENABLE_DISABLED; - mark "unstable" tests which have failed since the beginning of the year in buildbot on main 10.0 branch; - mark "unstable" tests which fail in Debian builds; - mark "unstable" tests which have been changed during the last two releases --- mysql-test/disabled.def | 67 +++++++++++++++++++- mysql-test/lib/mtr_cases.pm | 47 ++++++++++++-- mysql-test/mysql-test-run.pl | 12 +++- mysql-test/suite/binlog/disabled.def | 14 ++++ mysql-test/suite/federated/disabled.def | 15 +++++ mysql-test/suite/funcs_1/disabled.def | 16 ++++- mysql-test/suite/innodb/disabled.def | 21 ++++++ mysql-test/suite/parts/disabled.def | 13 ++++ mysql-test/suite/perfschema/disabled.def | 24 +++++++ mysql-test/suite/rpl/disabled.def | 39 ++++++++++++ mysql-test/suite/sys_vars/disabled.def | 14 ++++ storage/connect/mysql-test/connect/disabled.def | 12 ++++ .../mysql-test/mroonga/storage/disabled.def | 30 +++++++++ storage/tokudb/mysql-test/rpl/disabled.def | 26 ++++++++ storage/tokudb/mysql-test/tokudb/disabled.def | 29 +++++++++ storage/tokudb/mysql-test/tokudb_bugs/disabled.def | 27 ++++++++ .../tokudb/mysql-test/tokudb_parts/disabled.def | 26 ++++++++ 17 files changed, 421 insertions(+), 11 deletions(-) diff --git a/mysql-test/disabled.def b/mysql-test/disabled.def index 5c4b742..f0cb6ec 100644 --- a/mysql-test/disabled.def +++ b/mysql-test/disabled.def @@ -6,17 +6,82 @@ # # <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> # +# To mark a test as unstable, put '+' in front of the testcase name: +# + <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# '*' wildcard in testcase names is supported. +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# # Do not use any TAB characters for whitespace. # ############################################################################## + tablespace : disabled in MariaDB (no TABLESPACE table attribute) events_time_zone : Test is not predictable as it depends on precise timing. read_many_rows_innodb : Bug#11748886 2010-11-15 mattiasj report already exists log_tables-big : Bug#11756699 2010-11-15 mattiasj report already exists mysql_embedded : Bug#12561297 2011-05-14 Anitha Dependent on PB2 changes - eventum#41836 -#show_explain : Psergey: random timeout in range-checked-for-each record query. ssl_crl_clients_valid : broken upstream ssl_crl : broken upstream ssl_crl_clrpath : broken upstream file_contents : MDEV-6526 these files are not installed anymore lowercase_fs_on : lower_case_table_names=0 is not an error until 10.1 + +########################## + ++ bootstrap : Modified on 2016-06-18 (MDEV-9969) ++ create_delayed : MDEV-10605 - failed with timeout ++ create_or_replace : Modified on 2016-06-23 (MDEV-9728) ++ ctype_recoding : Modified on 2016-06-10 (MDEV-10181) ++ ctype_utf8 : Modified on 2016-06-21 (merge) ++ ctype_utf8mb4 : Modified on 2016-06-21 (merge) ++ events_1 : Modified on 2016-06-21 (MDEV-9524) ++ func_group : Modified on 2016-08-08 (MDEV-10468) ++ func_in : Modified on 2016-06-20 (MDEV-10020) ++ func_math : Modified on 2016-08-10 (merge) ++ func_misc : Modified on 2016-08-10 (merge) ++ grant2 : Modified on 2016-07-18 (MDEV-8569) ++ help : Modified on 2016-06-21 (MDEV-9524) ++ host_cache_size_functionality : MDEV-10606 - sporadic failure on shutdown ++ index_intersect_innodb : MDEV-10643 - failed with timeout ++ index_merge_innodb : MDEV-7142 - sporadic wrong execution plan ++ information_schema_stats : Modified on 2016-07-25 (MDEV-10428) ++ innodb_mysql_lock : MDEV-7861 - sporadic lock detection failure ++ insert_innodb : Modified on 2016-06-14 (merge from upstream) ++ loaddata : Modified on 2016-08-10 (merge) ++ locale : Modified on 2016-06-21 (merge) ++ mdev-504 : MDEV-10607 - sporadic "can't connect" ++ mdev375 : MDEV-10607 - sporadic "can't connect" ++ merge : MDEV-10607 - sporadic "can't connect" ++ multi_update : Modified on 2016-06-20 (MDEV-5973) ++ myisam_enable_keys-10506 : Added on 2016-08-10 (MDEV-10506) ++ mysqlcheck : Modified on 2016-08-10 (merge) ++ mysqldump : MDEV-10512 - sporadic assertion failure ++ mysqltest : MDEV-9269 - fails on Alpha ++ named_pipe : Modified on 2016-08-02 (MDEV-10383) ++ openssl_1 : Modified on 2016-07-11 (MDEV-10211) ++ parser : Modified on 2016-06-21 (merge) ++ pool_of_threads : MDEV-10100 - sporadic error on detecting max connections ++ ps_1general : Modified on 2016-07-12 (merge) ++ range : Modified on 2016-08-10 (merge) ++ range_mrr_icp : Modified on 2016-08-10 (merge) ++ query_cache : MDEV-10611 - sporadic mutex problem ++ shutdown : MDEV-10612 - sporadic crashes ++ sp-prelocking : Modified on 2016-08-10 (merge) ++ sp-security : MDEV-10607 - sporadic "can't connect" ++ ssl : MDEV-10211 - different ciphers on some platforms ++ ssl_ca : Modified on 2016-07-11 (MDEV-10211) ++ ssl_compress : Modified on 2016-07-11 (MDEV-10211) ++ ssl_timeout : Modified on 2016-07-11 (MDEV-10211) ++ stat_tables_par_innodb : MDEV-10515 - sporadic wrong results ++ status_user : Modified on 2016-06-20 (MDEV-8633) ++ subselect_innodb : MDEV-10614 - sporadic wrong results ++ temp_table : Modified on 2016-06-18 (MDEV-8569) ++ type_date : Modified on 2016-08-10 (merge) ++ type_datetime : Modified on 2016-06-16 (MDEV-9374) ++ view : Modified on 2016-08-10 (merge) ++ xtradb_mrr : Modified on 2016-08-04 (MDEV-9946) diff --git a/mysql-test/lib/mtr_cases.pm b/mysql-test/lib/mtr_cases.pm index b76b10d..886e5a6 100644 --- a/mysql-test/lib/mtr_cases.pm +++ b/mysql-test/lib/mtr_cases.pm @@ -36,6 +36,7 @@ our $do_test; our $skip_test; our $binlog_format; our $enable_disabled; +our $enable_unstable; our $default_storage_engine; our $opt_with_ndbcluster_only; @@ -313,6 +314,9 @@ sub combinations_from_file($$) } our %disabled; +our %disabled_wildcards; +our %unstable; +our %unstable_wildcards; sub parse_disabled { my ($filename, $suitename) = @_; @@ -321,10 +325,21 @@ sub parse_disabled { chomp; next if /^\s*#/ or /^\s*$/; mtr_error("Syntax error in $filename line $.") - unless /^\s*(?:([-0-9A-Za-z_\/]+)\.)?([-0-9A-Za-z_]+)\s*:\s*(.*?)\s*$/; + unless /^\s*(\+)?\s*(?:([-0-9A-Za-z_\/]+)\.)?([-0-9A-Za-z_\*]+)\s*:\s*(.*?)\s*$/; mtr_error("Wrong suite name in $filename line $.") - if defined $1 and defined $suitename and $1 ne $suitename; - $disabled{($1 || $suitename || '') . ".$2"} = $3; + if defined $2 and defined $suitename and $2 ne $suitename; + $suitename= $2 || $suitename || ''; + my ($casename, $unstable_flag, $text)= ($3, $1, $4); + + if ($casename =~ /\*/) { + # Wildcard + $disabled_wildcards{"$casename"}= $text; + $unstable_wildcards{"$casename"}= 1 if defined $unstable_flag; + } + else { + $disabled{$suitename . ".$casename"}= $text; + $unstable{$suitename . ".$casename"}= 1 if defined $unstable_flag; + } } close DISABLED; } @@ -721,16 +736,36 @@ sub collect_one_test_case { # Check for disabled tests # ---------------------------------------------------------------------- my $disable = $disabled{".$tname"} || $disabled{$name}; + if (not $disable) { + foreach my $w (keys %disabled_wildcards) { + if ($tname =~ /^$w/) { + $disable= $disabled_wildcards{$w}; + last; + } + } + } + my $unstable = $unstable{".$tname"} || $unstable{$name}; + if (not $unstable) { + foreach my $w (keys %unstable_wildcards) { + if ($tname =~ /^$w/) { + $unstable= $unstable_wildcards{$w}; + last; + } + } + } if (not defined $disable and $suite->{parent}) { $disable = $disabled{$suite->{parent}->{name} . ".$tname"}; } + if (not defined $unstable and $suite->{parent}) { + $unstable = $unstable{$suite->{parent}->{name} . ".$tname"}; + } if (defined $disable) { $tinfo->{comment}= $disable; - if ( $enable_disabled ) + if ( $enable_disabled or ( $enable_unstable and $unstable ) ) { - # User has selected to run all disabled tests - mtr_report(" - $tinfo->{name} wil be run although it's been disabled\n", + # User has selected to run disabled tests + mtr_report(" - $tinfo->{name} will be run although it's been disabled" . ($unstable ? " as unstable\n" : "\n"), " due to '$disable'"); } else diff --git a/mysql-test/mysql-test-run.pl b/mysql-test/mysql-test-run.pl index 5af19c8..82e84ec 100755 --- a/mysql-test/mysql-test-run.pl +++ b/mysql-test/mysql-test-run.pl @@ -1203,6 +1203,7 @@ sub command_line_setup { 'force-restart' => \$opt_force_restart, 'reorder!' => \$opt_reorder, 'enable-disabled' => \&collect_option, + 'enable-unstable' => \&collect_option, 'verbose+' => \$opt_verbose, 'verbose-restart' => \&report_option, 'sleep=i' => \$opt_sleep, @@ -1597,6 +1598,14 @@ sub command_line_setup { $ENV{'STAGING_RUN'}= $opt_staging_run; # -------------------------------------------------------------------------- + # Unstable tests + # -------------------------------------------------------------------------- + if ( $ENV{'MTR_ENABLE_UNSTABLE'} ) + { + collect_option('enable-unstable',1); + } + + # -------------------------------------------------------------------------- # Gcov flag # -------------------------------------------------------------------------- if ( ($opt_gcov or $opt_gprof) and ! $source_dist ) @@ -6455,7 +6464,8 @@ Options to control what test suites or cases to run twice to run only "big" tests. staging-run Run a limited number of tests (no slow tests). Used for running staging trees with valgrind. - enable-disabled Run also tests marked as disabled + enable-disabled Run also tests marked as disabled or unstable + enable-unstable Run also tests marked as unstable print-testcases Don't run the tests but print details about all the selected tests, in the order they would be run. skip-test-list=FILE Skip the tests listed in FILE. Each line in the file diff --git a/mysql-test/suite/binlog/disabled.def b/mysql-test/suite/binlog/disabled.def index 424e554..7a42685 100644 --- a/mysql-test/suite/binlog/disabled.def +++ b/mysql-test/suite/binlog/disabled.def @@ -6,8 +6,22 @@ # # <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> # +# To mark a test as unstable, put '+' in front of the testcase name: +# +<testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# # Do not use any TAB characters for whitespace. # ############################################################################## binlog_truncate_innodb : BUG#11764459 2010-10-20 anitha Originally disabled due to BUG#42643. Product bug fixed, but test changes needed binlog_spurious_ddl_errors : BUG#11761680 2013-01-18 astha Fixed on mysql-5.6 and trunk + +########################## + ++ binlog_commit_wait : sporadic failures: MDEV-10150 ++ binlog_dmls_on_tmp_tables_readonly : Added on 2016-05-04 (upstream) ++ binlog_xa_recover : sporadic failures: MDEV-8517 diff --git a/mysql-test/suite/federated/disabled.def b/mysql-test/suite/federated/disabled.def index bb2e4dd..279947b 100644 --- a/mysql-test/suite/federated/disabled.def +++ b/mysql-test/suite/federated/disabled.def @@ -6,6 +6,21 @@ # # <testcasename> : BUG#<xxxx> <comment> # +# To mark a test as unstable, put '+' in front of the testcase name: +# +<testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# # Do not use any TAB characters for whitespace. # ############################################################################## + +########################## + ++ federatedx : sporadic failures: MDEV-10617 ++ federated_innodb : sporadic failures: MDEV-10617, MDEV-8691 ++ federated_transactions : sporadic failures: MDEV-10617, MDEV-8691 ++ federated_innodb : sporadic failures: MDEV-8691 diff --git a/mysql-test/suite/funcs_1/disabled.def b/mysql-test/suite/funcs_1/disabled.def index 9bfe956..3cdc1c6 100644 --- a/mysql-test/suite/funcs_1/disabled.def +++ b/mysql-test/suite/funcs_1/disabled.def @@ -1,12 +1,22 @@ ############################################################################## # -# List the test cases that are to be disabled temporarely. +# List the test cases that are to be disabled temporarily. # # Separate the test case name and the comment with ':'. # -# <testcasename> : Comment test +# <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> # -# Don't use any TAB characters for whitespace. +# To mark a test as unstable, put '+' in front of the testcase name: +# +<testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# +# Do not use any TAB characters for whitespace. # ############################################################################## ++ processlist_priv_no_prot : Include file modified on 2016-07-12 (merge) ++ processlist_priv_ps : Include file modified on 2016-07-12 (merge) diff --git a/mysql-test/suite/innodb/disabled.def b/mysql-test/suite/innodb/disabled.def index 8cae44a..49e87c4 100644 --- a/mysql-test/suite/innodb/disabled.def +++ b/mysql-test/suite/innodb/disabled.def @@ -6,7 +6,28 @@ # # <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> # +# To mark a test as unstable, put '+' in front of the testcase name: +# +<testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# # Do not use any TAB characters for whitespace. # ############################################################################## +########################## + ++ innodb.binlog_consistent : sporadic failures: MDEV-10618 ++ innodb.innodb-alter-table : sporadic failures: MDEV-10619 ++ innodb.innodb-wl5522-debug-zip : sporadic failures: MDEV-10427 ++ innodb.innodb_bug30423 : sporadic failures: MDEV-7311 ++ innodb-alter-table : Modified on 2016-08-11 (MDEV-10535) ++ innodb-alter-tempfile : Modified on 2016-08-09 (MDEV-10469) ++ innodb-fk-warnings : Modified on 2016-07-18 (MDEV-8569) ++ innodb-fkcheck : Modified on 2016-06-13 (MDEV-10083) ++ innodb-wl5522 : rdiff file modified on 2016-08-10 (merge) ++ innodb_corrupt_bit : Modified on 2016-06-21 (merge) + diff --git a/mysql-test/suite/parts/disabled.def b/mysql-test/suite/parts/disabled.def index abbdbf7..17e2855 100644 --- a/mysql-test/suite/parts/disabled.def +++ b/mysql-test/suite/parts/disabled.def @@ -6,6 +6,14 @@ # # <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> # +# To mark a test as unstable, put '+' in front of the testcase name: +# +<testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# # Do not use any TAB characters for whitespace. # ############################################################################## @@ -15,3 +23,8 @@ parts.partition_mgm_lc1_archive : MDEV-5077 2013-09-27 svoj Cannot exchange part parts.partition_exchange_archive: MDEV-5077 2013-09-27 svoj Cannot exchange partition with archive table parts.partition_mgm_lc0_archive : MDEV-5077 2013-09-27 svoj Cannot exchange partition with archive table parts.partition_mgm_lc2_archive : MDEV-5077 2013-09-27 svoj Cannot exchange partition with archive table + +########################## + ++ partition_float_myisam : sporadic failures: MDEV-10621 ++ partition_int_myisam : sporadic failures: MDEV-10621 diff --git a/mysql-test/suite/perfschema/disabled.def b/mysql-test/suite/perfschema/disabled.def index d9e9e5b..8b4e222 100644 --- a/mysql-test/suite/perfschema/disabled.def +++ b/mysql-test/suite/perfschema/disabled.def @@ -6,6 +6,16 @@ # # <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> # +# To mark a test as unstable, put '+' in front of the testcase name: +# +<testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# '*' wildcard in testcase names is supported. +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# # Do not use any TAB characters for whitespace. # ############################################################################## @@ -22,3 +32,17 @@ binlog_edge_stmt: needs slave_relay_log_info binlog_ok_mix: needs slave_relay_log_info binlog_ok_row: needs slave_relay_log_info binlog_ok_stmt: needs slave_relay_log_info + +########################## + ++ digest_table_full : Modified on 2016-06-21 (merge) ++ func_file_io : MDEV-5708 - fails for s390x ++ func_mutex : MDEV-5708 - fails for s390x ++ rpl_gtid_func : Modified on 2016-06-21 (merge) ++ sizing_low : Modified on 2016-04-26 (5.6.30 merge) ++ socket_summary_by_event_name_func : sporadic failures: MDEV-10622 ++ start_server_low_digest : Modified on 2016-06-21 (merge) ++ statement_digest : Modified on 2016-06-21 (merge) ++ statement_digest_consumers : Modified on 2016-06-21 (merge) ++ statement_digest_long_query : Modified on 2016-06-21 (merge) ++ table_name : Added on 2016-04-26 (5.6.30 merge) diff --git a/mysql-test/suite/rpl/disabled.def b/mysql-test/suite/rpl/disabled.def index de3091a..1d68be9 100644 --- a/mysql-test/suite/rpl/disabled.def +++ b/mysql-test/suite/rpl/disabled.def @@ -6,6 +6,14 @@ # # <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> # +# To mark a test as unstable, put '+' in front of the testcase name: +# +<testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# # Do not use any TAB characters for whitespace. # ############################################################################## @@ -13,3 +21,34 @@ rpl_spec_variables : BUG#11755836 2009-10-27 jasonh rpl_spec_variables fails on PB2 hpux rpl_get_master_version_and_clock : Bug#11766137 Jan 05 2011 joro Valgrind warnings rpl_get_master_version_and_clock rpl_partition_archive : MDEV-5077 2013-09-27 svoj Cannot exchange partition with archive table + +########################## + ++ last_insert_id : sporadic failures: MDEV-10625 ++ rpl_auto_increment : sporadic failures: MDEV-8691 ++ rpl_auto_increment_bug45679 : sporadic failures: MDEV-8691 ++ rpl_auto_increment_update_failure : sporadic failures: MDEV-10625 ++ rpl_binlog_index : sporadic failures: MDEV-9501 ++ rpl_checksum_cache : sporadic failures: MDEV-10626 ++ rpl_circular_for_4_hosts : sporadic failures: MDEV-10627 ++ rpl_gtid_crash : sporadic failures: MDEV-9501 ++ rpl_gtid_master_promote : sporadic failures: MDEV-10628 ++ rpl_gtid_stop_start : sporadic failures: MDEV-10629 ++ rpl_gtid_until : sporadic failures: MDEV-10625 ++ rpl_ignore_table : Modified on 2016-06-22 ++ rpl_insert : sporadic failures: MDEV-9329 ++ rpl_insert_delayed : sporadic failures: MDEV-9329 ++ rpl_invoked_features : sporadic failures: MDEV-8691 ++ rpl_mdev6020 : sporadic failures: MDEV-1630 ++ rpl_mdev6386 : sporadic failures: MDEV-10631 ++ rpl_parallel : sporadic failures: MDEV-10632 ++ rpl_parallel_temptable : sporadic failures: MDEV-10356 ++ rpl_row_drop_create_temp_table : sporadic failures: MDEV-10626 ++ rpl_row_sp001 : sporadic failures: MDEV-9329 ++ rpl_semi_sync_uninstall_plugin : sporadic failures: MDEV-7140 ++ rpl_slave_grp_exec : sporadic failures: MDEV-10514 ++ rpl_switch_stm_row_mixed : sporadic failures: MDEV-10611 ++ rpl_sync : sporadic failures: MDEV-10633 ++ rpl_temporary_error2 : sporadic failures: MDEV-10634 ++ sec_behind_master-5114 : sporadic failures: MDEV-8518 ++ rpl_skip_replication : sporadic failures: MDEV-9268 diff --git a/mysql-test/suite/sys_vars/disabled.def b/mysql-test/suite/sys_vars/disabled.def index 7582152..06c394c 100644 --- a/mysql-test/suite/sys_vars/disabled.def +++ b/mysql-test/suite/sys_vars/disabled.def @@ -6,9 +6,23 @@ # # <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> # +# To mark a test as unstable, put '+' in front of the testcase name: +# +<testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# # Do not use any TAB characters for whitespace. # ############################################################################## innodb_flush_checkpoint_debug_basic: removed from XtraDB-26.0 table_open_cache_instances_basic: no such variable in MariaDB + +########################## + ++ autocommit_func2 : sporadic failures: MDEV-9329 ++ general_log_file_basic : Modified on 2016-08-09 (MDEV-10465) ++ slow_query_log_file_basic : Modified on 2016-08-09 (MDEV-10465) diff --git a/storage/connect/mysql-test/connect/disabled.def b/storage/connect/mysql-test/connect/disabled.def index 9b45709..0ce6169 100644 --- a/storage/connect/mysql-test/connect/disabled.def +++ b/storage/connect/mysql-test/connect/disabled.def @@ -6,6 +6,14 @@ # # <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> # +# To mark a test as unstable, put '+' in front of the testcase name: +# +<testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# # Do not use any TAB characters for whitespace. # ############################################################################## @@ -14,3 +22,7 @@ jdbc : Variable settings depend on machine configuration jdbc_new : Variable settings depend on machine configuration jdbc_oracle : Variable settings depend on machine configuration jdbc_postgresql : Variable settings depend on machine configuration + +########################## + ++ tbl : MDEV-9844, MDEV-10179 - sporadic crashes, valgrind warnings, wrong results diff --git a/storage/mroonga/mysql-test/mroonga/storage/disabled.def b/storage/mroonga/mysql-test/mroonga/storage/disabled.def index 6866adc..c5c44db 100644 --- a/storage/mroonga/mysql-test/mroonga/storage/disabled.def +++ b/storage/mroonga/mysql-test/mroonga/storage/disabled.def @@ -1,3 +1,25 @@ +############################################################################## +# +# List the test cases that are to be disabled temporarily. +# +# Separate the test case name and the comment with ':'. +# +# <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# To mark a test as unstable, put '+' in front of the testcase name: +# +<testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# '*' wildcard in testcase names is supported. +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# +# Do not use any TAB characters for whitespace. +# +############################################################################## + alter_table_add_index_token_filters_one_token_filter : Bundled Mroonga does not support token filter yet. alter_table_change_token_filter : Bundled Mroonga does not support token filter yet. fulltext_token_filters_skip : Bundled Mroonga does not support token filter yet. @@ -8,3 +30,11 @@ create_table_token_filters_table_comment_one_token_filter : Bundled Mroonga does foreign_key_create : Bundled Mroonga does not support this test yet. partition_insert : Bundled Mroonga does not support this test yet. +########################## + ++ column_datetime_32bit_2038 : sporadic failures: MDEV-9270 ++ column_datetime_32bit_before_unix_epoch : sporadic failures: MDEV-9270 ++ column_datetime_32bit_max : sporadic failures: MDEV-9270 ++ column_datetime_32bit_out_of_range : sporadic failures: MDEV-9270 ++ index_multiple_column_unique_date_32bit_equal : sporadic failures: MDEV-9270 ++ index_multiple_column_unique_date_order_32bit_desc : sporadic failures: MDEV-9270 diff --git a/storage/tokudb/mysql-test/rpl/disabled.def b/storage/tokudb/mysql-test/rpl/disabled.def index 4c1a9a3..c58408e 100644 --- a/storage/tokudb/mysql-test/rpl/disabled.def +++ b/storage/tokudb/mysql-test/rpl/disabled.def @@ -1,3 +1,25 @@ +############################################################################## +# +# List the test cases that are to be disabled temporarily. +# +# Separate the test case name and the comment with ':'. +# +# <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# To mark a test as unstable, put '+' in front of the testcase name: +# + <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# '*' wildcard in testcase names is supported. +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# +# Do not use any TAB characters for whitespace. +# +############################################################################## + rpl_tokudb_delete_pk: unreliable, uses timestamp differences rpl_tokudb_delete_pk_lookup1: unreliable, uses timestamp differences rpl_tokudb_update_pk_uc0_lookup0: unreliable, uses timestamp differences @@ -13,3 +35,7 @@ rpl_tokudb_write_unique_uc1: unreliable, uses timestamp differences rpl_tokudb_read_only_ff: unreliable, uses timestamp differences rpl_tokudb_read_only_tf: unreliable, uses timestamp differences rpl_tokudb_read_only_tt: unreliable, uses timestamp differences + +########################## + ++ * : MDEV-9891 - massive crashes on shutdown diff --git a/storage/tokudb/mysql-test/tokudb/disabled.def b/storage/tokudb/mysql-test/tokudb/disabled.def index c98a8aa..3545544 100644 --- a/storage/tokudb/mysql-test/tokudb/disabled.def +++ b/storage/tokudb/mysql-test/tokudb/disabled.def @@ -1,3 +1,25 @@ +############################################################################## +# +# List the test cases that are to be disabled temporarily. +# +# Separate the test case name and the comment with ':'. +# +# <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# To mark a test as unstable, put '+' in front of the testcase name: +# +<testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# '*' wildcard in testcase names is supported. +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# +# Do not use any TAB characters for whitespace. +# +############################################################################## + fast_update_binlog_mixed: No UPSERT in MariaDB 5.5 fast_update_binlog_row: No UPSERT in MariaDB 5.5 fast_update_binlog_statement: No UPSERT in MariaDB 5.5 @@ -28,3 +50,10 @@ type_timestamp_explicit: cluster_key_part: engine options on partitioned tables i_s_tokudb_lock_waits_released: unstable, race conditions i_s_tokudb_locks_released: unstable, race conditions + +########################## + ++ background_job_manager : sporadic failures: MDEV-10327 ++ cluster_filter_unpack_varchar : sporadic failures: MDEV-10636 + ++ * : MDEV-9891 - massive crashes on shutdown diff --git a/storage/tokudb/mysql-test/tokudb_bugs/disabled.def b/storage/tokudb/mysql-test/tokudb_bugs/disabled.def index d0d1a47..6554b9a 100644 --- a/storage/tokudb/mysql-test/tokudb_bugs/disabled.def +++ b/storage/tokudb/mysql-test/tokudb_bugs/disabled.def @@ -1,2 +1,29 @@ +############################################################################## +# +# List the test cases that are to be disabled temporarily. +# +# Separate the test case name and the comment with ':'. +# +# <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# To mark a test as unstable, put '+' in front of the testcase name: +# +<testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# +# Do not use any TAB characters for whitespace. +# +############################################################################## + 5585: times out, too many huge insert...selects db233: different execution path in mariadb, debug_sync point is not hit + +########################## + ++ checkpoint_lock : sporadic failures: MDEV-10637 ++ checkpoint_lock_3 : sporadic failures: MDEV-10637 + ++ * : MDEV-9891 - massive crashes on shutdown diff --git a/storage/tokudb/mysql-test/tokudb_parts/disabled.def b/storage/tokudb/mysql-test/tokudb_parts/disabled.def index 3252a46..7df2ac7 100644 --- a/storage/tokudb/mysql-test/tokudb_parts/disabled.def +++ b/storage/tokudb/mysql-test/tokudb_parts/disabled.def @@ -1,3 +1,25 @@ +############################################################################## +# +# List the test cases that are to be disabled temporarily. +# +# Separate the test case name and the comment with ':'. +# +# <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# To mark a test as unstable, put '+' in front of the testcase name: +# + <testcasename> : BUG#<xxxx> <date disabled> <disabler> <comment> +# +# '*' wildcard in testcase names is supported. +# +# The difference between disabled and unstable tests is that the former +# have to be skipped (either fail always, or are not applicable), +# while the latter fail sporadically, but can still have value. +# Importantly, unstable tests will be run in buildbot. +# +# Do not use any TAB characters for whitespace. +# +############################################################################## + partition_max_parts_hash_tokudb: 5.6 test not merged yet partition_max_parts_inv_tokudb: 5.6 test not merged yet partition_max_parts_key_tokudb: 5.6 test not merged yet @@ -7,3 +29,7 @@ partition_max_sub_parts_key_list_tokudb: 5.6 test not merged yet partition_max_sub_parts_key_range_tokudb: 5.6 test not merged yet partition_max_sub_parts_list_tokudb: 5.6 test not merged yet partition_max_sub_parts_range_tokudb: 5.6 test not merged yet + +########################## + ++ * : MDEV-9891 - massive crashes on shutdown _______________________________________________ commits mailing list commits(a)mariadb.org https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

2 1

Re: [Maria-developers] Patch for parallel replication DBUG_ASSERT check on DROP TABLE et al
by Kristian Nielsen 20 Aug '16

20 Aug '16

Monty writes: > Can you please check if this looks correct? > diff --git a/sql/mdl.cc b/sql/mdl.cc > index 28d2006..3b1f9f2 100644 > --- a/sql/mdl.cc > +++ b/sql/mdl.cc > @@ -17,6 +17,7 @@ > #include "sql_class.h" > #include "debug_sync.h" > #include "sql_array.h" > +#include "rpl_rli.h" > #include <hash.h> > #include <mysqld_error.h> > #include <mysql/plugin.h> > @@ -442,6 +443,7 @@ class MDL_lock > virtual void notify_conflicting_locks(MDL_context *ctx) = 0; > > virtual bitmap_t hog_lock_types_bitmap() const = 0; > + bool check_if_conflicting_replication_locks(MDL_context *ctx); This should probably be #ifdef NDEBUG, since you're only using it in debug build? > > /** List of granted tickets for this lock. */ > Ticket_list m_granted; > @@ -2290,6 +2292,44 @@ void MDL_scoped_lock::notify_conflicting_locks(MDL_context *ctx) > } > } > > +/** > + Check if there is any conflicting lock that could cause this thread > + to wait for another thread which is not ready to commit. > + This is always an error, as the upper level of parallel replication > + should not allow a scheduling of a conflicting DDL until all earlier > + transactions has commited. > + > + This function is only called for a slave using parallel replication > + and trying to get an exclusive lock for the table. > +*/ > + > +bool MDL_lock::check_if_conflicting_replication_locks(MDL_context *ctx) > +{ > + Ticket_iterator it(m_granted); > + MDL_ticket *conflicting_ticket; > + > + while ((conflicting_ticket= it++)) > + { > + if (conflicting_ticket->get_ctx() != ctx) > + { > + MDL_context *conflicting_ctx= conflicting_ticket->get_ctx(); > + > + /* > + If the conflicting thread is another parallel replication > + thread for the same master and it's not in commit stage, then > + the current transaction has started too early and something is > + seriously wrong. > + */ > + if (conflicting_ctx->get_thd()->rgi_slave && > + conflicting_ctx->get_thd()->rgi_slave->rli == > + ctx->get_thd()->rgi_slave->rli && > + !conflicting_ctx->get_thd()->rgi_slave->did_mark_start_commit) > + return 1; // Fatal error No, this is not correct. For example, the threads might be in different domains, or using different multi-source connections. You need a check like in thd_report_wait_for(). _If_ T1 and T2 are in the same parallel replication domain, _and_ T1 goes before T2, _and_ there is a DDL lock conflict but T1 has not started to commit, then it indicates a potential problem. Also, the comments are too vague, "Something is seriously wrong" is not a good description. What is wrong? Please describe the issue being checked more precisely. What is the role of each of the two threads involved? What is the situation being checked? Why is it wrong? Also, use some temporaries for conflicting_ctx->get_thd()->rgi_slave etc, so the condition becomes more readable. > + } > + } > + return 0; > +} > + > > /** > Acquire one lock with waiting for conflicting locks to go away if needed. > @@ -2355,6 +2395,11 @@ MDL_context::acquire_lock(MDL_request *mdl_request, ulong lock_wait_timeout) > if (lock->needs_notification(ticket) && lock_wait_timeout) > lock->notify_conflicting_locks(this); > > + DBUG_ASSERT(mdl_request->type != MDL_INTENTION_EXCLUSIVE || > + !(get_thd()->rgi_slave && > + get_thd()->rgi_slave->is_parallel_exec && > + lock->check_if_conflicting_replication_locks(this))); > + > mysql_prlock_unlock(&lock->m_rwlock); > > will_wait_for(ticket); > diff --git a/sql/mdl.h b/sql/mdl.h > index c4d792a..13de602 100644 > --- a/sql/mdl.h > +++ b/sql/mdl.h > @@ -910,7 +910,6 @@ class MDL_context > */ > MDL_wait_for_subgraph *m_waiting_for; > private: > - THD *get_thd() const { return m_owner->get_thd(); } > MDL_ticket *find_ticket(MDL_request *mdl_req, > enum_mdl_duration *duration); > void release_locks_stored_before(enum_mdl_duration duration, MDL_ticket *sentinel); > @@ -919,6 +918,7 @@ class MDL_context > MDL_ticket **out_ticket); > > public: > + THD *get_thd() const { return m_owner->get_thd(); } > void find_deadlock(); > > ulong get_thread_id() const { return thd_get_thread_id(get_thd()); } - Kristian.

2 4

Re: [Maria-developers] [Maria-discuss] Known limitation with TokuDB in Read Free Replication & parallel replication ?
by Rich Prohaska 18 Aug '16

18 Aug '16

Hello Kristian, I am working on a second variant of the kill query design that will only kill the pending lock request if any for the thd being killed. The previous design had a problem when killing the query that triggered the wait for call On Aug 15, 2016 7:26 PM, "Rich Prohaska" <prohaska7(a)gmail.com> wrote: Hello Kristian, I have a prototype of the TokuFT code that will cause ALL lock waiters to call their killed callback here: https://github.com/ prohaska7/tokuft/tree/kill_lockers On Mon, Aug 15, 2016 at 11:51 AM, Rich Prohaska <prohaska7(a)gmail.com> wrote: > Hello Kristian, > See attached snapshot of slave threads and tokudb locks. Thread 16 is > waiting for a tokudb lock held by thread 16, which is waiting for a tokudb > lock held by thread 14. Thread 14 is waiting for a prior transaction to > complete, presumably either thread 15 or 16. So, we have a deadlock that > tokudb can not detect because the ordering constraint is not available to > tokudb. I assume that the optimistic scheduler killed thread 16, but since > tokudb does not implement the kill_query function, the deadlock is only > resolved when the tokudb lock timer pops. > > On Mon, Aug 15, 2016 at 8:16 AM, Rich Prohaska <prohaska7(a)gmail.com> > wrote: > >> Hello Kristian, >> The simplest kill_query implementation for tokudb would just signal all >> of the pending lock request's condition variables. This would cause the >> killed callback to be called. A performance refinement, if necessary, >> would allow thread A (executing the kill_query function) to identify and >> signal a condition variable for a blocked thread B. >> >> On Mon, Aug 15, 2016 at 5:42 AM, Kristian Nielsen < >> knielsen(a)knielsen-hq.org> wrote: >> >>> Rich Prohaska <prohaska7(a)gmail.com> writes: >>> >>> > tokudb lock timeouts are resolving the replication stall. >>> unfortunately, >>> > the tokudb lock timeout is 4 seconds, so the throughput is almost zero. >>> >>> Yes. Sorry for not making it clear that my proof-of-concept patch was >>> incomplete... >>> >>> >> > I suspect that the poor slave replication performance for optimistic >>> >> > replication occurs because TokuDB does not implement the kill_query >>> >> > handlerton function. kill_handlerton gets called to resolve lock >>> wait >>> >>> >> Possibly, but I'm not sure it's that important. The kill will be >>> effective >>> >> as soon as the wait is over. >>> >>> No, you're absolutely right, after testing (and thinking) some more, I >>> realise that indeed the kill_query functionality is important. >>> >>> A possible scenario is, given transactions T1, T2, and T3 in that order: >>> >>> T3 acquires a lock on row R3, T2 similarly acquires R2. >>> Now T3 tries to acquire R2, but has to wait for T2 to release it. >>> Later T1 tries to acquire R3, also has to wait. >>> >>> At this point, we kill T3, since it is holding a lock (R3) needed by an >>> earlier transaction T1. However, T3 will not notice the kill until its >>> own >>> wait (on R2 held by T2) times out. T2 cannot release the lock because it >>> is >>> waiting for T1 to commit first. So we have a deadlock :-/ >>> >>> With InnoDB, the kill causes T3 to wake up immediately and roll back, so >>> that T1 can proceed without much delay. >>> >>> Ok, so something more is needed here. I see there is a killed_callback() >>> which seems to check for the kill, so I'm hoping that can be used with a >>> suitable wakeup of the offending lock_request (or all requests, >>> perhaps). But as I'm completely new to TokuDB, I still need some more >>> time >>> to read the code and try to understand how everything fits together... >>> >>> > TokuFT implements pessimistic locking and 2 phase locking algorithms. >>> This >>> > wiki describes locking and concurrency in a little more detail: >>> > https://github.com/percona/tokudb-engine/wiki/Transactions-a >>> nd-Concurrency. >>> >>> Thanks, this was quite helpful. >>> >>> > Yes, I think they are false positives since the thd_report_wait_for >>> API is >>> > called but it does NOT call the THD::awake function. >>> >>> Ah. Then it's probably normal, caused by the group-commit optimisation. >>> In >>> conservative mode, if two transactions T1 and T2 did not group commit on >>> the >>> master, then cannot be started in parallel on the slave. But T2 can >>> start as >>> soon as T1 has reached COMMIT. Thus, if T2 happens to conflict with T1, >>> there is a small window where T2 can need to wait on T1 until T1 has >>> completed its commit. >>> >>> Thanks, >>> >>> - Kristian. >>> >> >> >

2 2

Re: [Maria-developers] Freeing options & wrappers.
by Kristian Nielsen 16 Aug '16

16 Aug '16

Oleksandr Byelkin <sanja(a)montyprogram.com> writes: >>> Making something more I think do not worth it because: >>> 1) in 10.2 C/C will replace the library and it has no the problem >> But Connector-C should have exactly the same problem. If it frees the >> options on failed connect, it will break async API. If it does not, code >> that skips mysql_close() will leak. How does C/C solve this problem? >> If we are making any changes in 10.0, we should preferably make things work >> the same as they will work in 10.2. > Don't it has rewritten async operation? (at least I got such impression). My understanding is that Georg implemented all the async API also in connector-c. It should be using much the same code (mysql_async.c and such). But this is not about the implementation. This is about the semantics of how libmysqlclient works. My understanding is that connector-c is supposed to be binary compatible with libmysqlclient.so? So either connector-c removes any mysql_options() set when mysql_real_connect() fails without CLIENT_REMEMBER_OPTIONS. Or it does not. In the former case, there is the problem with MYSQL_OPT_NONBLOCK. In the latter case, the problem with leak when mysql_close() is not called. I really think you need to understand the problem fully, including how this is going to work in 10.2, before you make any changes to how the client library works... - Kristian.

2 3

Re: [Maria-developers] c56c966: MDEV-10563 Crash during shutdown in Master_info_index::any_slave_sql_running
by Kristian Nielsen 16 Aug '16

16 Aug '16

Vicențiu Ciorbaru <vicentiu(a)mariadb.org> writes: > This is undefined behavior: > // In GCC v6 this function _always_ returns 1. > ClassA::member_function() { > if (!this) > return 0; > return 1; > } Yeah, we shouldn't do this, agree. > make use of this construct. I have not changed the logic of the code, > however I am not sure whether "give_error_if_slave_running()" is supposed > to return TRUE if master_info_index is deleted (aka, we are performing a > shutdown). Probably there is no good reason to return TRUE. Unfortunately, there is a lot of code like this in replication, where it is not clear what the purpose is, but also very hard to be sure it's not needed. My guess would be - without knowing for sure - that the return value doesn't matter. This should only happen during shutdown anyway, and it seems to be used only for changing the values of some system-variables. If server is shutting down, then values of replication-related system-variables do not really matter. As to the patch, you're basically moving the check for NULL into all the callers. I'm not sure if the check is even needed for all callers, but then, I also do not think it's worth it to try and determine where it is needed and where not. But wouldn't it be nicer to instead change the give_error_if_slave_running() to be a normal function that takes the Master_info_index pointer as an argument (or a static method, if you want)? This way, you can keep the null-pointer check in the function. Also helps avoid a new bug if another caller is added. - Kristian.

2 1

[Maria-developers] Freeing options & wrappers.
by Oleksandr Byelkin 16 Aug '16

16 Aug '16

Hi! (sorry I forgot mailing list so resending the e-mail) So it is about https://jira.mariadb.org/browse/MDEV-10559 Original problem was that database drivers in Python & Perl (probably other) do not call mysql_close for MYSQL object if there was no connection happing (memory leak). MySQL has solution for this - freeing option on failed connection if user did not asked something else (we even has the same flag, but it did not work). So I took MySQL solution. But it appeared not so compatible with our async wrappers. Frankly speaking the code is difficult to read and debug because everything made over macros nor I have found decent description/overview of how it is done (thank you if somebody point to it) so I can not say that understand what it is doing, and all my suggestion based on how I understood it. So what we have: we allocate memory for some options set between mysql_init and mysql_real_connect, instead of "reall connect" called some wrapper which store some pointers to the options then call mysql_real_connect and if "real_connect" free them it fail on exit. I think that best solution is that wrapper 1) store current state of CLIENT_REMEMBER_OPTIONS 2) set the flag 3) call mysql_real_connect 4) if connect fail and there was not "remember options" then wrapper free options by itself Again, I am not 100% sure that it is possible nor really know the place were this should be added (if idea above ok, I probably find ho to make it).

2 4

Re: [Maria-developers] [Maria-discuss] Known limitation with TokuDB in Read Free Replication & parallel replication ?
by Kristian Nielsen 15 Aug '16

15 Aug '16

[Moving the discussion to maria-developers@, hope that is ok/makes sense...] Ok, so here is a proof-of-concept patch for this, which seems to make TokuDB work with optimistic parallel replication. The core of the patch is this line in lock_request.cc lock_wait_callback(callback_data, m_txnid, conflicts.get(i)); which ends up doing this: thd_report_wait_for (requesting_thd, blocking_thd); All the rest of the patch is just getting the right information around between the different parts of the code. I put this on top of Jocelyn Fournier's tokudb_rpl.rpl_parallel_optimistic patches, and pushed it on my github: https://github.com/knielsen/server/tree/toku_opr2 With this patch, the test case passes! So that's promising. Some things still left to do for this to be a good patch: - I think the callback needs to trigger also for an already waiting transaction, in case another transaction arrives later to contend for the same lock, but happens to get the lock earlier. I can look into this. - This patch needs linear time (in number of active transactions) per callback to find the THD from the TXNID, maybe that could be optimised. - Probably the new callback etc. needs some cleanup to better match TokuDB code organisation and style. - And testing, of course. I'll definitely need some help there, as I'm not familiar with how to run TokuDB efficiently. Any thoughts or comments? - Kristian.

2 8

Re: [Maria-developers] Status of InnoDB 5.7 merge to 10.2
by Kristian Nielsen 13 Aug '16

13 Aug '16

Jan Lindström <jan.lindstrom(a)mariadb.com> writes: > I have reached to situation where I need help fixing remaining test > failures and mentoring is current status good enough for 10.2 Beta. I need > help on test failures on parts where I do not follow why test is failing or > server crashes I do not know how to fix. > > First of all current merge result is on bb-10.2-jan with > commit ca4f194460da469e29624d930e6d0027349d047c > > Higher level of status is on https://jira.mariadb.org/browse/MDEV-10024 > > Note that mtr thinks currently that innodb_plugin compiled using static > linking is xtradb (but not have_xtradb as xtradb is not compiled). > Furthermore galera is not compiled (waiting for codership). > > Here are current issues where I need help and possible persons who could > help: > > (1) Parallel replication lock wait/deadlock handling on InnoDB 5.7 with > refactored lock0lock.cc does not work > https://jira.mariadb.org/browse/MDEV-10550 (Monty, Kristian) >From a quick look at the patch, this seems to be because of code removed from innobase_kill_query(). Specifically: > @@ -4799,65 +5511,11 @@ innobase_kill_query( > DBUG_ENTER("innobase_kill_query"); > DBUG_ASSERT(hton == innodb_hton_ptr); > - if (!wsrep_thd_is_BF(trx->mysql_thd, FALSE) && > - trx->abort_type == TRX_SERVER_ABORT) { > - ut_ad(!lock_mutex_own()); > - lock_mutex_enter(); When parallel replication discovers a case where a transaction T1 goes to wait on a row lock held by a later (in the replication stream) transaction T2, T2 will be killed to avoid a deadlock. In this situation, we are already holding lock mutex. So this code is there in 10.1 to prevent re-taking the mutex, code that is removed here in your patch. > + if (trx != NULL) { > + /* Cancel a pending lock request if there are any */ > + lock_trx_handle_wait(trx); > +lock_trx_handle_wait( > +/*=================*/ > + trx_t* trx) /*!< in/out: trx lock state */ > { > dberr_t err; > - ulint heap_no; > > - ut_ad(!dict_index_is_clust(index)); > - ut_ad(!dict_index_is_online_ddl(index)); > - ut_ad(block->frame == page_align(rec)); > - ut_ad(page_rec_is_user_rec(rec) || page_rec_is_supremum(rec)); > - ut_ad(rec_offs_validate(rec, index, offsets)); > - ut_ad(mode == LOCK_X || mode == LOCK_S); > + lock_mutex_enter(); But in the new code here, the protection on lock_mutex_enter is no longer there, lock_mutex_enter() is done unconditionally. The code which sets the flag TRX_SERVER_ABORT seems to be still there, though: > +void > +lock_report_waiters_to_mysql( > +/*=======================*/ > + struct thd_wait_reports* waitee_buf_ptr, /*!< in: set of trxs */ > + THD* mysql_thd, /*!< in: THD */ > + trx_id_t victim_trx_id) /*!< in: Trx selected > + as deadlock victim, if > + any */ > +{ > + struct thd_wait_reports* p; > + struct thd_wait_reports* q; > + ulint i; > + > + p = waitee_buf_ptr; > + while (p) { > + i = 0; > + while (i < p->used) { > + trx_t *w_trx = p->waitees[i]; > + /* There is no need to report waits to a trx already > + selected as a victim. */ > + if (w_trx->id != victim_trx_id) { > + /* If thd_report_wait_for() decides to kill the > + transaction, then we will get a call back into > + innobase_kill_query. We mark this by setting > + current_lock_mutex_owner, so we can avoid trying > + to recursively take lock_sys->mutex. */ > + w_trx->abort_type = TRX_REPLICATION_ABORT; > + thd_report_wait_for(mysql_thd, w_trx->mysql_thd); > + w_trx->abort_type = TRX_SERVER_ABORT; abort_type is set to TRX_REPLICATION_ABORT around thd_report_wait_for(), during which innobase_kill_query() may be invoked with lock mutex already held. So it would appear that a first attempt should be to restore the protection around lock_mutex_enter()... Note that, if w_trx->abort_type is not functional in your current patch, the parallel replication code was originally working without this, you can see in the snippet above the comment about current_lock_mutex_owner, which was (mistakenly) not removed when this code was changed to use Galera stuff. So you can check git history and go back to the original code (using current_lock_mutex_owner) if necessary, perhaps. Or maybe w_trx->abort_type is fine, I'm not familiar with that change. > (2) mysqld: /home/jan/mysql/10.2/sql/handler.cc:2692: int > handler::ha_index_first(uchar*): Assertion `table_share->tmp_table != > NO_TMP_TABLE || m_lock_type != 2' failed. > https://jira.mariadb.org/browse/MDEV-10549 (Serg) > > (3) Some of the debug sync waits do not work InnoDB 5.7 > https://jira.mariadb.org/browse/MDEV-10548 (Monty, Serg) > > (4) Test multi_update_innodb fails with InnoDB 5.7 > https://jira.mariadb.org/browse/MDEV-10550 (Petrunia) > > (5) Test innodb.defrag_mdl-9155 hangs on InnoDB 5.7 > https://jira.mariadb.org/browse/MDEV-10551 (Jan, Serg) > > (6) IS tables are not found on 10.2 InnoDB 5.7 > https://jira.mariadb.org/browse/MDEV-10200 (Jan) Hope this helps, - Kristian.

2 1

[Maria-developers] MariaDB Developers Meetup
by Ian Gilfillan 10 Aug '16

10 Aug '16

The MariaDB Developers Meetup will be taking place from the 6th to the 8th of October in Amsterdam. https://mariadb.org/2016-mariadb-developers-meetup/ Details of where to sign up, and where to suggest topics for the meetup are on that page. Please feel free to suggest topic you're interested in presenting, as well as topics you'd like to see covered. The meetup is scheduled to be just after Percona Live, which also takes place in Amsterdam from the 3rd to the 5th. The Foundation has arranged a discount so that anyone signing up for the MariaDB Developers Meetup and that also wants to attend Percona Live will receive a discount for that event (the Developers Meetup has no charge). Percona Live are also looking for talks or sessions, so if you have a topic that would be of interest to the broader community, feel free to submit it at https://www.percona.com/live/plam16/program

1 1

[Maria-developers] Please review MDEV-10468 Assertion `nr >= 0.0' failed in Item_sum_std::val_real()
by Alexander Barkov 08 Aug '16

08 Aug '16

Hello Alexey, Can you please review a patch for MDEV-10468? Thanks!

2 3

[Maria-developers] Please review MDEV-10500 CASE/IF Statement returns multiple values and shifts further result values to the next column
by Alexander Barkov 08 Aug '16

08 Aug '16

Hi Serg, Please review a patch for MDEV-10500. Thanks.

2 2

Re: [Maria-developers] 634e130: MDEV-10424 - Assertion `ticket == __null' failed in MDL_request::set_type
by Sergei Golubchik 08 Aug '16

08 Aug '16

Hi, Sergey! On Aug 04, Sergey Vojtovich wrote: > revision-id: 634e130b00bc900d1b7b8a8df6c13b1701f84b30 (mariadb-5.5.50-24-g634e130) > parent(s): 2f55f0b010619a09870ba704b55dd1426a158f1d > committer: Sergey Vojtovich > timestamp: 2016-08-04 15:51:17 +0400 > message: > > MDEV-10424 - Assertion `ticket == __null' failed in MDL_request::set_type > > Reexecution of prepared "ANALYZE TABLE merge_table, table" may miss to > reinitialize "table" for subsequent execution and trigger assertion failure. > > This happens because MERGE engine may adjust table->next_global chain, which > gets cleared by close_thread_tables()/ha_myisammrg::detach_children() later. > Since reinitilization iterates next_global chain, it won't see tables following > merge table. > > Fixed by appending saved next_global chain after merge children. looks ok to me, but let elena do some extra testing of this, please Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] GSoC weekly reports (Unique indexes for blobs)
by Shubham Barai 03 Aug '16

03 Aug '16

GSoC (week 5) Hello everyone, Here is the list of things I have done in the 5th week of GSoC. 1.implemented unique key violation with a hash collision. (actual row comparison is remaining ). 2.modified hash function for data types like varchar and binary data types. 3.fixed a bug which was causing a server to crash for complex unique keys. 4.added support to allow any number of nulls which will not cause any unique key violation. 5.added test cases for above features. On 22 June 2016 at 16:36, Shubham Barai <shubhambaraiss(a)gmail.com> wrote: > Hi, > > can we discuss on IRC first? > > > Regards, > Shubham > > On 22 June 2016 at 13:21, Jan Lindström <jan.lindstrom(a)mariadb.com> wrote: > >> Hi, >> >> Please commit and push these changes to your git branch I have not yet >> seen them, in my opinion as this is only a working branch you can push >> often. I still fail to see any test cases on InnoDB branch, do you have >> more than one branch and if you have why ? Depending on extent of these >> changes my estimate is that you are behind schedule to complete project in >> time. Based on your progress report you are still missing update and delete >> and redo-logging. For alter table you should start from forcing copy-method >> and then if time permits develop on-line method. This naturally only after >> everything else >> has been completed and tested. >> >> R: Jan >> >> On Tue, Jun 21, 2016 at 11:06 PM, Shubham Barai <shubhambaraiss(a)gmail.com >> > wrote: >> >>> GSoC (week 4) >>> >>> Hello everyone, >>> >>> After working on create table operation,next thing I had to work on was >>> insert operations.So I explored some of the functions like >>> row_ins_scan_index_for_duplicates, btr_pcur_get_rec to get clear >>> understanding about how to implement duplicate search on hash index. >>> There was a problem in hash function that I wrote .It would calculate >>> same hash value for two different keys if the prefix length of blob key >>> part was zero. Now it seems to be working after I checked it in debugger.I >>> still have to modify it for data types like varchar etc. >>> I have added test cases for insert operations in myisam. >>> In MyIsam, I found one problem in update operation. When updating a >>> row,if the key is conflicting then server crashes because some pointer goes >>> invalid in compare_record. I haven't fixed this issue yet. >>> >>> I also modified some functions in dict0load.cc to adjust some members >>> of dict_index_t for a new index type.The main problem is that index entry >>> for hash based index cointains only two fields(hash value and row id) while >>> dict_index_t contains hash field and other user defined fields which are >>> used to calculate hash value.Some of the operations like alter table( e.g. >>> rename column) needs to get access to all fields while other functions like >>> rec_get_offsets and row_build_index_entry_low needs to get access to only >>> hash field and row id. I am still working on this to find efficient >>> solution to this problem. >>> >>> On 16 June 2016 at 23:29, Sergei Golubchik <vuvova(a)gmail.com> wrote: >>> >>>> Hi, Shubham! >>>> >>>> What I wanted to say on IRC was: >>>> >>>> here's what the comment of cmp_dtuple_rec_with_match_low() says: >>>> >>>> ............... If rec has an externally stored field we do not >>>> compare it but return with value 0 if such a comparison should be >>>> made. >>>> >>>> Note that blobs are externally stored fields in InnoDB, so, I think, >>>> this means that you cannot use cmp_dtuple_rec() to compare blobs. >>>> >>>> Regards, >>>> Sergei >>>> Chief Architect MariaDB >>>> and security(a)mariadb.org >>>> >>> >>> >>> _______________________________________________ >>> Mailing list: https://launchpad.net/~maria-developers >>> Post to : maria-developers(a)lists.launchpad.net >>> Unsubscribe : https://launchpad.net/~maria-developers >>> More help : https://help.launchpad.net/ListHelp >>> >>> >> >

1 5

Re: [Maria-developers] bdef42c: MDEV-10365 - Race condition in error handling of INSERT DELAYED
by Sergei Golubchik 03 Aug '16

03 Aug '16

Hi, Sergey! On Aug 03, Sergey Vojtovich wrote: > revision-id: bdef42c54f8938d28d9a421271b11e09e88735a4 (mariadb-5.5.50-19-gbdef42c) > parent(s): 45ec5d673a1f5c736baec8ea9e2065bca4350324 > committer: Sergey Vojtovich > timestamp: 2016-08-03 11:49:35 +0400 > message: > > MDEV-10365 - Race condition in error handling of INSERT DELAYED > > Shared variables of Delayed_insert may be updated without mutex protection > when delayed insert thread gets an error. > > Re-acquire mutex earlier, so that shared variables are protected. ok to push Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] fc42991: MDEV-7901: re-implement analyze table for low impact
by Sergei Golubchik 03 Aug '16

03 Aug '16

Hi, Oleksandr! Looks mostly ok, see few comments below On Aug 01, Oleksandr Byelkin wrote: > revision-id: fc42991720838e165ad448b6707602cded92faa4 (mariadb-10.2.1-9-gfc42991) > parent(s): 08683a726773f8cdf16a4a3dfb3920e5f7842481 > committer: Oleksandr Byelkin > timestamp: 2016-08-01 19:25:45 +0200 > message: > > MDEV-7901: re-implement analyze table for low impact > > Table before collecting engine independent statistics now is reopened in read mode, > InnoDB allow write operations in this case. > > --- > mysql-test/r/stat_tables_innodb_debug.result | 27 +++ > mysql-test/t/stat_tables_innodb_debug.test | 37 ++++ > sql/sql_admin.cc | 295 +++++++++++++++------------ > storage/innobase/handler/ha_innodb.cc | 1 + > storage/xtradb/handler/ha_innodb.cc | 2 +- > 5 files changed, 227 insertions(+), 135 deletions(-) > > diff --git a/mysql-test/r/stat_tables_innodb_debug.result b/mysql-test/r/stat_tables_innodb_debug.result > new file mode 100644 > index 0000000..b02848d > --- /dev/null > +++ b/mysql-test/r/stat_tables_innodb_debug.result > @@ -0,0 +1,27 @@ > +# > +# MDEV-7901: re-implement analyze table for low impact > +# > +create table t1(a int primary key) engine=innodb; > +insert into t1 values (0),(1),(2),(3),(4),(5),(6),(7),(8),(9); you use t1 only to populate t2. You don't need to create a table for thatr, use seq_0_to_9 - https://mariadb.com/kb/en/mariadb/sequence/ > +create table t2 (a int, b int, c int, key IDX1(a), key IDX2(a,b)) engine=innodb; > +insert into t2 select a/10, a/2, a from test.t1; > +SET DEBUG_SYNC='statistics_collection_start1 SIGNAL analyzing WAIT_FOR written'; > +analyze table t2 persistent for all; > +connect con1, localhost, root,,; > +connection con1; > +SET DEBUG_SYNC= 'now WAIT_FOR analyzing'; > +select count(*) from t2; > +count(*) > +10 > +insert into t2 values (333,333,333); > +update t2 set a=1; > +SET DEBUG_SYNC= 'now SIGNAL written'; > +connection default; > +Table Op Msg_type Msg_text > +test.t2 analyze status Engine-independent statistics collected > +test.t2 analyze status OK > +select count(*) from t2; > +count(*) > +11 > +set debug_sync='RESET'; > +drop table t1,t2; good direct test > diff --git a/sql/sql_admin.cc b/sql/sql_admin.cc > index 578b78d..933f3d0 100644 > --- a/sql/sql_admin.cc > +++ b/sql/sql_admin.cc > @@ -812,7 +773,73 @@ static bool mysql_admin_table(THD* thd, TABLE_LIST* tables, > > if (compl_result_code == HA_ADMIN_OK && collect_eis) > { ok, I've quickly checked the code and I think (not sure) that there is no need to reopen for innodb. But as you do, please, show the benefits of it - add a test case for MyISAM too (remember, TL_READ allows concurrent inserts). besides, this reopening code needs a comment explaining why you're doing it. > + enum_sql_command save_sql_command= lex->sql_command; > + trans_commit_stmt(thd); > + trans_commit(thd); > + thd->open_options|= extra_open_options; > + close_thread_tables(thd); > + table->table= NULL; > + thd->mdl_context.release_transactional_locks(); > + lex->sql_command= save_sql_command; where is lex->sql_command changed? > + table->mdl_request.init(MDL_key::TABLE, table->db, table->table_name, > + MDL_SHARED_NO_READ_WRITE, MDL_TRANSACTION); > + table->mdl_request.set_type(MDL_SHARED_READ); > + > + table->lock_type= TL_READ; > + open_error= open_only_one_table(thd, table, > + repair_table_use_frm, > + (view_operator_func != NULL)); can one analyze a view? how does that work? > + thd->open_options&= ~extra_open_options; ... Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 1

[Maria-developers] MariaDB memory usage when rebuilding a large table
by Mark Wadham 03 Aug '16

03 Aug '16

Hi, Apologies if this is the wrong place to ask this question but it seemed like the most apt. We are working on operational processes for online schema changes with MariaDB 10.1.16 and have encountered memory issues which are proving difficult to reliably solve. If anyone on this list could explain to me in detail how to calculate the memory footprint of modifying a large table with DDL statements I would be very grateful. Our test case is a 3-node galera cluster on which we want to alter a ~160GB table that has . Each node has 64GB ram and (originally) no swap space allocated. Even with the innodb buffer pool lowered to 16GB we consistently reached an OOM condition trying to add a column to the table. If we lower the innodb buffer pool to 16GB *and* provision 16GB of swap, the DDL completes. With the innodb buffer pool left at 50GB and allocated swap of 16GB, same OOM condition. Since this is an operational procedure we are testing the process under heavy load, but the node that is applying the DDL is first placed into RSU mode so the writesets simply build up in the gcache while it's executing (and obviously traffic from the production environment would be directed away from the node while this is happening). We are also increasing the gcache to 800GB on the node as part of this process to ensure that we have plenty of time for it to complete. I'm fairly sure that the gcache is simply a continuous write-to-disk process rather than anything involving significant memory, so *hope* that's not a contributing factor here. I am however quite surprised that executing DDL on a table can be so memory-intensive. Our next test run will be with the innodb buffer pool set to 0 on the node executing the DDL with no swap allocated to see if it can complete purely in-memory. Logically it should be able to if this is a memory constraint rather than a bug. But fundamentally we need to be able to determine with confidence how much memory we need to successfully execute DDL on a table of {n} size. Thanks, Mark

1 1

Re: [Maria-developers] e252dae: MDEV-9872 - Add common optimized CRC32 function interface
by Sergei Golubchik 03 Aug '16

03 Aug '16

Hi, Sergey! So, what did you do here? Did you change InnoDB's version of crc32 or just moved it to extra/ ? You removed mysys/my_crc32.c - where will crc32 come from on x86? Why did you name new function crc32_vpmsum instead of just naming it crc32 and avoding #ifdef's elsewhere? On Jul 05, Sergey Vojtovich wrote: > revision-id: e252dae86392a2b50d647a380db36fb8b01bf405 (mariadb-10.2.1-5-ge252dae) > parent(s): 72c94872835f27b536f617978dbc664ddb57da06 > committer: Sergey Vojtovich > timestamp: 2016-07-05 18:09:23 +0400 > message: > > MDEV-9872 - Add common optimized CRC32 function interface > > Move crc32-vpmsum to extra. > Compile static crc32-vpmsum instead of adding sources directly. > Make use of crc32-vpmsum via my_checksum(). > > Based on contribution by Daniel Black. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 1

Re: [Maria-developers] f89d1f5: MDEV-10421 duplicate CHECK CONSTRAINTs.
by Sergei Golubchik 02 Aug '16

02 Aug '16

Hi, Alexey! Thanks! See below On Jul 31, Alexey Botchkov wrote: > revision-id: f89d1f55576136fb229392cb9a2997b9c0131241 (mariadb-10.2.1-9-gf89d1f5) > parent(s): 08683a726773f8cdf16a4a3dfb3920e5f7842481 > committer: Alexey Botchkov > timestamp: 2016-07-31 14:04:52 +0400 > message: > > MDEV-10421 duplicate CHECK CONSTRAINTs. > > mysql_prepare_create_table fixed so it doesn't let duplicating > constraint names. Syntax for CONSTRAINT IF NOT EXISTS added > and handled in mysql_alter_table. > > diff --git a/mysql-test/r/alter_table.result b/mysql-test/r/alter_table.result > index 3461038..5c1da48 100644 > --- a/mysql-test/r/alter_table.result > +++ b/mysql-test/r/alter_table.result > @@ -2057,3 +2057,23 @@ t1 CREATE TABLE `t1` ( > KEY `i1` (`a`) COMMENT 'comment2' > ) ENGINE=MyISAM DEFAULT CHARSET=latin1 > DROP TABLE t1; > +# > +# MDEV-10421 duplicate CHECK CONSTRAINTs > +# > +CREATE TABLE t1 (a INT, b INT) engine=myisam; > +ALTER TABLE t1 ADD CONSTRAINT IF NOT EXISTS `min` CHECK (a+b > 100); > +ALTER TABLE t1 ADD CONSTRAINT `min` CHECK (a+b > 100); > +ERROR HY000: Duplicate constraint name 'min' > +ALTER TABLE t1 ADD CONSTRAINT IF NOT EXISTS `min` CHECK (a+b > 100); > +Warnings: > +Note 4027 Duplicate constraint name 'min' > +ALTER TABLE t1 ADD CONSTRAINT `mini` CHECK (a+b > 100); please also add a test like create table (a int, b int, constraint min check (a>5), constraint min check (b>5)); > +SHOW CREATE TABLE t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` int(11) DEFAULT NULL, > + `b` int(11) DEFAULT NULL, > + CONSTRAINT `min` CHECK (a+b > 100), > + CONSTRAINT `mini` CHECK (a+b > 100) > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +DROP TABLE t1; > diff --git a/sql/field.h b/sql/field.h > index 13a0e91..7b271b0 100644 > --- a/sql/field.h > +++ b/sql/field.h > @@ -606,6 +606,7 @@ class Virtual_column_info: public Sql_alloc > /* Flag indicating that the field is physically stored in the database */ > bool stored_in_db; > bool utf8; /* Already in utf8 */ > + bool create_if_not_exists; Hmm, I don't think it's a good idea. Virtual_column_info is used for storing run-time information about generated columns, defaults, and check constraints. It's kind of wrong to put the a parser-time check-constraint-only flag. > /* The expression to compute the value of the virtual column */ > Item *expr_item; > /* Text representation of the defining expression */ > diff --git a/sql/share/errmsg-utf8.txt b/sql/share/errmsg-utf8.txt > index 8dfa519..261021b 100644 > --- a/sql/share/errmsg-utf8.txt > +++ b/sql/share/errmsg-utf8.txt > @@ -7214,3 +7214,5 @@ ER_CALCULATING_DEFAULT_VALUE > eng "Got an error when calculating default value for %`s" > ER_EXPRESSION_REFERS_TO_UNINIT_FIELD 01000 > eng "Expression for field %`-.64s is refering to uninitialized field %`s" > +ER_DUP_CONSTRAINT_NAME > + eng "Duplicate constraint name '%-.192s'" Say "Duplicate CHECK constraint" here, because there is already ER_FK_DUP_NAME with the text "Duplicate foreign key constraint name" Alternatively, (better, I think) generalize ER_FK_DUP_NAME to say Duplicate %s constraint name: %`s (note correct quoting %` and no need to limit the length, it's an identifier anyway, it cannot be arbitrarily long). Then it can be used like my_error(ER_DUP_CONSTRAINT_NAME, MYF(0), "FOREIGN KEY", name); my_error(ER_DUP_CONSTRAINT_NAME, MYF(0), "CHECK", name); That also means renaming the error from ER_FK_DUP_NAME to ER_DUP_CONSTRAINT_NAME, and putting somewhere a compatibility define #define ER_FK_DUP_NAME ER_DUP_CONSTRAINT_NAME > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index 9d7e735..71f3692 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -6070,7 +6070,11 @@ key_def: > constraint_def: > opt_constraint check_constraint > { > - Lex->add_constraint(&$1, $2); > + Lex->add_constraint(&$1, $2, FALSE); > + } > + | CONSTRAINT IF_SYM not EXISTS field_ident check_constraint > + { > + Lex->add_constraint(&$5, $6, TRUE); Eh, not here please. IF NOT EXIST should not be recognized inside CREATE TABLE :) > } > ; > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] 5486f34: MDEV-10341: InnoDB: Failing assertion: mutex_own(mutex) - mutex_exit_func
by Sergey Vojtovich 02 Aug '16

02 Aug '16

Hi Vicentiu, Ok to push. This patch is against 10.2, do you plan to fix previous versions separately? On Mon, Aug 01, 2016 at 02:46:03PM +0300, vicentiu wrote: > revision-id: 5486f3458859fa4bd161150b9ac0e9c0f633a0e3 (mariadb-10.2.1-9-g5486f34) > parent(s): 08683a726773f8cdf16a4a3dfb3920e5f7842481 > author: Vicențiu Ciorbaru > committer: Vicențiu Ciorbaru > timestamp: 2016-08-01 14:43:41 +0300 > message: > > MDEV-10341: InnoDB: Failing assertion: mutex_own(mutex) - mutex_exit_func > > Fix memory barrier issues on releasing mutexes. We must have Looks like a cutted comment. > > --- > storage/innobase/include/os0sync.h | 17 ++++++++--------- > storage/innobase/include/sync0sync.ic | 2 ++ > storage/xtradb/include/os0sync.h | 16 ++++++++-------- > storage/xtradb/include/sync0sync.ic | 2 ++ > 4 files changed, 20 insertions(+), 17 deletions(-) Thanks, Sergey

2 1

Re: [Maria-developers] Sachin weekly report
by Sachin Setia 02 Aug '16

02 Aug '16

Hello Sergei! Sir I am stuck at one problem. Consider the case create table t1 (abc blob unique, xyz int unique); /* Insert data */ update t1 set abc = 33 where xyz =23; This is not working because xyz = 23 will require index scan and at the time of ha_update_row the inited != NONE and if inited != NONE then ha_index_read_idx_map would fail so how can I update row in case of blob unique ? In this case I think the only option left is sequential scan but that will defeat the purpose of blob unique. Regards sachin On Thu, Jul 28, 2016 at 1:10 PM, Sergei Golubchik <serg(a)mariadb.org> wrote: > Hi, Sachin! > > On Jul 27, Sergei Golubchik wrote: >> > >> > Please review branch >> > https://github.com/SachinSetiya/server/tree/unique_index_where >> >> Sure. Will do. > > Sorry. Correction :( > > I'm on vacations two weeks in August, 9th to 22nd. And I have an awful > lot to do for 10.2. Naturally I won't work on 10.2 on vacations, but I > will take my laptop with me and I will answer your questions and review > your code. Which also means that I'd better use every minute before > vacations to work on 10.2, you see... > > So, I'll review you patch in a couple of weeks. > Until then I can only answer questions, sorry :( > And when on vacations, mind it, I will be only rarely on irc, so if I'm > not there, do not wait for me, send an email, please. I will reply to > your emails and I will do my reviews. > > By the way, did you do a full mysql-test run? Like this: > > ./mtr --force --parallel 5 > > In fact, what I do is: > > cmake -DCMAKE_BUILD_TYPE=Debug -DWITH_EMBEDDED_SERVER=ON > make -j5 > cd mysql-test-run > script -c './mtr --force --parallel 5; ./mtr --force --parallel 5 --ps; ./mtr --force --parallel 3 --embed' > > and that last command takes 4-5 hours on my laptop. I don't do that very > often :) but it's throurough. You can run it overnight. Because of > 'script' the complete log will be in the typescript file, so there's no > need to monitor manually anything. > > Regards, > Sergei > Chief Architect MariaDB > and security(a)mariadb.org

2 1

Re: [Maria-developers] [Commits] 5486f34: MDEV-10341: InnoDB: Failing assertion: mutex_own(mutex) - mutex_exit_func
by Sergey Vojtovich 01 Aug '16

01 Aug '16

Hi Vicentiu, I'm back from vacation, but quite busy with paperwork etc. I'll review your patch tomorrow. Regards, Sergey On Mon, Aug 01, 2016 at 02:46:03PM +0300, vicentiu wrote: > revision-id: 5486f3458859fa4bd161150b9ac0e9c0f633a0e3 (mariadb-10.2.1-9-g5486f34) > parent(s): 08683a726773f8cdf16a4a3dfb3920e5f7842481 > author: Vicențiu Ciorbaru > committer: Vicențiu Ciorbaru > timestamp: 2016-08-01 14:43:41 +0300 > message: > > MDEV-10341: InnoDB: Failing assertion: mutex_own(mutex) - mutex_exit_func > > Fix memory barrier issues on releasing mutexes. We must have > > --- > storage/innobase/include/os0sync.h | 17 ++++++++--------- > storage/innobase/include/sync0sync.ic | 2 ++ > storage/xtradb/include/os0sync.h | 16 ++++++++-------- > storage/xtradb/include/sync0sync.ic | 2 ++ > 4 files changed, 20 insertions(+), 17 deletions(-) > > diff --git a/storage/innobase/include/os0sync.h b/storage/innobase/include/os0sync.h > index 1cf4e9c..08911e7 100644 > --- a/storage/innobase/include/os0sync.h > +++ b/storage/innobase/include/os0sync.h > @@ -479,20 +479,13 @@ os_atomic_test_and_set(volatile lock_word_t* ptr) > } > > /** Do an atomic release. > - > -In theory __sync_lock_release should be used to release the lock. > -Unfortunately, it does not work properly alone. The workaround is > -that more conservative __sync_lock_test_and_set is used instead. > - > -Performance regression was observed at some conditions for Intel > -architecture. Disable release barrier on Intel architecture for now. > @param[in,out] ptr Memory location to write to > @return the previous value */ > inline > -lock_word_t > +void > os_atomic_clear(volatile lock_word_t* ptr) > { > - return(__sync_lock_test_and_set(ptr, 0)); > + __sync_lock_release(ptr); > } > > # elif defined(HAVE_IB_GCC_ATOMIC_TEST_AND_SET) > @@ -861,6 +854,12 @@ for synchronization */ > architecture. Disable memory barrier for Intel architecture for now. */ > # define os_rmb do { } while(0) > # define os_wmb do { } while(0) > +# ifndef __WIN__ > +# define os_mb __sync_synchronize() > +# else > + /* Windows atomic primitives imply a full memory barrier themselves. */ > +# define os_mb do { } while(0) > +# endif > # define os_isync do { } while(0) > # define IB_MEMORY_BARRIER_STARTUP_MSG \ > "Memory barrier is not used" > diff --git a/storage/innobase/include/sync0sync.ic b/storage/innobase/include/sync0sync.ic > index 85c7f75..19474e7 100644 > --- a/storage/innobase/include/sync0sync.ic > +++ b/storage/innobase/include/sync0sync.ic > @@ -178,6 +178,8 @@ mutex_exit_func( > to wake up possible hanging threads if > they are missed in mutex_signal_object. */ > > + os_mb; > + > if (mutex_get_waiters(mutex) != 0) { > > mutex_signal_object(mutex); > diff --git a/storage/xtradb/include/os0sync.h b/storage/xtradb/include/os0sync.h > index 0f93f3f..f06aa9d 100644 > --- a/storage/xtradb/include/os0sync.h > +++ b/storage/xtradb/include/os0sync.h > @@ -531,19 +531,13 @@ os_atomic_test_and_set(volatile lock_word_t* ptr) > > /** Do an atomic release. > > -In theory __sync_lock_release should be used to release the lock. > -Unfortunately, it does not work properly alone. The workaround is > -that more conservative __sync_lock_test_and_set is used instead. > - > -Performance regression was observed at some conditions for Intel > -architecture. Disable release barrier on Intel architecture for now. > @param[in,out] ptr Memory location to write to > @return the previous value */ > inline > -lock_word_t > +void > os_atomic_clear(volatile lock_word_t* ptr) > { > - return(__sync_lock_test_and_set(ptr, 0)); > + __sync_lock_release(ptr); > } > > # elif defined(HAVE_IB_GCC_ATOMIC_TEST_AND_SET) > @@ -912,6 +906,12 @@ for synchronization */ > architecture. Disable memory barrier for Intel architecture for now. */ > # define os_rmb do { } while(0) > # define os_wmb do { } while(0) > +# ifndef __WIN__ > +# define os_mb __sync_synchronize() > +# else > + /* Windows atomic primitives imply a full memory barrier themselves. */ > +# define os_mb do { } while(0) > +# endif > # define os_isync do { } while(0) > # define IB_MEMORY_BARRIER_STARTUP_MSG \ > "Memory barrier is not used" > diff --git a/storage/xtradb/include/sync0sync.ic b/storage/xtradb/include/sync0sync.ic > index 83f28bf..dbecc79 100644 > --- a/storage/xtradb/include/sync0sync.ic > +++ b/storage/xtradb/include/sync0sync.ic > @@ -181,6 +181,8 @@ mutex_exit_func( > to wake up possible hanging threads if > they are missed in mutex_signal_object. */ > > + os_mb; > + > if (mutex_get_waiters(mutex) != 0) { > > mutex_signal_object(mutex); > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

Re: [Maria-developers] 60d3489: MDEV-9143 JSON_xxx functions.
by Sergei Golubchik 27 Jul '16

27 Jul '16

Hi, Alexey! Summary: no big problems found, no conceptual changes needed :) Still - I feel there was too much code copied around in item_jsonfunc.cc, better to reduce the code duplication. And in sql_json.cc - lots of complains about naming and absense of comments. I was mostly able to figure out what it does regardless, but one should not need to spend a better half of the day just to understand one small function. Please add comments and rename functions/methods/variables, as I've noted in the review. On Jul 25, Alexey Botchkov wrote: > revision-id: 60d3489c96a4a95bd249b11972a132e321d15f29 (mariadb-10.2.1-4-g60d3489) > parent(s): 848d211c5c4df00b819cd84d7530cf7d29bb0524 > committer: Alexey Botchkov > timestamp: 2016-07-25 13:37:01 +0400 > message: > > MDEV-9143 JSON_xxx functions. an empty line after the commit subject, please > A number of JSON functions implemented. be a bit more verbose here. list all functions here. specify which are SQL standard, which are compatible with MySQL, etc and please explain (not in the commit comment, here, in the email) why did you decide to implement exactly these functions. SQL Standard - that's clear, but MySQL has many more, why did you choose only these few? > > --- > mysql-test/r/func_json.result | 66 +++ > mysql-test/t/func_json.test | 26 + > sql/CMakeLists.txt | 4 +- > sql/item.h | 15 + > sql/item_create.cc | 163 +++++++ > sql/item_jsonfunc.cc | 654 +++++++++++++++++++++++++ > sql/item_jsonfunc.h | 132 ++++++ > sql/item_xmlfunc.cc | 17 +- > sql/sql_json.cc | 1055 +++++++++++++++++++++++++++++++++++++++++ > sql/sql_json.h | 120 +++++ > sql/sql_yacc.yy | 4 +- > strings/ctype-ucs2.c | 12 +- > 12 files changed, 2246 insertions(+), 22 deletions(-) > > diff --git a/mysql-test/r/func_json.result b/mysql-test/r/func_json.result > new file mode 100644 > index 0000000..f22b935 > --- /dev/null > +++ b/mysql-test/r/func_json.result > @@ -0,0 +1,66 @@ > +select json_valid('[1, 2]'); > +json_valid('[1, 2]') > +1 > +select json_valid('"mama"}'); > +json_valid('"mama"}') > +0 > +select json_valid('{"mama":1, "myla":[2,3]}'); heh I'm surprised, you haven't added the third key :) > +json_valid('{"mama":1, "myla":[2,3]}') > +1 > +select json_valid('[false, true, null]'); > +json_valid('[false, true, null]') > +1 > +select json_value('{"key1":123}', '$.key2'); > +json_value('{"key1":123}', '$.key2') > +NULL > +select json_value('{"key1":123}', '$.key1'); > +json_value('{"key1":123}', '$.key1') > +123 > +select json_value('{"key1":[1,2,3]}', '$.key1'); > +json_value('{"key1":[1,2,3]}', '$.key1') > +NULL > +select json_value('{"key1": [1,2,3], "key1":123}', '$.key1'); > +json_value('{"key1": [1,2,3], "key1":123}', '$.key1') please, also implement json_query(). it's trivial to do and it's part of SQL standard. there's no reason not to support it :) > +123 > +select json_array_append('["a", "b"]', '$', FALSE); > +json_array_append('["a", "b"]', '$', FALSE) > +["a", "b", false] > +select json_array_append('{"k1":1, "k2":["a", "b"]}', '$.k2', 2); > +json_array_append('{"k1":1, "k2":["a", "b"]}', '$.k2', 2) test with multiple values to append? > +{"k1":1, "k2":["a", "b", 2]} > +select json_contains_path('{"key1":1, "key2":[2,3]}', "oNE", "$.key2[1]"); what's sql standard way of doing it? > +json_contains_path('{"key1":1, "key2":[2,3]}', "oNE", "$.key2[1]") > +1 > +select json_contains_path('{"key1":1, "key2":[2,3]}', "oNE", "$.key2[10]"); > +json_contains_path('{"key1":1, "key2":[2,3]}', "oNE", "$.key2[10]") > +0 > +select json_contains_path('{"key1":1, "key2":[2,3]}', "oNE", "$.ma"); > +json_contains_path('{"key1":1, "key2":[2,3]}', "oNE", "$.ma") > +0 > +select json_contains_path('{"key1":1, "key2":[2,3]}', "one", "$.key1"); > +json_contains_path('{"key1":1, "key2":[2,3]}', "one", "$.key1") > +1 > +select json_contains_path('{"key1":1, "key2":[2,3]}', "one", "$.key1", "$.ma"); > +json_contains_path('{"key1":1, "key2":[2,3]}', "one", "$.key1", "$.ma") > +1 > +select json_contains_path('{"key1":1, "key2":[2,3]}', "aLl", "$.key1", "$.ma"); > +json_contains_path('{"key1":1, "key2":[2,3]}', "aLl", "$.key1", "$.ma") > +0 > +select json_contains_path('{"key1":1, "key2":[2,3]}', "aLl", "$.key1", "$.key2"); > +json_contains_path('{"key1":1, "key2":[2,3]}', "aLl", "$.key1", "$.key2") > +1 > +select json_extract('{"key1":"asd", "key2":[2,3]}', "$.key1"); > +json_extract('{"key1":"asd", "key2":[2,3]}', "$.key1") > +asd > +select json_extract('{"key1":"asd", "key2":[2,3]}', "$.keyX", "$.keyY"); > +json_extract('{"key1":"asd", "key2":[2,3]}', "$.keyX", "$.keyY") > +NULL > +select json_extract('{"key1":"asd", "key2":[2,3]}', "$.key1", "$.key2"); > +json_extract('{"key1":"asd", "key2":[2,3]}', "$.key1", "$.key2") > +["asd", [2,3]] > +select json_extract('{"key1":5, "key2":[2,3]}', "$.key1", "$.key2"); > +json_extract('{"key1":5, "key2":[2,3]}', "$.key1", "$.key2") > +[5, [2,3]] > +select json_extract('{"key0":true, "key1":"qwe"}', "$.key1"); > +json_extract('{"key0":true, "key1":"qwe"}', "$.key1") > +qwe > diff --git a/sql/CMakeLists.txt b/sql/CMakeLists.txt > index 089d793..db11391 100644 > --- a/sql/CMakeLists.txt > +++ b/sql/CMakeLists.txt > @@ -134,12 +134,12 @@ SET (SQL_SOURCE > opt_table_elimination.cc sql_expression_cache.cc > gcalc_slicescan.cc gcalc_tools.cc > threadpool_common.cc ../sql-common/mysql_async.c > - my_apc.cc my_apc.h mf_iocache_encr.cc > + my_apc.cc my_apc.h mf_iocache_encr.cc item_jsonfunc.cc > my_json_writer.cc my_json_writer.h > rpl_gtid.cc rpl_parallel.cc > sql_type.cc sql_type.h > item_windowfunc.cc sql_window.cc > - sql_cte.cc sql_cte.h > + sql_cte.cc sql_cte.h sql_json.cc sql_json.h do I get correctly that item_jsonfunc.cc is json functions and sql_json.cc is json parsing library? in that case, please, rename the file simply to json.cc, there's not much "sql" in it. And, btw, there's no need to add *.h files to the SQL_SOURCE list. > ${WSREP_SOURCES} > table_cache.cc encryption.cc temporary_tables.cc > ${CMAKE_CURRENT_BINARY_DIR}/sql_builtin.cc > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index 9d7e735..26786ba 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -13767,13 +13767,13 @@ literal: > } > | FALSE_SYM > { > - $$= new (thd->mem_root) Item_int(thd, (char*) "FALSE",0,1); > + $$= new (thd->mem_root) Item_bool(thd, (char*) "FALSE",0); > if ($$ == NULL) > MYSQL_YYABORT; > } > | TRUE_SYM > { > - $$= new (thd->mem_root) Item_int(thd, (char*) "TRUE",1,1); > + $$= new (thd->mem_root) Item_bool(thd, (char*) "TRUE",1); This commit of yours does not change any result files. Does it mean that you have run the complete test suite and did not find any differences in result files caused by this change? > if ($$ == NULL) > MYSQL_YYABORT; > } > diff --git a/strings/ctype-ucs2.c b/strings/ctype-ucs2.c > index 06dab08..c52177a 100644 > --- a/strings/ctype-ucs2.c > +++ b/strings/ctype-ucs2.c > @@ -1190,10 +1190,13 @@ my_lengthsp_mb2(CHARSET_INFO *cs __attribute__((unused)), > #endif /* HAVE_CHARSET_mb2*/ > > > +/* > + Next part is actually HAVE_CHARSET_utf16-specific, > + but it the JSON functions needed my_utf16_uni() typo: "it the" > + so the #ifdef was moved lower. > +*/ > > > -#ifdef HAVE_CHARSET_utf16 > - > /* > D800..DB7F - Non-provate surrogate high (896 pages) > DB80..DBFF - Private surrogate high (128 pages) > diff --git a/sql/item_create.cc b/sql/item_create.cc > index 82f6bbd..585b706 100644 > --- a/sql/item_create.cc > +++ b/sql/item_create.cc > @@ -1708,6 +1708,71 @@ class Create_func_issimple : public Create_func_arg1 > #endif > > > +class Create_func_json_valid : public Create_func_arg1 > +{ > +public: > + virtual Item *create_1_arg(THD *thd, Item *arg1); > + > + static Create_func_json_valid s_singleton; > + > +protected: > + Create_func_json_valid() {} > + virtual ~Create_func_json_valid() {} > +}; > + > + > +class Create_func_json_value : public Create_func_arg2 > +{ > +public: > + virtual Item *create_2_arg(THD *thd, Item *arg1, Item *arg2); > + > + static Create_func_json_value s_singleton; > + > +protected: > + Create_func_json_value() {} > + virtual ~Create_func_json_value() {} > +}; okay, although I suspect that we (you?) will need to remove that later and create Item_func_json_value directly in sql_yacc.yy - if we ever want to do the complete SQL standard syntax for it. > + > + > +class Create_func_json_contains_path : public Create_native_func > +{ > +public: > + virtual Item *create_native(THD *thd, LEX_STRING name, List<Item> *item_list); > + > + static Create_func_json_contains_path s_singleton; > + > +protected: > + Create_func_json_contains_path() {} > + virtual ~Create_func_json_contains_path() {} > +}; > + > + > +class Create_func_json_extract : public Create_native_func > +{ > +public: > + virtual Item *create_native(THD *thd, LEX_STRING name, List<Item> *item_list); > + > + static Create_func_json_extract s_singleton; > + > +protected: > + Create_func_json_extract() {} > + virtual ~Create_func_json_extract() {} > +}; > + > + > +class Create_func_json_array_append : public Create_native_func > +{ > +public: > + virtual Item *create_native(THD *thd, LEX_STRING name, List<Item> *item_list); > + > + static Create_func_json_array_append s_singleton; > + > +protected: > + Create_func_json_array_append() {} > + virtual ~Create_func_json_array_append() {} > +}; > + > + > class Create_func_last_day : public Create_func_arg1 > { > public: > diff --git a/sql/sql_json.h b/sql/sql_json.h > new file mode 100644 > index 0000000..e937d3d > --- /dev/null > +++ b/sql/sql_json.h > @@ -0,0 +1,120 @@ > +#ifndef SQL_JSON_INCLUDED > +#define SQL_JSON_INCLUDED > + > +#include <m_ctype.h> > + > +static const int json_depth_limit= 32; uppercase, please > + > +enum json_errors { > + JE_BAD= -1, /* Invalid character - cannot read the string. */ > + JE_IMP= -2, /* This character disallowed in JSON. */ like what? and why "IMP"? > + JE_EOS= -3, /* Unexpected end of string. */ > + JE_UEX= -4, /* Syntax error - unexpected character. */ "UEX" isn't clear at all. JE_UNEXPECTED, please > + JE_IMS= -5, /* This character disallowed in string constant. */ like what? and why "IMS"? > + JE_ESCAPING= -6, /* Wrong escaping. */ > + JE_TOODEEP= -7, /* The limit on the depth was overrun. */ > +}; > + > + This is a json library. it needs *lots* of comments. I'll mark below what's not obvious > +class json_string > +{ > +public: > + const uchar *c_str; > + const uchar *str_end; > + my_wc_t c_next; > + int error; > + > + CHARSET_INFO *cs; > + my_charset_conv_mb_wc wc; > + int get_next_char() > + { > + return wc(cs, &c_next, c_str, str_end); > + } > + my_wc_t next_chr() const { return c_next; } comment, explaining the difference between get_next_char and next_chr > + bool eos() const { return c_str >= str_end; } > + int handle_esc(); comment > + void skip_s_getchar(int &t_next, int &c_len); comment > + void set_cs(CHARSET_INFO *i_cs); > + void set_str(const uchar *str, const uchar *end) > + { > + c_str= str; > + str_end= end; > + } > + void setup(CHARSET_INFO *i_cs, const uchar *str, const uchar *end) > + { > + set_cs(i_cs); > + set_str(str, end); > + } > + int read_str_chr(); comment > +}; > + > + > +struct json_path_step comment > +{ > + enum types { KEY=0, ARRAY=1 }; > + > + types type; > + int wild; > + const uchar *key; > + const uchar *key_end; > + uint n_item; comment > +}; > + > + > +class json_path : public json_string > +{ > +public: > + json_path_step steps[json_depth_limit]; > + json_path_step *last_step; > + > + bool mode_strict; > + int setup(CHARSET_INFO *i_cs, const uchar *str, const uchar *end); Hm, so the actual search is not a method of json_path and not in sql_json.cc at all. Why is that? > +}; > + > + > +class json_engine : public json_string > +{ > +public: > + enum states { > + VALUE, /* value met */ > + DONE, /* ok to finish */ > + KEY, /* key met */ s/met/found/ > + OB_B, /* object */ s/OB_B/OBJ_START/ > + OB_E, /* object ended */ OBJ_END > + OB_C, /* object continues */ OBJ_CONT > + AR_B, /* array */ ARRAY_START > + AR_E, /* array ended */ ARRAY_END > + AR_C, /* array continues */ you guessed it, ARRAY_CONT > + READ_VALUE, /* value is beeing read */ s/beeing/being/ > + NR_STATES > + }; > + > + enum value_types > + { OBJECT=0, ARRAY=1, STRING, NUMBER, V_TRUE, V_FALSE, V_NULL }; > + > + int sav_c_len; comment > + value_types value_type; > + const uchar *value; > + const uchar *value_begin; > + const uchar *value_end; > + int value_len; comment. what value? this is json_engine object, not json_value > + > + int stack[json_depth_limit]; > + int *stack_p; > + > + int state; comment. current state? > + > + int scan_start(CHARSET_INFO *i_cs, const uchar *str, const uchar *end); > + int scan_next(); > + > + int read_keyname_chr(); comment > + > + int read_value(); comment > + int skip_key(); comment > + int skip_level(); comment > + int skip_array_item() { return skip_key(); } comment > + bool value_scalar() const { return value_type > ARRAY; } > +}; > + > +#endif /* SQL_JSON_INCLUDED */ > + > diff --git a/sql/sql_json.cc b/sql/sql_json.cc > new file mode 100644 > index 0000000..3880a04 > --- /dev/null > +++ b/sql/sql_json.cc > @@ -0,0 +1,1055 @@ > +#include <my_global.h> > + > +#include "sql_json.h" > + > + > +/* Copied from ctype-ucs2.c. */ no, don't do that, please. move defines to some header file included into both. and you've change my_utf16_uni() to be no longer static, you don't seem to need to copy it here. Same should be done for arrays that you need. Another question. Why do you treat json as utf16, instead of being it just a regular string in whatever character set it might happen to be? > +/* > + D800..DB7F - Non-provate surrogate high (896 pages) > + DB80..DBFF - Private surrogate high (128 pages) > + DC00..DFFF - Surrogate low (1024 codes in a page) > +*/ > +#define MY_UTF16_SURROGATE_HIGH_FIRST 0xD800 > +#define MY_UTF16_SURROGATE_HIGH_LAST 0xDBFF > +#define MY_UTF16_SURROGATE_LOW_FIRST 0xDC00 > +#define MY_UTF16_SURROGATE_LOW_LAST 0xDFFF > + > +#define MY_UTF16_HIGH_HEAD(x) ((((uchar) (x)) & 0xFC) == 0xD8) > +#define MY_UTF16_LOW_HEAD(x) ((((uchar) (x)) & 0xFC) == 0xDC) > +/* Test if a byte is a leading byte of a high or low surrogate head: */ > +#define MY_UTF16_SURROGATE_HEAD(x) ((((uchar) (x)) & 0xF8) == 0xD8) > +/* Test if a Unicode code point is a high or low surrogate head */ > +#define MY_UTF16_SURROGATE(x) (((x) & 0xF800) == 0xD800) > + > +#define MY_UTF16_WC2(a, b) ((a << 8) + b) > + > +/* > + a= 110110?? (<< 18) > + b= ???????? (<< 10) > + c= 110111?? (<< 8) > + d= ???????? (<< 0) > +*/ > +#define MY_UTF16_WC4(a, b, c, d) (((a & 3) << 18) + (b << 10) + \ > + ((c & 3) << 8) + d + 0x10000) > + > +int > +my_utf16_uni(CHARSET_INFO *cs __attribute__((unused)), > + my_wc_t *pwc, const uchar *s, const uchar *e) > +{ > + if (s + 2 > e) > + return MY_CS_TOOSMALL2; > + > + /* > + High bytes: 0xD[89AB] = B'110110??' > + Low bytes: 0xD[CDEF] = B'110111??' > + Surrogate mask: 0xFC = B'11111100' > + */ > + > + if (MY_UTF16_HIGH_HEAD(*s)) /* Surrogate head */ > + { > + if (s + 4 > e) > + return MY_CS_TOOSMALL4; > + > + if (!MY_UTF16_LOW_HEAD(s[2])) /* Broken surrigate pair */ > + return MY_CS_ILSEQ; > + > + *pwc= MY_UTF16_WC4(s[0], s[1], s[2], s[3]); > + return 4; > + } > + > + if (MY_UTF16_LOW_HEAD(*s)) /* Low surrogate part without high part */ > + return MY_CS_ILSEQ; > + > + *pwc= MY_UTF16_WC2(s[0], s[1]); > + return 2; > +} > + > +/* End of ctype-ucs2.c code. */ > + > +enum json_c_classes { comment. or rename to json_char_classes > + C_EOS, /* end of string */ > + C_LCURB, /* { */ > + C_RCURB, /* } */ > + C_LSQRB, /* [ */ > + C_RSQRB, /* ] */ > + C_COLON, /* : */ > + C_COMMA, /* , */ > + C_QUOTE, /* " */ > + C_DIGIT, /* -0123456789 */ > + C_LOW_F, /* f */ > + C_LOW_N, /* n */ > + C_LOW_T, /* t */ clarify these comment, like: + C_LOW_T, /* t (for "true") */ otherwise it's not at all clear why f/n/t are special here. > + C_ETC, /* everything else */ > + C_ERR, /* character disallowed in JSON*/ > + C_BAD, /* invalid character */ > + NR_C_CLASSES, > + C_SPACE /* space */ why space is after NR_C_CLASSES? > +}; > + > + > +/* > + This array maps first 128 Unicode Code Points into classes. > + The remaining Unicode characters should be mapped to C_ETC. > +*/ > + > +static int json_chr_map[128] = { > + C_ERR, C_ERR, C_ERR, C_ERR, C_ERR, C_ERR, C_ERR, C_ERR, > + C_ERR, C_SPACE, C_SPACE, C_ERR, C_ERR, C_SPACE, C_ERR, C_ERR, > + C_ERR, C_ERR, C_ERR, C_ERR, C_ERR, C_ERR, C_ERR, C_ERR, > + C_ERR, C_ERR, C_ERR, C_ERR, C_ERR, C_ERR, C_ERR, C_ERR, > + > + C_SPACE, C_ETC, C_QUOTE, C_ETC, C_ETC, C_ETC, C_ETC, C_ETC, > + C_ETC, C_ETC, C_ETC, C_ETC, C_COMMA, C_DIGIT, C_ETC, C_ETC, > + C_DIGIT, C_DIGIT, C_DIGIT, C_DIGIT, C_DIGIT, C_DIGIT, C_DIGIT, C_DIGIT, > + C_DIGIT, C_DIGIT, C_COLON, C_ETC, C_ETC, C_ETC, C_ETC, C_ETC, > + > + C_ETC, C_ETC, C_ETC, C_ETC, C_ETC, C_ETC, C_ETC, C_ETC, > + C_ETC, C_ETC, C_ETC, C_ETC, C_ETC, C_ETC, C_ETC, C_ETC, > + C_ETC, C_ETC, C_ETC, C_ETC, C_ETC, C_ETC, C_ETC, C_ETC, > + C_ETC, C_ETC, C_ETC, C_LSQRB, C_ETC, C_RSQRB, C_ETC, C_ETC, > + > + C_ETC, C_ETC, C_ETC, C_ETC, C_ETC, C_ETC, C_LOW_F, C_ETC, > + C_ETC, C_ETC, C_ETC, C_ETC, C_ETC, C_ETC, C_LOW_N, C_ETC, > + C_ETC, C_ETC, C_ETC, C_ETC, C_LOW_T, C_ETC, C_ETC, C_ETC, > + C_ETC, C_ETC, C_ETC, C_LCURB, C_ETC, C_RCURB, C_ETC, C_ETC > +}; > + > + > +typedef int (*json_state_handler)(json_engine *); > + > + > +/* The string is broken. */ > +static int er_en(json_engine *j) > +{ > + j->error= JE_EOS; > + return 1; > +} > + > + > +/* This symbol here breaks the JSON syntax. */ > +static int __(json_engine *j) > +{ > + j->error= JE_UEX; > + return 1; > +} > + > + > +/* Value of object. */ > +static int ky_ob(json_engine *j) dn' abbr lk tht use longer readable names, please > +{ > + j->state= json_engine::OB_B; > + *(++j->stack_p)= json_engine::OB_C; > + return 0; > +} > + > +/* Read value of object. */ > +static int rd_ob(json_engine *j) ok, in the context of the array below, rd_ob is kind of readable. but I have no idea what "ky" means. If it's "key" then I'd suggest to rename "ky" -> "key" and "rd" -> "val". and "obj", "arr" too. > +{ > + j->state= json_engine::OB_B; > + j->value_type= json_engine::OBJECT; > + j->value= j->value_begin; > + *(++j->stack_p)= json_engine::OB_C; > + return 0; > +} > + > + > +/* Value of array. */ > +static int ky_ar(json_engine *j) > +{ > + j->state= json_engine::AR_B; > + *(++j->stack_p)= json_engine::AR_C; > + j->value= j->value_begin; > + return 0; > +} > + > +/* Read value of object. */ > +static int rd_ar(json_engine *j) > +{ > + j->state= json_engine::AR_B; > + j->value_type= json_engine::ARRAY; > + j->value= j->value_begin; > + *(++j->stack_p)= json_engine::AR_C; > + return 0; > +} > + > + > + > +/* \b 8 \f 12 \n 10 \r 13 \t 9 */ > +enum json_s_classes { what does that mean? this needs a comment for the enum and even a comment for a comment :) a I don't see how this your comment applies to this your enum. > + S_0= 0, > + S_1= 1, > + S_2= 2, > + S_3= 3, > + S_4= 4, > + S_5= 5, > + S_6= 6, > + S_7= 7, > + S_8= 8, > + S_9= 9, > + S_A= 10, > + S_B= 11, > + S_C= 12, > + S_D= 13, > + S_E= 14, > + S_F= 15, > + S_ETC= 36, /* rest of characters. */ > + S_QUOTE= 37, > + S_SOLID= 38, /* \ */ why "solid" ? because it's "reverse solidus"? better call it "backslash", that's far less confusing. > + S_ERR= 100, /* disallowed */ > +}; > + > + > +/* This maps characters to their types inside a string constant. */ > +static const int json_instr_chr_map[128] = { > + S_ERR, S_ERR, S_ERR, S_ERR, S_ERR, S_ERR, S_ERR, S_ERR, > + S_ERR, S_ERR, S_ERR, S_ERR, S_ERR, S_ERR, S_ERR, S_ERR, > + S_ERR, S_ERR, S_ERR, S_ERR, S_ERR, S_ERR, S_ERR, S_ERR, > + S_ERR, S_ERR, S_ERR, S_ERR, S_ERR, S_ERR, S_ERR, S_ERR, > + > + S_ETC, S_ETC, S_QUOTE, S_ETC, S_ETC, S_ETC, S_ETC, S_ETC, > + S_ETC, S_ETC, S_ETC, S_ETC, S_ETC, S_ETC, S_ETC, S_ETC, > + S_0, S_1, S_2, S_3, S_4, S_5, S_6, S_7, > + S_8, S_9, S_ETC, S_ETC, S_ETC, S_ETC, S_ETC, S_ETC, > + > + S_ETC, S_A, S_B, S_C, S_D, S_E, S_F, S_ETC, > + S_ETC, S_ETC, S_ETC, S_ETC, S_ETC, S_ETC, S_ETC, S_ETC, > + S_ETC, S_ETC, S_ETC, S_ETC, S_ETC, S_ETC, S_ETC, S_ETC, > + S_ETC, S_ETC, S_ETC, S_ETC, S_SOLID, S_ETC, S_ETC, S_ETC, > + > + S_ETC, S_A, S_B, S_C, S_D, S_E, S_F, S_ETC, > + S_ETC, S_ETC, S_ETC, S_ETC, S_ETC, S_ETC, S_ETC, S_ETC, > + S_ETC, S_ETC, S_ETC, S_ETC, S_ETC, S_ETC, S_ETC, S_ETC, > + S_ETC, S_ETC, S_ETC, S_ETC, S_ETC, S_ETC, S_ETC, S_ETC > +}; > + > + > +static int read_4digit(json_string *j, uchar *s) s/read_4digit/read_4_hexdigits/ > +{ > + int i, t, c_len; > + for (i=0; i<4; i++) > + { > + if ((c_len= j->get_next_char()) <= 0) > + return j->error= j->eos() ? JE_EOS : JE_BAD; > + > + if (j->c_next >= 128 || (t= json_instr_chr_map[j->c_next]) >= S_F) > + return j->error= JE_UEX; > + > + j->c_str+= c_len; > + s[i/2]+= (i % 2) ? t : t*16; > + } > + return 0; > +} > + > + > +/* \b 8 \f 12 \n 10 \r 13 \t 9 */ > +int json_string::handle_esc() > +{ > + int t, c_len; > + > + if ((c_len= get_next_char()) <= 0) > + return error= eos() ? JE_EOS : JE_BAD; > + > + c_str+= c_len; > + switch (c_next) > + { > + case 'b': > + c_next= 8; > + return 0; > + case 'f': > + c_next= 12; > + return 0; > + case 'n': > + c_next= 10; > + return 0; > + case 'r': > + c_next= 13; > + return 0; > + case 't': > + c_next= 9; > + return 0; > + } > + > + if (c_next < 128 && (t= json_instr_chr_map[c_next]) == S_ERR) > + { > + c_str-= c_len; > + return error= JE_ESCAPING; > + } where did you read that the behavior should be like that? > + > + > + if (c_next != 'u') > + return 0; where did you read that the behavior should be like that? > + > + /* > + Read the four-hex-digits code. > + If symbol is not in the Basic Multilingual Plane, we're reading > + the string for the next four digits to compose the UTF-16 surrogate pair. > + */ > + uchar s[4]= {0,0,0,0}; > + > + if (read_4digit(this, s)) > + return 1; > + > + if ((c_len= my_utf16_uni(0, &c_next, s, s+2)) == 2) > + return 0; > + > + if (c_len != MY_CS_TOOSMALL4) > + return error= JE_BAD; > + > + if ((c_len= get_next_char()) <= 0) > + return error= eos() ? JE_EOS : JE_BAD; > + if (c_next != '\\') > + return error= JE_UEX; > + > + if ((c_len= get_next_char()) <= 0) > + return error= eos() ? JE_EOS : JE_BAD; > + if (c_next != 'u') > + return error= JE_UEX; > + > + if (read_4digit(this, s+2)) > + return 1; > + > + if ((c_len= my_utf16_uni(0, &c_next, s, s+4)) == 2) > + return 0; > + > + return error= JE_BAD; > +} > + > + > +int json_string::read_str_chr() at least, it's not "rd_str_chr", but I still don't understand what "read_str_chr" does. please, rename > +{ > + int c_len; > + > + if ((c_len= get_next_char()) > 0) > + { > + c_str+= c_len; > + return (c_next == '\\') ? handle_esc() : 0; > + } > + error= eos() ? JE_EOS : JE_BAD; > + return 1; > +} > + > + > +inline int pass_str(json_engine *j) what is "pass_str"? please, rename > +{ > + int t, c_len; > + for (;;) > + { > + if ((c_len= j->get_next_char()) > 0) > + { > + j->c_str+= c_len; > + if (j->c_next >= 128 || ((t=json_instr_chr_map[j->c_next]) <= S_ETC)) > + continue; > + > + if (j->c_next == '"') > + break; ok, so it apparently skips over a string, moving the cursor from the opening to the closing double quote > + if (j->c_next == '\\') > + { > + if (j->handle_esc()) > + return 1; > + continue; > + } > + /* Symbol not allowed in JSON. */ > + return j->error= JE_IMP; > + } > + else > + return j->error= j->eos() ? JE_EOS : JE_BAD; > + } > + > + j->state= *j->stack_p; > + return 0; > +} > + > + > +/* Scalar string. */ > +static int scr_s(json_engine *j) five letters per name is difficult, beter make the array wider but more readable. > +{ > + return pass_str(j) || j->scan_next(); > +} > + > + > +/* Read scalar string. */ > +static int rd_s(json_engine *j) > +{ > + j->value= j->c_str; > + > + if (pass_str(j)) > + return 1; > + > + j->state= *j->stack_p; > + j->value_type= json_engine::STRING; > + j->value_len= (j->c_str - j->value) - 1; > + return 0; > +} > + > + > +enum json_num_classes { > + N_MINUS, > + N_PLUS, > + N_ZERO, > + N_DIGIT, > + N_POINT, > + N_E, > + N_END, > + N_EEND, > + N_ERR, > + N_NUM_CLASSES > +}; comments! > + > + > +static int json_num_chr_map[128] = { > + N_ERR, N_ERR, N_ERR, N_ERR, N_ERR, N_ERR, N_ERR, N_ERR, > + N_ERR, N_END, N_END, N_ERR, N_ERR, N_END, N_ERR, N_ERR, > + N_ERR, N_ERR, N_ERR, N_ERR, N_ERR, N_ERR, N_ERR, N_ERR, > + N_ERR, N_ERR, N_ERR, N_ERR, N_ERR, N_ERR, N_ERR, N_ERR, > + > + N_END, N_EEND, N_EEND, N_EEND, N_EEND, N_EEND, N_EEND, N_EEND, > + N_EEND, N_EEND, N_EEND, N_PLUS, N_END, N_MINUS, N_POINT, N_EEND, > + N_ZERO, N_DIGIT, N_DIGIT, N_DIGIT, N_DIGIT, N_DIGIT, N_DIGIT, N_DIGIT, > + N_DIGIT, N_DIGIT, N_EEND, N_EEND, N_EEND, N_EEND, N_EEND, N_EEND, > + > + N_EEND, N_EEND, N_EEND, N_EEND, N_EEND, N_E, N_EEND, N_EEND, > + N_EEND, N_EEND, N_EEND, N_EEND, N_EEND, N_EEND, N_EEND, N_EEND, > + N_EEND, N_EEND, N_EEND, N_EEND, N_EEND, N_EEND, N_EEND, N_EEND, > + N_EEND, N_EEND, N_EEND, N_EEND, N_EEND, N_END, N_EEND, N_EEND, > + > + N_EEND, N_EEND, N_EEND, N_EEND, N_EEND, N_E, N_EEND, N_EEND, > + N_EEND, N_EEND, N_EEND, N_EEND, N_EEND, N_EEND, N_EEND, N_EEND, > + N_EEND, N_EEND, N_EEND, N_EEND, N_EEND, N_EEND, N_EEND, N_EEND, > + N_EEND, N_EEND, N_EEND, N_EEND, C_ETC, N_END, N_EEND, N_EEND, what is C_ETC doing here? if you'd declared your array of json_num_classes type, you wouldn't be able to put value from another enum there. this applies everywhere. many your arrays are int. the stack is int. they all need to be of their corresponding enum types. > +}; > + > +enum json_num_states { > + NS_OK, > + NS_GO, > + NS_GO1, > + NS_Z, > + NS_Z1, > + NS_INT, > + NS_FRAC, > + NS_EX, > + NS_EX1, > + NS_NUM_STATES > +}; > + > + > +static int json_num_states[NS_NUM_STATES][N_NUM_CLASSES]= > +{ > +/* - + 0 1..9 POINT E END_OK ERROR */ > +/* OK */ { JE_UEX, JE_UEX, JE_UEX, JE_UEX, JE_UEX, JE_UEX, JE_UEX, JE_BAD }, > +/* GO */ { NS_GO1, JE_UEX, NS_Z, NS_INT, JE_UEX, JE_UEX, JE_UEX, JE_BAD }, > +/* GO1 */ { JE_UEX, JE_UEX, NS_Z1, NS_INT, JE_UEX, JE_UEX, JE_UEX, JE_BAD }, > +/* ZERO */ { JE_UEX, JE_UEX, JE_UEX, JE_UEX, NS_FRAC, JE_UEX, NS_OK, JE_BAD }, > +/* ZE1 */ { JE_UEX, JE_UEX, JE_UEX, JE_UEX, NS_FRAC, JE_UEX, JE_UEX, JE_BAD }, > +/* INT */ { JE_UEX, JE_UEX, NS_INT, NS_INT, NS_FRAC, NS_EX, NS_OK, JE_BAD }, > +/* FRAC */ { JE_UEX, JE_UEX, NS_FRAC, NS_FRAC,JE_UEX, NS_EX, NS_OK, JE_BAD }, > +/* EX */ { NS_EX1, NS_EX1, NS_EX1, NS_EX1, JE_UEX, JE_UEX, JE_UEX, JE_BAD }, > +/* EX1 */ { JE_UEX, JE_UEX, NS_EX1, NS_EX1, JE_UEX, JE_UEX, JE_UEX, JE_BAD } > +}; > + > + > +inline int pass_n(json_engine *j) > +{ > + int state= json_num_states[NS_GO][json_num_chr_map[j->c_next]]; > + int c_len; > + > + for (;;) > + { > + if ((c_len= j->get_next_char()) > 0) > + { > + if ((state= json_num_states[state][json_num_chr_map[j->c_next]]) > 0) > + { > + j->c_str+= c_len; > + continue; > + } > + break; > + } > + > + if ((j->error= j->eos() ? json_num_states[state][N_END] : JE_BAD) < 0) > + return 1; > + else > + break; > + } > + > + j->state= *j->stack_p; > + return 0; > +} > + > + > +/* Scalar numeric. */ > +static int scr_n(json_engine *j) > +{ > + return pass_n(j) || j->scan_next(); > +} > + > + > +/* Read number. */ > +static int rd_n(json_engine *j) > +{ > + j->value= j->value_begin; > + if (pass_n(j) == 0) > + { > + j->value_type= json_engine::NUMBER; > + j->value_len= j->c_str - j->value_begin; > + return 0; > + } > + return 1; > +} > + > + > +static int assure_string(json_string *j, const char *str) better "match_string", or "match_verbatim" or "skip_verbatim", something like that > +{ > + int c_len; > + while (*str) > + { > + if ((c_len= j->get_next_char()) > 0) > + { > + if (j->c_next == (my_wc_t) *(str++)) > + { > + j->c_str+= c_len; > + continue; > + } > + return JE_UEX; > + } > + return j->eos() ? JE_EOS : JE_BAD; > + } > + > + return 0; > +} > + > + > +/* Scalar false. */ > +static int scr_f(json_engine *j) > +{ > + if (assure_string(j, "alse")) > + return 1; > + j->state= *j->stack_p; > + return j->scan_next(); > +} > + > + > +/* Scalar null. */ > +static int scr_l(json_engine *j) > +{ > + if (assure_string(j, "ull")) > + return 1; > + j->state= *j->stack_p; > + return j->scan_next(); > +} > + > + > +/* Scalar true. */ > +static int scr_t(json_engine *j) > +{ > + if (assure_string(j, "rue")) > + return 1; > + j->state= *j->stack_p; > + return j->scan_next(); > +} > + > + > +/* Read false. */ > +static int rd_f(json_engine *j) > +{ > + j->value_type= json_engine::V_FALSE; > + j->value= j->value_begin; > + j->state= *j->stack_p; > + j->value_len= 5; > + return assure_string(j, "alse"); > +} > + > + > +/* Read null. */ > +static int rd_l(json_engine *j) > +{ > + j->value_type= json_engine::V_NULL; > + j->value= j->value_begin; > + j->state= *j->stack_p; > + j->value_len= 4; > + return assure_string(j, "ull"); > +} > + > + > +/* Read true. */ > +static int rd_t(json_engine *j) > +{ > + j->value_type= json_engine::V_TRUE; > + j->value= j->value_begin; > + j->state= *j->stack_p; > + j->value_len= 4; > + return assure_string(j, "rue"); > +} > + > + > +/* Disallowed character. */ > +static int e_chr(json_engine *j) > +{ > + j->error= JE_IMP; > + return 1; > +} > + > + > +/* Bad character. */ > +static int b_chr(json_engine *j) > +{ > + j->error= JE_BAD; > + return 1; > +} > + > + > +/* Correct finish. */ > +static int done(json_engine *j) > +{ > + return 1; > +} > + > + > +/* End of the object. */ > +static int en_ob(json_engine *j) > +{ > + j->stack_p--; > + j->state= json_engine::OB_E; > + return 0; > +} > + > + > +/* End of the array. */ > +static int en_ar(json_engine *j) > +{ > + j->stack_p--; > + j->state= json_engine::AR_E; > + return 0; > +} > + > + > +/* Start reading key name. */ > +static int get_k(json_engine *j) > +{ > + j->state= json_engine::KEY; > + return 0; > +} > + > + > +inline void json_string::skip_s_getchar(int &t_next, int &c_len) comment. the name is not clear at all, please rename > +{ > + do > + { > + if ((c_len= get_next_char()) <= 0) > + t_next= eos() ? C_EOS : C_BAD; > + else > + { > + t_next= (c_next < 128) ? json_chr_map[c_next] : C_ETC; > + c_str+= c_len; > + } > + } while (t_next == C_SPACE); ok, so it's "get first non-space character"? > +} > + > + > +/* Next key name. */ > +static int nex_k(json_engine *j) > +{ > + int t_next, c_len; > + j->skip_s_getchar(t_next, c_len); > + > + if (t_next == C_QUOTE) > + { > + j->state= json_engine::KEY; > + return 0; > + } > + > + j->error= (t_next == C_EOS) ? JE_EOS : > + ((t_next == C_BAD) ? JE_BAD : > + JE_UEX); > + return 1; > +} > + > + > +/* Forward declarations. */ > +static int skp_c(json_engine *j); > +static int skp_k(json_engine *j); > +static int ee_cb(json_engine *j); > +static int ee_qb(json_engine *j); > +static int ee_cm(json_engine *j); > +static int ee_dn(json_engine *j); > + > + > +static int ar_c(json_engine *j) comment > +{ > + j->state= json_engine::VALUE; > + return 0; > +} comment > + > + > +static int arb(json_engine *j) comment > +{ > + j->state= json_engine::VALUE; > + j->c_str-= j->sav_c_len; > + return 0; > +} > + > + > +static json_state_handler json_actions[json_engine::NR_STATES][NR_C_CLASSES]= > +/* EOS { } [ ] : , " -0..9 f n t ETC ERR BAD*/ > +{ > +/*VALUE*/{er_en, ky_ob, __, ky_ar, __, __, __, scr_s, scr_n, scr_f, scr_l, scr_t, __, e_chr, b_chr}, > +/*DONE*/ {done, __, __, __, __, __, __, __, __, __, __, __, __, e_chr, b_chr}, > +/*KEY*/ {er_en, skp_k, skp_k, skp_k, skp_k, skp_k, skp_k, skp_c, skp_k, skp_k, skp_k, skp_k, skp_k, e_chr, b_chr}, } ] and some other characters - that's surely an error, why don't you use an error action for them? and why one skp_c ? > +/*OB_B*/ {er_en, __, en_ob, __, __, __, __, get_k, __, __, __, __, __, e_chr, b_chr}, > +/*OB_E*/ {ee_dn, __, ee_cb, __, ee_qb, __, ee_cm, __, __, __, __, __, __, e_chr, b_chr}, > +/*OB_C*/ {er_en, __, en_ob, __, en_ar, __, nex_k, __, __, __, __, __, __, e_chr, b_chr}, > +/*AR_B*/ {er_en, arb, __, arb, en_ar, __, __, arb, arb, arb, arb, arb, __, e_chr, b_chr}, > +/*AR_E*/ {ee_dn, __, ee_cb, __, ee_qb, __, ee_cm, __, __, __, __, __, __, e_chr, b_chr}, > +/*AR_C*/ {er_en, __, __, __, en_ar, __, ar_c, __, __, __, __, __, __, e_chr, b_chr}, > +/*RD_V*/ {er_en, rd_ob, __, rd_ar, __, __, __, rd_s, rd_n, rd_f, rd_l, rd_t, __, e_chr, b_chr}, > +}; > + > + > +void json_string::set_cs(CHARSET_INFO *i_cs) > +{ > + cs= i_cs; > + error= 0; > + wc= i_cs->cset->mb_wc; > +} > + > + > +int json_engine::scan_start(CHARSET_INFO *i_cs, const uchar *str, const uchar *end) > +{ > + json_string::setup(i_cs, str, end); > + stack[0]= json_engine::DONE; > + stack_p= stack; > + state= json_engine::VALUE; > + return 0; > +} > + > + > +/* Skip colon and the value. */ > +static int skp_c(json_engine *j) > +{ > + int t_next, c_len; > + > + j->skip_s_getchar(t_next, c_len); > + > + if (t_next == C_COLON) > + { > + j->skip_s_getchar(t_next, c_len); > + return json_actions[json_engine::VALUE][t_next](j); please check how compiler optimizes that. it is a call/ret or a jump? > + } > + > + j->error= (t_next == C_EOS) ? JE_EOS : > + ((t_next == C_BAD) ? JE_BAD : > + JE_UEX); > + > + return 1; > +} > + > + > +/* Skip colon and the value. */ same comment as for the previous function? > +static int skp_k(json_engine *j) > +{ > + int t_next, c_len; > + while (j->read_keyname_chr() == 0) {} > + > + if (j->error) > + return 1; > + > + j->skip_s_getchar(t_next, c_len); > + return json_actions[json_engine::VALUE][t_next](j); > +} > + > + > +/* Continue after the end of the structure. */ > +static int ee_dn(json_engine *j) > +{ return json_actions[*j->stack_p][C_EOS](j); } You really need a comment explaining how this state machine works. > + > + > +/* Continue after the end of the structure. */ > +static int ee_cb(json_engine *j) > +{ return json_actions[*j->stack_p][C_RCURB](j); } > + > + > +/* Continue after the end of the structure. */ > +static int ee_qb(json_engine *j) > +{ return json_actions[*j->stack_p][C_RSQRB](j); } > + > + > +/* Continue after the end of the structure. */ > +static int ee_cm(json_engine *j) > +{ return json_actions[*j->stack_p][C_COMMA](j); } > + > + > +int json_engine::read_keyname_chr() comment. if the first quote already read by now? > +{ > + int c_len, t; > + > + if ((c_len= get_next_char()) > 0) > + { > + c_str+= c_len; > + if (c_next>= 128 || (t= json_instr_chr_map[c_next]) <= S_ETC) > + return 0; > + > + switch (t) > + { > + case S_QUOTE: > + for (;;) /* Skip spaces until ':'. */ why? you can use skip_s_getchar() to read whatever comes after the key. > + { > + if ((c_len= get_next_char() > 0)) > + { > + if (c_next == ':') > + { > + c_str+= c_len; > + state= VALUE; > + return 1; > + } > + > + if (c_next < 128 && json_chr_map[c_next] == C_SPACE) > + { > + c_str+= c_len; > + continue; > + } > + error= JE_UEX; > + break; > + } > + error= eos() ? JE_EOS : JE_BAD; > + break; > + } > + return 1; > + case S_SOLID: > + return handle_esc(); > + case S_ERR: > + c_str-= c_len; > + error= JE_IMS; > + return 1; > + } > + } > + error= eos() ? JE_EOS : JE_BAD; > + return 1; > +} > + > + > +int json_engine::read_value() > +{ > + int t_next, c_len, res; > + > + if (state == KEY) > + { > + while (read_keyname_chr() == 0) {} > + > + if (error) > + return 1; > + } > + > + skip_s_getchar(t_next, c_len); here, in fact you *do* use skip_s_getchar() after a key. > + > + value_begin= c_str-c_len; > + res= json_actions[READ_VALUE][t_next](this); > + value_end= c_str; > + return res; > +} > + > + > +int json_engine::scan_next() > +{ > + int t_next; > + > + skip_s_getchar(t_next, sav_c_len); > + return json_actions[state][t_next](this); > +} > + > + > +enum json_path_c_classes { > + P_EOS, /* end of string */ > + P_USD, /* $ */ > + P_ASTER, /* * */ > + P_LSQRB, /* [ */ > + P_RSQRB, /* ] */ > + P_POINT, /* . */ > + P_ZERO, /* 0 */ > + P_DIGIT, /* 123456789 */ > + P_L, /* l */ > + P_S, /* s */ better comments for P_L and P_S (for "lax" / for "strict") > + P_SPACE, /* space */ > + P_SOLID, /* \ */ > + P_ETC, /* everything else */ > + P_ERR, /* character disallowed in JSON*/ > + P_BAD, /* invalid character */ > + N_PATH_CLASSES, > +}; > + > + > +static int json_path_chr_map[128] = { > + P_ERR, P_ERR, P_ERR, P_ERR, P_ERR, P_ERR, P_ERR, P_ERR, > + P_ERR, P_SPACE, P_SPACE, P_ERR, P_ERR, P_SPACE, P_ERR, P_ERR, > + P_ERR, P_ERR, P_ERR, P_ERR, P_ERR, P_ERR, P_ERR, P_ERR, > + P_ERR, P_ERR, P_ERR, P_ERR, P_ERR, P_ERR, P_ERR, P_ERR, > + > + P_SPACE, P_ETC, P_ETC, P_ETC, P_USD, P_ETC, P_ETC, P_ETC, > + P_ETC, P_ETC, P_ASTER, P_ETC, P_ETC, P_ETC, P_POINT, P_ETC, > + P_ZERO, P_DIGIT, P_DIGIT, P_DIGIT, P_DIGIT, P_DIGIT, P_DIGIT, P_DIGIT, > + P_DIGIT, P_DIGIT, P_ETC, P_ETC, P_ETC, P_ETC, P_ETC, P_ETC, > + > + P_ETC, P_ETC, P_ETC, P_ETC, P_ETC, P_ETC, P_ETC, P_ETC, > + P_ETC, P_ETC, P_ETC, P_ETC, P_L, P_ETC, P_ETC, P_ETC, > + P_ETC, P_ETC, P_S, P_ETC, P_ETC, P_ETC, P_ETC, P_ETC, > + P_ETC, P_ETC, P_ETC, P_LSQRB, P_SOLID, P_RSQRB, P_ETC, P_ETC, > + > + P_ETC, P_ETC, P_ETC, P_ETC, P_ETC, P_ETC, P_ETC, P_ETC, > + P_ETC, P_ETC, P_ETC, P_ETC, P_L, P_ETC, P_ETC, P_ETC, > + P_ETC, P_ETC, P_S, P_ETC, P_ETC, P_ETC, P_ETC, P_ETC, > + P_ETC, P_ETC, P_ETC, P_ETC, P_ETC, P_ETC, P_ETC, P_ETC > +}; > + > + > +enum json_path_states { > + PS_GO, > + PS_LAX, > + PS_PT, > + PS_AR, > + PS_AWD, > + PS_Z, > + PS_INT, > + PS_AS, > + PS_KEY, > + PS_KNM, > + PS_KWD, > + N_PATH_STATES, > + PS_SCT, > + PS_EKY, > + PS_EAR, > + PS_ESC, > + PS_OK, > + PS_KOK > +}; comment! explain every state, please! > + > + > +static int json_path_transitions[N_PATH_STATES][N_PATH_CLASSES]= > +{ > +/* EOS $, * [ ] . 0 1..9 L S SPACE SOLID ETC ERR BAD */ > +/* GO */ { JE_EOS, PS_PT, JE_UEX, JE_UEX, JE_UEX, JE_UEX, JE_UEX, JE_UEX, PS_LAX, PS_SCT, PS_GO, JE_UEX, JE_UEX, JE_IMP, JE_BAD}, > +/* LAX */ { JE_EOS, JE_UEX, JE_UEX, JE_UEX, JE_UEX, JE_UEX, JE_UEX, JE_UEX, PS_LAX, JE_UEX, PS_GO, JE_UEX, JE_UEX, JE_IMP, JE_BAD}, > +/* PT */ { PS_OK, JE_UEX, JE_UEX, PS_AR, JE_UEX, PS_KEY, JE_UEX, JE_UEX, JE_UEX, JE_UEX, JE_UEX, JE_UEX, JE_UEX, JE_IMP, JE_BAD}, > +/* AR */ { JE_EOS, JE_UEX, PS_AWD, JE_UEX, PS_PT, JE_UEX, PS_Z, PS_INT, JE_UEX, JE_UEX, PS_AR, JE_UEX, JE_UEX, JE_IMP, JE_BAD}, > +/* AWD */ { JE_EOS, JE_UEX, JE_UEX, JE_UEX, PS_PT, JE_UEX, JE_UEX, JE_UEX, JE_UEX, JE_UEX, PS_AS, JE_UEX, JE_UEX, JE_IMP, JE_BAD}, > +/* Z */ { JE_EOS, JE_UEX, JE_UEX, JE_UEX, PS_PT, JE_UEX, JE_UEX, JE_UEX, JE_UEX, JE_UEX, PS_AS, JE_UEX, JE_UEX, JE_IMP, JE_BAD}, > +/* INT */ { JE_EOS, JE_UEX, JE_UEX, JE_UEX, PS_PT, JE_UEX, PS_INT, PS_INT, JE_UEX, JE_UEX, PS_AS, JE_UEX, JE_UEX, JE_IMP, JE_BAD}, > +/* AS */ { JE_EOS, JE_UEX, JE_UEX, JE_UEX, PS_PT, JE_UEX, JE_UEX, JE_UEX, JE_UEX, JE_UEX, PS_AS, JE_UEX, JE_UEX, JE_IMP, JE_BAD}, > +/* KEY */ { JE_EOS, PS_KNM, PS_KWD, JE_UEX, PS_KNM, JE_UEX, PS_KNM, PS_KNM, PS_KNM, PS_KNM, PS_KNM, JE_UEX, PS_KNM, JE_IMP, JE_BAD}, > +/* KNM */ { PS_KOK, PS_KNM, PS_KNM, PS_EAR, PS_KNM, PS_EKY, PS_KNM, PS_KNM, PS_KNM, PS_KNM, PS_KNM, PS_ESC, PS_KNM, JE_IMP, JE_BAD}, > +/* KWD */ { PS_OK, JE_UEX, JE_UEX, PS_AR, JE_UEX, PS_EKY, JE_UEX, JE_UEX, JE_UEX, JE_UEX, JE_UEX, JE_UEX, JE_UEX, JE_IMP, JE_BAD} > +}; > + > + > +int json_path::setup(CHARSET_INFO *i_cs, const uchar *str, const uchar *end) > +{ > + int c_len, t_next, state= PS_GO; > + > + json_string::setup(i_cs, str, end); > + > + steps[0].type= json_path_step::ARRAY; > + steps[0].wild= 1; > + last_step= steps; > + mode_strict= false; > + > + do > + { > + if ((c_len= get_next_char()) <= 0) > + t_next= eos() ? P_EOS : P_BAD; > + else > + t_next= (c_next >= 128) ? P_ETC : json_path_chr_map[c_next]; > + > + if ((state= json_path_transitions[state][t_next]) < 0) > + return error= state; > + > + c_str+= c_len; > + > + switch (state) > + { > + case PS_LAX: > + if ((error= assure_string(this, "ax"))) > + return 1; > + mode_strict= false; > + continue; > + case PS_SCT: > + if ((error= assure_string(this, "rict"))) > + return 1; > + mode_strict= true; > + state= PS_LAX; > + continue; I'd personally done lax/strict without a state machine, they can only be in the beginning of the string right? Two states and two path character classes less. > + case PS_AWD: > + last_step->wild= 1; > + continue; > + case PS_INT: > + last_step->n_item*= 10; > + last_step->n_item+= c_next - '0'; > + continue; > + case PS_EKY: > + last_step->key_end= c_str - c_len; > + state= PS_KEY; > + /* Note no 'continue' here. */ > + case PS_KEY: > + last_step++; > + last_step->type= json_path_step::KEY; > + last_step->wild= 0; > + last_step->key= c_str; > + continue; > + case PS_EAR: > + last_step->key_end= c_str - c_len; > + state= PS_AR; > + /* Note no 'continue' here. */ > + case PS_AR: > + last_step++; > + last_step->type= json_path_step::ARRAY; > + last_step->wild= 0; > + last_step->n_item= 0; > + continue; > + case PS_KWD: > + last_step->wild= 1; > + continue; > + case PS_ESC: > + if (handle_esc()) > + return 1; > + continue; > + case PS_KOK: > + last_step->key_end= c_str - c_len; > + state= PS_OK; > + break; > + }; > + } while (state != PS_OK); > + > + return 0; > +} > + > + > +int json_engine::skip_level() > +{ > + int ct= 0; > + > + while (scan_next() == 0) > + { > + switch (state) { > + case OB_B: > + case AR_B: > + ct++; > + break; > + case OB_E: > + case AR_E: > + if (ct == 0) > + return 0; > + ct--; > + break; > + } > + } > + > + return 1; > +} > + > + > +int json_engine::skip_key() comment! is it what standard says? that the search should continue over non-scalars instead of aborting with an error? > +{ > + if (read_value()) > + return 1; > + > + if (value_scalar()) > + return 0; > + > + return skip_level(); > +} > diff --git a/sql/item_jsonfunc.h b/sql/item_jsonfunc.h > new file mode 100644 > index 0000000..7aea3f5 > --- /dev/null > +++ b/sql/item_jsonfunc.h > @@ -0,0 +1,132 @@ > +#ifndef ITEM_JSONFUNC_INCLUDED > +#define ITEM_JSONFUNC_INCLUDED > + > +/* Copyright (c) 2016, MariaDB > + > + This program is free software; you can redistribute it and/or modify > + it under the terms of the GNU General Public License as published by > + the Free Software Foundation; version 2 of the License. > + > + This program is distributed in the hope that it will be useful, > + but WITHOUT ANY WARRANTY; without even the implied warranty of > + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + GNU General Public License for more details. > + > + You should have received a copy of the GNU General Public License > + along with this program; if not, write to the Free Software > + Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA */ > + > + > +/* This file defines all XML functions */ JSON > + > + > +#ifdef USE_PRAGMA_INTERFACE > +#pragma interface /* gcc class implementation */ > +#endif don't bother (and feel free to remove it from item_xmlfunc.h too :) > + > +#include "item_cmpfunc.h" // Item_bool_func > +#include "item_strfunc.h" // Item_str_func > +#include "sql_json.h" > + > + > +class json_path_with_flags : public json_path > +{ > +public: > + bool constant; > + bool parsed; > + json_path_step *cur_step; > + void set_constant_flag(bool s_constant) > + { > + constant= s_constant; > + parsed= FALSE; > + } > +}; > + > + > +class Item_func_json_valid: public Item_int_func > +{ > +protected: > + String tmp_value; > + > +public: > + Item_func_json_valid(THD *thd, Item *json) : Item_int_func(thd, json) {} > + longlong val_int(); > + const char *func_name() const { return "json_valid"; } > + void fix_length_and_dec() > + { > + Item_int_func::fix_length_and_dec(); > + maybe_null= 1; > + } > +}; > + > + > +class Item_func_json_value: public Item_str_func > +{ > +protected: > + json_path_with_flags path; > + String tmp_js, tmp_path; > + > +public: > + Item_func_json_value(THD *thd, Item *js, Item *path): > + Item_str_func(thd, js, path) {} > + const char *func_name() const { return "json_value"; } > + void fix_length_and_dec(); > + String *val_str(String *); > +}; > + > + > +class Item_func_json_extract: public Item_str_func > +{ > +protected: > + String tmp_js; > + json_path_with_flags *paths; > + String **tmp_paths; > + > +public: > + Item_func_json_extract(THD *thd, List<Item> &list): > + Item_str_func(thd, list) {} > + const char *func_name() const { return "json_extract"; } > + bool fix_fields(THD *thd, Item **ref); > + void fix_length_and_dec(); > + String *val_str(String *); > + longlong val_int(); > +}; > + > + > +class Item_func_json_contains_path: public Item_int_func > +{ > +protected: > + String tmp_js; > + json_path_with_flags *paths; > + String **tmp_paths; > + bool mode_one; > + bool ooa_constant, ooa_parsed; > + > +public: > + Item_func_json_contains_path(THD *thd, List<Item> &list): > + Item_int_func(thd, list) {} > + const char *func_name() const { return "json_contains_path"; } > + bool fix_fields(THD *thd, Item **ref); > + void fix_length_and_dec(); > + longlong val_int(); > +}; > + > + > +class Item_func_json_array_append: public Item_str_func > +{ > +protected: > + String tmp_js; > + String tmp_val; > + json_path_with_flags *paths; > + String **tmp_paths; > +public: > + Item_func_json_array_append(THD *thd, List<Item> &list): > + Item_str_func(thd, list) {} > + bool fix_fields(THD *thd, Item **ref); > + void fix_length_and_dec(); > + String *val_str(String *); > + const char *func_name() const { return "json_array_append"; } > +}; > + > + > +#endif /* ITEM_JSONFUNC_INCLUDED */ > diff --git a/sql/item_jsonfunc.cc b/sql/item_jsonfunc.cc > new file mode 100644 > index 0000000..989fc6e > --- /dev/null > +++ b/sql/item_jsonfunc.cc > @@ -0,0 +1,654 @@ > +/* Copyright (c) 2016, Monty Program Ab. > + > + This program is free software; you can redistribute it and/or modify > + it under the terms of the GNU General Public License as published by > + the Free Software Foundation; version 2 of the License. > + > + This program is distributed in the hope that it will be useful, > + but WITHOUT ANY WARRANTY; without even the implied warranty of > + MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the > + GNU General Public License for more details. > + > + You should have received a copy of the GNU General Public License > + along with this program; if not, write to the Free Software > + Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA */ > + > +#ifdef __GNUC__ > +#pragma implementation > +#endif don't bother (and feel free to remove it from item_xmlfunc.cc too :) > + > +#include <my_global.h> > +#include "sql_priv.h" > +#include "sql_class.h" > +#include "item.h" > + > + function comment? > +static bool eq_ascii_string(const CHARSET_INFO *cs, > + const char *ascii, > + const char *s, uint32 s_len) > +{ > + const char *s_end= s + s_len; > + > + while (*ascii && s < s_end) > + { > + my_wc_t wc; > + int wc_len; > + > + wc_len= cs->cset->mb_wc(cs, &wc, (uchar *) s, (uchar *) s_end); > + if (wc_len <= 0 || (wc | 0x20) != (my_wc_t) *ascii) > + return 0; > + > + ascii++; > + s+= wc_len; > + } > + > + return *ascii == 0 && s >= s_end; > +} Why did you need a special function for that? And you're only using it for "one" and "all". It's totally not worth optimizing for, just use strnncoll. You'll most likely be doing it only once anyway. > + > + > +longlong Item_func_json_valid::val_int() > +{ > + String *js= args[0]->val_str(&tmp_value); > + json_engine je; > + > + if ((null_value= args[0]->null_value) || js == NULL) > + return 0; > + > + je.scan_start(js->charset(), (const uchar *) js->ptr(), > + (const uchar *) js->ptr()+js->length()); > + > + while (je.scan_next() == 0) {} > + > + return je.error == 0; > +} > + > + > +void Item_func_json_value::fix_length_and_dec() > +{ > + collation.set(args[0]->collation); > + max_length= args[0]->max_length; > + path.constant= args[1]->const_item(); > + path.parsed= FALSE; path.set_constant_flag() > +} > + > + > +static int handle_match(json_engine *je, json_path_with_flags *p, uint *n_arrays) comment! I don't understand what this function does > +{ > + DBUG_ASSERT(p->cur_step < p->last_step); > + > + if (je->read_value()) > + return 1; > + > + if (je->value_scalar()) > + return 0; > + > + p->cur_step++; > + n_arrays[p->cur_step - p->steps]= 0; > + > + if ((int) je->value_type != (int) p->cur_step->type) > + { > + p->cur_step--; > + return je->skip_level(); > + } > + > + return 0; > +} > + > + > +static int json_key_matches(json_engine *je, json_string *k) > +{ > + while (je->read_keyname_chr() == 0) > + { > + if (k->read_str_chr() || > + je->next_chr() != k->next_chr()) > + return 0; > + } > + > + if (k->read_str_chr()) > + return 1; > + > + return 0; > +} > + > + > +static int find_value(json_engine *je, String *js, json_path_with_flags *p) > +{ > + json_string kn; > + uint n_arrays[json_depth_limit]; > + > + kn.set_cs(p->cs); what does 'kn' mean? key name? then call it 'key_name' or 'key' or at least add a comment > + > + do > + { > + json_path_step *cur_step= p->cur_step; > + switch (je->state) > + { > + case json_engine::KEY: > + DBUG_ASSERT(cur_step->type == json_path_step::KEY); > + if (!cur_step->wild) > + { > + kn.set_str(cur_step->key, cur_step->key_end); > + if (!json_key_matches(je, &kn)) > + { > + if (je->skip_key()) > + goto exit; > + continue; > + } > + } > + if (cur_step == p->last_step || > + handle_match(je, p, n_arrays)) > + goto exit; > + break; > + case json_engine::VALUE: > + DBUG_ASSERT(cur_step->type == json_path_step::ARRAY); > + if (cur_step->wild || > + cur_step->n_item == n_arrays[cur_step - p->steps]) > + { > + /* Array item matches. */ > + if (cur_step == p->last_step || > + handle_match(je, p, n_arrays)) > + goto exit; > + } > + else > + { > + je->skip_array_item(); > + n_arrays[cur_step - p->steps]++; > + } > + break; > + case json_engine::OB_E: > + case json_engine::AR_E: > + p->cur_step--; > + break; > + default: > + DBUG_ASSERT(0); > + break; > + }; > + } while (je->scan_next() == 0); > + > + /* No luck. */ > + return 1; > + > +exit: > + return je->error; > +} > + > + > +String *Item_func_json_value::val_str(String *str) > +{ > + json_engine je; > + String *js= args[0]->val_str(&tmp_js); > + > + if (!path.constant || !path.parsed) > + { > + String *s_p= args[1]->val_str(&tmp_path); > + if (s_p && > + path.setup(s_p->charset(), (const uchar *) s_p->ptr(), > + (const uchar *) s_p->ptr() + s_p->length())) > + goto error; > + path.parsed= TRUE; better: path.parsed= path.constant; and only if(path.parsed) in the if() above. > + } > + > + if ((null_value= args[0]->null_value || args[1]->null_value)) > + return NULL; > + > + je.scan_start(js->charset(),(const uchar *) js->ptr(), > + (const uchar *) js->ptr() + js->length()); > + > + path.cur_step= path.steps; > +continue_search: > + if (find_value(&je, js, &path)) > + { > + if (je.error) > + goto error; > + > + null_value= 1; > + return 0; > + } ^^^ this { .... } can be replaced with "goto error" > + > + if (je.read_value()) > + goto error; > + > + if (!je.value_scalar()) > + { > + /* We only look for scalar values! */ > + if (je.skip_level() || je.scan_next()) > + goto error; > + goto continue_search; > + } ^^^ I don't understand that part. 1. what is je.skip_level() || je.scan_next() ? 2. is it what standard says? that the search should continue over non-scalars instead of aborting with an error? > + > + str->set((const char *) je.value, je.value_len, je.cs); is that right? json_value is supposed to the return a value, not json. doesn't it mean that you should convert this json string to its actual string value? remove all escapes, that is? > + return str; > + > +error: > + null_value= 1; > + return 0; > +} > + > + > +static int alloc_tmp_paths(THD *thd, uint n_paths, > + json_path_with_flags **paths,String ***tmp_paths) > +{ > + uint n; > + *paths= (json_path_with_flags *) alloc_root(thd->mem_root, > + sizeof(json_path_with_flags) * n_paths); > + *tmp_paths= (String **) alloc_root(thd->mem_root, sizeof(String *) * n_paths); > + if (*paths == 0 || *tmp_paths == 0) > + return 1; > + > + for (n=0; n<n_paths; n++) > + { > + if (((*tmp_paths)[n]= new(thd->mem_root) String) == 0) > + return 1; > + } > + return 0; > +} Uh. String allocates the memory on the heap. Now, you allocate the String itself on a memroot. That's fine, but I don't see where you destroy these Strings, they must be explicitly destroyed, even if they're in a memroot. > + > + > +static void mark_constant_paths(json_path_with_flags *p, > + Item** args, uint n_args) > +{ > + uint n; > + for (n= 0; n < n_args; n++) > + p[n].set_constant_flag(args[n]->const_item()); > +} perhaps a cleaner option would be to create, say, class Item_func_json_many_paths : public Item_str_func { String tmp_js; json_path_with_flags *paths; String **tmp_paths; void mark_constant_paths(); int alloc_tmp_paths(); and inherit Item_func_json_extract and Item_func_json_contains_path from it. > + > + > +bool Item_func_json_extract::fix_fields(THD *thd, Item **ref) > +{ > + return alloc_tmp_paths(thd, arg_count-1, &paths, &tmp_paths) || > + Item_str_func::fix_fields(thd, ref); > +} > + > + > +void Item_func_json_extract::fix_length_and_dec() > +{ > + collation.set(args[0]->collation); > + max_length= args[0]->max_length * (arg_count - 1); > + > + mark_constant_paths(paths, args+1, arg_count-1); > +} > + > + > +String *Item_func_json_extract::val_str(String *str) > +{ > + String *js= args[0]->val_str(&tmp_js); > + json_engine je; > + bool multiple_values_found= FALSE; > + const uchar *value; > + const char *first_value= NULL, *first_p_value; > + uint n_arg, v_len, first_len, first_p_len; > + > + if ((null_value= args[0]->null_value)) > + return 0; > + > + str->set_charset(js->charset()); > + str->length(0); > + > + for (n_arg=1; n_arg < arg_count; n_arg++) > + { > + json_path_with_flags *c_path= paths + n_arg - 1; > + if (!c_path->constant || !c_path->parsed) > + { > + String *s_p= args[n_arg]->val_str(tmp_paths[n_arg-1]); > + if (s_p && > + c_path->setup(s_p->charset(), (const uchar *) s_p->ptr(), > + (const uchar *) s_p->ptr() + s_p->length())) > + goto error; > + c_path->parsed= TRUE; > + } same as in Item_func_json_value, could be moved to a common parent method, Item_func_json::setup_path(); it doesn't need to be virtual, so compiler will inline it and you'll have the same end result with less copy-pasted code > + > + je.scan_start(js->charset(),(const uchar *) js->ptr(), > + (const uchar *) js->ptr() + js->length()); > + > + c_path->cur_step= c_path->steps; > + > + if (find_value(&je, js, c_path)) > + { > + /* Path wasn't found. */ > + if (je.error) > + goto error; > + > + continue; > + } > + > + if (je.read_value()) > + goto error; > + > + value= je.value_begin; > + if (je.value_scalar()) > + v_len= je.value_end - value; > + else > + { > + if (je.skip_level()) > + goto error; > + v_len= je.c_str - value; > + } > + > + if (!multiple_values_found) > + { > + if (first_value == NULL) > + { > + /* > + Just remember the first value as we don't know yet > + if we need to create an array out of it or not. > + */ > + first_value= (const char *) value; > + first_len= v_len; > + /* > + We need this as we have to preserve quotes around string > + constants if we use the value to create an array. Otherwise > + we get the value without the quotes. > + */ > + first_p_value= (const char *) je.value; > + first_p_len= je.value_len; > + continue; > + } > + else > + { > + multiple_values_found= TRUE; /* We have to make an JSON array. */ > + if ( str->append("[") || > + str->append(first_value, first_len)) > + goto error; /* Out of memory. */ > + } > + > + } > + if (str->append(", ", 2) || > + str->append((const char *) value, v_len)) > + goto error; /* Out of memory. */ > + } > + > + if (first_value == NULL) > + { > + /* Nothing was found. */ > + null_value= 1; > + return 0; > + } > + > + if (multiple_values_found ? > + str->append("]") : > + str->append(first_p_value, first_p_len)) > + goto error; /* Out of memory. */ > + > + return str; > + > +error: > + /* TODO: launch error messages. */ > + null_value= 1; > + return 0; > +} > + > + > +longlong Item_func_json_extract::val_int() > +{ > + String *js= args[0]->val_str(&tmp_js); > + json_engine je; > + uint n_arg; > + > + if ((null_value= args[0]->null_value)) > + return 0; > + > + for (n_arg=1; n_arg < arg_count; n_arg++) > + { > + json_path_with_flags *c_path= paths + n_arg - 1; > + if (!c_path->constant || !c_path->parsed) > + { > + String *s_p= args[n_arg]->val_str(tmp_paths[n_arg-1]); > + if (s_p && > + c_path->setup(s_p->charset(), (const uchar *) s_p->ptr(), > + (const uchar *) s_p->ptr() + s_p->length())) > + goto error; > + c_path->parsed= TRUE; > + } > + > + je.scan_start(js->charset(),(const uchar *) js->ptr(), > + (const uchar *) js->ptr() + js->length()); > + > + c_path->cur_step= c_path->steps; > + > + if (find_value(&je, js, c_path)) > + { > + /* Path wasn't found. */ > + if (je.error) > + goto error; > + > + continue; > + } > + > + if (je.read_value()) > + goto error; > + > + if (je.value_scalar()) > + { > + int err; > + char *v_end= (char *) je.value_end; > + return (je.cs->cset->strtoll10)(je.cs, (const char *) je.value_begin, > + &v_end, &err); > + } > + else > + break; > + } > + > + /* Nothing was found. */ > + null_value= 1; > + return 0; > + > +error: > + /* TODO: launch error messages. */ > + null_value= 1; > + return 0; > +} > + > + > +bool Item_func_json_contains_path::fix_fields(THD *thd, Item **ref) > +{ > + return alloc_tmp_paths(thd, arg_count-2, &paths, &tmp_paths) || > + Item_int_func::fix_fields(thd, ref); > +} > + > + > +void Item_func_json_contains_path::fix_length_and_dec() > +{ > + ooa_constant= args[1]->const_item(); > + ooa_parsed= FALSE; > + mark_constant_paths(paths, args+2, arg_count-2); > + Item_int_func::fix_length_and_dec(); > +} > + > + > +longlong Item_func_json_contains_path::val_int() > +{ > + String *js= args[0]->val_str(&tmp_js); > + json_engine je; > + uint n_arg; > + longlong result; > + > + if ((null_value= args[0]->null_value)) > + return 0; > + > + if (!ooa_constant || !ooa_parsed) > + { > + char buff[20]; > + String *res, tmp(buff, sizeof(buff), &my_charset_bin); > + res= args[1]->val_str(&tmp); > + mode_one=eq_ascii_string(res->charset(), "one", > + res->ptr(), res->length()); > + if (!mode_one) > + { > + if (!eq_ascii_string(res->charset(), "all", res->ptr(), res->length())) > + goto error; > + } > + ooa_parsed= TRUE; > + } > + > + result= !mode_one; > + for (n_arg=2; n_arg < arg_count; n_arg++) > + { > + json_path_with_flags *c_path= paths + n_arg - 2; > + if (!c_path->constant || !c_path->parsed) > + { > + String *s_p= args[n_arg]->val_str(tmp_paths[n_arg-2]); > + if (s_p && > + c_path->setup(s_p->charset(), (const uchar *) s_p->ptr(), > + (const uchar *) s_p->ptr() + s_p->length())) > + goto error; > + c_path->parsed= TRUE; > + } > + > + je.scan_start(js->charset(),(const uchar *) js->ptr(), > + (const uchar *) js->ptr() + js->length()); > + > + c_path->cur_step= c_path->steps; > + if (find_value(&je, js, c_path)) > + { > + /* Path wasn't found. */ > + if (je.error) > + goto error; > + > + if (!mode_one) > + { > + result= 0; > + break; > + } > + } > + else if (mode_one) > + { > + result= 1; > + break; > + } > + } > + > + > + return result; > + > +error: > + null_value= 1; > + return 0; > +} > + > + > +bool Item_func_json_array_append::fix_fields(THD *thd, Item **ref) > +{ > + return alloc_tmp_paths(thd, arg_count/2, &paths, &tmp_paths) || > + Item_str_func::fix_fields(thd, ref); > +} > + > + > +void Item_func_json_array_append::fix_length_and_dec() > +{ > + uint n_arg; > + ulonglong char_length; > + > + collation.set(args[0]->collation); > + char_length= args[0]->max_char_length(); > + > + for (n_arg= 1; n_arg < arg_count; n_arg+= 2) > + { > + paths[n_arg-1].set_constant_flag(args[n_arg]->const_item()); > + char_length+= args[n_arg+1]->max_char_length() + 4; > + } > + > + fix_char_length_ulonglong(char_length); > +} > + > + > +static int append_json(String *str, Item *item, String *tmp_val) > +{ > + if (item->is_bool_type()) > + { > + longlong v_int= item->val_int(); > + if (item->null_value) > + goto append_null; > + > + const char *t_f; > + t_f= v_int ? "true" : "false"; > + return str->append(t_f, strlen(t_f)); > + } > + { > + String *sv= item->val_str(tmp_val); > + if (item->null_value) > + goto append_null; > + return str->append(*sv); It seems that you believe the item to be a number. Because I'd expect you to quote and properly escape strings here. > + } > + > +append_null: > + return str->append("null", 4); > +} > + > + > +String *Item_func_json_array_append::val_str(String *str) > +{ > + json_engine je; > + String *js= args[0]->val_str(&tmp_js); > + uint n_arg, n_path, str_rest_len; > + const uchar *ar_end; > + > + DBUG_ASSERT(fixed == 1); > + > + if ((null_value= args[0]->null_value)) > + return 0; > + > + for (n_arg=1, n_path=0; n_arg < arg_count; n_arg+=2, n_path++) > + { > + json_path_with_flags *c_path= paths + n_path; > + if (!c_path->constant || !c_path->parsed) > + { > + String *s_p= args[n_arg]->val_str(tmp_paths[n_path]); > + if (s_p && > + c_path->setup(s_p->charset(), (const uchar *) s_p->ptr(), > + (const uchar *) s_p->ptr() + s_p->length())) > + goto error; > + c_path->parsed= TRUE; > + } > + if (args[n_arg]->null_value) > + { > + null_value= 1; > + return 0; > + } > + > + je.scan_start(js->charset(),(const uchar *) js->ptr(), > + (const uchar *) js->ptr() + js->length()); > + c_path->cur_step= c_path->steps; > + > + if (find_value(&je, js, c_path)) > + { > + if (je.error) > + goto error; > + null_value= 1; > + return 0; > + } > + > + if (je.read_value()) > + goto error; > + > + if (je.value_type != json_engine::ARRAY) > + { > + /* Must be an array. */ > + goto error; > + } > + > + if (je.skip_level()) > + goto error; > + > + str->length(0); > + str->set_charset(js->charset()); > + if (str->reserve(js->length() + 8, 512)) > + goto error; /* Out of memory. */ > + ar_end= je.c_str - je.sav_c_len; > + str_rest_len= js->length() - (ar_end - (const uchar *) js->ptr()); > + str->q_append(js->ptr(), ar_end-(const uchar *) js->ptr()); > + str->append(", ", 2); > + if (append_json(str, args[n_arg+1], &tmp_val)) > + goto error; /* Out of memory. */ > + > + if (str->reserve(str->length() + str_rest_len, 512)) > + goto error; /* Out of memory. */ > + str->q_append((const char *) ar_end, str_rest_len); > + } > + > + return str; > + > +error: > + null_value= 1; > + return 0; > +} > + Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] review for MDEV-9143 (JSON).
by Alexey Botchkov 26 Jul '16

26 Jul '16

Hi, Sergei. I've requested your review on this, here are few additional notes. I ran some benchmarks comparing against MySQL5.7's JSON field type. And it seems that the difference in performance is negligible. For instance here are results for 20M rows with rather complicated JSON: Space occupied: MariaDB -rw-rw----. 1 hf hf 9520000000 Jul 25 13:08 jt1.MYD MySQL -rw-r-----. 1 hf hf 9440000000 Jul 25 13:28 jt1.MYD Query time: MariaDB [test]> select count(*) from jt1 where json_extract(j, "$.key1[1].key2")='IT'; +----------+ | count(*) | +----------+ | 20000000 | +----------+ 1 row in set (7.65 sec) MySQL [test]> select count(*) from jt1 where json_extract(j, "$.key1[1].key2")='IT'; +----------+ | count(*) | +----------+ | 20000000 | +----------+ 1 row in set (7.56 sec) So again i can't think now of usecases where we have to store the JSON in some special binary representation. Only thing i can figure out is that we'd like to be able to read the MySQL database without any specific conversion. Best regards. HF

1 1

Re: [Maria-developers] [MariaDB/server] MDEV-10271: Log stop of slave SQL thread on errors, too (#188)
by Kristian Nielsen 25 Jul '16

25 Jul '16

Hartmut Holzgraefe <hartmut(a)php.net> writes: > there is an according internal escalation request to engineering > @mariadb.com ... as this is a customer request a launchpad list > is not really the right point to discuss this at this point ... Well, since you are asking to get changes in stable 10.0 and 10.1 releases, that certainly is something that should be discussed on maria-developers@. This change will affect most everyone, as security fixes are only available by upgrading to latest stable version, so needs to be discussed. The "internal escalation request" is not relevant... - Kristian.

1 1

Re: [Maria-developers] [Commits] 0f3aaf1: MDEV-10045: Server crashes in Time_and_counter_tracker::incr_loops
by Sergey Petrunia 21 Jul '16

21 Jul '16

Hi Sanja, Please find some review feeback below (please also check my reasoning while addressing it). I was unable to find any other issues, ok to push after the feedback is addressed. We should also ask Elena to run testing with correlated/non-correlated subqueries, EXPLAINs and ANALYZE. On Mon, Jun 27, 2016 at 04:43:26PM +0200, Oleksandr Byelkin wrote: > revision-id: 0f3aaf1439a22ff5e4db5374b2eefe702e1b246c (mariadb-10.1.14-27-g0f3aaf1) > parent(s): 6f6692008617d789b581971541dd9a1377c8dc5c > committer: Oleksandr Byelkin > timestamp: 2016-06-27 16:43:26 +0200 > message: > > MDEV-10045: Server crashes in Time_and_counter_tracker::incr_loops > > Do not set 'optimized' flag until whole optimization procedure is finished. > > --- > mysql-test/r/subselect.result | 16 +++++++++++ > mysql-test/r/subselect_no_exists_to_in.result | 16 +++++++++++ > mysql-test/r/subselect_no_mat.result | 16 +++++++++++ > mysql-test/r/subselect_no_opts.result | 16 +++++++++++ > mysql-test/r/subselect_no_scache.result | 16 +++++++++++ > mysql-test/r/subselect_no_semijoin.result | 16 +++++++++++ > mysql-test/t/subselect.test | 17 ++++++++++++ > sql/item_subselect.cc | 39 ++++++++++++++++++++------- > sql/sql_select.cc | 24 +++++++++-------- > sql/sql_select.h | 5 ++-- > 10 files changed, 159 insertions(+), 22 deletions(-) ... > diff --git a/sql/item_subselect.cc b/sql/item_subselect.cc > index 1812110..944aa59 100644 > --- a/sql/item_subselect.cc > +++ b/sql/item_subselect.cc > @@ -557,6 +557,21 @@ void Item_subselect::recalc_used_tables(st_select_lex *new_parent, > bool Item_subselect::is_expensive() > { > double examined_rows= 0; > + bool all_are_simple= true; > + > + /* check extremely simple select */ This is a vague term. Does this mean we're checking for a single "(SELECT constant)", without UNION? > + if (!unit->first_select()->next_select()) // no union > + { > + /* > + such single selects works even without optimization because > + can not makes loops > + */ > + SELECT_LEX *sl= unit->first_select(); > + JOIN *join = sl->join; > + if (join && !join->tables_list && !sl->first_inner_unit()) > + return false; > + } > + > > for (SELECT_LEX *sl= unit->first_select(); sl; sl= sl->next_select()) > { > @@ -566,23 +581,27 @@ bool Item_subselect::is_expensive() > if (!cur_join) > return true; > > - /* very simple subquery */ > - if (!cur_join->tables_list && !sl->first_inner_unit()) > - return false; > - > /* > If the subquery is not optimised or in the process of optimization > it supposed to be expensive > */ > - if (!cur_join->optimized) > + if (cur_join->optimization_state != JOIN::OPTIMIZATION_DONE) > return true; > > + if (!cur_join->tables_list && !sl->first_inner_unit()) > + continue; > + > /* > Subqueries whose result is known after optimization are not expensive. > Such subqueries have all tables optimized away, thus have no join plan. > */ > if ((cur_join->zero_result_cause || !cur_join->tables_list)) > - return false; > + continue; > + > + /* > + This is not simple SELECT in union so we can not go by simple condition > + */ > + all_are_simple= false; > > /* > If a subquery is not optimized we cannot estimate its cost. A subquery is > @@ -603,7 +622,8 @@ bool Item_subselect::is_expensive() > examined_rows+= cur_join->get_examined_rows(); > } > > - return (examined_rows > thd->variables.expensive_subquery_limit); > + return !all_are_simple && > + (examined_rows > thd->variables.expensive_subquery_limit); > } > > > @@ -3663,7 +3683,7 @@ int subselect_single_select_engine::exec() > SELECT_LEX *save_select= thd->lex->current_select; > thd->lex->current_select= select_lex; > > - if (!join->optimized) > + if (join->optimization_state == JOIN::NOT_OPTIMIZED) > { > SELECT_LEX_UNIT *unit= select_lex->master_unit(); > > @@ -5311,7 +5331,8 @@ int subselect_hash_sj_engine::exec() > */ > thd->lex->current_select= materialize_engine->select_lex; > /* The subquery should be optimized, and materialized only once. */ > - DBUG_ASSERT(materialize_join->optimized && !is_materialized); > + DBUG_ASSERT(materialize_join->optimization_state == JOIN::OPTIMIZATION_DONE && > + !is_materialized); > materialize_join->exec(); > if ((res= MY_TEST(materialize_join->error || thd->is_fatal_error || > thd->is_error()))) > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index 4825726..775684c 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -694,7 +694,7 @@ JOIN::prepare(Item ***rref_pointer_array, > DBUG_ENTER("JOIN::prepare"); > > // to prevent double initialization on EXPLAIN > - if (optimized) > + if (optimization_state != JOIN::NOT_OPTIMIZED) > DBUG_RETURN(0); > > conds= conds_init; > @@ -1032,15 +1032,20 @@ bool JOIN::prepare_stage2() > > int JOIN::optimize() > { > - bool was_optimized= optimized; > + bool was_optimized= (optimization_state == JOIN::OPTIMIZATION_DONE); > + // to prevent double initialization on EXPLAIN > + if (optimization_state != JOIN::NOT_OPTIMIZED) > + return FALSE; Ok so, if we didn't do 'return FALSE' above, then optimization_state==NOT_OPTIMIZED, and this means that was_optimized==FALSE? > + optimization_state= JOIN::OPTIMIZATION_IN_PROGRESS; > + > int res= optimize_inner(); > /* > - If we're inside a non-correlated subquery, this function may be > + If we're inside a non-correlated subquery, this function may be > called for the second time after the subquery has been executed > and deleted. The second call will not produce a valid query plan, it will > - short-circuit because optimized==TRUE. > + short-circuit because optimization_state != JOIN::NOT_OPTIMIZED. > > - "was_optimized != optimized" is here to handle this case: > + "!was_optimized" is here to handle this case: > - first optimization starts, gets an error (from a const. cheap > subquery), returns 1 > - another JOIN::optimize() call made, and now join->optimize() will (*) > @@ -1049,7 +1054,7 @@ int JOIN::optimize() > Can have QEP_NOT_PRESENT_YET for degenerate queries (for example, > SELECT * FROM tbl LIMIT 0) > */ > - if (was_optimized != optimized && !res && have_query_plan != QEP_DELETED) > + if (!was_optimized && !res && have_query_plan != QEP_DELETED) was_optimized==FALSE, so there is no reason to check for this variable. It only makes the code confusing. If we don't check that variable, then there is no use for it, and it can be removed completely. Once was_optimized was removed, the comment near (*) should also be removed. The code after the patch seems to be able to handle the case it is talking about (error on first optimization, another join->optimize() call returning false). > { > create_explain_query_if_not_exists(thd->lex, thd->mem_root); > have_query_plan= QEP_AVAILABLE; > @@ -1058,6 +1063,7 @@ int JOIN::optimize() > !skip_sort_order && !no_order && (order || group_list), > select_distinct); > } > + optimization_state= JOIN::OPTIMIZATION_DONE; > return res; > } > > @@ -1083,10 +1089,6 @@ JOIN::optimize_inner() > DBUG_ENTER("JOIN::optimize"); > > do_send_rows = (unit->select_limit_cnt) ? 1 : 0; > - // to prevent double initialization on EXPLAIN > - if (optimized) > - DBUG_RETURN(0); > - optimized= 1; > DEBUG_SYNC(thd, "before_join_optimize"); > > THD_STAGE_INFO(thd, stage_optimizing); > @@ -2060,7 +2062,7 @@ int JOIN::init_execution() > { > DBUG_ENTER("JOIN::init_execution"); > > - DBUG_ASSERT(optimized); > + DBUG_ASSERT(optimization_state == JOIN::OPTIMIZATION_DONE); > DBUG_ASSERT(!(select_options & SELECT_DESCRIBE)); > initialized= true; > > diff --git a/sql/sql_select.h b/sql/sql_select.h > index 89ee63e..dfa96f1 100644 > --- a/sql/sql_select.h > +++ b/sql/sql_select.h > @@ -1290,7 +1290,8 @@ class JOIN :public Sql_alloc > enum join_optimization_state { NOT_OPTIMIZED=0, > OPTIMIZATION_IN_PROGRESS=1, > OPTIMIZATION_DONE=2}; > - bool optimized; ///< flag to avoid double optimization in EXPLAIN > + // state of JOIN optimization > + enum join_optimization_state optimization_state; > bool initialized; ///< flag to avoid double init_execution calls > > Explain_select *explain; > @@ -1378,7 +1379,7 @@ class JOIN :public Sql_alloc > ref_pointer_array= items0= items1= items2= items3= 0; > ref_pointer_array_size= 0; > zero_result_cause= 0; > - optimized= 0; > + optimization_state= JOIN::NOT_OPTIMIZED; > have_query_plan= QEP_NOT_PRESENT_YET; > initialized= 0; > cleaned= 0; > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Innodb tablespace corrupted and mariadb crashed.
by 김경남 19 Jul '16

19 Jul '16

Hi, We're use mariadb 10.1.3 as a main database of our system. We migrated data from develpment server to product server a week ago. Below is our error case : MariaDB shutdown with signal 6 when some query executes. (insert or update...) The error log is below. ------------------------------------------------------------------------ ---------------------------------- ... 2016-07-18 9:44:14 139882584566528 [ERROR] InnoDB: Block in space_id 0 in file /data001/masvc01/ibdata1 encrypted. 2016-07-18 9:44:14 139882584566528 [ERROR] InnoDB: However key management plugin or used key_id 286 is not found or used encryption algorithm or method does not match. 2016-07-18 9:44:14 139882584566528 [ERROR] InnoDB: Marking tablespace as missing. You may drop this table or install correct key management plugin and key file. 2016-07-18 9:44:14 139882584566528 [ERROR] InnoDB: Block in space_id 0 in file /data001/masvc01/ibdata1 encrypted. 2016-07-18 9:44:14 139882584566528 [ERROR] InnoDB: However key management plugin or used key_id 286 is not found or used encryption algorithm or method does not match. 2016-07-18 9:44:14 139882584566528 [ERROR] InnoDB: Marking tablespace as missing. You may drop this table or install correct key management plugin and key file. 2016-07-18 09:44:14 7f38f3c36b00 InnoDB: Assertion failure in thread 139882584566528 in file buf0buf.cc line 4803 ... ... ------------------------------------------------------------------------ ----------------------------------------------- We check out system innodb table space with innochecksum. The result is below. ------------------------------------------------------------------------ ----- /engn001/masvc01/mysql/bin/innochecksum -v ibdata1 InnoDB offline file checksum utility. Variables (--variable-name=value) and boolean options {FALSE|TRUE} Value (after reading options) --------------------------------- ---------------------------------------- verbose TRUE debug FALSE skip-corrupt FALSE count FALSE start-page 0 end-page 0 page 0 per-page-details FALSE leaf FALSE merge 0 Table is uncompressed Page size is 16384 file ibdata1_20160718 = 3221225472 bytes (196608 pages)... InnoChecksum; checking pages in range 0 to 196607 Fail; page 0 invalid (fails innodb and crc32 checksum) ------------------------------------------------------------------------ ---- We hava ibdata1, ibdata2, ibdata3 and ibdata4 and results of innochecksum are same. We compiled mariadb with debug option and test under same enviroment. Below is the mysql.trace when db crash. ------------------------------------------------- ======================================================================== ================================================== T@10 : | | | | | | | | | | | | | | | | | >print_buffer_to_file T@10 : | | | | | | | | | | | | | | | | | | enter: buffer: InnoDB: Marking tablespace as missing. You may drop this table or install correct key management plugin and key file. T@10 : | | | | | | | | | | | | | | | | | | mutex: LOCK_error_log (0x15b4b60) locking T@10 : | | | | | | | | | | | | | | | | | | mutex: LOCK_error_log (0x15b4b60) locked T@10 : | | | | | | | | | | | | | | | | | | mutex: LOCK_error_log (0x15b4b60) unlocking T@10 : | | | | | | | | | | | | | | | | | <print_buffer_to_file T@10 : | | | | | | | | | | | | | | | | <vprint_msg_to_log T@10 : | | | | | | | | | | | | | | | <sql_print_error T@10 : | | | | | | | | | | | | | | | >my_malloc T@10 : | | | | | | | | | | | | | | | | my: size: 4096 my_flags: 16 T@10 : | | | | | | | | | | | | | | | | exit: ptr: 0x7f9e84159690 T@10 : | | | | | | | | | | | | | | | <my_malloc T@10 : | | | | | | | | | | | | | | | >push_warning_printf T@10 : | | | | | | | | | | | | | | | | enter: warning: 192 T@10 : | | | | | | | | | | | | | | | | >push_warning T@10 : | | | | | | | | | | | | | | | | | enter: code: 192, msg: Table in tablespace 0 encrypted.However key management plugin or used key_id 290 is not found or used encryption algorithm or method does not match. Can't continue opening the table. T@10 : | | | | | | | | | | | | | | | | | >THD::raise_condition T@10 : | | | | | | | | | | | | | | | | | | >mysql_audit_acquire_plugins T@10 : | | | | | | | | | | | | | | | | | | <mysql_audit_acquire_plugins T@10 : | | | | | | | | | | | | | | | | | | mutex: l_perm->lock (0x5dcaeb00) locking T@10 : | | | | | | | | | | | | | | | | | | >my_multi_malloc T@10 : | | | | | | | | | | | | | | | | | | | >my_malloc T@10 : | | | | | | | | | | | | | | | | | | | | my: size: 208 my_flags: 24 T@10 : | | | | | | | | | | | | | | | | | | | | exit: ptr: 0x7f9e8415a710 T@10 : | | | | | | | | | | | | | | | | | | | <my_malloc T@10 : | | | | | | | | | | | | | | | | | | <my_multi_malloc T@10 : | | | | | | | | | | | | | | | | | | >my_hash_init T@10 : | | | | | | | | | | | | | | | | | | | enter: hash: 0x7f9e8415a710 size: 128 T@10 : | | | | | | | | | | | | | | | | | | | >init_dynamic_array2 T@10 : | | | | | | | | | | | | | | | | | | | | >my_malloc ------------------------------------------------------------------------ --------------------------------------------------- So we export data from mariadb and import to new table space. After migration, we shutdown mariadb and restart. but signal 11 occurred and db crash. Below is our error log. ------------------------------------------------------------------------ -------------------------- 2016-07-19 8:55:09 140556344748800 [Note] /engn001/masvc01/mysql/bin/mysqld: Normal shutdown 2016-07-19 8:55:09 140556344748800 [Note] Event Scheduler: Killing the scheduler thread, thread id 1 2016-07-19 8:55:09 140556344748800 [Note] Event Scheduler: Waiting for the scheduler thread to reply 2016-07-19 8:55:09 140556344748800 [Note] Event Scheduler: Stopped 2016-07-19 8:55:09 140556344748800 [Note] Event Scheduler: Purging the queue. 0 events 160719 8:55:09 server_audit: STOPPED 2016-07-19 8:55:09 140556627830528 [Note] InnoDB: FTS optimize thread exiting. 2016-07-19 8:55:09 140556344748800 [Note] InnoDB: Starting shutdown... 2016-07-19 8:55:09 140556648810240 [Warning] InnoDB: Dumping buffer pool(s) to /data001/masvc01/ib_buffer_pool 2016-07-19 8:55:10 140556648810240 [Warning] InnoDB: Buffer pool(s) dump completed at 160719 8:55:10 2016-07-19 8:55:15 140556344748800 [Note] InnoDB: Shutdown completed; log sequence number 399145649446 2016-07-19 8:55:15 140556344748800 [Note] /engn001/masvc01/mysql/bin/mysqld: Shutdown complete 160719 08:55:16 mysqld_safe mysqld from pid file /engn001/masvc01/mysql/mysqld.pid ended 160719 08:57:05 mysqld_safe Starting mysqld daemon with databases from /data001/masvc01 2016-07-19 8:57:05 140304014350176 [Note] /engn001/masvc01/mysql/bin/mysqld (mysqld 10.1.14-MariaDB) starting as process 2608 ... 2016-07-19 8:57:06 140304014350176 [Note] InnoDB: Using mutexes to ref count buffer pool pages 2016-07-19 8:57:06 140304014350176 [Note] InnoDB: The InnoDB memory heap is disabled 2016-07-19 8:57:06 140304014350176 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2016-07-19 8:57:06 140304014350176 [Note] InnoDB: Memory barrier is not used 2016-07-19 8:57:06 140304014350176 [Note] InnoDB: Compressed tables use zlib 1.2.3 2016-07-19 8:57:06 140304014350176 [Note] InnoDB: Using Linux native AIO 2016-07-19 8:57:06 140304014350176 [Note] InnoDB: Using SSE crc32 instructions 2016-07-19 8:57:06 140304014350176 [Note] InnoDB: Initializing buffer pool, size = 24.0G 2016-07-19 8:57:08 140304014350176 [Note] InnoDB: Completed initialization of buffer pool 2016-07-19 8:57:10 140304014350176 [Note] InnoDB: Highest supported file format is Barracuda. 2016-07-19 8:57:13 140304014350176 [Note] InnoDB: 128 rollback segment(s) are active. 2016-07-19 8:57:13 140304014350176 [Note] InnoDB: Waiting for purge to start 2016-07-19 8:57:13 140304014350176 [Note] InnoDB: Percona XtraDB (http://www.percona.com <http://www.percona.com/> ) 5.6.29-76.2 started; log sequence number 399145649446 2016-07-19 8:57:13 140273638147840 [Note] InnoDB: Dumping buffer pool(s) not yet started 2016-07-19 08:57:13 7f94005f8700 InnoDB: Loading buffer pool(s) from /data001/masvc01/ib_buffer_pool 2016-07-19 8:57:13 140304014350176 [Note] Plugin 'FEEDBACK' is disabled. 160719 8:57:13 server_audit: MariaDB Audit Plugin version 1.4.0 STARTED. 2016-07-19 8:57:13 140304014350176 [Note] Server socket created on IP: '::'. 2016-07-19 8:57:13 140304014350176 [Warning] 'user' entry 'root@lgedglap02v' ignored in --skip-name-resolve mode. 2016-07-19 8:57:13 140304014350176 [Warning] 'user' entry '@lgedglap02v' ignored in --skip-name-resolve mode. 2016-07-19 8:57:13 140304014350176 [Warning] 'proxies_priv' entry '@% root@lgedglap02v' ignored in --skip-name-resolve mode. 2016-07-19 8:57:13 140273595853568 [Note] Event Scheduler: scheduler thread started with id 1 2016-07-19 8:57:13 140304014350176 [Note] Reading of all Master_info entries succeded 2016-07-19 8:57:13 140304014350176 [Note] Added new Master_info '' to hash table 2016-07-19 8:57:13 140304014350176 [Note] /engn001/masvc01/mysql/bin/mysqld: ready for connections. Version: '10.1.14-MariaDB' socket: '/engn001/masvc01/mysql/mysqld.sock' port: 3310 MariaDB Server 2016-07-19 8:57:47 140273638147840 [ERROR] InnoDB: Block in space_id 0 in file /data001/masvc01/ibdata1 encrypted. 2016-07-19 8:57:47 140273638147840 [ERROR] InnoDB: However key management plugin or used key_id 92 is not found or used encryption algorithm or method does not match. 2016-07-19 8:57:47 140273638147840 [ERROR] InnoDB: Marking tablespace as missing. You may drop this table or install correct key management plugin and key file. 2016-07-19 8:57:47 140273638147840 [ERROR] InnoDB: Block in space_id 0 in file /data001/masvc01/ibdata1 encrypted. 2016-07-19 8:57:47 140273638147840 [ERROR] InnoDB: However key management plugin or used key_id 92 is not found or used encryption algorithm or method does not match. 2016-07-19 8:57:47 140273638147840 [ERROR] InnoDB: Marking tablespace as missing. You may drop this table or install correct key management plugin and key file. 160719 8:57:47 [ERROR] mysqld got signal 11 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. To report this bug, see https://mariadb.com/kb/en/reporting-bugs We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Server version: 10.1.14-MariaDB key_buffer_size=67108864 read_buffer_size=50331648 max_used_connections=3 max_threads=102 thread_count=4 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 10094612 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x0x0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 0x0 thread_stack 0x48400 /engn001/masvc01/mysql/bin/mysqld(my_print_stacktrace+0x2e)[0xc028ae] /engn001/masvc01/mysql/bin/mysqld(handle_fatal_signal+0x464)[0x7651f4] /lib64/libpthread.so.0[0x3ffe40f7e0] /engn001/masvc01/mysql/bin/mysqld[0x8fa6d3] /engn001/masvc01/mysql/bin/mysqld[0xa3ba47] /engn001/masvc01/mysql/bin/mysqld[0xa51a2e] /engn001/masvc01/mysql/bin/mysqld[0xa52bc9] /engn001/masvc01/mysql/bin/mysqld[0xa433ff] /engn001/masvc01/mysql/bin/mysqld[0xa440b7] /lib64/libpthread.so.0[0x3ffe407aa1] /lib64/libc.so.6(clone+0x6d)[0x3ffe0e893d] The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. ..... ------------------------------------------------------------------------ -------------------------------------------------- I can't find solution for it. Is it a bug or hardware fault ? Regards. 김경남 차장 CTO 아키텍처담당 Post Architecture팀 02-2099-6177 / 010-2208-3006 _____ 상기 메시지와 첨부화일 내에는 비밀정보가 포함되어 있을 수 있으며, 지정된 수신자에 한하여 조회 및 사용될 수 있습니다. 만약 송신자의 실수로 인하여 상기 메시지를 수신하였다면, 송신자에게 메시지를 반송해 주시고, 원본 메시지와 모든 사본을 폐기해 주시기 바랍니다. 상기 메시지의 전체 또는 일부에 대해 무단 열람, 사용, 공개, 배포하는 것은 금지되어 있습니다.(주)LG CNS. This message and its attachments may contain confidential information, and they are intended to be viewed or used by only the individuals specified in the message. If you have received this message in an error from the sender, please contact the sender immediately to notify the error and delete all of the message and its copies. It is prohibited to view, use, make public and/or distribute part or whole of this message without written permission.

1 0

Re: [Maria-developers] [Commits] 3b6eff5: MDEV-8569 build_table_filename() doesn't support temporary tables.
by Sergei Golubchik 12 Jul '16

12 Jul '16

Hi, Alexey! On Jul 12, Alexey Botchkov wrote: > revision-id: 3b6eff5954c70a5795e1162b381a1b88af390764 (mariadb-10.0.26-8-g3b6eff5) > parent(s): 3e8ae6ef4f08937332e6ad317f37a9c0c9a717d3 > committer: Alexey Botchkov > timestamp: 2016-07-12 16:19:26 +0400 > message: > > MDEV-8569 build_table_filename() doesn't support temporary tables. > > Temporary tables support added for RENAME and ALTER TABLE. Ok, so instead of fixing build_table_filename, you've fixed RENAME and ALTER. Did you verify that all other build_table_filename() uses are safe? Ok to push, if yes. Still, it's rather fragile code, any idea how to improve it? Is it possible to add an assert to build_table_filename() to catch incorrect usage? Can build_table_filename() be changed somehow to make incorrect usage impossible (e.g. passing TABLE as an argument)? Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] Running tests in multi-node env
by Sandeep Jangra 07 Jul '16

07 Jul '16

Hey guys, I am curious to know how we run tests in a multi-node environment for Mariadb. I did build/install the server based on instructions given here: https://mariadb.com/kb/en/mariadb/generic-build-instructions/ But I am curious to know how does this test runs: https://github.com/MariaDB/server/blob/2783fc7d14bc8ad16acfeb509d3b19615023… Thanks, Sandeep

1 1

Re: [Maria-developers] [ME-585] Integrate existing Community Patch “Flashback” into MariaDB Server.
by Kristian Nielsen 07 Jul '16

07 Jul '16

Lixun Peng <lixun.peng(a)mariadb.com> writes: > *Currently, flashback will continue to run when faced a error, because the > usage is using flashback option to output the flashback events into a file, > and then import the file into mysql.* > > *Do you think we should stop to process when we found DDL?* > *Because flashback feature just output the events that can rollback the > database, so if we found the query event, we can stop to output the events > anymore.* > *But in my opinion, I think flashback it's a geek feature, the users know > what they are doing. We can provide a option that if continue to process > when faced a query event, what do you think?* Yes, I agree. It's definitely fine to have an option to continue to process even in case of query event. As you explained, the result of flashback is at first just an SQL script, only when run will this script modify anything. So it might be ok go give the output by default even in case when query events are encountered. But I think there should be at least a warning produced, and or an optional error. That should be easy to do (just set a flag if query event seen), and would help user to detect the problem. > *Yes, the test cases are not contain the --review option, because in > Alibaba the DBAs consider this option is useless in most cases, then I'm > not maintain this part of code so long, we should evaluate is this feature Right, I understand. My general opinion is that features pushed into the main tree should be reasonably complete, or not pushed at all. But I agree that some feature might not be priority at the moment, and so could be omitted rather than spend a lot of time polishing. >> Did you test this feature with binlog checksums enabled? >> > > *Now, that's a problem, if we support checksum, I should re-calculate the > checksum after modified the events, that maybe cost about half of month. I Well, at least it should be tested so that it doesn't crash etc. Binlog checksums is a constant source of pain, unfortunately. One thing that can go wrong is to compute the length of events incorrectly, that is easy to fix if the case (just subtract 4 in case of checksum). I don't think flashback needs to do extra work to eg. verify the checksums or any such. If there is a problem with the code rejecting the modified events, then I think it's not hard just to call the function that re-computes the checksum. But if it turns out to be harder, I'm fine with just putting in the documentation that flashback is not supported, and detecting checksums and giving an error. Just things should still behave somehow reasonable in case of checksums, not like crash or silently produce garbage results. And it should be tested. >> ^^^ "Check if the table was already created" ? >> > > *Yes, you're right. Sorry for my poor English.* Sure, actually your English is just fine, I have no problems understanding :-) I'm not a native English speaker myself either, so just when I notice something that seems off a bit, I may suggest better wording. >> > +SET binlog_format= ROW; >> >> This should not be needed, since you have the proper >> have_binlog_format_row.inc. Or is there some specific reason it is needed, >> and if so, what? >> > > *I don't know why, in my environment, event I sourced > have_binlog_format_row.inc, mtr will also run with statement format.* > *So I added the set command.* Ok, in any case, there is no harm in having SET binlog_format=ROW, I was just wondering... not important. >> Can you come up with a better name for "only_parse"? It is not clear to me >> what it means, I think it is an option if the function needs to instead >> generate the SQL for filling-in the flashback review table? >> >> > *Yes, you're right.* > *How do you think "no_fill_output"?* Yes, that's fine, thanks. > *In my understanding, `` can always be used.* Yes, I think you are right. Probably just my memory playing tricks on me. > These comments are very very useful for me, learned so many things! Ok, great, glad that you found them useful. Thanks, - Kristian.

1 0

Re: [Maria-developers] [Commits] 3bbf3d2: MDEV-10216: Assertion `strcmp(share->unique_file_name, filename) ||
by Sergey Vojtovich 05 Jul '16

05 Jul '16

Hi Nirbhay, Looks good, thanks! On Fri, Jul 01, 2016 at 02:47:29PM -0400, Nirbhay Choubey wrote: > revision-id: 3bbf3d2e246da1a2fd458b634f30cbe35afb5668 (mariadb-10.2.0-142-g3bbf3d2) > parent(s): 76f492e26d8d45c678c570be2cf4ca5d238edceb > author: Nirbhay Choubey > committer: Nirbhay Choubey > timestamp: 2016-07-01 14:47:27 -0400 > message: > > MDEV-10216: Assertion `strcmp(share->unique_file_name,filename) || > > .. share->last_version' failed in myisam/mi_open.c:67: test_if_reopen > > During the RENAME operation since the renamed temporary table is also > opened and added to myisam_open_list/maria_open_list, resetting the > last_version at the end of operation (HA_EXTRA_PREPARE_FOR_RENAME) > will cause an assertion failure when a subsequent query tries to open > an additional temporary table instance and thus attempts to reuse it > from the open table list. > > This commit fixes the issue by skipping flush/close operations executed > toward the end of ALTER for temporary tables. It also enables a shortcut > for simple ALTERs (like rename, disable/enable keys) on temporary > tables. > > As safety checks, added some assertions at code points that should not > be hit for temporary tables. ... Regards, Sergey

2 1

[Maria-developers] MariaDB 10.1.15
by Antonio Fernandes 04 Jul '16

04 Jul '16

Hi all, What happened to MariaDB 10.1.15? I've upgraded my slave server, waited a few days (like I always do before master) and now, no MariaDB 10.1.15 :) Ohh, btw, 10.1.14 changelog (https://mariadb.com/kb/en/mariadb/mariadb-10114-changelog/) still displays that 10.1.15 is the latest. Regards, António

2 1

[Maria-developers] MDEV-8919 Wrong result for CAST(9999999999999999999.0)
by Alexander Barkov 04 Jul '16

04 Jul '16

Hi Sergei, Please review a patch for mdev-8919. Thanks!

1 0

[Maria-developers] no slave left behind
by Colin Charles 02 Jul '16

02 Jul '16

Hi Kristian, I notice that MDEV-8112 [no slave left behind](https://jira.mariadb.org/browse/MDEV-8112) seems to be stalled. You posted a review over a year ago, so is all that is waiting for fixing the test failure AND documenting the feature? Thanks -- Colin Charles, http://bytebot.net/blog/ twitter: @bytebot | skype: colincharles "First they ignore you, then they laugh at you, then they fight you, then you win." -- Mohandas Gandhi

1 0

Re: [Maria-developers] InnoDB blob for primary key
by Sachin Setia 02 Jul '16

02 Jul '16

Hello Sir Weekly Report for second week of gsoc 1. Implemented unique in the case of null 2. Implemented unique(A,b,c....) Currently working on tests , field hiding Regards sachin On Tue, May 31, 2016 at 10:49 PM, Sergei Golubchik <serg(a)mariadb.org> wrote: > Hi, Sachin! > > On May 31, Sachin Setia wrote: > > Hello Sir > > Weekly Report For Gsoc > > 1. Implemented hash function in parser for one column > > 2. Implemenetd hash function for unlimited columns in parser > > 3. Implemented unique checking in case of hash collusion(it retrieves the > > row and compare data) but this only woks for unique blob column > > > > Currently Working on > > unique checking in case of unique(a,b,c) > > Thanks. This is ok. > > Next time, please, send it to maria-developers list, not to me only :) > > Do you have questions? Any code you want me to look at? > > Regards, > Sergei > Chief Architect MariaDB > and security(a)mariadb.org >

2 3

Re: [Maria-developers] [Commits] d1de640: MDEV-10216: Assertion `strcmp(share->unique_file_name, filename) ||
by Sergey Vojtovich 01 Jul '16

01 Jul '16

Hi Nirbhay, This solution looks better than MyISAM/Aria based one. Still a few doubts inline. I still suggest someone should do second review. On Tue, Jun 21, 2016 at 01:47:15PM -0400, Nirbhay Choubey wrote: > revision-id: d1de6402c8734e9a1929c2384c0cfc07c2ec48c0 (mariadb-10.2.0-83-gd1de640) > parent(s): b2ae32aafdd2787ad456f38833f630182ded25e8 > author: Nirbhay Choubey > committer: Nirbhay Choubey > timestamp: 2016-06-21 13:47:12 -0400 > message: > > MDEV-10216: Assertion `strcmp(share->unique_file_name,filename) || ...skip... > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index 0be329e..07d4b42 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -8151,12 +8151,15 @@ static bool fk_prepare_copy_alter_table(THD *thd, TABLE *table, Looks like git confused function names here and for most of other hunks of this patch. I vaguely remember this bug was affecting really old git versions like pre-2.x. > > if (keys_onoff != Alter_info::LEAVE_AS_IS) > { > - if (wait_while_table_is_used(thd, table, extra_func)) > - DBUG_RETURN(true); > + if (!table->s->tmp_table) > + { > + if (wait_while_table_is_used(thd, table, extra_func)) > + DBUG_RETURN(true); > > - // It's now safe to take the table level lock. > - if (lock_tables(thd, table_list, alter_ctx->tables_opened, 0)) > - DBUG_RETURN(true); > + // It's now safe to take the table level lock. > + if (lock_tables(thd, table_list, alter_ctx->tables_opened, 0)) > + DBUG_RETURN(true); > + } > > error= alter_table_manage_keys(table, > table->file->indexes_are_disabled(), Are you sure lock_tables() is not necessary for temporary tables here? It would be nice to make sure this code is covered by a test case. > @@ -8166,43 +8169,57 @@ static bool fk_prepare_copy_alter_table(THD *thd, TABLE *table, > if (!error && alter_ctx->is_table_renamed()) > { > THD_STAGE_INFO(thd, stage_rename); > - handlerton *old_db_type= table->s->db_type(); > - /* > - Then do a 'simple' rename of the table. First we need to close all > - instances of 'source' table. > - Note that if wait_while_table_is_used() returns error here (i.e. if > - this thread was killed) then it must be that previous step of > - simple rename did nothing and therefore we can safely return > - without additional clean-up. > - */ > - if (wait_while_table_is_used(thd, table, extra_func)) > - DBUG_RETURN(true); > - close_all_tables_for_name(thd, table->s, HA_EXTRA_PREPARE_FOR_RENAME, NULL); > - > LEX_STRING old_db_name= { alter_ctx->db, strlen(alter_ctx->db) }; > LEX_STRING old_table_name= > { alter_ctx->table_name, strlen(alter_ctx->table_name) }; > LEX_STRING new_db_name= { alter_ctx->new_db, strlen(alter_ctx->new_db) }; > LEX_STRING new_table_name= > { alter_ctx->new_alias, strlen(alter_ctx->new_alias) }; > - (void) rename_table_in_stat_tables(thd, &old_db_name, &old_table_name, > - &new_db_name, &new_table_name); > > - if (mysql_rename_table(old_db_type, alter_ctx->db, alter_ctx->table_name, > - alter_ctx->new_db, alter_ctx->new_alias, 0)) > - error= -1; > - else if (Table_triggers_list::change_table_name(thd, > - alter_ctx->db, > - alter_ctx->alias, > - alter_ctx->table_name, > - alter_ctx->new_db, > - alter_ctx->new_alias)) > - { > - (void) mysql_rename_table(old_db_type, > - alter_ctx->new_db, alter_ctx->new_alias, > - alter_ctx->db, alter_ctx->table_name, > - NO_FK_CHECKS); > - error= -1; > + if (!table->s->tmp_table) > + { > + handlerton *old_db_type= table->s->db_type(); > + /* > + Then do a 'simple' rename of the table. First we need to close all > + instances of 'source' table. > + Note that if wait_while_table_is_used() returns error here (i.e. if > + this thread was killed) then it must be that previous step of > + simple rename did nothing and therefore we can safely return > + without additional clean-up. > + */ > + if (wait_while_table_is_used(thd, table, extra_func)) > + DBUG_RETURN(true); > + close_all_tables_for_name(thd, table->s, HA_EXTRA_PREPARE_FOR_RENAME, > + NULL); > + > + (void) rename_table_in_stat_tables(thd, &old_db_name, &old_table_name, > + &new_db_name, &new_table_name); > + > + if (mysql_rename_table(old_db_type, alter_ctx->db, alter_ctx->table_name, > + alter_ctx->new_db, alter_ctx->new_alias, 0)) > + error= -1; > + else if (Table_triggers_list::change_table_name(thd, > + alter_ctx->db, > + alter_ctx->alias, > + alter_ctx->table_name, > + alter_ctx->new_db, > + alter_ctx->new_alias)) > + { > + (void) mysql_rename_table(old_db_type, > + alter_ctx->new_db, alter_ctx->new_alias, > + alter_ctx->db, alter_ctx->table_name, > + NO_FK_CHECKS); > + error= -1; > + } > + } > + else > + { > + if ((thd->rename_temporary_table(table, alter_ctx->new_db, > + alter_ctx->new_alias))) > + { > + /* Allocation error, no need to rename it back to the original name. */ > + error= -1; > + } > } > } > A few lines below we adjust MDL locks, is it safe for temporary tables? Previously we didn't binlog temporary table creation if binlog format was row. Now we binlog it independently of binlog format. Previously we did "new_table->s->table_creation_was_logged= table->s->table_creation_was_logged;". Now we don't. > @@ -9623,7 +9639,8 @@ bool mysql_trans_commit_alter_copy_data(THD *thd) > to->file->ha_release_auto_increment(); > if (to->file->ha_external_lock(thd,F_UNLCK)) > error=1; > - if (error < 0 && to->file->extra(HA_EXTRA_PREPARE_FOR_RENAME)) > + if (error < 0 && !to->s->tmp_table && > + to->file->extra(HA_EXTRA_PREPARE_FOR_RENAME)) > error= 1; > thd_progress_end(thd); > DBUG_RETURN(error > 0 ? -1 : 0); Isn't to->s->tmp_table always temporary? Shouldn't we check from->s->tmp_table instead? Regards, Sergey

2 1

Re: [Maria-developers] MDEV-9423: FTWRL and Binlog checkpoint
by Kristian Nielsen 29 Jun '16

29 Jun '16

Nirbhay Choubey <nirbhay(a)mariadb.com> writes: [Cc: maria-developers@, please always keep these discussions on the mailing list] > In Galera cluster, the state transfer scripts perform FTWRL and > copy data along with the last of all available binlog files to the > joiner node. > > After MDEV-181, I understand that the binlog checkpoint can be > in any of the binary log files (and not necessarily the last one). > > This seemingly has caused MDEV-9423, in which the joiner node > complains of the missing binlog file. > > Now the question is : Is FTWRL not sufficient to ensure that the > checkpoint is always the last binlog file? So if I understand correctly, the issue is related to having binlog files available during XA crash recovery. When the binlog file is rotated, there is a small window where both the latest and the previous binlog files are needed for crash recovery. The binlog checkpoint is the earliest binlog file that is needed for crash recovery, and it can be seen from the binlog checkpoint event. So the problem here is that a copy is made just after binlog rotation, and Galera only copies the most recent, mostly-empty binlog file, leaving insufficient information for XA recovery, right? One option to solve this is to always copy the last two binlog files. While it is theoretically possible to have the binlog checkpoint more than two files back, I think it will not occur in practice. Another option is to wait for the binlog checkpoint to reach the current binlog file. You can see this done in the test suite: mysql-test/include/wait_for_binlog_checkpoint.inc The binlog checkpointing happens asynchroneously, I *think* it can complete even while FTWRL is active, but I am not 100% sure though. The checkpoint happens after InnoDB has made its commits durable with fsync() or similar - only after that is it safe to discard the old binlog data and still have correct crash recovery. - Kristian.

3 11

Re: [Maria-developers] [MariaDB/server] MDEV-10298: Systemd hardening (#195)
by Kristian Nielsen 29 Jun '16

29 Jun '16

Sergey Vojtovich <notifications(a)github.com> writes: > Similar to other open source projects, the MariaDB Foundation needs to > have shared ownership of all code that is included in the MariaDB > distribution. The easiest way to achieve this is by submitting your > code under the BSD-new license. > > The other alternative is to sign the code contribution agreement which > can be found here: https://mariadb.com/kb/en/mariadb/mca/ As was concluded in this mailing thread, https://lists.launchpad.net/maria-developers/msg09606.html the third option is to contribute the code under GPLv2. Personally, I would strongly recommend to contribute under GPLv2 only. There are good and important reason why the GPL exists and why it is used by many of the most successful free software projects. Most of the code by core developers is contributed to the project under GPLv2 only, as is most of the major external contributions. Svoj, please update your "contributer request template" to mention prominently the GPLv2 option, and use it in all future cases. eg: ----------------------------------------------------------------------- Similar to other open source projects, the MariaDB Foundation needs to have the right to include and re-distribute under the GPL all code that is included in the MariaDB distribution. The easiest way to achieve this is by submitting your code under the same license as the MariaDB source code, GPLv2. The other alternatives are to submit under the BSD-new license, or to sign the code contribution agreement which can be found here: https://mariadb.com/kb/en/mariadb/mca/. These alternatives both provide further rights for the MariaDB Foundation and its members to use the code in your contribution, including the rights to submit it for inclusion in Oracle MySQL or to use it commercially in proprietary products. Please indicate in a comment below that you are contributing your new code of the whole pull request, including one or several files that are either new files or modified ones, under the GPLv2 license, under the BSD-new license, or that you have filled out the contribution agreement and sent it. ----------------------------------------------------------------------- Thanks, - Kristian.

2 1

Re: [Maria-developers] [MariaDB/server] MDEV-10303 - rsync_sst_rsync: ignore lost+found during database directory transfer (#196)
by Kristian Nielsen 29 Jun '16

29 Jun '16

Sergey Vojtovich <notifications(a)github.com> writes: > Hi Nicolas, > > Thanks for your contribution. JIRA task has been created to track this > pull request: https://jira.mariadb.org/browse/MDEV-10303 > > This task was added to 10.2.2 backlog, which hasn't been scheduled yet. > > Similar to other open source projects, the MariaDB Foundation needs to > have shared ownership of all code that is included in the MariaDB > distribution. The easiest way to achieve this is by submitting your > code under the BSD-new license. > > The other alternative is to sign the code contribution agreement which > can be found here: https://mariadb.com/kb/en/mariadb/mca/ As was concluded in this mailing thread, https://lists.launchpad.net/maria-developers/msg09606.html the third option is to contribute the code under GPLv2. Personally, I would strongly recommend to contribute under GPLv2 only. There are good and important reason why the GPL exists and why it is used by many of the most successful free software projects. Most of the code by core developers is contributed to the project under GPLv2 only, as is most of the major external contributions. - Kristian.

1 0

Re: [Maria-developers] eb9fab7: MDEV-9531: GROUP_CONCAT with ORDER BY inside takes a lot of memory while it's executed
by Sergei Golubchik 28 Jun '16

28 Jun '16

Hi, Sanja! On May 04, Olaksandr Byslkin wrote: still haven't fixed the typo? :) > revision-id: eb9fab76fb4322a7821e6502e6303ed381624429 (mariadb-10.1.13-18-geb9fab7) > parent(s): 732adec0a4c75d99389230feeb0deca0ad668de7 > committer: Oleksandr Byelkin > timestamp: 2016-05-04 19:15:31 +0200 > message: > > MDEV-9531: GROUP_CONCAT with ORDER BY inside takes a lot of memory while it's executed > > Limitation added to Red-Black tree. > > diff --git a/include/my_tree.h b/include/my_tree.h > index f8be55f..f1916b9 100644 > --- a/include/my_tree.h > +++ b/include/my_tree.h > @@ -57,11 +57,14 @@ typedef struct st_tree_element { > } TREE_ELEMENT; > > #define ELEMENT_CHILD(element, offs) (*(TREE_ELEMENT**)((char*)element + offs)) > +#define R_ELEMENT_CHILD(element, offs) ((TREE_ELEMENT**)((char*)element + offs)) what does "R_ELEMENT_CHILD" mean? What's "R"? > > typedef struct st_tree { > TREE_ELEMENT *root,null_element; > TREE_ELEMENT **parents[MAX_TREE_HEIGHT]; > + TREE_ELEMENT *free_element; why do you need a free_element pointer? > uint offset_to_key,elements_in_tree,size_of_element; > + uint elements_limit, del_direction; why do you need to support different directions here? > size_t memory_limit, allocated; > qsort_cmp2 compare; > void *custom_arg; > diff --git a/mysql-test/r/mysqld--help.result b/mysql-test/r/mysqld--help.result > index bef0390..3699b18 100644 > --- a/mysql-test/r/mysqld--help.result > +++ b/mysql-test/r/mysqld--help.result > @@ -240,6 +240,9 @@ The following options may be given as the first argument: > --group-concat-max-len=# > The maximum length of the result of function > GROUP_CONCAT() > + --group-concat-max-mem=# > + The maximum memory used to calculate result of function > + GROUP_CONCAT() No, please. There is group_concat_max_len already, it's all that user needs. There is no reason why one might need to configure a second limit on for the GROUP_CONCAT. > --gtid-domain-id=# Used with global transaction ID to identify logically > independent replication streams. When events can > propagate through multiple parallel paths (for example Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 3

Re: [Maria-developers] 0752333: MDEV-10161: wsrep_sync_wait not enabled when set to 1 in config file
by Sergei Golubchik 27 Jun '16

27 Jun '16

Hi, Nirbhay! On Jun 08, Nirbhay Choubey wrote: > revision-id: 075233312cc5bc6c242f4629f8293d4c67937f02 (mariadb-10.1.14-10-g0752333) > parent(s): c9f05974e618a869563f2360ef1ca910974a4f1c > author: Nirbhay Choubey > committer: Nirbhay Choubey > timestamp: 2016-06-08 16:03:01 -0400 > message: > > MDEV-10161: wsrep_sync_wait not enabled when set to 1 in config file > > Since wsrep_sync_wait & wsrep_causal_reads variables are related, > they are always kept in sync whenever one of them changes. > Same is tried on server start, where wsrep_sync_wait get updated > based on wsrep_causal_reads' value. But, since wsrep_causal_reads > is OFF by default, wsrep_sync_wait's value gets modified and loses > its WSREP_SYNC_WAIT_BEFORE_READ bit. > > Fixed by giving higher precedence to one that's explicitly set to > a non-default value and later to wsrep_sync_wait in case both are > set. > > diff --git a/sql/wsrep_mysqld.cc b/sql/wsrep_mysqld.cc > index d0798ef..fffb0cc 100644 > --- a/sql/wsrep_mysqld.cc > +++ b/sql/wsrep_mysqld.cc > @@ -567,7 +567,39 @@ int wsrep_init() > > wsrep_sst_auth_init(wsrep_sst_auth); > > - wsrep_causal_reads_update(&global_system_variables); > + /* > + On server start, there is no way to detect whether the default value > + for a system variable was set explicitly by the user. > + Now, since wsrep_causal_reads & wsrep_sync_wait are used for same > + purpose, a rule of precedence must be defined to keep their values > + in sync. It goes as following: This is wrong. First, in 10.1 there is a way to detect whether the value was set by a user. I_S.SYSTEM_VARIABLES can show that. But even if you'd detected that, it won't help you do distinguish between mysqld --wsrep-sync-wait=1 --wsrep-causal-reads=0 mysqld --wsrep-sync-wait=0 --wsrep-causal-reads=1 The only correct solution is to handle this in the mysqld_get_one_option switch, updating the dependent option when one of these two is set. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 1

[Maria-developers] Please review MDEV-10236 Where expression with NOT function gives incorrect result
by Alexander Barkov 27 Jun '16

27 Jun '16

Hi Serg. Please review a patch for MDEV-10236. Thanks.

2 3

Re: [Maria-developers] 3e61493: MDEV-10010 - Recursive call to mysql_rwlock_rdlock for LOCK_system_variables_hash
by Sergei Golubchik 27 Jun '16

27 Jun '16

Hi, Sergey! On Jun 27, Sergey Vojtovich wrote: > revision-id: 3e61493dff068c86b37f2e2ec0d0b6cbe9a255b6 (mariadb-10.1.14-28-g3e61493) > parent(s): 305bbbce6152b0909937659ecba4d51e5b21c094 > committer: Sergey Vojtovich > timestamp: 2016-06-27 15:01:22 +0400 > message: > > MDEV-10010 - Recursive call to mysql_rwlock_rdlock for LOCK_system_variables_hash > > Avoid recursive LOCK_system_variables_hash acquisition in > intern_sys_var_ptr() by pre-syncing dynamic session variables. Looks good, thanks! Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] 79716e0: [MDEV-9127] Crash reporter often fails to show the query that crashed
by Sergei Golubchik 27 Jun '16

27 Jun '16

Hi, Vicentiu! On May 30, Vicentiu Ciorbaru wrote: > revision-id: 79716e07f15e8dabca4d2647ff5dbf189935581a (mariadb-10.0.25-6-g79716e0) > parent(s): eaf5376f808b3d71c442fb6b95048c4dcd684b08 > author: Vicențiu Ciorbaru > committer: Vicențiu Ciorbaru > timestamp: 2016-05-30 23:21:22 +0300 > message: > > [MDEV-9127] Crash reporter often fails to show the query that crashed > > Addreses are not necessarily between heap_start && heap_end. Malloc > calls using mmap can place pointers outside these bounds. In this case, > we'll warn the user that the query pointer is potentially invalid. > However, we'll attempt to print the data anyway after we're done > printing everything else. ok to push Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] d1a7ae2: MDEV-9618 solaris sparc build fails on 10.1.
by Sergei Golubchik 26 Jun '16

26 Jun '16

Hi, Alexey! On Jun 26, Alexey Botchkov wrote: > revision-id: d1a7ae232ef6e35e4f81871e3c9d0d318a5dcef1 (mariadb-10.1.14-28-gd1a7ae2) > parent(s): 305bbbce6152b0909937659ecba4d51e5b21c094 > committer: Alexey Botchkov > timestamp: 2016-06-26 00:11:30 +0400 > message: > > MDEV-9618 solaris sparc build fails on 10.1. > > Compiler on Sparc is strict about the 'const' modifiers > in function declarations and definitions. Meaning > they should be exactly same. ok to push Interestingly enough, I've just read about it few days ago - that const modifiers do *not* have to be exactly the same, the compiler can be lax about it, because these const modifiers do not mean anything for the caller. So, this strictness could be considered a bug in that Solaris C++ compiler. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] When SP DEFINER is empty.
by Alexey Botchkov 25 Jun '16

25 Jun '16

Hi, Sergey, all. I'm in doubts about how this one should be fixed: https://jira.mariadb.org/browse/MDEV-10119 The story goes like this: - mysql_install_db script runs 'mysqld --skip-grant-tables' In this case the 'current_user()' seems to be empty. - so the CREATE PROCEDURE command in the bootstrap creates the procedure with the empty DEFINER. - as a result, the 'SHOW CREATE PROCEDURE' returns query started with 'CREATE DEFINER=`` PROCEDURE...', - that DEFINER=`` gives an error when feed to the server. That can be fixed on any stage. We can do any of these: - set some 'current_user()' to be not empty even with the --skip-grant-tables option - specify some non-empty DEFINER for the CREATE PROCEDURE statement (in both options it's not that clear what user could that be) - fix the SHOW CREATE PROCEDURE statement so it doesn't add the errorneous DEFINER=`` to the query. - make server handling the 'DEFINER=``' with no error. Maybe assigning the 'current_user()' in this case. So, what can You recommend as a fix in this case? Best regards. HF

2 1

Re: [Maria-developers] [Commits] 773ce40: MDEV-10045: Server crashes in Time_and_counter_tracker::incr_loops
by Sergey Petrunia 24 Jun '16

24 Jun '16

Hi Sanja, O the patch fixes the crash. However when I am debugging it, I see the execution to go like this: (gdb) wher #0 JOIN::optimize_inner (this=0x7fff900087d0) at /home/psergey/dev-git/10.1-dbg6/sql/sql_select.cc:1095 #1 0x0000555555a8e5f8 in JOIN::optimize (this=0x7fff900087d0) at /home/psergey/dev-git/10.1-dbg6/sql/sql_select.cc:1036 #2 0x0000555555b277ce in st_select_lex_unit::optimize (this=0x7fff900068a8) at /home/psergey/dev-git/10.1-dbg6/sql/sql_union.cc:747 #3 0x0000555555b279cb in st_select_lex_unit::exec (this=0x7fff900068a8) at /home/psergey/dev-git/10.1-dbg6/sql/sql_union.cc:780 #4 0x0000555555d3855f in subselect_union_engine::exec (this=0x7fff90007970) at /home/psergey/dev-git/10.1-dbg6/sql/item_subselect.cc:3785 #5 0x0000555555d2eb12 in Item_subselect::exec (this=0x7fff9000b028) at /home/psergey/dev-git/10.1-dbg6/sql/item_subselect.cc:684 #6 0x0000555555d3057e in Item_singlerow_subselect::val_str (this=0x7fff9000b028, str=0x7ffff43b3a40) at /home/psergey/dev-git/10.1-dbg6/sql/item_subselect.cc:1295 #7 0x0000555555cbca5e in Item_func_conv_charset::Item_func_conv_charset (this=0x7fff90021aa0, thd=0x555557ffda80, a=0x7fff9000b028, cs=0x555556c18b80, cache_if_const=true) at /home/psergey/dev-git/10.1-dbg6/sql/item_strfunc.h:975 #8 0x0000555555ca1a7b in Item::safe_charset_converter (this=0x7fff9000b028, thd=0x555557ffda80, tocs=0x555556c18b80) at /home/psergey/dev-git/10.1-dbg6/sql/item.cc:1084 #9 0x0000555555ca4452 in Item_func_or_sum::agg_item_set_converter (this=0x7fff90021828, coll=..., fname=0x555556460377 "<", args=0x7fff900218b0, nargs=2, flags=7, item_sep=1) at /home/psergey/dev-git/10.1-dbg6/sql/item.cc:2141 #10 0x0000555555cd4106 in Item_func_or_sum::agg_arg_charsets (this=0x7fff90021828, c=..., items=0x7fff900218b0, nitems=2, flags=7, item_sep=1) at /home/psergey/dev-git/10.1-dbg6/sql/item.h:3714 #11 0x0000555555cd414e in Item_func_or_sum::agg_arg_charsets_for_comparison (this=0x7fff90021828, c=..., items=0x7fff900218b0, nitems=2, item_sep=1) at /home/psergey/dev-git/10.1-dbg6/sql/item.h:3760 #12 0x0000555555cc10ac in Item_func::setup_args_and_comparator (this=0x7fff90021828, thd=0x555557ffda80, cmp=0x7fff900218e0) at /home/psergey/dev-git/10.1-dbg6/sql/item_cmpfunc.cc:509 #13 0x0000555555cc11b5 in Item_bool_rowready_func2::fix_length_and_dec (this=0x7fff90021828) at /home/psergey/dev-git/10.1-dbg6/sql/item_cmpfunc.cc:531 #14 0x0000555555cf022b in Item_func::fix_fields (this=0x7fff90021828, thd=0x555557ffda80, ref=0x7ffff43b3eb0) at /home/psergey/dev-git/10.1-dbg6/sql/item_func.cc:234 #15 0x0000555555d322fb in Item_allany_subselect::transform_into_max_min (this=0x7fff900077a8, join=0x7fff900087d0) at /home/psergey/dev-git/10.1-dbg6/sql/item_subselect.cc:1969 #16 0x0000555555bd557d in JOIN::transform_max_min_subquery (this=0x7fff900087d0) at /home/psergey/dev-git/10.1-dbg6/sql/opt_subselect.cc:901 #17 0x0000555555a8e9c6 in JOIN::optimize_inner (this=0x7fff900087d0) at /home/psergey/dev-git/10.1-dbg6/sql/sql_select.cc:1131 #18 0x0000555555a8e5f8 in JOIN::optimize (this=0x7fff900087d0) at /home/psergey/dev-git/10.1-dbg6/sql/sql_select.cc:1036 #19 0x0000555555a44f98 in st_select_lex::optimize_unflattened_subqueries (this=0x555558001b98, const_only=false) at /home/psergey/dev-git/10.1-dbg6/sql/sql_lex.cc:3760 #20 0x0000555555bdeaee in JOIN::optimize_unflattened_subqueries (this=0x7fff90007ca8) at /home/psergey/dev-git/10.1-dbg6/sql/opt_subselect.cc:5051 #21 0x0000555555a91dd7 in JOIN::optimize_inner (this=0x7fff90007ca8) at /home/psergey/dev-git/10.1-dbg6/sql/sql_select.cc:2043 #22 0x0000555555a8e5f8 in JOIN::optimize (this=0x7fff90007ca8) at /home/psergey/dev-git/10.1-dbg6/sql/sql_select.cc:1036 #23 0x0000555555a96bf2 in mysql_select (thd=0x555557ffda80, rref_pointer_array=0x555558001e10, tables=0x7fff900054b0, wild_num=1, fields=..., conds=0x7fff90007a80, og_num=0, order=0x0, group=0x0, having=0x0, proc_param=0x0, select_options=2147748608, result=0x7fff90007c88, unit=0x555558001498, select_lex=0x555558001b98) at /home/psergey/dev-git/10.1-dbg6/sql/sql_select.cc:3437 #24 0x0000555555a8c7fd in handle_select (thd=0x555557ffda80, lex=0x5555580013d0, result=0x7fff90007c88, setup_tables_done_option=0) at /home/psergey/dev-git/10.1-dbg6/sql/sql_select.cc:384 #25 0x0000555555a5cd61 in execute_sqlcom_select (thd=0x555557ffda80, all_tables=0x7fff900054b0) at /home/psergey/dev-git/10.1-dbg6/sql/sql_parse.cc:5894 #26 0x0000555555a52a21 in mysql_execute_command (thd=0x555557ffda80) at /home/psergey/dev-git/10.1-dbg6/sql/sql_parse.cc:2960 #27 0x0000555555a604b1 in mysql_parse (thd=0x555557ffda80, rawbuf=0x7fff90005258 "SELECT * FROM t1, t2 WHERE f3 = f2 AND f1 > ANY ( SELECT 'foo' UNION SELECT 'bar' )", length=83, parser_state=0x7ffff43b54f0) at /home/psergey/dev-git/10.1-dbg6/sql/sql_parse.cc:7314 #28 0x0000555555a4ec23 in dispatch_command (command=COM_QUERY, thd=0x555557ffda80, packet=0x555558004031 "SELECT * FROM t1, t2 WHERE f3 = f2 AND f1 > ANY ( SELECT 'foo' UNION SELECT 'bar' )", packet_length=83) at /home/psergey/dev-git/10.1-dbg6/sql/sql_parse.cc:1486 #29 0x0000555555a4d944 in do_command (thd=0x555557ffda80) at /home/psergey/dev-git/10.1-dbg6/sql/sql_parse.cc:1107 Note frame #0 and frame #17: JOIN::optimize_inner() has invoked a few functions which in turn invoked JOIN::optimize (and JOIN::optimize_inner()) for the same join. Before your patch this didn't happen, as the inner call to JOIN::optimize would find join->optimized= true and return immediately. In this particular example, the double-optimized join a degenerate "SELECT 'foo'", double-optimization has no effect. I tried to create an example with a non-degenerate join but didn't succeed. Looking at the code, I have no certainity that it's impossible to have two JOIN::optimize calls nest for a non-degenerate join. If you can explain why this is not possible, please do. If you can't prove this, I would suggest adding JOIN::inside_optimize_call and have "if (inside_optimize_call) return 0;" as the first line of JOIN::optimize. On Wed, Jun 22, 2016 at 11:17:44AM +0200, Oleksandr Byelkin wrote: > revision-id: 773ce408762b5f8256d4053b6d0d418d15657b92 (mariadb-10.1.14-24-g773ce40) > parent(s): 63120090f994cc78876944e9f7a76f53337fa46e > committer: Oleksandr Byelkin > timestamp: 2016-06-22 11:17:44 +0200 > message: > > MDEV-10045: Server crashes in Time_and_counter_tracker::incr_loops > > Do not set 'optimized' flag until whole optimization procedure is finished. > > --- > mysql-test/r/subselect.result | 11 +++++++++++ > mysql-test/r/subselect_no_exists_to_in.result | 11 +++++++++++ > mysql-test/r/subselect_no_mat.result | 11 +++++++++++ > mysql-test/r/subselect_no_opts.result | 11 +++++++++++ > mysql-test/r/subselect_no_scache.result | 11 +++++++++++ > mysql-test/r/subselect_no_semijoin.result | 11 +++++++++++ > mysql-test/t/subselect.test | 15 +++++++++++++++ > sql/sql_select.cc | 6 +++--- > 8 files changed, 84 insertions(+), 3 deletions(-) > > diff --git a/mysql-test/r/subselect.result b/mysql-test/r/subselect.result > index 6a53199..1ea7000 100644 > --- a/mysql-test/r/subselect.result > +++ b/mysql-test/r/subselect.result > @@ -7159,3 +7159,14 @@ f > foo > drop table t1; > SET NAMES default; > +# > +# MDEV-10045: Server crashes in Time_and_counter_tracker::incr_loops > +# > +SET NAMES utf8; > +CREATE TABLE t1 (f1 VARCHAR(3), f2 INT UNSIGNED) ENGINE=MyISAM; > +CREATE TABLE t2 (f3 INT) ENGINE=MyISAM; > +SELECT * FROM t1, t2 WHERE f3 = f2 AND f1 > ANY ( SELECT 'foo' UNION SELECT 'bar' ); > +f1 f2 f3 > +DROP TABLE t1, t2; > +SET NAMES default; > +End of 10.1 tests > diff --git a/mysql-test/r/subselect_no_exists_to_in.result b/mysql-test/r/subselect_no_exists_to_in.result > index aa68434..be5e5b6 100644 > --- a/mysql-test/r/subselect_no_exists_to_in.result > +++ b/mysql-test/r/subselect_no_exists_to_in.result > @@ -7159,6 +7159,17 @@ f > foo > drop table t1; > SET NAMES default; > +# > +# MDEV-10045: Server crashes in Time_and_counter_tracker::incr_loops > +# > +SET NAMES utf8; > +CREATE TABLE t1 (f1 VARCHAR(3), f2 INT UNSIGNED) ENGINE=MyISAM; > +CREATE TABLE t2 (f3 INT) ENGINE=MyISAM; > +SELECT * FROM t1, t2 WHERE f3 = f2 AND f1 > ANY ( SELECT 'foo' UNION SELECT 'bar' ); > +f1 f2 f3 > +DROP TABLE t1, t2; > +SET NAMES default; > +End of 10.1 tests > set optimizer_switch=default; > select @@optimizer_switch like '%exists_to_in=off%'; > @@optimizer_switch like '%exists_to_in=off%' > diff --git a/mysql-test/r/subselect_no_mat.result b/mysql-test/r/subselect_no_mat.result > index 754aec1..6dd227c 100644 > --- a/mysql-test/r/subselect_no_mat.result > +++ b/mysql-test/r/subselect_no_mat.result > @@ -7152,6 +7152,17 @@ f > foo > drop table t1; > SET NAMES default; > +# > +# MDEV-10045: Server crashes in Time_and_counter_tracker::incr_loops > +# > +SET NAMES utf8; > +CREATE TABLE t1 (f1 VARCHAR(3), f2 INT UNSIGNED) ENGINE=MyISAM; > +CREATE TABLE t2 (f3 INT) ENGINE=MyISAM; > +SELECT * FROM t1, t2 WHERE f3 = f2 AND f1 > ANY ( SELECT 'foo' UNION SELECT 'bar' ); > +f1 f2 f3 > +DROP TABLE t1, t2; > +SET NAMES default; > +End of 10.1 tests > set optimizer_switch=default; > select @@optimizer_switch like '%materialization=on%'; > @@optimizer_switch like '%materialization=on%' > diff --git a/mysql-test/r/subselect_no_opts.result b/mysql-test/r/subselect_no_opts.result > index e05dd4d..389c0f5 100644 > --- a/mysql-test/r/subselect_no_opts.result > +++ b/mysql-test/r/subselect_no_opts.result > @@ -7150,4 +7150,15 @@ f > foo > drop table t1; > SET NAMES default; > +# > +# MDEV-10045: Server crashes in Time_and_counter_tracker::incr_loops > +# > +SET NAMES utf8; > +CREATE TABLE t1 (f1 VARCHAR(3), f2 INT UNSIGNED) ENGINE=MyISAM; > +CREATE TABLE t2 (f3 INT) ENGINE=MyISAM; > +SELECT * FROM t1, t2 WHERE f3 = f2 AND f1 > ANY ( SELECT 'foo' UNION SELECT 'bar' ); > +f1 f2 f3 > +DROP TABLE t1, t2; > +SET NAMES default; > +End of 10.1 tests > set @optimizer_switch_for_subselect_test=null; > diff --git a/mysql-test/r/subselect_no_scache.result b/mysql-test/r/subselect_no_scache.result > index 71ade62..a90278c 100644 > --- a/mysql-test/r/subselect_no_scache.result > +++ b/mysql-test/r/subselect_no_scache.result > @@ -7165,6 +7165,17 @@ f > foo > drop table t1; > SET NAMES default; > +# > +# MDEV-10045: Server crashes in Time_and_counter_tracker::incr_loops > +# > +SET NAMES utf8; > +CREATE TABLE t1 (f1 VARCHAR(3), f2 INT UNSIGNED) ENGINE=MyISAM; > +CREATE TABLE t2 (f3 INT) ENGINE=MyISAM; > +SELECT * FROM t1, t2 WHERE f3 = f2 AND f1 > ANY ( SELECT 'foo' UNION SELECT 'bar' ); > +f1 f2 f3 > +DROP TABLE t1, t2; > +SET NAMES default; > +End of 10.1 tests > set optimizer_switch=default; > select @@optimizer_switch like '%subquery_cache=on%'; > @@optimizer_switch like '%subquery_cache=on%' > diff --git a/mysql-test/r/subselect_no_semijoin.result b/mysql-test/r/subselect_no_semijoin.result > index 43d191b..16aa7b8 100644 > --- a/mysql-test/r/subselect_no_semijoin.result > +++ b/mysql-test/r/subselect_no_semijoin.result > @@ -7150,5 +7150,16 @@ f > foo > drop table t1; > SET NAMES default; > +# > +# MDEV-10045: Server crashes in Time_and_counter_tracker::incr_loops > +# > +SET NAMES utf8; > +CREATE TABLE t1 (f1 VARCHAR(3), f2 INT UNSIGNED) ENGINE=MyISAM; > +CREATE TABLE t2 (f3 INT) ENGINE=MyISAM; > +SELECT * FROM t1, t2 WHERE f3 = f2 AND f1 > ANY ( SELECT 'foo' UNION SELECT 'bar' ); > +f1 f2 f3 > +DROP TABLE t1, t2; > +SET NAMES default; > +End of 10.1 tests > set @optimizer_switch_for_subselect_test=null; > set @join_cache_level_for_subselect_test=NULL; > diff --git a/mysql-test/t/subselect.test b/mysql-test/t/subselect.test > index 3599b52..c9c89a2 100644 > --- a/mysql-test/t/subselect.test > +++ b/mysql-test/t/subselect.test > @@ -6011,3 +6011,18 @@ INSERT INTO t1 VALUES ('foo'); > SELECT f FROM t1 WHERE f > ALL ( SELECT 'bar' UNION SELECT 'baz' ); > drop table t1; > SET NAMES default; > + > +--echo # > +--echo # MDEV-10045: Server crashes in Time_and_counter_tracker::incr_loops > +--echo # > +SET NAMES utf8; > + > +CREATE TABLE t1 (f1 VARCHAR(3), f2 INT UNSIGNED) ENGINE=MyISAM; > +CREATE TABLE t2 (f3 INT) ENGINE=MyISAM; > + > +SELECT * FROM t1, t2 WHERE f3 = f2 AND f1 > ANY ( SELECT 'foo' UNION SELECT 'bar' ); > + > +DROP TABLE t1, t2; > +SET NAMES default; > + > +--echo End of 10.1 tests > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index 4825726..ded59eb 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -1040,7 +1040,7 @@ int JOIN::optimize() > and deleted. The second call will not produce a valid query plan, it will > short-circuit because optimized==TRUE. > > - "was_optimized != optimized" is here to handle this case: > + "!was_optimized" is here to handle this case: > - first optimization starts, gets an error (from a const. cheap > subquery), returns 1 > - another JOIN::optimize() call made, and now join->optimize() will > @@ -1049,7 +1049,7 @@ int JOIN::optimize() > Can have QEP_NOT_PRESENT_YET for degenerate queries (for example, > SELECT * FROM tbl LIMIT 0) > */ > - if (was_optimized != optimized && !res && have_query_plan != QEP_DELETED) > + if (!was_optimized && !res && have_query_plan != QEP_DELETED) > { > create_explain_query_if_not_exists(thd->lex, thd->mem_root); > have_query_plan= QEP_AVAILABLE; > @@ -1058,6 +1058,7 @@ int JOIN::optimize() > !skip_sort_order && !no_order && (order || group_list), > select_distinct); > } > + optimized= 1; > return res; > } > > @@ -1086,7 +1087,6 @@ JOIN::optimize_inner() > // to prevent double initialization on EXPLAIN > if (optimized) > DBUG_RETURN(0); > - optimized= 1; > DEBUG_SYNC(thd, "before_join_optimize"); > > THD_STAGE_INFO(thd, stage_optimizing); > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] [Commits] 475cab8: MDEV-10050: Crash in subselect
by Sergey Petrunia 24 Jun '16

24 Jun '16

Hi Sanja, On Wed, Jun 22, 2016 at 02:17:06PM +0200, Oleksandr Byelkin wrote: > revision-id: 475cab835fb48c91d5cca649ab93917ec1718d75 (mariadb-5.5.50-6-g475cab8) > parent(s): a482e76e65a4fee70479e877929381c86b1ec62f > committer: Oleksandr Byelkin > timestamp: 2016-06-22 14:17:06 +0200 > message: > > MDEV-10050: Crash in subselect > > thd should not be taken earlier then fix_field and reset on fix_fields if it is needed. > > --- > sql/item_subselect.cc | 15 ++++++++++++--- > 1 file changed, 12 insertions(+), 3 deletions(-) > > diff --git a/sql/item_subselect.cc b/sql/item_subselect.cc > index ba67474..60cdd3f 100644 > --- a/sql/item_subselect.cc > +++ b/sql/item_subselect.cc > @@ -79,7 +79,9 @@ void Item_subselect::init(st_select_lex *select_lex, > DBUG_PRINT("enter", ("select_lex: 0x%lx this: 0x%lx", > (ulong) select_lex, (ulong) this)); > unit= select_lex->master_unit(); > - thd= unit->thd; > +#ifndef DBUG_OFF > + thd= 0; > +#endif So I've applied the patch, and I'm debugging this statement: prepare s from 'select a, (select max(one_k.a) from one_k where one_k.a <ten.a) from ten'; (this is just the first subquery I tried, nothing special about it). We arrive at the above #ifndef, and I have thd=0xa5a5a5a5a5, that is, it's uninitialized data. I let it execute further... > if (unit->item) > { > /* > Item can be changed in JOIN::prepare while engine in JOIN::optimize > => we do not copy old_engine here > */ > engine= unit->item->engine; > own_engine= FALSE; > parsing_place= unit->item->parsing_place; > unit->thd->change_item_tree((Item**)&unit->item, this); > engine->change_result(this, result, TRUE); > } > else > { > SELECT_LEX *outer_select= unit->outer_select(); > /* > do not take into account expression inside aggregate functions because > they can access original table fields > */ > parsing_place= (outer_select->in_sum_expr ? > NO_MATTER : > outer_select->parsing_place); > if (unit->is_union()) > engine= new subselect_union_engine(thd, unit, result, this); > else > engine= new subselect_single_select_engine(thd, select_lex, result, this); and it arrives here. Stepping into this call, I eventually reach where this stack trace shows: (gdb) wher #0 select_result::set_thd (this=0x7fffb002c130, thd_arg=0x0) at /home/psergey/dev-git/5.5/sql/sql_class.h:3316 #1 0x000000000086fd30 in subselect_engine::set_thd (this=0x7fffb002c150, thd_arg=0x0) at /home/psergey/dev-git/5.5/sql/item_subselect.cc:2892 #2 0x00000000007648e9 in subselect_engine::subselect_engine (this=0x7fffb002c150, thd_arg=0x0, si=0x7fffb002c010, res=0x7fffb002c130) at /home/psergey/dev-git/5.5/sql/item_subselect.h:726 #3 0x000000000086fd66 in subselect_single_select_engine::subselect_single_select_engine (this=0x7fffb002c150, thd_arg=0x0, select=0x7fffb002a8e0, result_arg=0x7fffb002c130, item_arg=0x7fffb002c010) at /home/psergey/dev-git/5.5/sql/item_subselect.cc:2902 #4 0x0000000000868232 in Item_subselect::init (this=0x7fffb002c010, select_lex=0x7fffb002a8e0, result=0x7fffb002c130) at /home/psergey/dev-git/5.5/sql/item_subselect.cc:111 #5 0x000000000086a0d1 in Item_singlerow_subselect::Item_singlerow_subselect (this=0x7fffb002c010, select_lex=0x7fffb002a8e0) at /home/psergey/dev-git/5.5/sql/item_subselect.cc:906 #6 0x000000000077d36f in MYSQLparse (thd=0x2fd99e0) at /home/psergey/dev-git/5.5/sql/sql_yacc.yy:8366 #7 0x00000000006214f2 in parse_sql (thd=0x2fd99e0, parser_state=0x7ffff7f81800, creation_ctx=0x0) at /home/psergey/dev-git/5.5/sql/sql_parse.cc:7818 #8 0x0000000000633dda in Prepared_statement::prepare (this=0x7fffb00167f0, packet=0x7fffb00063f8 "select a, (select max(one_k.a) from one_k where one_k.a <ten.a) from ten", packet_len=72) at /home/psergey/dev-git/5.5/sql/sql_prepare.cc:3353 #9 0x0000000000631a89 in mysql_sql_stmt_prepare (thd=0x2fd99e0) at /home/psergey/dev-git/5.5/sql/sql_prepare.cc:2458 #10 0x000000000061324d in mysql_execute_command (thd=0x2fd99e0) at /home/psergey/dev-git/5.5/sql/sql_parse.cc:2239 #11 0x000000000061d58d in mysql_parse (thd=0x2fd99e0, rawbuf=0x7fffb00062c8 "prepare s from 'select a, (select max(one_k.a) from one_k where one_k.a <ten.a) from ten'", length=89, parser_state=0x7ffff7f82540) at /home/psergey/dev-git/5.5/sql/sql_parse.cc:5934 Note that thd_arg is passing around un-initialized data all of this time. Is this how it is intended to work? This looks scary, a lot of functions are passed garbage as parameter. I assume the test suite passed with this patch, so these function don't do anything with the parameter (they probably save away the THD pointer, and then somebody assigns the pointer a valid value before it is used). Still, this looks really scary, we need to either fix this (strongly preferable), or add a lot of comments about it (acceptable as a last resort measure). > if (unit->item) > { > @@ -90,7 +92,7 @@ void Item_subselect::init(st_select_lex *select_lex, > engine= unit->item->engine; > own_engine= FALSE; > parsing_place= unit->item->parsing_place; > - thd->change_item_tree((Item**)&unit->item, this); > + unit->thd->change_item_tree((Item**)&unit->item, this); > engine->change_result(this, result, TRUE); > } > else > @@ -220,6 +222,10 @@ bool Item_subselect::fix_fields(THD *thd_param, Item **ref) > uint8 uncacheable; > bool res; > > + thd= thd_param; > + > + DBUG_ASSERT(unit->thd == thd); > + > status_var_increment(thd_param->status_var.feature_subquery); > > DBUG_ASSERT(fixed == 0); > @@ -2651,7 +2657,10 @@ bool Item_in_subselect::fix_fields(THD *thd_arg, Item **ref) > { > uint outer_cols_num; > List<Item> *inner_cols; > - char const *save_where= thd->where; > + char const *save_where= thd_arg->where; > + > + thd= thd_arg; > + DBUG_ASSERT(unit->thd == thd); > > if (test_strategy(SUBS_SEMI_JOIN)) > return !( (*ref)= new Item_int(1)); > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

2 1

[Maria-developers] Executing query inside mysql server
by knizhnik 24 Jun '16

24 Jun '16

Hi, I wonder what is the simplest way to execute SQL query inside server? In PostgreSQL there is SPI API, SQLite allows to use client API,... but I failed to find right way of running queries inside mySQL server. My task is the following: I want to patch mySQL server by redirecting execution of some queries to some external executor. The text of this queries will be stored inside database in some special table. So when some query is executed by mySQL server, I want to perform lookup in this table. What is the simplest way to do it in mySQL?

2 1

[Maria-developers] Why/how does the MariaDB client depend on libdbd-mysql-perl?
by Otto Kekäläinen 24 Jun '16

24 Jun '16

I am returning to this age-old discussion about libdbd-mysql-perl as dependency in Debian: 2015-07-21 19:50 GMT+03:00 Sergei Golubchik <vuvova(a)gmail.com>: >> + libdbi-perl, >> + libterm-readkey-perl, >> + mariadb-client-core-10.0 (>= ${source:Version}), >> + ${misc:Depends}, >> + ${perl:Depends}, >> + ${shlibs:Depends} >> +Recommends: libdbd-mysql-perl (>= 1.2202) > > really? why libdbi-perl is required but libdbd-perl is optional? > the one makes no sense without the other. and libterm-readkey-perl > is required, why it's really optional, everything works without it. In MySQL 5.6 this dependency also still exists: http://anonscm.debian.org/cgit/pkg-mysql/mysql-5.6.git/tree/debian/control Package: mysql-client-5.6 Architecture: any Depends: debianutils (>=1.6), libdbd-mysql-perl (>= 1.2202), libdbi-perl, libterm-readkey-perl, But it has been dropped in MySQL 5.7 packaging. In 5.7 it is not even a Recommends or Suggests, but completely removed. http://anonscm.debian.org/cgit/pkg-mysql/mysql-5.7.git/tree/debian/control I don't know what to do about this because I don't really understand what or how the mysql client does with perl library dependencies. It is a C program, does it really use the perl library in any way? So auxilary script does something? Can we drop the dependency on libdbd-mysql-perl, and starting from what version? - Otto

2 1

[Maria-developers] MDEV-8502 DECIMAL accepts out of range DEFAULT values
by Alexander Barkov 24 Jun '16

24 Jun '16

Hello Sergei, Please review a patch fixing: MDEV-8502 DECIMAL accepts out of range DEFAULT values MDEV-10277 Redundant NOTE when inserting '0.00001 ' into a DECIMAL(2,1) column Thanks.

1 0

Re: [Maria-developers] [MariaDB/server] MDEV-10271: Log stop of slave SQL thread on errors, too (#188)
by Kristian Nielsen 23 Jun '16

23 Jun '16

Hartmut Holzgraefe <notifications(a)github.com> writes: > -- Commit Summary -- > > * MDEV-10271: Log stop of slave SQL thread on errors, too > * MDEV-10272: add master host/port info to slave thread exit messages So one comment. With the first patch, there are two cases where the error message "Slave SQL thread exiting, replication stopped ..." will be logged even though the thread fails before it gets to log the corresponding start message "Slave SQL thread initialized, starting replication ..." Was this intended, it seems perhaps a bit confusing if the log has spurious messages about threads stopping, when there are no corresponding start messages? If not intended, it is simple to change by adding an extra label err_after_start: before the message, and using that or err: as appropriate in the gotos. What do you think? But I see you filed pull requests against 10.0 and 10.1, so it seems you were intending these for the stable releases? I do not have any strong opinion on this myself, but for stable releases, you should write an email to maria-developers(a)lists.launchpad.net to solicit opinions about making an exception for this case. Include justification of why it should go in stable releases (and not just in 10.2), and discussion of possible impact on existing applications/setups. If there are no objections, I'll push them. - Kristian.

1 0

[Maria-developers] Formatting rules for double => varchar assignment
by Jesse Schalken 23 Jun '16

23 Jun '16

Hi MariaDB Developers, I have an application which depends on the formatting MariaDB applies to a double when assigned to a char/varchar column of different lengths, and am trying to reproduce the behaviour in PHP. I have gotten some way but just can't seem to get the rules right. I'm having trouble finding a specification for this in the docs or the code that is responsible for this in the MariaDB source. Can someone point me in the right direction? Thanks

2 4

Re: [Maria-developers] GSoC weekly reports (Unique indexes for blobs)
by Shubham Barai 22 Jun '16

22 Jun '16

GSoC (week 4) Hello everyone, After working on create table operation,next thing I had to work on was insert operations.So I explored some of the functions like row_ins_scan_index_for_duplicates, btr_pcur_get_rec to get clear understanding about how to implement duplicate search on hash index. There was a problem in hash function that I wrote .It would calculate same hash value for two different keys if the prefix length of blob key part was zero. Now it seems to be working after I checked it in debugger.I still have to modify it for data types like varchar etc. I have added test cases for insert operations in myisam. In MyIsam, I found one problem in update operation. When updating a row,if the key is conflicting then server crashes because some pointer goes invalid in compare_record. I haven't fixed this issue yet. I also modified some functions in dict0load.cc to adjust some members of dict_index_t for a new index type.The main problem is that index entry for hash based index cointains only two fields(hash value and row id) while dict_index_t contains hash field and other user defined fields which are used to calculate hash value.Some of the operations like alter table( e.g. rename column) needs to get access to all fields while other functions like rec_get_offsets and row_build_index_entry_low needs to get access to only hash field and row id. I am still working on this to find efficient solution to this problem. On 16 June 2016 at 23:29, Sergei Golubchik <vuvova(a)gmail.com> wrote: > Hi, Shubham! > > What I wanted to say on IRC was: > > here's what the comment of cmp_dtuple_rec_with_match_low() says: > > ............... If rec has an externally stored field we do not > compare it but return with value 0 if such a comparison should be > made. > > Note that blobs are externally stored fields in InnoDB, so, I think, > this means that you cannot use cmp_dtuple_rec() to compare blobs. > > Regards, > Sergei > Chief Architect MariaDB > and security(a)mariadb.org >

2 1

Re: [Maria-developers] [Commits] 511bd2c: MDEV-10216: Assertion `strcmp(share->unique_file_name, filename) ||
by Nirbhay Choubey 21 Jun '16

21 Jun '16

Hi Monty, On Tue, Jun 21, 2016 at 9:30 AM, Michael Widenius <monty(a)mariadb.org> wrote: > > Hi! > > >>>>> "Nirbhay" == Nirbhay Choubey <nirbhay(a)mariadb.com> writes: > > <cut> > > Nirbhay> ALTER TABLE implementation takes a shortcut for operations not > affecting > Nirbhay> .frm files, > Nirbhay> which includes simple RENAMEs. But this is skipped for temp > tables and thus > Nirbhay> copying > Nirbhay> always takes place for temp tables. > > Nirbhay> The following commit (from 2002) added this exception for temp > tables: > Nirbhay> > https://github.com/MariaDB/server/commit/a7798dfd0a6472bf65fffc2ade543605e1… > > The above old patch is not really working for the current code. > > Nirbhay> I have created a patch to also include temp tables to this > shortcut : > Nirbhay> https://gist.github.com/nirbhayc/442a0c269ce48b283543cac434aaf44e > > This solves the problem of doing a simple rename, but not for things > like: > > alter table t1 add column c int, rename to t2; > > Which will still produce crashes. > > When it comes to last_version, is still vote for that we should be using > THD::rename_temporary_table when renaming temporary tables instead of > doing any handler calls. > Right. > > Maybe we could move THD::rename_temporary_tables to ha_rename_table to > get all temporary tables handled in one place ? > handler::ha_rename_table() is currently being used only for non-temporary tables at places which do not support temporary tables like partitions, DDL logs, RENAME TABLE, etc. So, moving rename_temporary_table to ha_rename_table seem overkill and possibly inappropriate. Here's the latest version of patch : http://lists.askmonty.org/pipermail/commits/2016-June/009477.html Best, Nirbhay > Regards, > Monty >

1 0

[Maria-developers] Different results from UNION and UNION DISTINCT
by Pantelis Theodosiou 21 Jun '16

21 Jun '16

According to this post at Stack Overflow, MariaDB 10.1.10 gives wrong results (duplicate rows) for UNION DISTINCT while the UNION version gives correct ones. (notice the comment: http://dba.stackexchange.com/questions/141799/select-distinct-values-of-two… ) The query in question is very simple: SELECT a AS result FROM tableXUNION DISTINCTSELECT b FROM tableX ; I couldn't find a related bug in the MDEV. Is this a known (or fixed) issue or should I go and add a bug report/ Pantelis Theodosiou

2 1

Re: [Maria-developers] 13f13b7: MDEV-5973: MySQL Bug#11757486:49539: NON-DESCRIPTIVE ERR (ERROR 0 FROM STORAGE ENGINE) WITH MULTI-TABLE UPDATE
by Sergei Golubchik 21 Jun '16

21 Jun '16

Hi, Oleksandr! On Jun 21, Oleksandr Byelkin wrote: > revision-id: 13f13b7b2bede46c90772a105a502c33fe93f4d1 (mariadb-10.0.25-11-g13f13b7) > parent(s): 70ad689b11bfbd8a30a777f4893a5384628c00e7 > committer: Oleksandr Byelkin > timestamp: 2016-06-21 18:37:22 +0200 > message: > > MDEV-5973: MySQL Bug#11757486:49539: NON-DESCRIPTIVE ERR (ERROR 0 FROM STORAGE ENGINE) WITH MULTI-TABLE UPDATE > > Condition in processing IGNORE clause for UPDATE & multi-table UPDATE made the same. ok to push, thanks! Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] MDEV-9524 Cannot load from mysql.event when sql_mode is set to PAD_CHAR_TO_FULL_LENGTH
by Alexander Barkov 21 Jun '16

21 Jun '16

Hello Sergei, Please review a patch for mdev-9524. Thanks.

2 3

Re: [Maria-developers] 9c253ae: MDEV-8636: REVOKE ALL PRIVILEGES, GRANT OPTION FROM CURRENT_ROLE breaks replication
by Sergei Golubchik 21 Jun '16

21 Jun '16

Hi, Vicentiu! On Jun 20, Vicentiu wrote: > revision-id: 9c253aeafdde8331b8d6f7ed11f1209ab4ce1d5b (mariadb-10.0.25-14-g9c253ae) > parent(s): 9301d5148d68ea7ae89918b87175c374028bfc5d > author: Vicențiu Ciorbaru > committer: Vicențiu Ciorbaru > timestamp: 2016-06-20 23:43:01 +0300 > message: > > MDEV-8636: REVOKE ALL PRIVILEGES, GRANT OPTION FROM CURRENT_ROLE breaks replication it's 8638, not 8636 > Fix the replication failure caused by incorect initialization of > THD::invoker_host && THD::invoker_user. > > Breakdown of the failure is this: > Query_log_event::host and Query_log_event::user can have their > LEX_STRING's set to length 0, but the actual str member points to > garbage. Code afterwards copies Query_log_event::host and user to > THD::invoker_host and THD::invoker_user. > Calling code for these members expects both members to be initialized. > Eg. the str member be a NULL terminated string and length have > appropriate size. > > diff --git a/sql/log_event.cc b/sql/log_event.cc > index 93704e7..4facd4f 100644 > --- a/sql/log_event.cc > +++ b/sql/log_event.cc > @@ -3631,10 +3631,8 @@ Query_log_event::Query_log_event(const char* buf, uint event_len, > if (time_zone_len) > copy_str_and_move(&time_zone_str, &start, time_zone_len); > > - if (user.length > 0) > - copy_str_and_move((const char **)&(user.str), &start, user.length); > - if (host.length > 0) > - copy_str_and_move((const char **)&(host.str), &start, host.length); > + copy_str_and_move((const char **)&(user.str), &start, user.length); > + copy_str_and_move((const char **)&(host.str), &start, host.length); I think the bug is that user.str or host.str may point to some garbage. I'd suggest the following fix instead: --- a/sql/log_event.cc +++ b/sql/log_event.cc @@ -3548,17 +3548,12 @@ Query_log_event::Query_log_event(const char* buf, uint e break; case Q_INVOKER: { - CHECK_SPACE(pos, end, 1); - user.length= *pos++; - CHECK_SPACE(pos, end, user.length); - user.str= (char *)pos; - pos+= user.length; - - CHECK_SPACE(pos, end, 1); - host.length= *pos++; - CHECK_SPACE(pos, end, host.length); - host.str= (char *)pos; - pos+= host.length; + if (get_str_len_and_pointer(&pos, &user.str, &user.length, end) || + get_str_len_and_pointer(&pos, &host.str. &host.length, end)) + { + query= 0; + DBUG_VOID_RETURN; + } break; } case Q_HRNOW: Ok ot push if you agree (and it works :) Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] dd32ca1: MDEV-5973: MySQL Bug#11757486:49539: NON-DESCRIPTIVE ERR (ERROR 0 FROM STORAGE ENGINE) WITH MULTI-TABLE UPDATE
by Sergei Golubchik 21 Jun '16

21 Jun '16

Hi, Oleksandr! On Jun 20, Oleksandr Byelkin wrote: > revision-id: dd32ca182246fccc7ba8d7c612af3ddd27fbf4af (mariadb-10.0.25-11-gdd32ca1) > parent(s): 70ad689b11bfbd8a30a777f4893a5384628c00e7 > committer: Oleksandr Byelkin > timestamp: 2016-06-20 14:35:58 +0200 > message: > > MDEV-5973: MySQL Bug#11757486:49539: NON-DESCRIPTIVE ERR (ERROR 0 FROM STORAGE ENGINE) WITH MULTI-TABLE UPDATE > > Condition in processing IGNORE clause for UPDATE & multi-table UPDATE made the same. > > diff --git a/mysql-test/r/multi_update.result b/mysql-test/r/multi_update.result > index 688184f..9e47fe2 100644 > --- a/mysql-test/r/multi_update.result > +++ b/mysql-test/r/multi_update.result > @@ -1039,4 +1039,24 @@ f1 f2 > 1 97 > DROP TABLE t1, t2; > DROP VIEW v1; > +# > +# MDEV-5973: MySQL Bug#11757486:49539: NON-DESCRIPTIVE ERR (ERROR 0 > +# FROM STORAGE ENGINE) WITH MULTI-TABLE UPDATE > +# > +CREATE TABLE table_11757486 (field1 tinyint) ENGINE=INNODB; > +INSERT INTO table_11757486 VALUES (0),(0); > +SET SESSION SQL_MODE='STRICT_ALL_TABLES'; > +UPDATE IGNORE (SELECT 128 as col1) x, table_11757486 SET field1=x.col1; Please add same tests in a single-table update version: UPDATE IGNORE table_11757486 SET field1=128; all three UPDATEs should be in both single and multi-table versions. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] b865428: MDEV-10258 - Valgrind warnings in buildbot after a set of mroonga tests
by Sergei Golubchik 21 Jun '16

21 Jun '16

Hi, Sergey! On Jun 20, Sergey Vojtovich wrote: > revision-id: b865428f1373cad45c2d92f7e659e09647bdbb31 (mariadb-10.0.25-13-gb865428) > parent(s): 99c8fb8d971c8e882b4f12296e322504f6b847cf > committer: Sergey Vojtovich > timestamp: 2016-06-20 13:54:19 +0400 > message: > > MDEV-10258 - Valgrind warnings in buildbot after a set of mroonga tests > > Fixed memory leak when mroonga fails to open index files. > Memory leak was detected by valgrind when running > mroonga/storage.repair_table_no_index_file. ok to push Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] 511bd2c: MDEV-10216: Assertion `strcmp(share->unique_file_name, filename) ||
by Sergey Vojtovich 20 Jun '16

20 Jun '16

Hi Nirbhay, last_version of MyISAM is used for debugging only and has no functional effect. last_version of Aria has some functional effect. Effect of HA_EXTRA_PREPARE_FOR_RENAME for MyISAM is flush key blocks and reset last_version. Effect of HA_EXTRA_PREPARE_FOR_RENAME for Aria is more complex. Other engines ignore HA_EXTRA_PREPARE_FOR_RENAME. I don't completely understand why "ALTER TABLE ... RENAME TO ..." has to copy data between tables. But it is probably subject of another bug (or even task). copy_data_between_tables() seem to be the only function that does HA_EXTRA_PREPARE_FOR_RENAME for temporary table (intermediate in this case). I don't think the above HA_EXTRA_PREPARE_FOR_RENAME is reasonable for MyISAM. There's no point in resetting last_version and flushing key blocks, since we're going to reuse this itermediate table anyway. I don't think the above can be reasonable for any engine. Said the above, I'd vote to remove this call. But please check with some Aria expert. Your fix is mostly alright, but I guess we shouldn't reset last_version in case of HA_EXTRA_PREPARE_FOR_DROP. Regards, Sergey On Wed, Jun 15, 2016 at 08:57:06AM -0400, Nirbhay Choubey wrote: > revision-id: 511bd2ca3269bc7bf80e30cceeab534f7f3e5666 (mariadb-10.2.0-83-g511bd2c) > parent(s): b2ae32aafdd2787ad456f38833f630182ded25e8 > author: Nirbhay Choubey > committer: Nirbhay Choubey > timestamp: 2016-06-15 08:57:04 -0400 > message: > > MDEV-10216: Assertion `strcmp(share->unique_file_name,filename) || > > .. share->last_version' failed in myisam/mi_open.c:67: test_if_reopen > > During the RENAME operation since the renamed temporary table is also > opened and added to myisam_open_list/maria_open_list, resetting the > last_version at the end of operation (HA_EXTRA_PREPARE_FOR_RENAME) > will cause an assertion failure when a subsequent query tries to open > an additional temporary table instance and thus attempts to reuse it > from the open table list. > > Fixed by not resetting share->last_version for temporary tables so that > the share gets reused when needed. > > --- > mysql-test/r/reopen_temp_table.result | 18 ++++++++++++++++++ > mysql-test/t/reopen_temp_table.test | 16 ++++++++++++++++ > storage/maria/ma_extra.c | 7 ++++++- > storage/myisam/mi_extra.c | 7 ++++++- > 4 files changed, 46 insertions(+), 2 deletions(-) > > diff --git a/mysql-test/r/reopen_temp_table.result b/mysql-test/r/reopen_temp_table.result > index 08affaa..e63a21e 100644 > --- a/mysql-test/r/reopen_temp_table.result > +++ b/mysql-test/r/reopen_temp_table.result > @@ -164,5 +164,23 @@ SELECT COUNT(*)=4 FROM t6; > COUNT(*)=4 > 1 > DROP TABLE t5, t6; > +# > +# MDEV-10216: Assertion `strcmp(share->unique_file_name,filename) || > +# share->last_version' failed in myisam/mi_open.c:67: test_if_reopen > +# > +CREATE TEMPORARY TABLE t7 (i INT) ENGINE=MYISAM; > +INSERT INTO t7 VALUES(1); > +ALTER TABLE t7 RENAME TO t; > +SELECT * FROM t a, t b; > +i i > +1 1 > +DROP TABLE t; > +CREATE TEMPORARY TABLE t7 (i INT) ENGINE=ARIA; > +INSERT INTO t7 VALUES(1); > +ALTER TABLE t7 RENAME TO t; > +SELECT * FROM t a, t b; > +i i > +1 1 > +DROP TABLE t; > # Cleanup > DROP DATABASE temp_db; > diff --git a/mysql-test/t/reopen_temp_table.test b/mysql-test/t/reopen_temp_table.test > index 98de983..83a5bbc 100644 > --- a/mysql-test/t/reopen_temp_table.test > +++ b/mysql-test/t/reopen_temp_table.test > @@ -159,5 +159,21 @@ SELECT COUNT(*)=6 FROM t5; > SELECT COUNT(*)=4 FROM t6; > DROP TABLE t5, t6; > > +--echo # > +--echo # MDEV-10216: Assertion `strcmp(share->unique_file_name,filename) || > +--echo # share->last_version' failed in myisam/mi_open.c:67: test_if_reopen > +--echo # > +CREATE TEMPORARY TABLE t7 (i INT) ENGINE=MYISAM; > +INSERT INTO t7 VALUES(1); > +ALTER TABLE t7 RENAME TO t; > +SELECT * FROM t a, t b; > +DROP TABLE t; > + > +CREATE TEMPORARY TABLE t7 (i INT) ENGINE=ARIA; > +INSERT INTO t7 VALUES(1); > +ALTER TABLE t7 RENAME TO t; > +SELECT * FROM t a, t b; > +DROP TABLE t; > + > --echo # Cleanup > DROP DATABASE temp_db; > diff --git a/storage/maria/ma_extra.c b/storage/maria/ma_extra.c > index fd21d28..5b33ad6 100644 > --- a/storage/maria/ma_extra.c > +++ b/storage/maria/ma_extra.c > @@ -399,7 +399,12 @@ int maria_extra(MARIA_HA *info, enum ha_extra_function function, > share->bitmap.changed_not_flushed= 0; > } > /* last_version must be protected by intern_lock; See collect_tables() */ > - share->last_version= 0L; /* Impossible version */ > + /* > + Temporary table has already been added to maria_open_list, so > + lets not reset the last_version. > + */ > + if (!share->temporary) > + share->last_version= 0L; /* Impossible version */ > mysql_mutex_unlock(&share->intern_lock); > break; > } > diff --git a/storage/myisam/mi_extra.c b/storage/myisam/mi_extra.c > index a47c198..bc5705a 100644 > --- a/storage/myisam/mi_extra.c > +++ b/storage/myisam/mi_extra.c > @@ -265,7 +265,12 @@ int mi_extra(MI_INFO *info, enum ha_extra_function function, void *extra_arg) > /* Fall trough */ > case HA_EXTRA_PREPARE_FOR_RENAME: > mysql_mutex_lock(&THR_LOCK_myisam); > - share->last_version= 0L; /* Impossible version */ > + /* > + Temporary table has already been added to myisam_open_list, so > + lets not reset the last_version. > + */ > + if (!share->temporary) > + share->last_version= 0L; /* Impossible version */ > mysql_mutex_lock(&share->intern_lock); > /* Flush pages that we don't need anymore */ > if (flush_key_blocks(share->key_cache, share->kfile, > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

2 1

Re: [Maria-developers] 7910c99: MDEV-10043 - main.events_restart fails sporadically in buildbot (crashes upon
by Sergei Golubchik 20 Jun '16

20 Jun '16

Hi, Sergey! On Jun 17, Sergey Vojtovich wrote: > revision-id: 7910c99bd1fa30911225e512997f466cb1b7996d (mariadb-5.5.49-51-g7910c99) > parent(s): 7ff86b49912df5de7b5a8b9c9134e72134b7ba5c > committer: Sergey Vojtovich > timestamp: 2016-06-17 18:54:11 +0400 > message: > > MDEV-10043 - main.events_restart fails sporadically in buildbot (crashes upon > shutdown) > > There was race condition between shutdown thread and event worker threads. > > Shutdown thread waits for thread_count to become 0 in close_connections(). It > may happen so that event worker thread was started but didn't increment > thread_count by this time. In this case shutdown thread may miss wait for this > working thread and continue deinitialization. Worker thread in turn may continue > execution and crash on deinitialized data. > > Fixed by incrementing thread_count before thread is actually created like it is > done for connection threads. > > Also let event scheduler not to inc/dec running threads counter for symmetry > with other "service" threads. ok to push. I'll have a questionable pleasure of merging it into 10.2 :) (it conflicts with 562c1df). Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] MDEV-10020 InnoDB NOT IN Query Crash When One Item Is NULL
by Alexander Barkov 20 Jun '16

20 Jun '16

Hi Sergei, Please review a patch for mdev-10020. The problem was that the loop in get_func_mm_tree() accessed an improperly initialized instance of String, which is a bzero'ed part of the in_vector::base array. Strings in in_vector::base are initialized in Item_func_in::fix_length_and_dec(): in in_vector::in_vector() using sql_calloc, rather than a String constructor, so its str_charset member of this String is NULL. Strings in in_vector::base are later initialized in Item_func_in::fix_length_and_dec(), using array->set(), in this code: uint j=0; for (uint i=1 ; i < arg_count ; i++) { array->set(j,args[i]); if (!args[i]->null_value) // Skip NULL values j++; else have_null= 1; } if ((array->used_count= j)) array->sort(); NULLs are not taken into account, so array->used_count can be smaller than array->count. This patch fixes the loop in opt_range.cc, in get_func_mm_tree(), to access only properly initialized elements in in_vector::base, preventing access to its bzero'ed non-initialized tail. Thanks.

2 1

Re: [Maria-developers] fa129a7: MDEV-9728 - Hard crash in metadata_lock_info
by Sergei Golubchik 20 Jun '16

20 Jun '16

Hi, Sergey! Looks ok to push, thanks! On Jun 17, Sergey Vojtovich wrote: > revision-id: fa129a7d22656eff9a7d1fb137d22976e0f0cec1 (mariadb-10.0.25-7-gfa129a7) > parent(s): 56ec3bdebd4a17ff59a487a988069f77bfc69aab > committer: Sergey Vojtovich > timestamp: 2016-06-17 12:52:35 +0400 > message: > > MDEV-9728 - Hard crash in metadata_lock_info > > metadata_lock_info plugin called MDL_context::find_ticket() to obtain lock > duration, which in turn iterates foreign thread private lists. These lists > can be updated by owner thread without protection. > > Fixed by iterating threads (instead of MDL locks and tickets) and obtaining > data through APC. > > Also fixed mdl_iterate_lock() to initialize iterator under prlock protection. > Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] b21e7af: MDEV-9969 mysql_install_db error processing ignore_db_dirs.
by Sergei Golubchik 16 Jun '16

16 Jun '16

Hi, Alexey! On Jun 16, Alexey Botchkov wrote: > revision-id: b21e7af20ed4e694376b6a2a246af9dd481d7bb0 (mariadb-10.0.25-7-gb21e7af) > parent(s): 0e50b924820d365bcc1659eec2cad606f6887597 > committer: Alexey Botchkov > timestamp: 2016-06-16 13:41:45 +0400 > message: > > MDEV-9969 mysql_install_db error processing ignore_db_dirs. > > Changes to the mysql_install_db scripts so they don't repeat > arguments twice. ok to push Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] Parallel replication slave stops with duplicate key errors
by Sergey Petrunia 16 Jun '16

16 Jun '16

Hi Kristian, I was trying out parallel replication and got an error that I am not sure what to do with. == Setup == I started a master with binlog_format=row, I ran sysbench there to load the dataset ("prepare" mode, about 10M rows IIRC) then I restarted the master and ran sysbench in "run" mode a few times. Then I restarted the master again an ran a "CREATE TABLE" statement manually. Just so that I have third binlog. The master state is: mysql> show binary logs; +-------------+-----------+ | Log_name | File_size | +-------------+-----------+ | pslp.000001 | 194918772 | | pslp.000002 | 28895350 | | pslp.000003 | 589 | +-------------+-----------+ == Non-parallel test run == Then I connected with a slave, with all default settings. The slave had empty datadir, so I made it a slave with this command: CHANGE MASTER TO MASTER_HOST='127.0.0.1', MASTER_PORT=3310, MASTER_USER='root'; All server settings were pretty much default. It took a slave one hour to catch up, but it did so, and encountered no errors. Ok, this was serial replication == Parallel test run == Then I killed the slave, deleted its data directory, put an empty directory instead I added this to slave.cnf file: slave_parallel_threads=4 slave_parallel_mode=aggressive and started the slave. It ran in parallel for a few minutes, I saw this: mysql> show processlist; +----+-------------+-----------+------+---------+------+-----------------------------------------------+------------------+----------+ | Id | User | Host | db | Command | Time | State | Info | Progress | +----+-------------+-----------+------+---------+------+-----------------------------------------------+------------------+----------+ | 3 | root | localhost | test | Query | 0 | init | show processlist | 0.000 | | 4 | system user | | NULL | Connect | 15 | Waiting for master to send event | NULL | 0.000 | | 5 | system user | | NULL | Connect | 0 | Write_rows_log_event::write_row(-1) | NULL | 0.000 | | 6 | system user | | NULL | Connect | 0 | Waiting for work from SQL thread | NULL | 0.000 | | 7 | system user | | NULL | Connect | 0 | Write_rows_log_event::write_row(-1) | NULL | 0.000 | | 8 | system user | | NULL | Connect | 0 | Waiting for work from SQL thread | NULL | 0.000 | | 9 | system user | | NULL | Connect | 14 | Waiting for room in worker thread event queue | NULL | 0.000 | +----+-------------+-----------+------+---------+------+-----------------------------------------------+------------------+----------+ 7 rows in set (0.00 sec) but then it stopped. The error log has this: 2016-06-15 15:09:58 140702586379008 [Note] Slave SQL thread initialized, starting replication in log 'FIRST' at position 0, relay log './psergey-desktop-relay-bin.000001' position: 4 2016-06-15 15:09:59 140702586985216 [Note] InnoDB: Online DDL : Start 2016-06-15 15:09:59 140702586985216 [Note] InnoDB: Online DDL : Start reading clustered index of the table and create temporary files 2016-06-15 15:09:59 140702586985216 [Note] InnoDB: Online DDL : End of reading clustered index of the table and create temporary files 2016-06-15 15:09:59 140702586985216 [Note] InnoDB: Online DDL : Start applying row log 2016-06-15 15:09:59 140702586985216 [Note] InnoDB: Online DDL : End of applying row log 2016-06-15 15:09:59 140702586985216 [Note] InnoDB: Online DDL : Completed 2016-06-15 15:11:18 140702586985216 [ERROR] Slave SQL: Error during XID COMMIT: failed to update GTID state in mysql.gtid_slave_pos: 1062: Duplicate entry '0-1021' for key 'PRIMARY', Gtid 0-1-1021, Internal MariaDB error code: 1942 2016-06-15 15:11:18 140702586985216 [ERROR] Slave (additional info): Duplicate entry '0-1021' for key 'PRIMARY' Error_code: 1062 2016-06-15 15:11:18 140702586985216 [Warning] Slave: Duplicate entry '0-1021' for key 'PRIMARY' Error_code: 1062 2016-06-15 15:11:18 140702586985216 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'pslp.000002' position 8266 2016-06-15 15:11:18 140702586682112 [ERROR] Slave (additional info): Commit failed due to failure of an earlier commit on which this one depends Error_code: 1964 2016-06-15 15:11:18 140702586682112 [Warning] Slave: Duplicate entry '821827' for key 'PRIMARY' Error_code: 1062 2016-06-15 15:11:18 140702586682112 [Warning] Slave: Deadlock found when trying to get lock; try restarting transaction Error_code: 1213 2016-06-15 15:11:18 140702586682112 [Warning] Slave: Commit failed due to failure of an earlier commit on which this one depends Error_code: 1964 2016-06-15 15:11:18 140702586682112 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'pslp.000002' position 8266 2016-06-15 15:11:18 140702602623744 [ERROR] Slave (additional info): Commit failed due to failure of an earlier commit on which this one depends Error_code: 1964 2016-06-15 15:11:18 140702602623744 [Warning] Slave: Commit failed due to failure of an earlier commit on which this one depends Error_code: 1964 2016-06-15 15:11:18 140702602623744 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'pslp.000002' position 8266 2016-06-15 15:11:18 140702602926848 [ERROR] Slave (additional info): Commit failed due to failure of an earlier commit on which this one depends Error_code: 1964 2016-06-15 15:11:18 140702602926848 [Warning] Slave: Commit failed due to failure of an earlier commit on which this one depends Error_code: 1964 2016-06-15 15:11:18 140702602926848 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'pslp.000002' position 8266 2016-06-15 15:11:18 140702586985216 [ERROR] Slave (additional info): Commit failed due to failure of an earlier commit on which this one depends Error_code: 1964 2016-06-15 15:11:18 140702586985216 [Warning] Slave: Duplicate entry '0-1021' for key 'PRIMARY' Error_code: 1062 2016-06-15 15:11:18 140702586985216 [Warning] Slave: Commit failed due to failure of an earlier commit on which this one depends Error_code: 1964 2016-06-15 15:11:18 140702586985216 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'pslp.000002' position 8266 2016-06-15 15:11:18 140702586682112 [ERROR] Slave (additional info): Commit failed due to failure of an earlier commit on which this one depends Error_code: 1964 2016-06-15 15:11:18 140702586682112 [Warning] Slave: Duplicate entry '821827' for key 'PRIMARY' Error_code: 1062 ... and there were lots of these errors. Eventually it stopped with: 2016-06-15 15:11:22 140702586682112 [Warning] Slave: Commit failed due to failure of an earlier commit on which this one depends Error_code: 1964 2016-06-15 15:11:22 140702586682112 [Warning] Slave: Commit failed due to failure of an earlier commit on which this one depends Error_code: 1964 2016-06-15 15:11:22 140702586682112 [Warning] Slave: Commit failed due to failure of an earlier commit on which this one depends Error_code: 1964 2016-06-15 15:11:22 140702586682112 [ERROR] Error running query, slave SQL thread aborted. Fix the problem, and restart the slave SQL thread with "SLAVE START". We stopped at log 'pslp.000002' position 8266 2016-06-15 15:11:22 140702586379008 [Note] Slave SQL thread exiting, replication stopped in log 'pslp.000002' at position 8266 The status is: mysql> show slave status\G *************************** 1. row *************************** Slave_IO_State: Waiting for master to send event Master_Host: 127.0.0.1 Master_User: root Master_Port: 3310 Connect_Retry: 60 Master_Log_File: pslp.000003 Read_Master_Log_Pos: 589 Relay_Log_File: psergey-desktop-relay-bin.000003 Relay_Log_Pos: 8549 Relay_Master_Log_File: pslp.000002 Slave_IO_Running: Yes Slave_SQL_Running: No Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 1942 Last_Error: Error during XID COMMIT: failed to update GTID state in mysql.gtid_slave_pos: 1062: Duplicate entry '0-1021' for key 'PRIMARY' Skip_Counter: 0 Exec_Master_Log_Pos: 8266 Relay_Log_Space: 223815659 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: NULL Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 1942 Last_SQL_Error: Error during XID COMMIT: failed to update GTID state in mysql.gtid_slave_pos: 1062: Duplicate entry '0-1021' for key 'PRIMARY' Replicate_Ignore_Server_Ids: Master_Server_Id: 1 Master_SSL_Crl: Master_SSL_Crlpath: Using_Gtid: No Gtid_IO_Pos: Replicate_Do_Domain_Ids: Replicate_Ignore_Domain_Ids: Parallel_Mode: aggressive Can you advise, what does this error mean? (is there any benefit in cc'ing this to maria-developers@ ? if yes, feel free to..) BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

2 1