- developers - lists.mariadb.org

Re: [Maria-developers] [Commits] 16298d9e82e: MDEV-7533: COLUMN_JSON() doesn't escape control characters in string values
by Sergei Golubchik 22 Jan '18

22 Jan '18

Hi, Oleksandr! On Jan 22, Oleksandr Byelkin wrote: > revision-id: 16298d9e82eb507c5e0ca6d097e6a7016582ccbc (mariadb-10.0.33-47-g16298d9e82e) > parent(s): 5e87f49a9927bf3fd0bd9af789bbdd329e4b4416 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2018-01-22 16:53:10 +0100 > message: > > MDEV-7533: COLUMN_JSON() doesn't escape control characters in string values > > escape all charecters less or equal 0x1F (control symbols) > (shorter sequence are not used to make code simple, long encoding is > always legal according to the rfc4627) ok to push Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] MDEV-14603 signal 11 with short stacktrace
by Alexander Barkov 22 Jan '18

22 Jan '18

Hi Sergei, can you please review a patch for MDEV-14603? Thanks!

2 5

Re: [Maria-developers] [JIRA] (MDEV-14185) CREATE TEMPORARY TABLE AS SELECT causes error 1290 with read_only and InnoDB
by Sergei Golubchik 22 Jan '18

22 Jan '18

Hi, Vesa! On Jan 16, Vesa Pentti (JIRA) wrote: > > Vesa Pentti updated MDEV-14185: > ------------------------------- > Status: In Review (was: In Progress) > Assignee: Sergei Golubchik (was: Vesa Pentti) > > commit d4366c4a2c9b41901b11ca180d3ddbf79e992490 Please send commit emails next time. This time I had to go and download the patch from github and paste it in an email. > diff --git a/sql/handler.cc b/sql/handler.cc > index 174034a7eb55..85b73719f0e8 100644 > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -4387,8 +4390,11 @@ handler::ha_create_partitioning_metadata(const char *name, const char *old_name, > partitions. > */ > DBUG_ASSERT(m_lock_type == F_UNLCK || > - (!old_name && strcmp(name, table_share->path.str))); > - mark_trx_read_write(); > + (!old_name && table_share && strcmp(name, table_share->path.str))); > + if (should_mark_rw) > + { > + mark_trx_read_write(); > + } Just remove mark_trx_read_write() from ha_create_partitioning_metadata(). It's a non-transactional operation anyway. It's always a part of DDL, so will have the same protection as DDL does. > > return create_partitioning_metadata(name, old_name, action_flag); > } Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] 47bb2dcbcc8: MDEV-14786: Server crashes in Item_cond::transform on 2nd execution of SP querying from a view
by Sergei Golubchik 22 Jan '18

22 Jan '18

Hi, Oleksandr! On Jan 21, Oleksandr Byelkin wrote: > revision-id: 47bb2dcbcc8c3cdca43baf6259f2434a746c7b03 (mariadb-5.5.58-42-g47bb2dcbcc8) > parent(s): c0e964dcf1787a101f6e9c115751c0d02b0fac1b > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2018-01-21 23:48:32 +0100 > message: > > MDEV-14786: Server crashes in Item_cond::transform on 2nd execution of SP querying from a view > > MDEV-14957: JOIN::prepare gets unusable "conds" as argument > > Do not touch merged derived (it is irreversible) > > Fix first argument of in_optimizer for calls possible before fix_fields() > > diff --git a/sql/item.cc b/sql/item.cc > index 332e027adf1..562c0bec78d 100644 > --- a/sql/item.cc > +++ b/sql/item.cc > @@ -10010,7 +10010,7 @@ const char *dbug_print_item(Item *item) > if (!item) > return "(Item*)NULL"; > item->print(&str ,QT_ORDINARY); > - if (str.c_ptr() == buf) > + if (str.ptr() == buf) c_ptr_safe() here. It's MDEV-14981. > return buf; > else > return "Couldn't fit into buffer"; > diff --git a/sql/item.h b/sql/item.h > index 4d0860af547..add408602fc 100644 > --- a/sql/item.h > +++ b/sql/item.h > @@ -4478,4 +4478,11 @@ class Item_iterator_row: public Item_iterator > void close() {} > }; > > +#ifdef DBUG_OFF > +static inline const char *dbug_print_item(Item *item) { return NULL; } > +#else > +extern const char *dbug_print_item(Item *item); > +#endif Put it not at the end, but at the beginning - above 'class Protocol;' That's where someone had it in 10.2, it'll be easier to merge if you add it in the same place. And no 'extern' in the #else branch, just as in 10.2. > #endif /* SQL_ITEM_INCLUDED */ > diff --git a/sql/item_cmpfunc.cc b/sql/item_cmpfunc.cc > index 62e76922c0e..a40340baae9 100644 > --- a/sql/item_cmpfunc.cc > +++ b/sql/item_cmpfunc.cc > @@ -1443,6 +1443,33 @@ bool Item_in_optimizer::eval_not_null_tables(uchar *opt_arg) > } > > > +void Item_in_optimizer::print(String *str, enum_query_type query_type) > +{ > + restore_first_argumet(); > + Item_func::print(str, query_type); > +} > + > + > +/** > + "Restore" first argument before fix_fields() call (after it is harmless). > + > + @Note: Main pointer to left part of IN/ALL/ANY subselect is subselect's > + lest_expr (see Item_in_optimizer::fix_left) so changes made during > + fix_fields will be rolled back there which can make > + Item_in_optimizer::args[0] unusable on second execution before fix_left() > + call. This call fix the pointer. > +*/ > + > +void Item_in_optimizer::restore_first_argumet() > +{ > + if (args[1]->type() == Item::SUBSELECT_ITEM && > + ((Item_subselect *)args[1])->is_in_predicate()) > + { > + args[0]= ((Item_in_subselect *)args[1])->left_expr; > + } > +} Any way to trigger the bug without dbug_print_item() ? Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] Mdev-14586 Assertion `0' failed in retrieve_auto_increment upon attempt to drop PK
by Sergei Golubchik 19 Jan '18

19 Jan '18

Hi, Sachin! I agree with the patch, I think it's correct. Good tests too. I wasn't able to understand the bug from the commit comment though, see below. Ok to push (preferrably with the improved commit message). On Jan 19, Sachin Setiya wrote: > Hi serg , please review the latest patch > > commit d9641478bb8f6045039a3d942f2925ada334907e > Author: Sachin Setiya <sachin.setiya(a)maridb.com> > Date: Fri Jan 19 19:33:45 2018 +0530 > > MDEV-14586 Assertion `0' failed in retrieve_auto_increment ... > > Problem:- > If we create table using myisam/aria then this crashes the server. > CREATE TABLE t1(a bit(1), b int auto_increment , index(a,b)); > insert into t1 values(1,1); > Or this query > CREATE TABLE t1 (b BIT(1), pk INTEGER AUTO_INCREMENT PRIMARY KEY); > ALTER TABLE t1 ADD INDEX(b,pk); > INSERT INTO t1 VALUES (1,b'1'); > ALTER TABLE t1 DROP PRIMARY KEY; > > Reason:- > The reason for this is > 1st- Since bit field is stored in record null bits itself , so > find_ref_key in open_binary_frm will wrongly set share->next_number_key_offset > and share->next_number_keypart (both set to 0). I would say that find_ref_key() finds what key an auto_increment field belongs to by comparing key_part->offset and field->ptr. But BIT fields might have zero length in the record, so a key might have many key parts with the same offset. That is, comparing offsets cannot uniquely identify the correct key part. > 2nd- Since next_number_key_offset is zero it myisam/aria will think that > auto_increment is in first part of key. > 3nd- myisam/aria will call retrieve_auto_key which will see first key_part > field as a bit field and call assert(0) > > Solution:- Many key parts might have the same offset, but BIT fields do not support auto_increment. So, we can skip all key parts over BIT fields, and then comparing offsets will be unambiguous. > Although the bit field is stored in record null_byte but in key file(.MYI) > this does not hold true. Bit(1-7) will take one byte of memory. So the correct > approach will be to never call retrieve_auto_increment in the case when > bit field is in the starting of index. So I have changed the find_ref_key > function to take account that if first N fields is Bit field then simply go > to next field which is not Bit field. > Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] 60d1803ec64: MDEV-6707: Wrong result (extra row) with group by, multi-part key
by Sergey Petrunia 18 Jan '18

18 Jan '18

On Thu, Jan 11, 2018 at 02:35:55AM +0530, Varun wrote: > revision-id: 60d1803ec645b6d388aac7125ac6381ecee4d548 (mariadb-5.5.56-115-g60d1803ec64) > parent(s): 9c9cf556a15af84f9133ef859b50596085f7ae8e > author: Varun Gupta > committer: Varun Gupta > timestamp: 2018-01-11 01:51:23 +0530 > message: > > MDEV-6707: Wrong result (extra row) with group by, multi-part key > > Range optimizer creates ranges according to the where conditions. So this case involves using a composite key, > few parts of which are involved in GROUP BY and few in the MIN/MAX functions in the select list. > So during the execution of such queries, we try to find a prefix using the fields involved in GROUP BY > and we then check if this newly returned prefix lies within the range we had calculated earlier. > After we find a row that satisfies the partial range(involving fields of GROUP BY), then we look at the other fields > of the where clause. We look in all the ranges if we can satisfy the above retured row. > > The issue is while calculating these prefixes we might encounter same partial ranges for multiple ranges. > An example would be > where condition is f2 ='c' AND f1 <> 9 > so the ranges would be > (NULL,c <= f1,f2 <= c,9) > (c,9 <= f1,f2 <= c,+infinity) > > In this case for calculating prefixes with the group by field we take up the > partial ranges involving field f2 those would be > c<= f2 <=c > c<= f2 <=c > So we lookup rows with the same prefix in all such ranges and then we check for the other > part(in this case f1) in all the ranges. So lets take a row of the table to be (4,'c'), that would have > a prefix 'c' and value for f1 to be 4, so it would lie in the first range mentioned above. But we would > get this same record again when we read the partial range and this would again satisfy the first range. > This issue can be fixed if we compare such partial ranges and don't lookup if we see the same prefix again. This is a huge and long comment. Please make it smaller, and move at least part of it into the code. Good places would be: - The function comment for QUICK_RANGE_SELECT::get_next_prefix - the new lines in QUICK_RANGE_SELECT::get_next_prefix which use/update the value of save_last_range > > --- > mysql-test/r/range.result | 14 ++++++++++++++ > mysql-test/r/range_mrr_icp.result | 14 ++++++++++++++ > mysql-test/t/range.test | 13 +++++++++++++ > sql/opt_range.cc | 20 +++++++++++++++++--- > sql/opt_range.h | 6 ++++-- > 5 files changed, 62 insertions(+), 5 deletions(-) > > diff --git a/mysql-test/r/range.result b/mysql-test/r/range.result > index 630a692cef6..72b31de4df4 100644 > --- a/mysql-test/r/range.result > +++ b/mysql-test/r/range.result > @@ -2144,3 +2144,17 @@ value1 1000685 12345 > value1 1003560 12345 > value1 1004807 12345 > drop table t1; > +# > +# MDEV-6707: Wrong result (extra row) with group by, multi-part key > +# > +CREATE TABLE t1 (f1 INT, f2 VARCHAR(1), KEY(f2,f1)) ENGINE=InnoDB; > +INSERT INTO t1 VALUES > +(7,'v'),(0,'s'),(9,'l'),(4,'c'); > +explain > +SELECT MAX(f1), f2 FROM t1 WHERE f2 LIKE 'c%' AND f1 <> 9 GROUP BY f2; > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 range f2 f2 9 NULL 2 Using where; Using index for group-by > +SELECT MAX(f1), f2 FROM t1 WHERE f2 LIKE 'c%' AND f1 <> 9 GROUP BY f2; > +MAX(f1) f2 > +4 c > +DROP TABLE t1; > diff --git a/mysql-test/r/range_mrr_icp.result b/mysql-test/r/range_mrr_icp.result > index 3f5de5b0189..1050bfcd887 100644 > --- a/mysql-test/r/range_mrr_icp.result > +++ b/mysql-test/r/range_mrr_icp.result > @@ -2146,4 +2146,18 @@ value1 1000685 12345 > value1 1003560 12345 > value1 1004807 12345 > drop table t1; > +# > +# MDEV-6707: Wrong result (extra row) with group by, multi-part key > +# > +CREATE TABLE t1 (f1 INT, f2 VARCHAR(1), KEY(f2,f1)) ENGINE=InnoDB; > +INSERT INTO t1 VALUES > +(7,'v'),(0,'s'),(9,'l'),(4,'c'); > +explain > +SELECT MAX(f1), f2 FROM t1 WHERE f2 LIKE 'c%' AND f1 <> 9 GROUP BY f2; > +id select_type table type possible_keys key key_len ref rows Extra > +1 SIMPLE t1 range f2 f2 9 NULL 2 Using where; Using index for group-by > +SELECT MAX(f1), f2 FROM t1 WHERE f2 LIKE 'c%' AND f1 <> 9 GROUP BY f2; > +MAX(f1) f2 > +4 c > +DROP TABLE t1; > set optimizer_switch=@mrr_icp_extra_tmp; > diff --git a/mysql-test/t/range.test b/mysql-test/t/range.test > index 393ca68e945..62e907b7b4a 100644 > --- a/mysql-test/t/range.test > +++ b/mysql-test/t/range.test > @@ -1718,3 +1718,16 @@ where (key1varchar='value1' AND (key2int <=1 OR key2int > 1)); > --echo # The following must show col1=12345 for all rows: > select * from t1; > drop table t1; > + > +--echo # > +--echo # MDEV-6707: Wrong result (extra row) with group by, multi-part key > +--echo # > + > +CREATE TABLE t1 (f1 INT, f2 VARCHAR(1), KEY(f2,f1)) ENGINE=InnoDB; > +INSERT INTO t1 VALUES > +(7,'v'),(0,'s'),(9,'l'),(4,'c'); > + > +explain > +SELECT MAX(f1), f2 FROM t1 WHERE f2 LIKE 'c%' AND f1 <> 9 GROUP BY f2; > +SELECT MAX(f1), f2 FROM t1 WHERE f2 LIKE 'c%' AND f1 <> 9 GROUP BY f2; > +DROP TABLE t1; > diff --git a/sql/opt_range.cc b/sql/opt_range.cc > index 25a9e729a8b..e985aba1140 100644 > --- a/sql/opt_range.cc > +++ b/sql/opt_range.cc > @@ -11249,6 +11249,7 @@ int QUICK_RANGE_SELECT::get_next() > @param prefix_length length of cur_prefix > @param group_key_parts The number of key parts in the group prefix > @param cur_prefix prefix of a key to be searched for > + @save_last_range Saving the last range we encountered You've meant "@param save_last_range" ? Also please use " INOUT " to indicate this parameter is used to pass info both from the caller and to the caller. > > Each subsequent call to the method retrieves the first record that has a > prefix with length prefix_length and which is different from cur_prefix, > @@ -11271,7 +11272,8 @@ int QUICK_RANGE_SELECT::get_next() > > int QUICK_RANGE_SELECT::get_next_prefix(uint prefix_length, > uint group_key_parts, > - uchar *cur_prefix) > + uchar *cur_prefix, > + QUICK_RANGE **save_last_range) > { > DBUG_ENTER("QUICK_RANGE_SELECT::get_next_prefix"); > const key_part_map keypart_map= make_prev_keypart_map(group_key_parts); > @@ -11301,8 +11303,19 @@ int QUICK_RANGE_SELECT::get_next_prefix(uint prefix_length, > last_range= 0; > DBUG_RETURN(HA_ERR_END_OF_FILE); > } > + > last_range= *(cur_range++); > > + if (*save_last_range) > + { > + if (!key_tuple_cmp(key_part_info, (*save_last_range)->min_key, > + last_range->min_key, prefix_length)) > + { > + last_range=NULL; > + continue; > + } > + } > + *save_last_range= last_range; > key_range start_key, end_key; > last_range->make_min_endpoint(&start_key, prefix_length, keypart_map); > last_range->make_max_endpoint(&end_key, prefix_length, keypart_map); > @@ -13315,7 +13328,7 @@ QUICK_GROUP_MIN_MAX_SELECT(TABLE *table, JOIN *join_arg, bool have_min_arg, > seen_first_key(FALSE), doing_key_read(FALSE), min_max_arg_part(min_max_arg_part_arg), > key_infix(key_infix_arg), key_infix_len(key_infix_len_arg), > min_functions_it(NULL), max_functions_it(NULL), > - is_index_scan(is_index_scan_arg) > + is_index_scan(is_index_scan_arg), save_last_range(NULL) > { > head= table; > index= use_index; > @@ -13968,7 +13981,8 @@ int QUICK_GROUP_MIN_MAX_SELECT::next_prefix() > uchar *cur_prefix= seen_first_key ? group_prefix : NULL; > if ((result= quick_prefix_select->get_next_prefix(group_prefix_len, > group_key_parts, > - cur_prefix))) > + cur_prefix, > + &save_last_range))) > DBUG_RETURN(result); > seen_first_key= TRUE; > } > diff --git a/sql/opt_range.h b/sql/opt_range.h > index b8b46ae5ab1..f4fb0642590 100644 > --- a/sql/opt_range.h > +++ b/sql/opt_range.h > @@ -455,6 +455,7 @@ class QUICK_RANGE_SELECT : public QUICK_SELECT_I > QUICK_RANGE **cur_range; /* current element in ranges */ > > QUICK_RANGE *last_range; > + QUICK_RANGE *save_last_range; So, now both QUICK_RANGE_SELECT and QUICK_GROUP_MIN_MAX_SELECT have 'save_last_range'. Does QUICK_RANGE_SELECT need one? > > KEY_PART *key_parts; > KEY_PART_INFO *key_part_info; > @@ -477,7 +478,7 @@ class QUICK_RANGE_SELECT : public QUICK_SELECT_I > int get_next(); > void range_end(); > int get_next_prefix(uint prefix_length, uint group_key_parts, > - uchar *cur_prefix); > + uchar *cur_prefix, QUICK_RANGE **save_last_range); > bool reverse_sorted() { return 0; } > bool unique_key_range(); > int init_ror_merged_scan(bool reuse_handler, MEM_ROOT *alloc); > @@ -916,7 +917,8 @@ class QUICK_GROUP_MIN_MAX_SELECT : public QUICK_SELECT_I > Use index scan to get the next different key instead of jumping into it > through index read > */ > - bool is_index_scan; > + bool is_index_scan; > + QUICK_RANGE *save_last_range; > public: > /* > The following two members are public to allow easy access from BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] [Commits] 6ffd535e415: MDEV-14229: Stack trace is not resolved for shared objects
by Sergei Golubchik 18 Jan '18

18 Jan '18

Hi, Vicentiu! Looks mostly ok. Does it mean, it will replace my changes in 10.0+ related to calculating the offset? struct link_map *lm = (struct link_map*) dlopen(0, RTLD_NOW); etc As far as LIBDL and dependencies are concerned... 1. Don't add LIBDL to the link list. The server has it and clients should not link with my_addr_resove (and it's good to get a linker error, if by some mistake they do) 2. Check for dladdr in configure.cmake: CHECK_FUNCTION_EXISTS (dladdr HAVE_DLADDR) 3. handle dladdr in my_global.h (search for dlopen or dlerror) But, I think, you won't need any ifdefs in my_addr_resolve.c, because my_global.h will take care of that. Ok to push after that, thanks! On Jan 16, vicentiu(a)mariadb.org wrote: > revision-id: 6ffd535e415028f414b9d1c9e9efbfb81bf25798 (mariadb-5.5.58-42-g6ffd535e415) > parent(s): 6267be460ab5147e3bc0fffd03e690b3650f2fe2 > author: Vicențiu Ciorbaru > committer: Vicențiu Ciorbaru > timestamp: 2018-01-16 22:57:52 +0200 > message: > > MDEV-14229: Stack trace is not resolved for shared objects > > Resolving a stacktrace including functions in dynamic libraries requires > us to look inside the libraries for the symbols. Addr2line needs to be > started with the correct binary for each address on the stack. To do this, > figure out which library it is using dladdr, then if the addr2line > binary was started with a different binary, fork it again with the > correct one. > > We only have one addr2line process running at any point during the > stacktrace resolving step. The maximum number of forks for addr2line should > generally be around 6. > > One for server stacktrace code, one for plugin code, one when going back > into server code, one for pthread library, one for libc, one for the > _start function in the server. More can come up if plugin calls server > function which goes back to a plugin, etc. > > --- > mysys/CMakeLists.txt | 2 +- > mysys/my_addr_resolve.c | 97 +++++++++++++++++++++++++++++++++---------------- > 2 files changed, 67 insertions(+), 32 deletions(-) > > diff --git a/mysys/CMakeLists.txt b/mysys/CMakeLists.txt > index cb86850c2de..15f3fbad9bf 100644 > --- a/mysys/CMakeLists.txt > +++ b/mysys/CMakeLists.txt > @@ -67,7 +67,7 @@ ENDIF() > > ADD_CONVENIENCE_LIBRARY(mysys ${MYSYS_SOURCES}) > TARGET_LINK_LIBRARIES(mysys dbug strings ${ZLIB_LIBRARY} > - ${LIBNSL} ${LIBM} ${LIBRT} ${LIBSOCKET} ${LIBEXECINFO}) > + ${LIBNSL} ${LIBM} ${LIBRT} ${LIBSOCKET} ${LIBEXECINFO} ${LIBDL}) > DTRACE_INSTRUMENT(mysys) > > IF(HAVE_BFD_H) > diff --git a/mysys/my_addr_resolve.c b/mysys/my_addr_resolve.c > index 90e6f43f390..f831ad5121f 100644 > --- a/mysys/my_addr_resolve.c > +++ b/mysys/my_addr_resolve.c > @@ -132,15 +132,79 @@ const char *my_addr_resolve_init() > > #include <m_string.h> > #include <ctype.h> > + > +#include <sys/wait.h> > + > static int in[2], out[2]; > -static int initialized= 0; > +static pid_t pid; > +static char addr2line_binary[1024]; > static char output[1024]; > + > +int start_addr2line_fork(const char *binary_path) > +{ > + > + if (pid > 0) > + { > + /* Don't leak FDs */ > + close(in[1]); > + close(out[0]); > + /* Don't create zombie processes. */ > + waitpid(pid, NULL, 0); > + } > + > + if (pipe(in) < 0) > + return 1; > + if (pipe(out) < 0) > + return 1; > + > + pid = fork(); > + if (pid == -1) > + return 1; > + > + if (!pid) /* child */ > + { > + dup2(in[0], 0); > + dup2(out[1], 1); > + close(in[0]); > + close(in[1]); > + close(out[0]); > + close(out[1]); > + execlp("addr2line", "addr2line", "-C", "-f", "-e", binary_path, NULL); > + exit(1); > + } > + > + close(in[0]); > + close(out[1]); > + > + return 0; > +} > + > int my_addr_resolve(void *ptr, my_addr_loc *loc) > { > char input[32], *s; > size_t len; > > - len= my_snprintf(input, sizeof(input), "%p\n", ptr); > + Dl_info info; > + void *offset; > + > + if (!dladdr(ptr, &info)) > + return 1; > + > + if (strcmp(addr2line_binary, info.dli_fname)) > + { > + /* We use dli_fname in case the path is longer than the length of our static > + string. We don't want to allocate anything dynamicaly here as we are in > + a "crashed" state. */ > + if (start_addr2line_fork(info.dli_fname)) > + { > + addr2line_binary[0] = '\0'; > + return 1; > + } > + /* Save result for future comparisons. */ > + strnmov(addr2line_binary, info.dli_fname, sizeof(addr2line_binary)); > + } > + offset = info.dli_fbase; > + len= my_snprintf(input, sizeof(input), "%p\n", ptr - offset); > if (write(in[1], input, len) <= 0) > return 1; > if (read(out[0], output, sizeof(output)) <= 0) > @@ -168,35 +232,6 @@ int my_addr_resolve(void *ptr, my_addr_loc *loc) > > const char *my_addr_resolve_init() > { > - if (!initialized) > - { > - pid_t pid; > - > - if (pipe(in) < 0) > - return "pipe(in)"; > - if (pipe(out) < 0) > - return "pipe(out)"; > - > - pid = fork(); > - if (pid == -1) > - return "fork"; > - > - if (!pid) /* child */ > - { > - dup2(in[0], 0); > - dup2(out[1], 1); > - close(in[0]); > - close(in[1]); > - close(out[0]); > - close(out[1]); > - execlp("addr2line", "addr2line", "-C", "-f", "-e", my_progname, NULL); > - exit(1); > - } > - > - close(in[0]); > - close(out[1]); > - initialized= 1; > - } > return 0; > } > #endif Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] a doubt about locks on supremum
by hongxiang jiang 18 Jan '18

18 Jan '18

2 1

Re: [Maria-developers] [Commits] 4b4267288cb: MDEV-14743: Server crashes in Item_func_match::init_search
by Sergei Golubchik 12 Jan '18

12 Jan '18

Hi, Oleksandr! Two thoughts. 1. Please use if (!ifm->fixed) then you'll only need one #ifdef. 2. How does it work with the iterator? if you remove the current element from the list and then iterator with jump to the next one, won't it cause one element to be skipped? On Jan 10, Oleksandr Byelkin wrote: > revision-id: 4b4267288cb2668717cc70537eb7e79cecaff58a (mariadb-5.5.58-24-g4b4267288cb) > parent(s): a408e881cf73d06fc92097fce6ef9584e16edf77 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2018-01-10 12:22:56 +0100 > message: > > MDEV-14743: Server crashes in Item_func_match::init_search > > Remove non prepared (and so belonging to removed clauses FT functions) from the list. > > in later version it will be fixed by building the list during preparation. > > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > index c06c4fcff29..26e302ded35 100644 > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -9550,7 +9550,19 @@ int init_ftfuncs(THD *thd, SELECT_LEX *select_lex, bool no_order) > DBUG_PRINT("info",("Performing FULLTEXT search")); > > while ((ifm=li++)) > - ifm->init_search(no_order); > +#if MYSQL_VERSION_ID < 100213 > + if (ifm->fixed) > +#endif > + ifm->init_search(no_order); > +#if MYSQL_VERSION_ID < 100213 > + else > + /* > + it mean that clause where was FT function was removed, so we have > + to remove the function from the list. > + */ > + li.remove(); > +#endif > + > } > return 0; > } Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 1

[Maria-developers] Eliminating or replacing the InnoDB row_prebuilt_t::fetch_cache
by Marko Mäkelä 12 Jan '18

12 Jan '18

The handler API in MySQL and MariaDB is operating on a single record at a time. There are some ‘batched’ primitives that in my understanding were developed for the NDB cluster, but generally InnoDB does not use them. The only nontrivial thing that InnoDB does during reads is the Index Condition Pushdown (ICP) and the end-of-range detection (DsMrr) that was ported from the never-released MySQL 6.0 or 5.2 to MySQL 5.6. To improve the performance of range scans, InnoDB uses a caching mechanism to prefetch results. Starting with the 4th consecutive read on the same ha_innobase handle, InnoDB would prefetch 8 records into row_prebuilt_t::fetch_cache. There are several drawbacks of this: - The parameters of this cache are hard-coded, and these caches can make wrong guesses (for SELECT * FROM t LIMIT 5, it would unnecessarily read 8 records, possibly causing unnecessary page accesses) - The caches waste memory, especially on partitioned tables, where each partition handle would have its own cache. (The main benefit of ha_innopart or InnoDB native partitioning in MySQL 5.7 is that the fetch_cache of the table is shared between partitions. It is only a benefit of saving memory, because the cache will be emptied when switching partitions.) As far as I understand, the entire purpose of the prebuilt->fetch_cache is to reduce the cost of acquiring the page latch and repositioning the cursor on each read request. If we did not call mtr_t::commit() between each request, maybe we could remove this cache altogether. When I was in the InnoDB team at MySQL, I did bring up a few times the idea of eliminating the fetch_cache, but the optimizer developers were in a separate team, and to do this, we would have needed support from both managers. I am hoping that with MariaDB this would merely be a technical challenge to first write a proof-of-concept prototype, and then to polish and test it. When I discussed this with the late Olav Sandstå some years ago, there was some thinking that we should stop hard-coding the TABLE::record[0] in various parts of the code, and the storage engine should be able to return multiple records to a larger TABLE::record[] buffer. I wonder if a simpler approach would work, and if we could do without any prefetching: 1. Remove row_sel_dequeue_cached_row_for_mysql(), prebuilt->fetch_cache, and related code. 2. Add prebuilt->mtr and let it be reused across multiple subsequent calls to row_search_mvcc(), until the query executor stops the current read batch. 3. A call to btr_pcur_store_position() and mtr_t::commit() will be necessary before any potentially long delay, such as when the result is being sent to a client connection, or when the query executor is going to switch to read records from another table. 4. The "end of read batch" could be signalled by invoking handler::extra() with a parameter. With such changes, less copying would take place, and there should be less contention on dict_index_t::lock and the upper-level buf_block_t::lock, due to fewer calls to btr_pcur_restore_position(). While that function does have a fast path ("optimistic restore"), it cannot switch to the next page without a key lookup. Range scans would be more efficient if we can always simply advance to the right leaf page and release the lock on the left page. I would love to do this in InnoDB. But I think that the effort should be lead from the optimizer side. Someone should first implement the "end of batched read" calls and some DBUG_ASSERT() that would catch any wrong-doing. That is, whenever we start to send data to a client or start to write into a temporary table, or switch to read records from another cursor, we must assert that the "end of batched read" flag is set on all cursor handles. Similarly, at the end and start of a statement, and at commit or rollback, all cursor handles must be at "end of batched read". Who would take the challenge? Best regards, Marko -- Marko Mäkelä, Lead Developer InnoDB MariaDB Corporation DON’T MISS M|18 MariaDB User Conference February 26 - 27, 2018 New York City https://m18.mariadb.com/

1 0

[Maria-developers] Check constraints
by Martijn Tonies (Upscene Productions) 10 Jan '18

10 Jan '18

Hi, Where can I find meta information for check constraints on tables? With regards, Martijn Tonies Upscene Productions http://www.upscene.com Database Workbench - developer tool for Oracle, MS SQL Server, PostgreSQL, SQL Anywhere, MySQL, InterBase, NexusDB and Firebird.

3 4

[Maria-developers] subscribe
by Martijn Tonies (Upscene Productions) 09 Jan '18

09 Jan '18

subscribe With regards, Martijn Tonies Upscene Productions http://www.upscene.com Database Workbench - developer tool for Oracle, MS SQL Server, PostgreSQL, SQL Anywhere, MySQL, InterBase, NexusDB and Firebird.

1 0

Re: [Maria-developers] [Commits] fbe0fc2: Fixed mdev-14879 Lost rows for query using recursive CTE
by Oleksandr Byelkin 09 Jan '18

09 Jan '18

Hi, Igor! Am 06.01.2018 um 18:32 schrieb IgorBabaev: > revision-id: fbe0fc2d87cb1bdb9f7040d44ad3c58a94857e20 (mariadb-10.2.12-14-gfbe0fc2) > parent(s): 15b1840f43be8c660382458826e9d5a47d148a67 > author: Igor Babaev > committer: Igor Babaev > timestamp: 2018-01-06 09:32:47 -0800 > message: > > Fixed mdev-14879 Lost rows for query using recursive CTE > with recursive reference in subquery > > If a recursive CTE uses a subquery with recursive reference then > the virtual function reset() must be called after each iteration > performed at the execution of the CTE. OK to push. [skip]

1 0

Re: [Maria-developers] [Commits] 81965f0: Fixed mdev-14852 Fails to reopen temp table within standard CTE
by Oleksandr Byelkin 05 Jan '18

05 Jan '18

Am 05.01.2018 um 08:40 schrieb IgorBabaev: > revision-id: 81965f03065b84322f2d03d784893179e8b79fd9 (mariadb-10.2.12-6-g81965f0) > parent(s): 0f253d3e644d3027face26371eb1bb59e8280728 > author: Igor Babaev > committer: Igor Babaev > timestamp: 2018-01-04 23:40:37 -0800 > message: > > Fixed mdev-14852 Fails to reopen temp table within standard CTE > > If the specification of a CTE contains a reference to a temporary table > then THD::open_temporary_table() must be called for this reference for > any occurrence of the CTE in the query. By mistake this was done only > for the first occurrences of CTEs. > The patch fixes this problem in With_element::clone_parsed_spec(). > It also moves there the call of check_dependencies_in_with_clauses() > to its proper place before the call of check_table_access(). > Additionally the patch optimizes the number of calls of the > function check_dependencies_in_with_clauses(). OK to push! > > --- > mysql-test/r/cte_nonrecursive.result | 30 ++++++++++++++++++++++++++++++ > mysql-test/t/cte_nonrecursive.test | 29 +++++++++++++++++++++++++++++ > sql/sql_cte.cc | 7 +++++++ > sql/sql_parse.cc | 22 +++++----------------- > sql/sql_prepare.cc | 11 +++-------- > 5 files changed, 74 insertions(+), 25 deletions(-) > > diff --git a/mysql-test/r/cte_nonrecursive.result b/mysql-test/r/cte_nonrecursive.result > index b1b41b1..c1f4c9f 100644 > --- a/mysql-test/r/cte_nonrecursive.result > +++ b/mysql-test/r/cte_nonrecursive.result > @@ -1265,3 +1265,33 @@ a > 4 > deallocate prepare stmt; > drop table t1; > +# > +# MDEV-14852: CTE using temporary table in query > +# with two references to the CTE > +# > +create temporary table t1 (i int); > +insert into t1 values (5),(4),(1),(2),(3); > +with > +c1 as (select i from t1), > +c2 as (select i from c1 where c1.i=2) > +select i from c1 where i > 3 union select i from c2; > +i > +5 > +4 > +2 > +drop table t1; > +create table t1 (term char(10)); > +create temporary table t2 (term char(10)); > +insert into t1 values ('TERM01'),('TERM02'),('TERM03'); > +insert into t2 values ('TERM02'),('TERM03'),('TERM04'); > +with c1 as (select * from t1), c2 as (select * from t2) > +(select * from c1 left outer join c2 on c1.term = c2.term) > +union all > +(select * from c1 right outer join c2 on c1.term = c2.term > +where c1.term is null); > +term term > +TERM02 TERM02 > +TERM03 TERM03 > +TERM01 NULL > +NULL TERM04 > +drop table t1,t2; > diff --git a/mysql-test/t/cte_nonrecursive.test b/mysql-test/t/cte_nonrecursive.test > index 1f21dbc..9436665 100644 > --- a/mysql-test/t/cte_nonrecursive.test > +++ b/mysql-test/t/cte_nonrecursive.test > @@ -853,3 +853,32 @@ execute stmt; > deallocate prepare stmt; > > drop table t1; > + > +--echo # > +--echo # MDEV-14852: CTE using temporary table in query > +--echo # with two references to the CTE > +--echo # > + > +create temporary table t1 (i int); > +insert into t1 values (5),(4),(1),(2),(3); > + > +with > +c1 as (select i from t1), > +c2 as (select i from c1 where c1.i=2) > +select i from c1 where i > 3 union select i from c2; > + > +drop table t1; > + > +create table t1 (term char(10)); > +create temporary table t2 (term char(10)); > + > +insert into t1 values ('TERM01'),('TERM02'),('TERM03'); > +insert into t2 values ('TERM02'),('TERM03'),('TERM04'); > + > +with c1 as (select * from t1), c2 as (select * from t2) > +(select * from c1 left outer join c2 on c1.term = c2.term) > +union all > +(select * from c1 right outer join c2 on c1.term = c2.term > + where c1.term is null); > + > +drop table t1,t2; > diff --git a/sql/sql_cte.cc b/sql/sql_cte.cc > index 601c192..d12948f 100644 > --- a/sql/sql_cte.cc > +++ b/sql/sql_cte.cc > @@ -817,12 +817,19 @@ st_select_lex_unit *With_element::clone_parsed_spec(THD *thd, > parse_status= parse_sql(thd, &parser_state, 0); > if (parse_status) > goto err; > + > + if (check_dependencies_in_with_clauses(lex->with_clauses_list)) > + goto err; > + > spec_tables= lex->query_tables; > spec_tables_tail= 0; > for (TABLE_LIST *tbl= spec_tables; > tbl; > tbl= tbl->next_global) > { > + if (!tbl->derived && !tbl->schema_table && > + thd->open_temporary_table(tbl)) > + goto err; > spec_tables_tail= tbl; > } > if (check_table_access(thd, SELECT_ACL, spec_tables, FALSE, UINT_MAX, FALSE)) > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > index f00c74b..6d068eb 100644 > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -2990,6 +2990,9 @@ mysql_execute_command(THD *thd) > thd->get_stmt_da()->opt_clear_warning_info(thd->query_id); > } > > + if (check_dependencies_in_with_clauses(thd->lex->with_clauses_list)) > + DBUG_RETURN(1); > + > #ifdef HAVE_REPLICATION > if (unlikely(thd->slave_thread)) > { > @@ -3446,14 +3449,6 @@ mysql_execute_command(THD *thd) > ulong privileges_requested= lex->exchange ? SELECT_ACL | FILE_ACL : > SELECT_ACL; > > - /* > - The same function must be called for DML commands > - when CTEs are supported in DML statements > - */ > - res= check_dependencies_in_with_clauses(thd->lex->with_clauses_list); > - if (res) > - break; > - > if (all_tables) > res= check_table_access(thd, > privileges_requested, > @@ -3879,8 +3874,7 @@ mysql_execute_command(THD *thd) > /* Copy temporarily the statement flags to thd for lock_table_names() */ > uint save_thd_create_info_options= thd->lex->create_info.options; > thd->lex->create_info.options|= create_info.options; > - if (!(res= check_dependencies_in_with_clauses(lex->with_clauses_list))) > - res= open_and_lock_tables(thd, create_info, lex->query_tables, TRUE, 0); > + res= open_and_lock_tables(thd, create_info, lex->query_tables, TRUE, 0); > thd->lex->create_info.options= save_thd_create_info_options; > if (res) > { > @@ -4493,8 +4487,7 @@ mysql_execute_command(THD *thd) > > unit->set_limit(select_lex); > > - if (!(res= check_dependencies_in_with_clauses(lex->with_clauses_list)) && > - !(res=open_and_lock_tables(thd, all_tables, TRUE, 0))) > + if (!(res=open_and_lock_tables(thd, all_tables, TRUE, 0))) > { > MYSQL_INSERT_SELECT_START(thd->query()); > /* > @@ -4821,9 +4814,6 @@ mysql_execute_command(THD *thd) > { > List<set_var_base> *lex_var_list= &lex->var_list; > > - if (check_dependencies_in_with_clauses(thd->lex->with_clauses_list)) > - goto error; > - > if ((check_table_access(thd, SELECT_ACL, all_tables, FALSE, UINT_MAX, FALSE) > || open_and_lock_tables(thd, all_tables, TRUE, 0))) > goto error; > @@ -6376,8 +6366,6 @@ static bool execute_sqlcom_select(THD *thd, TABLE_LIST *all_tables) > new (thd->mem_root) Item_int(thd, > (ulonglong) thd->variables.select_limit); > } > - if (check_dependencies_in_with_clauses(lex->with_clauses_list)) > - return 1; > > if (!(res= open_and_lock_tables(thd, all_tables, TRUE, 0))) > { > diff --git a/sql/sql_prepare.cc b/sql/sql_prepare.cc > index be78a76..390b708 100644 > --- a/sql/sql_prepare.cc > +++ b/sql/sql_prepare.cc > @@ -1573,8 +1573,6 @@ static int mysql_test_select(Prepared_statement *stmt, > lex->select_lex.context.resolve_in_select_list= TRUE; > > ulong privilege= lex->exchange ? SELECT_ACL | FILE_ACL : SELECT_ACL; > - if (check_dependencies_in_with_clauses(lex->with_clauses_list)) > - goto error; > if (tables) > { > if (check_table_access(thd, privilege, tables, FALSE, UINT_MAX, FALSE)) > @@ -1841,9 +1839,6 @@ static bool mysql_test_create_table(Prepared_statement *stmt) > if (create_table_precheck(thd, tables, create_table)) > DBUG_RETURN(TRUE); > > - if (check_dependencies_in_with_clauses(lex->with_clauses_list)) > - DBUG_RETURN(TRUE); > - > if (select_lex->item_list.elements) > { > /* Base table and temporary table are not in the same name space. */ > @@ -2234,9 +2229,6 @@ static bool mysql_test_insert_select(Prepared_statement *stmt, > if (insert_precheck(stmt->thd, tables)) > return 1; > > - if (check_dependencies_in_with_clauses(lex->with_clauses_list)) > - return 1; > - > /* store it, because mysql_insert_select_prepare_tester change it */ > first_local_table= lex->select_lex.table_list.first; > DBUG_ASSERT(first_local_table != 0); > @@ -2339,6 +2331,9 @@ static bool check_prepared_statement(Prepared_statement *stmt) > if (tables) > thd->get_stmt_da()->opt_clear_warning_info(thd->query_id); > > + if (check_dependencies_in_with_clauses(thd->lex->with_clauses_list)) > + goto error; > + > if (sql_command_flags[sql_command] & CF_HA_CLOSE) > mysql_ha_rm_tables(thd, tables); > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

Re: [Maria-developers] [Commits] de21526: This is a full cost-based implementation of the optimization that employs
by Oleksandr Byelkin 29 Dec '17

29 Dec '17

Hi, Igor! Am 29.12.2017 um 01:47 schrieb IgorBabaev: > revision-id: de215264682a9f3291ca028fb3ae052ecb191ca3 (mariadb-10.3.3-4-gde21526) > parent(s): 7a66e0ab8f52f3bd32850463daa05f9a2401e6b1 > author: Igor Babaev > committer: Igor Babaev > timestamp: 2017-12-28 16:47:25 -0800 > message: > > This is a full cost-based implementation of the optimization that employs > splitting technique for equi-joins of materialized derived tables/views/CTEs. > (see mdev-13369 and mdev-13389). I have to make special mention of really good description. As we agreed OK to push after adding test of double execution. [skip]

1 0

Re: [Maria-developers] a01f9ff: MDEV-11071 - Assertion `thd->transaction.stmt.is_empty()' failed in
by Sergei Golubchik 27 Dec '17

27 Dec '17

Hi, Sergey! On Dec 26, Sergey Vojtovich wrote: > revision-id: a01f9ff80b892bf32dd2754da17438e8884c003a (mariadb-10.2.11-51-ga01f9ff) > parent(s): c3bd0b0301936d0dcd0918bb8c966c272610d61b > committer: Sergey Vojtovich > timestamp: 2017-12-26 14:38:45 +0400 > message: > > MDEV-11071 - Assertion `thd->transaction.stmt.is_empty()' failed in > Locked_tables_list::unlock_locked_tables > > Similarly to regular DROP TABLE, don't leave locked tables mode if CREATE OR > REPLACE dropped temporary table but failed to cerate new one. > > The problem is that there's no track of which temporary table was "locked" by > LOCK TABLES. Looks reasonable. Could you also add a test where t2 is locked and t2 is dropped? Just to show that pos_in_locked_tables works for temporary tables. Ok to push after that. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] Please document HOUR() vs EXTRACT(HOUR)
by Alexander Barkov 23 Dec '17

23 Dec '17

Hello Ian, Can you please document the difference between functions HOUR() and EXTRACT(HOUR)? Please see details here: https://jira.mariadb.org/browse/MDEV-14731 Thanks!

1 0

[Maria-developers] mysql and mariadb got different result
by 落寞回头不如华丽转身 21 Dec '17

21 Dec '17

mysql but mariadb

2 1

[Maria-developers] Fwd: Need Help
by Deepak Gupta 20 Dec '17

20 Dec '17

Hi There, I am working on a project where I have some wired requirement for my project. Issue : I am looking for the generic script that I can run with the help of Admin rights on the data base and I want the return output in such a way that it will give me the complete data base structure along with the table relation ship. For an example , there is one HR schema , with in that schema there might be 100 tables example , Employee / Salary / Address / Department etc and every table has primary and foreign key concept. So I need the script should be such a way that it will fetch all these information and as soon as I run the same script in my local system it will replicate the same data base structure for me in my local system. Kindly suggest how can I achieve this requirement ? Any help or suggestions are more than welcome. Looking foreword to here you soon. Thanks D

2 1

Re: [Maria-developers] [Commits] 203a658: MDEV-11084 Select statement with partition selection against MyISAM table opens all partitions.
by Sergei Golubchik 19 Dec '17

19 Dec '17

Hi, Alexey! On Dec 18, Alexey Botchkov wrote: > revision-id: 203a6585d3deedb36e30d5ebf388de44b6810403 (mariadb-10.1.23-51-g203a658) > parent(s): e4d10e09cf318aad237143254c45458d16009f70 > committer: Alexey Botchkov > timestamp: 2017-12-18 19:31:26 +0400 > message: > > MDEV-11084 Select statement with partition selection against MyISAM table opens all partitions. > > Now SELECT FROM t PARTITION(x) only opens the 'x' file. > The table->partition_names parameter is sent to ha_open > so it only opens the required partitions. > If the table is reopened, the > change_partition_to_open(partition_names) is called that > closes and opens partitions specified for the query. Looks ok to me, thanks! Please add some tests, can status variables show the difference in the number of opened partitions? And make sure that the whole tree compiles without partitioning. Then ok to push into 10.3 Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] 7a7ad82: MDEV-13478 Full SST sync fails because of the error in the cleaning part
by Sergei Golubchik 19 Dec '17

19 Dec '17

Hi, Sachin! Unfortunately, your comment is rather difficult to understand. What about this one: Problem: The command was: find $paths -mindepth 1 -regex $cpat -prune -o -exec rm -rf {} \+ Which was supposed to work as * skipping $paths directories themselves (-mindepth 1) * see if the dir/file name matches $cpat (-regex) * if yes - don't dive into the directory, skip it (-prune) * otherwise (-o) * remove it and everything inside (-exec) Now -exec ... \+ works like this: every new found path is appended to the end of the command line. when accumulated command line length reaches `getconf ARG_MAX` (~2Gb) it's executed, and find continues, appending to a new command line. What happens here, find appends some directory to the command line, then dives into it, and starts appending files from that directory. At some point command line overflows, rm -rf gets executed and removes the whole directory. Now find tries to continue scanning the directory that was already removed. Fix: don't dive into directories that will be recursively removed anyway, use -prune for them. Basically, we should be pruning both paths that have matched $cpat and paths that have not matched it. This is achived by pruning unconditionally, before the regex is tested: find $paths -mindepth 1 -prune -regex $cpat -o -exec rm -rf {} \+ On Dec 19, sachin wrote: > revision-id: 7a7ad82029a6c78d31d6736a562c12d02c4d968c (mariadb-galera-5.5.58-3-g7a7ad82) > parent(s): e6e026ae51a77969749de201d491a176483bbc69 > author: Sachin Setiya > committer: Sachin Setiya > timestamp: 2017-12-19 22:30:43 +0530 > message: > > MDEV-13478 Full SST sync fails because of the error in the cleaning part > > Problem:- The problem is in wsrep_sst_xtrabackup-v2.sh we use > find $ib_home_dir $ib_log_dir $ib_undo_dir $DATA -mindepth 1 -regex $cpat -prune -o -exec rm -rfv {} 1>&2 \+ > the problem is that since we have '\+' in end that means all output > will be expanded after rm -rfv . If we have really large database( > quite a no of tables with big names) then this create a problem. > This will result in calling 'rm -rvf' two time(or may be more). So non > deterministicly this might that upto directory name went to rm and > remaining was truncated. For example consider a folder xyz with lots > of files. We executed > find xyz -exec rm -rfv {} \+ > Since we have like millions of file in rm, So it will be greater then > ARG_MAX , so there will be multiple rm invocation. Say in nth invocation > this might happen rm ....... xyz/ {So it get truncated at xyz/} > Above will remove the whole xyz directory and remaining invocation > of rm will return error. How ever this type of error is non deterministic > so is bug. > Solution:- > In above example if instead of removing each file in xyz if we remove > xyz then we have our solution :). > Actually if we shift the -prune term in find we can get to solution. > Why currently find is working like this > find ( -regex && -prune) || -exec > that means if -regex is true (-prune will always return true) then exec > (and hence rm )wont work. But if regex fails then exec is applied with > out -prune which makes rm delete each single file instead of folder. > So we change the position of regex and prune then prune will always be > applied whether regex is true or not. > find (-prune && -regex) || -exec > > Patch Credit:- Serg > > --- > scripts/wsrep_sst_xtrabackup-v2.sh | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/scripts/wsrep_sst_xtrabackup-v2.sh b/scripts/wsrep_sst_xtrabackup-v2.sh > index 327c92f..00d8fe2 100644 > --- a/scripts/wsrep_sst_xtrabackup-v2.sh > +++ b/scripts/wsrep_sst_xtrabackup-v2.sh > @@ -863,7 +863,7 @@ then > > > wsrep_log_info "Cleaning the existing datadir and innodb-data/log directories" > - find $ib_home_dir $ib_log_dir $ib_undo_dir $DATA -mindepth 1 -regex $cpat -prune -o -exec rm -rfv {} 1>&2 \+ > + find $ib_home_dir $ib_log_dir $ib_undo_dir $DATA -mindepth 1 -prune -regex $cpat -o -exec rm -rfv {} 1>&2 \+ > > tempdir=$(parse_cnf mysqld log-bin "") > if [[ -n ${tempdir:-} ]];then Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 1

Re: [Maria-developers] 665402a: MDEV-14265 - RPMLint warning: shared-lib-calls-exit
by Sergei Golubchik 18 Dec '17

18 Dec '17

Hi, Sergey! On Dec 18, Sergey Vojtovich wrote: > revision-id: 665402aa9b83565c1cef3caa06cd998d3808f0d7 (mariadb-10.1.29-31-g665402a) > parent(s): ef9e78c9d41a5ac644f08068e3dabad948b0e30a > committer: Sergey Vojtovich > timestamp: 2017-12-18 20:23:36 +0400 > message: > > MDEV-14265 - RPMLint warning: shared-lib-calls-exit > > find_type_with_warning() client helper did exit(1) on error, exit(1) moved to > clients. It was find_type_or_exit(), and you could've had less changes if you'd replaced it with an inline function. Basically, copied the function into the header and made it static inline. > mysql_read_default_options() did exit(1) on error, error is passed through and > handled now. ok > my_str_malloc_default() did exit(1) on error, replaced my_str_ allocator > functions with normal my_malloc()/my_realloc()/my_free(). I think you need to replace my_str_malloc_default() with my_malloc(..., MYF(MY_FAE)), not my_malloc(..., MYF(0)) > sql_connect.cc did many exit(1) on hash initialisation failure. Removed error > check since my_hash_init() never fails. > > my_malloc() did exit(1) on error. Replaced with abort(). > > my_load_defaults() did exit(1) on error, replaced with abort(). I'd say, return 2 instead of exit/abort. my_load_defaults already does it, because my_search_option_files returns 2 for OOM. > my_load_defaults() still does exit(0) when invoked with --print-defaults. dunno, return -1, for example, and exit in the caller? Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 1

[Maria-developers] MDEV-14031 Password policy causes replication failure
by Alexander Barkov 18 Dec '17

18 Dec '17

Hi Sergei, Please review a patch for MDEV-14031. Thanks!

1 0

Re: [Maria-developers] [Commits] 2603dc0: MDEV-11084 Select statement with partition selection against MyISAM table opens all partitions.
by Sergei Golubchik 15 Dec '17

15 Dec '17

Hi, Alexey! That looked reasonable enough. I didn't find anything big. Small comments - see below. On Jun 01, Alexey Botchkov wrote: > revision-id: 2603dc0d56ec90f308678a3c002f1b75af69f554 (mariadb-10.1.23-51-g2603dc0) > parent(s): e4d10e09cf318aad237143254c45458d16009f70 > committer: Alexey Botchkov > timestamp: 2017-06-01 17:33:34 +0400 > message: > > MDEV-11084 Select statement with partition selection against MyISAM table opens all partitions. > > Now SELECT FROM t PARTITION(x) only opens the 'x' file. > The table->partition_names parameter is sent to ha_open > so it only opens the required partitions. > If the table is reopened, the > change_partition_to_open(partition_names) is called that > closes and opens partitions specified for the query. > > diff --git a/sql/ha_partition.cc b/sql/ha_partition.cc > index 29054c7..53a128c 100644 > --- a/sql/ha_partition.cc > +++ b/sql/ha_partition.cc > @@ -3481,12 +3498,16 @@ int ha_partition::open(const char *name, int mode, uint test_if_locked) > } > name_buffer_ptr+= strlen(name_buffer_ptr) + 1; > } > + file_sample= m_file[0]; are you sure that m_file[0] is opened? > } > else > { > file= m_file; > do > { > + if (!bitmap_is_set(&m_opened_partitions, file - m_file)) > + continue; > + > create_partition_name(name_buff, name, name_buffer_ptr, NORMAL_PART_NAME, > FALSE); > table->s->connect_string = m_connect_string[(uint)(file-m_file)]; > @@ -3494,20 +3515,21 @@ int ha_partition::open(const char *name, int mode, uint test_if_locked) > test_if_locked | HA_OPEN_NO_PSI_CALL))) > goto err_handler; > bzero(&table->s->connect_string, sizeof(LEX_STRING)); > - if (m_file == file) > - m_num_locks= (*file)->lock_count(); > + if (!file_sample) > + m_num_locks= (file_sample= (*file))->lock_count(); > DBUG_ASSERT(m_num_locks == (*file)->lock_count()); > - name_buffer_ptr+= strlen(name_buffer_ptr) + 1; > - } while (*(++file)); > + } while (name_buffer_ptr+= strlen(name_buffer_ptr) + 1, *(++file)); why? > } > > file= m_file; > - ref_length= (*file)->ref_length; > - check_table_flags= (((*file)->ha_table_flags() & > + ref_length= file_sample->ref_length; > + check_table_flags= ((file_sample->ha_table_flags() & > ~(PARTITION_DISABLED_TABLE_FLAGS)) | > (PARTITION_ENABLED_TABLE_FLAGS)); > while (*(++file)) > { > + if (!bitmap_is_set(&m_opened_partitions, file - m_file)) > + continue; > /* MyISAM can have smaller ref_length for partitions with MAX_ROWS set */ > set_if_bigger(ref_length, ((*file)->ref_length)); > /* > @@ -6782,6 +6809,63 @@ void ha_partition::get_dynamic_partition_info(PARTITION_STATS *stat_info, > } > > > +void ha_partition::set_partitions_to_open(List<String> *partition_names) > +{ > + m_partitions_to_open= partition_names; > +} > + > + > +int ha_partition::change_partitions_to_open(List<String> *partition_names) > +{ > + char name_buff[FN_REFLEN]; > + handler **file; > + char *name_buffer_ptr; > + int error= 0; > + > + if (m_is_clone_of) > + return 0; > + > + m_partitions_to_open= partition_names; > + if ((error= m_part_info->set_partition_bitmaps(partition_names))) > + goto err_handler; > + > + if (bitmap_cmp(&m_opened_partitions, &m_part_info->read_partitions) != 0) > + return 0; > + > + if ((error= read_par_file(table->s->normalized_path.str))) > + goto err_handler; > + > + name_buffer_ptr= m_name_buffer_ptr; > + file= m_file; > + do > + { > + int n_file= file-m_file; > + int is_open= bitmap_is_set(&m_opened_partitions, n_file); > + int should_be_open= bitmap_is_set(&m_part_info->read_partitions, n_file); > + > + if (is_open && !should_be_open) > + (*file)->ha_close(); > + else if (!is_open && should_be_open) > + { > + create_partition_name(name_buff, table->s->normalized_path.str, > + name_buffer_ptr, NORMAL_PART_NAME, FALSE); > + table->s->connect_string = m_connect_string[(uint)(file-m_file)]; > + if ((error= (*file)->ha_open(table, name_buff, m_mode, > + m_open_test_lock | HA_OPEN_NO_PSI_CALL))) > + goto err_handler; > + bzero(&table->s->connect_string, sizeof(LEX_STRING)); > + } > + } while (name_buffer_ptr+= strlen(name_buffer_ptr) + 1, *(++file)); Could you reuse that, instead of copying from ::open? Perhaps ::open() can reset the bitmap to be empty and then call ::change_partitions_to_open() ? Or the common code could be moved into a third function that these both will use... > + > + bitmap_copy(&m_opened_partitions, &m_part_info->read_partitions); > + > + clear_handler_file(); > + > +err_handler: > + return error; > +} > + > + > /** > General function to prepare handler for certain behavior. > > diff --git a/sql/partition_info.cc b/sql/partition_info.cc > index bc0db9e..7fb8f70 100644 > --- a/sql/partition_info.cc > +++ b/sql/partition_info.cc > @@ -283,6 +281,27 @@ bool partition_info::set_partition_bitmaps(TABLE_LIST *table_list) > > > /** > + Set read/lock_partitions bitmap over non pruned partitions > + > + @param table_list Possible TABLE_LIST which can contain > + list of partition names to query > + > + @return Operation status > + @retval FALSE OK > + @retval TRUE Failed to allocate memory for bitmap or list of partitions > + did not match > + > + @note OK to call multiple times without the need for free_bitmaps. > +*/ > +bool partition_info::set_partition_bitmaps_from_table(TABLE_LIST *table_list) > +{ > + List<String> *partition_names= table_list ? > + NULL : table_list->partition_names; > + return set_partition_bitmaps(partition_names); > +} This looks unused. > + > + > +/** > Checks if possible to do prune partitions on insert. > > @param thd Thread context > diff --git a/sql/sql_base.cc b/sql/sql_base.cc > index 9c56b7d..2f9ec3e 100644 > --- a/sql/sql_base.cc > +++ b/sql/sql_base.cc > @@ -2533,6 +2533,12 @@ bool open_table(THD *thd, TABLE_LIST *table_list, Open_table_context *ot_ctx) > { > DBUG_ASSERT(table->file != NULL); > MYSQL_REBIND_TABLE(table->file); > + if (table->part_info) > + { > + /* Set all [named] partitions as used. */ > + if (table->file->change_partitions_to_open(table_list->partition_names)) > + DBUG_RETURN(true); > + } > } > else > { > @@ -2549,7 +2555,8 @@ bool open_table(THD *thd, TABLE_LIST *table_list, Open_table_context *ot_ctx) > HA_TRY_READ_ONLY), > (READ_KEYINFO | COMPUTE_TYPES | > EXTRA_RECORD), > - thd->open_options, table, FALSE); > + thd->open_options, table, FALSE, > + table_list->partition_names); Please, make sure you code compiles without partitioning too. (in this case I wouldn't make the last argument of open_table_from_share conditional, but I'd rather make table_list->partition_names unconditional). > > if (error) > { > @@ -2598,9 +2605,11 @@ bool open_table(THD *thd, TABLE_LIST *table_list, Open_table_context *ot_ctx) > #ifdef WITH_PARTITION_STORAGE_ENGINE > if (table->part_info) > { > +#ifdef CODE_TO_REMOVE > /* Set all [named] partitions as used. */ > if (table->part_info->set_partition_bitmaps(table_list)) > DBUG_RETURN(true); > +#endif /*CODE_TO_REMOVE*/ why not to remove at once? > } > else if (table_list->partition_names) > { Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] 14acb68: MDEV-14593 human-readable XA RECOVER.
by Sergei Golubchik 14 Dec '17

14 Dec '17

Hi, Alexey! On Dec 12, Alexey Botchkov wrote: > revision-id: 14acb687199f0d30444f8c0849c0d1c1284f3e2b (mariadb-10.2.11-26-g14acb68) > parent(s): 4d016e6ed2d32298977a66e125cbfcae39e23847 > committer: Alexey Botchkov > timestamp: 2017-12-12 14:31:41 +0400 > message: > > MDEV-14593 human-readable XA RECOVER. > > The 'data' field in the XA RECOVER resultset changed > to be charset_bin. It seems to me right and also > --binary-as-hex starts working. > The XA RECOVER FORMAT='SQL' option implemented. > It returns the XID string that fits to be an argument for the > XA ... statements. Looks ok, just a couple of comments: > +xa recover format='SQL'; Please add a test for format=SQL (without quotes). We normally document EXPLAIN FORMAT=JSON without quotes (although it works with quotes too), so XA RECOVER should work the same way. > +ER_UNKNOWN_XID_FORMAT > + eng "Unknown XID format name: '%s'" Don't add a new error message, reuse ER_UNKNOWN_EXPLAIN_FORMAT: - eng "Unknown EXPLAIN format name: '%s'" - rus "Неизвестное имя формата команды EXPLAIN: '%s'" + eng "Unknown %s format name: '%s'" + rus "Неизвестное имя формата команды %s: '%s'" note, you'll need either to remove/rephrase the russian error message, or use "XA RECOVER" as the first argument, not "XID". Ok to push (into bb-10.2-ext) with these changes. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] 8048670: MDEV-10715 Galera: Replicate MariaDB GTID to other nodes in the cluster
by Jan Lindström 12 Dec '17

12 Dec '17

On Mon, Dec 11, 2017 at 9:40 AM, sachin <sachin.setiya(a)mariadb.com> wrote: > revision-id: 8048670e25e5cda915ad7ce16dbd66afa6227aa5 > (mariadb-10.1.29-24-g8048670) > parent(s): de76cbdcb0728b80c0f7b453b24b0b81f653e450 > author: Sachin Setiya > committer: Sachin Setiya > timestamp: 2017-12-11 13:05:08 +0530 > message: > > MDEV-10715 Galera: Replicate MariaDB GTID to other nodes in the cluster > > Problem:- Gtid are not transferred in Galera Cluster. > > Solution:- We need to transfer gtid in the case on either when cluster is > slave/master in async replication. In normal Gtid replication gtid are > generated on > recieving node itself and it is always on sync with other nodes. Because > galera keeps > node in sync , So all nodes get same no of event groups. So the issue > arises when > say galera is slave in async replication. > A > | (Async replication) > D <-> E <-> F {Galera replication} > So what should happen is that all node should apply the master gtid but > this does > node happen, becuase node E, F does not recieve gtid from D in write set , > So what E(or F) > does is that it applies wsrep_gtid_domain_id, D server-id , E gtid next > seq no. This > generated gtid does not always work when say A has different domain id. > > So In this commit, on galera node when we see that this event is recieved > from master > we simply write Gtid_Log_Event in write_set and send it to other nodes. > > --- > mysql-test/suite/galera/r/galera_gtid_slave.result | 26 +++ > .../r/galera_gtid_slave_sst_mysqldump.result | 133 +++++++++++++ > .../galera/r/galera_gtid_slave_sst_rsync.result | 130 +++++++++++++ > mysql-test/suite/galera/t/galera_gtid_slave.cnf | 18 ++ > mysql-test/suite/galera/t/galera_gtid_slave.test | 78 ++++++++ > .../galera/t/galera_gtid_slave_sst_mysqldump.cnf | 19 ++ > .../galera/t/galera_gtid_slave_sst_mysqldump.test | 208 > +++++++++++++++++++++ > .../suite/galera/t/galera_gtid_slave_sst_rsync.cnf | 18 ++ > .../galera/t/galera_gtid_slave_sst_rsync.test | 207 > ++++++++++++++++++++ > mysql-test/suite/galera_3nodes/galera_2x3nodes.cnf | 122 ++++++++++++ > .../galera_3nodes/r/galera_gtid_2_cluster.result | 117 ++++++++++++ > .../galera_3nodes/t/galera_gtid_2_cluster.cnf | 28 +++ > .../galera_3nodes/t/galera_gtid_2_cluster.test | 143 ++++++++++++++ > sql/log.cc | 51 ++++- > sql/wsrep_mysqld.cc | 10 + > 15 files changed, 1306 insertions(+), 2 deletions(-) > > diff --git a/mysql-test/suite/galera/r/galera_gtid_slave.result > b/mysql-test/suite/galera/r/galera_gtid_slave.result > new file mode 100644 > index 0000000..40f3f1c > --- /dev/null > +++ b/mysql-test/suite/galera/r/galera_gtid_slave.result > @@ -0,0 +1,26 @@ > +START SLAVE; > +CREATE TABLE t1 (f1 INTEGER PRIMARY KEY) ENGINE=InnoDB; > +CREATE TABLE t2 (f1 INTEGER PRIMARY KEY) ENGINE=InnoDB; > +INSERT INTO t1 VALUES(1); > +begin; > +insert into t2 values(21); > +insert into t2 values(22); > +commit; > +SELECT @@global.gtid_binlog_state; > +@@global.gtid_binlog_state > +1-1-4 > +INSERT INTO t1 VALUES(2); > +INSERT INTO t1 VALUES(3); > +SELECT @@global.gtid_binlog_state; > +@@global.gtid_binlog_state > +1-1-4,2-2-2 > +INSERT INTO t1 VALUES(4); > +SELECT @@global.gtid_binlog_state; > +@@global.gtid_binlog_state > +1-1-4,2-2-2,2-3-3 > +DROP TABLE t1,t2; > +reset master; > +STOP SLAVE; > +RESET SLAVE ALL; > +reset master; > +reset master; > diff --git a/mysql-test/suite/galera/r/galera_gtid_slave_sst_mysqldump.result > b/mysql-test/suite/galera/r/galera_gtid_slave_sst_mysqldump.result > new file mode 100644 > index 0000000..78a0a17 > --- /dev/null > +++ b/mysql-test/suite/galera/r/galera_gtid_slave_sst_mysqldump.result > @@ -0,0 +1,133 @@ > +#Connection 2 > +START SLAVE; > +include/wait_for_slave_to_start.inc > +#Connection 1 > +CREATE TABLE t2 (f1 INTEGER PRIMARY KEY, f2 int unique) ENGINE=InnoDB; > +INSERT INTO t2 VALUES(1,11); > +INSERT INTO t2 VALUES(2,22); > +INSERT INTO t2 VALUES(3,33); > +SELECT @@global.gtid_binlog_state; > +@@global.gtid_binlog_state > +1-1-4 > +include/save_master_gtid.inc > +#Connection 2 > +include/sync_with_master_gtid.inc > +SELECT @@global.gtid_binlog_state; > +@@global.gtid_binlog_state > +1-1-4 > +INSERT INTO t2 VALUES(4,44); > +INSERT INTO t2 VALUES(5,55); > +INSERT INTO t2 VALUES(6,66); > +SELECT @@global.gtid_binlog_state; > +@@global.gtid_binlog_state > +1-1-4,2-2-3 > +#Connection 3 > +INSERT INTO t2 VALUES(7,77); > +INSERT INTO t2 VALUES(8,88); > +SELECT @@global.gtid_binlog_state; > +@@global.gtid_binlog_state > +1-1-4,2-2-3,2-3-5 > +#Connection 1 > +CREATE TABLE t1 (f1 CHAR(255)) ENGINE=InnoDB; > +SET AUTOCOMMIT=OFF; > +START TRANSACTION; > +INSERT INTO t1 VALUES ('node1_committed_before'); > +INSERT INTO t1 VALUES ('node1_committed_before'); > +COMMIT; > +include/save_master_gtid.inc > +#Connection 2 > +include/sync_with_master_gtid.inc > +SET AUTOCOMMIT=OFF; > +START TRANSACTION; > +INSERT INTO t1 VALUES ('node2_committed_before'); > +INSERT INTO t1 VALUES ('node2_committed_before'); > +COMMIT; > +#Connection 3 > +Shutting down server ... > +#Connection 2 > +SET AUTOCOMMIT=OFF; > +START TRANSACTION; > +INSERT INTO t1 VALUES ('node1_committed_during'); > +INSERT INTO t1 VALUES ('node1_committed_during'); > +COMMIT; > +#Connection 3 > +Starting server ... > +SET AUTOCOMMIT=OFF; > +START TRANSACTION; > +INSERT INTO t1 VALUES ('node3_committed_after'); > +INSERT INTO t1 VALUES ('node3_committed_after'); > +COMMIT; > +#Connection 2 > +Select * from t1 order by f1; > +f1 > +node1_committed_before > +node1_committed_before > +node1_committed_during > +node1_committed_during > +node2_committed_before > +node2_committed_before > +node3_committed_after > +node3_committed_after > +#Connection 3 > +Select * from t1 order by f1; > +f1 > +node1_committed_before > +node1_committed_before > +node1_committed_during > +node1_committed_during > +node2_committed_before > +node2_committed_before > +node3_committed_after > +node3_committed_after > +#Connection 2 > +SELECT @@global.gtid_binlog_state; > +@@global.gtid_binlog_state > +1-1-6,2-2-7,2-3-8 > +#Connection 3 > +SELECT @@global.gtid_binlog_state; > +@@global.gtid_binlog_state > +1-1-6,2-2-7,2-3-8 > +#Connection 1 > +SET AUTOCOMMIT=ON; > +#Connection 2 > +SET AUTOCOMMIT=ON; > +#Connection 3 > +SET AUTOCOMMIT=ON; > +#Connection 2 > +STOP slave; > +include/wait_for_slave_to_stop.inc > +INSERT INTO t1 VALUES ('node2_slave_stoped'); > +#Connection 1 > +INSERT INTO t1 VALUES ('node1_normal_entry'); > +include/save_master_gtid.inc > +#Connection 2 > +INSERT INTO t1 VALUES ('node2_slave_stoped_inserted'); > +start slave; > +include/wait_for_slave_to_start.inc > +include/sync_with_master_gtid.inc > +INSERT INTO t1 VALUES ('node2_slave_started'); > +SELECT count(*) from t1; > +count(*) > +12 > +SELECT @@global.gtid_binlog_state; > +@@global.gtid_binlog_state > +1-1-7,2-3-8,2-2-11 > +#Connection 3 > +SELECT count(*) from t1; > +count(*) > +12 > +SELECT @@global.gtid_binlog_state; > +@@global.gtid_binlog_state > +1-1-7,2-3-8,2-2-11 > +#Connection 1 > +DROP TABLE t2,t1; > +#Connection 2 > +#Connection 3 > +#Connection 2 > +STOP SLAVE; > +RESET SLAVE; > +reset master; > +#Connection 3 > +reset master; > +#Connection 1 > +reset master; > diff --git a/mysql-test/suite/galera/r/galera_gtid_slave_sst_rsync.result > b/mysql-test/suite/galera/r/galera_gtid_slave_sst_rsync.result > new file mode 100644 > index 0000000..81fae57 > --- /dev/null > +++ b/mysql-test/suite/galera/r/galera_gtid_slave_sst_rsync.result > @@ -0,0 +1,130 @@ > +#Connection 2 > +START SLAVE; > +#Connection 1 > +CREATE TABLE t2 (f1 INTEGER PRIMARY KEY, f2 int unique) ENGINE=InnoDB; > +INSERT INTO t2 VALUES(1,11); > +INSERT INTO t2 VALUES(2,22); > +INSERT INTO t2 VALUES(3,33); > +SELECT @@global.gtid_binlog_state; > +@@global.gtid_binlog_state > +1-1-4 > +include/save_master_gtid.inc > +#Connection 2 > +include/sync_with_master_gtid.inc > +SELECT @@global.gtid_binlog_state; > +@@global.gtid_binlog_state > +1-1-4 > +INSERT INTO t2 VALUES(4,44); > +INSERT INTO t2 VALUES(5,55); > +INSERT INTO t2 VALUES(6,66); > +SELECT @@global.gtid_binlog_state; > +@@global.gtid_binlog_state > +1-1-4,2-2-3 > +#Connection 3 > +INSERT INTO t2 VALUES(7,77); > +INSERT INTO t2 VALUES(8,88); > +SELECT @@global.gtid_binlog_state; > +@@global.gtid_binlog_state > +1-1-4,2-2-3,2-3-5 > +#Connection 1 > +CREATE TABLE t1 (f1 CHAR(255)) ENGINE=InnoDB; > +SET AUTOCOMMIT=OFF; > +START TRANSACTION; > +INSERT INTO t1 VALUES ('node1_committed_before'); > +INSERT INTO t1 VALUES ('node1_committed_before'); > +COMMIT; > +include/save_master_gtid.inc > +#Connection 2 > +include/sync_with_master_gtid.inc > +SET AUTOCOMMIT=OFF; > +START TRANSACTION; > +INSERT INTO t1 VALUES ('node2_committed_before'); > +INSERT INTO t1 VALUES ('node2_committed_before'); > +COMMIT; > +#Connection 3 > +Shutting down server ... > +#Connection 2 > +SET AUTOCOMMIT=OFF; > +START TRANSACTION; > +INSERT INTO t1 VALUES ('node1_committed_during'); > +INSERT INTO t1 VALUES ('node1_committed_during'); > +COMMIT; > +#Connection 3 > +Starting server ... > +SET AUTOCOMMIT=OFF; > +START TRANSACTION; > +INSERT INTO t1 VALUES ('node3_committed_after'); > +INSERT INTO t1 VALUES ('node3_committed_after'); > +COMMIT; > +#Connection 2 > +Select * from t1 order by f1; > +f1 > +node1_committed_before > +node1_committed_before > +node1_committed_during > +node1_committed_during > +node2_committed_before > +node2_committed_before > +node3_committed_after > +node3_committed_after > +#Connection 3 > +Select * from t1 order by f1; > +f1 > +node1_committed_before > +node1_committed_before > +node1_committed_during > +node1_committed_during > +node2_committed_before > +node2_committed_before > +node3_committed_after > +node3_committed_after > +#Connection 2 > +SELECT @@global.gtid_binlog_state; > +@@global.gtid_binlog_state > +1-1-6,2-2-7,2-3-8 > +#Connection 3 > +SELECT @@global.gtid_binlog_state; > +@@global.gtid_binlog_state > +1-1-6,2-2-7,2-3-8 > +#Connection 1 > +SET AUTOCOMMIT=ON; > +#Connection 2 > +SET AUTOCOMMIT=ON; > +#Connection 3 > +SET AUTOCOMMIT=ON; > +#Connection 2 > +STOP slave; > +INSERT INTO t1 VALUES ('node2_slave_stoped'); > +#Connection 1 > +INSERT INTO t1 VALUES ('node1_normal_entry'); > +include/save_master_gtid.inc > +#Connection 2 > +INSERT INTO t1 VALUES ('node2_slave_stoped_inserted'); > +start slave; > +include/sync_with_master_gtid.inc > +INSERT INTO t1 VALUES ('node2_slave_started'); > +SELECT count(*) from t1; > +count(*) > +12 > +SELECT @@global.gtid_binlog_state; > +@@global.gtid_binlog_state > +1-1-7,2-3-8,2-2-11 > +#Connection 3 > +SELECT count(*) from t1; > +count(*) > +12 > +SELECT @@global.gtid_binlog_state; > +@@global.gtid_binlog_state > +1-1-7,2-3-8,2-2-11 > +#Connection 1 > +DROP TABLE t2,t1; > +#Connection 2 > +#Connection 3 > +#Connection 2 > +STOP SLAVE; > +RESET SLAVE ALL; > +reset master; > +#Connection 3 > +reset master; > +#Connection 1 > +reset master; > diff --git a/mysql-test/suite/galera/t/galera_gtid_slave.cnf > b/mysql-test/suite/galera/t/galera_gtid_slave.cnf > new file mode 100644 > index 0000000..409d0d1 > --- /dev/null > +++ b/mysql-test/suite/galera/t/galera_gtid_slave.cnf > @@ -0,0 +1,18 @@ > +!include ../galera_2nodes_as_slave.cnf > + > +[mysqld] > +log-bin=mysqld-bin > +log-slave-updates > +binlog-format=ROW > + > +[mysqld.1] > +gtid-domain-id=1 > +[mysqld.2] > +gtid-domain-id=2 > +wsrep_gtid_mode=1 > +wsrep_gtid_domain_id=2 > +[mysqld.3] > +gtid-domain-id=2 > +wsrep_gtid_mode=1 > +wsrep_gtid_domain_id=2 > + > diff --git a/mysql-test/suite/galera/t/galera_gtid_slave.test > b/mysql-test/suite/galera/t/galera_gtid_slave.test > new file mode 100644 > index 0000000..8ef8745 > --- /dev/null > +++ b/mysql-test/suite/galera/t/galera_gtid_slave.test > @@ -0,0 +1,78 @@ > +# > +# Test Galera as a slave to a MariaDB master using GTIDs > +# > +# suite/galera/galera_2nodes_as_slave.cnf describes the setup of the > nodes > +# suite/galera/t/galera_as_slave_gtid.cnf has the GTID options > +# > +# In addition to performing DDL and DML, we check that the gtid of the > master is preserved inside the cluster > +# > + > +--source include/have_innodb.inc > + > +# As node #1 is not a Galera node, we connect to node #2 in order to run > include/galera_cluster.inc > +--connect node_2a, 127.0.0.1, root, , test, $NODE_MYPORT_2 > +--source include/galera_cluster.inc > + > +--connection node_2 > +--disable_query_log > +--eval CHANGE MASTER TO MASTER_HOST='127.0.0.1', MASTER_USER='root', > MASTER_PORT=$NODE_MYPORT_1; > +--enable_query_log > +START SLAVE; > + > +--connection node_1 > +CREATE TABLE t1 (f1 INTEGER PRIMARY KEY) ENGINE=InnoDB; > +CREATE TABLE t2 (f1 INTEGER PRIMARY KEY) ENGINE=InnoDB; > +INSERT INTO t1 VALUES(1); > + > +#multi stmt trans > +begin; > +insert into t2 values(21); > +insert into t2 values(22); > +commit; > + > +SELECT @@global.gtid_binlog_state; > + > +--connection node_2 > +--let $wait_condition = SELECT COUNT(*) = 1 FROM > INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME = 't1'; > +--source include/wait_condition.inc > + > +--let $wait_condition = SELECT COUNT(*) = 1 FROM t1; > +--source include/wait_condition.inc > +--sleep 1 > +INSERT INTO t1 VALUES(2); > +INSERT INTO t1 VALUES(3); > +SELECT @@global.gtid_binlog_state; > + > +--connect node_3, 127.0.0.1, root, , test, $NODE_MYPORT_3 > +--let $wait_condition = SELECT COUNT(*) = 3 FROM t1; > +--source include/wait_condition.inc > + > +INSERT INTO t1 VALUES(4); > +SELECT @@global.gtid_binlog_state; > + > +--connection node_1 > +DROP TABLE t1,t2; > +reset master; > +# > +# Unfortunately without the sleep below the following statement fails > with "query returned no rows", which > +# is difficult to understand given that it is an aggregate query. A > "query execution was interrupted" > +# warning is also reported by MTR, which is also weird. > +# > + > +--sleep 1 > + > +--connection node_2 > +--let $wait_condition = SELECT COUNT(*) = 0 FROM > INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME = 't1'; > +--source include/wait_condition.inc > + > +--connection node_3 > +--let $wait_condition = SELECT COUNT(*) = 0 FROM > INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME = 't1'; > +--source include/wait_condition.inc > + > +--connection node_2 > +STOP SLAVE; > +RESET SLAVE ALL; > +reset master; > + > +--connection node_3 > +reset master; > diff --git a/mysql-test/suite/galera/t/galera_gtid_slave_sst_mysqldump.cnf > b/mysql-test/suite/galera/t/galera_gtid_slave_sst_mysqldump.cnf > new file mode 100644 > index 0000000..0b4a776 > --- /dev/null > +++ b/mysql-test/suite/galera/t/galera_gtid_slave_sst_mysqldump.cnf > @@ -0,0 +1,19 @@ > +!include ../galera_2nodes_as_slave.cnf > + > +[mysqld] > +log-bin=mysqld-bin > +log-slave-updates > +binlog-format=ROW > +wsrep_sst_method=xtrabackup > + > +[mysqld.1] > +gtid-domain-id=1 > +[mysqld.2] > +gtid-domain-id=2 > +wsrep_gtid_mode=1 > +wsrep_gtid_domain_id=2 > +[mysqld.3] > +gtid-domain-id=2 > +wsrep_gtid_mode=1 > +wsrep_gtid_domain_id=2 > + > diff --git a/mysql-test/suite/galera/t/galera_gtid_slave_sst_mysqldump.test > b/mysql-test/suite/galera/t/galera_gtid_slave_sst_mysqldump.test > new file mode 100644 > index 0000000..b507698 > --- /dev/null > +++ b/mysql-test/suite/galera/t/galera_gtid_slave_sst_mysqldump.test > @@ -0,0 +1,208 @@ > +# > +# Test Galera as a slave to a MariaDB master using GTIDs > +# > +# suite/galera/galera_2nodes_as_slave.cnf describes the setup of the > nodes > +# > +# In addition to performing DDL and DML, we check that the gtid of the > master is preserved inside the cluster > +# > + > +--source include/big_test.inc > +--source include/have_innodb.inc > +# As node #1 is not a Galera node, we connect to node #2 in order to run > include/galera_cluster.inc > +--connect node_2a, 127.0.0.1, root, , test, $NODE_MYPORT_2 > +--source include/galera_cluster.inc > + > +--echo #Connection 2 > +--connection node_2 > +--disable_query_log > +--eval CHANGE MASTER TO MASTER_HOST='127.0.0.1', MASTER_USER='root', > MASTER_PORT=$NODE_MYPORT_1,master_use_gtid=slave_pos; > +--enable_query_log > +START SLAVE; > +--source include/wait_for_slave_to_start.inc > + > +--echo #Connection 1 > +--connection node_1 > +CREATE TABLE t2 (f1 INTEGER PRIMARY KEY, f2 int unique) ENGINE=InnoDB; > +INSERT INTO t2 VALUES(1,11); > +INSERT INTO t2 VALUES(2,22); > +INSERT INTO t2 VALUES(3,33); > + > +SELECT @@global.gtid_binlog_state; > +--source include/save_master_gtid.inc > + > +--echo #Connection 2 > +--connection node_2 > +--source include/sync_with_master_gtid.inc > +SELECT @@global.gtid_binlog_state; > +INSERT INTO t2 VALUES(4,44); > +INSERT INTO t2 VALUES(5,55); > +INSERT INTO t2 VALUES(6,66); > +SELECT @@global.gtid_binlog_state; > + > +--echo #Connection 3 > +--connect node_3, 127.0.0.1, root, , test, $NODE_MYPORT_3 > +--let $wait_condition = SELECT COUNT(*) = 1 FROM > INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME= 't2'; > +--source include/wait_condition.inc > +--let $wait_condition = SELECT COUNT(*) = 6 FROM t2; > +--source include/wait_condition.inc > + > +INSERT INTO t2 VALUES(7,77); > +INSERT INTO t2 VALUES(8,88); > +SELECT @@global.gtid_binlog_state; > + > +#Perform SST > +--echo #Connection 1 > +--connection node_1 > +CREATE TABLE t1 (f1 CHAR(255)) ENGINE=InnoDB; > +SET AUTOCOMMIT=OFF; > +START TRANSACTION; > +INSERT INTO t1 VALUES ('node1_committed_before'); > +INSERT INTO t1 VALUES ('node1_committed_before'); > +COMMIT; > +--source include/save_master_gtid.inc > + > +--echo #Connection 2 > +--connection node_2 > +--source include/sync_with_master_gtid.inc > +SET AUTOCOMMIT=OFF; > +START TRANSACTION; > +INSERT INTO t1 VALUES ('node2_committed_before'); > +INSERT INTO t1 VALUES ('node2_committed_before'); > +COMMIT; > + > +--echo #Connection 3 > +--connection node_3 > +--let $wait_condition = SELECT COUNT(*) = 1 FROM > INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME= 't1'; > +--source include/wait_condition.inc > +--let $wait_condition = SELECT COUNT(*) = 4 FROM t1; > +--source include/wait_condition.inc > +--let $node_1= node_2 > +--let $node_2= node_3 > +--source include/auto_increment_offset_save.inc > +--echo Shutting down server ... > +--source include/shutdown_mysqld.inc > + > + > +--echo #Connection 2 > +--connection node_2 > +--let $wait_condition = SELECT VARIABLE_VALUE = 1 FROM > INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = > 'wsrep_cluster_size' > +--source include/wait_condition.inc > +SET AUTOCOMMIT=OFF; > +START TRANSACTION; > +INSERT INTO t1 VALUES ('node1_committed_during'); > +INSERT INTO t1 VALUES ('node1_committed_during'); > +COMMIT; > + > +--echo #Connection 3 > +--connection node_3 > +--echo Starting server ... > +--source include/start_mysqld.inc > +--source include/wait_until_ready.inc > +--source include/auto_increment_offset_restore.inc > + > +SET AUTOCOMMIT=OFF; > +START TRANSACTION; > +INSERT INTO t1 VALUES ('node3_committed_after'); > +INSERT INTO t1 VALUES ('node3_committed_after'); > +COMMIT; > + > +--echo #Connection 2 > +--connection node_2 > +--let $wait_condition = SELECT COUNT(*) = 8 FROM t1; > +--source include/wait_condition.inc > +Select * from t1 order by f1; > + > +--echo #Connection 3 > +--connection node_3 > +Select * from t1 order by f1; > + > +#SST Done > +--sleep 1 > +--echo #Connection 2 > +--connection node_2 > +SELECT @@global.gtid_binlog_state; > + > +--echo #Connection 3 > +--connection node_3 > +SELECT @@global.gtid_binlog_state; > + > +--echo #Connection 1 > +--connection node_1 > +SET AUTOCOMMIT=ON; > +#drop table t1; > +#CREATE TABLE t1 (f1 CHAR(255)) ENGINE=InnoDB; > + > +--echo #Connection 2 > +--connection node_2 > +SET AUTOCOMMIT=ON; > +--echo #Connection 3 > +--connection node_3 > +SET AUTOCOMMIT=ON; > + > +# > +#stop slave on node 2 > +--echo #Connection 2 > +--connection node_2 > +STOP slave; > +--source include/wait_for_slave_to_stop.inc > +--sleep 1 > +INSERT INTO t1 VALUES ('node2_slave_stoped'); > + > +--echo #Connection 1 > +--connection node_1 > +INSERT INTO t1 VALUES ('node1_normal_entry'); > +--source include/save_master_gtid.inc > + > +#start slave > +--echo #Connection 2 > +--connection node_2 > +INSERT INTO t1 VALUES ('node2_slave_stoped_inserted'); > +start slave; > +--source include/wait_for_slave_to_start.inc > +--source include/sync_with_master_gtid.inc > +INSERT INTO t1 VALUES ('node2_slave_started'); > +SELECT count(*) from t1; > +SELECT @@global.gtid_binlog_state; > + > +--echo #Connection 3 > +--connection node_3 > +--let $wait_condition = SELECT COUNT(*) = 12 FROM t1; > +--source include/wait_condition.inc > +SELECT count(*) from t1; > +SELECT @@global.gtid_binlog_state; > + > +--echo #Connection 1 > +--connection node_1 > +DROP TABLE t2,t1; > + > +# Unfortunately without the sleep below the following statement fails > with "query returned no rows", which > +# is difficult to understand given that it is an aggregate query. A > "query execution was interrupted" > +# warning is also reported by MTR, which is also weird. > +# > + > +--sleep 3 > + > +--echo #Connection 2 > +--connection node_2 > +--let $wait_condition = SELECT COUNT(*) = 0 FROM > INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME = 't2'; > +--source include/wait_condition.inc > + > +--echo #Connection 3 > +--connection node_3 > +--let $wait_condition = SELECT COUNT(*) = 0 FROM > INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME = 't1'; > +--source include/wait_condition.inc > + > +--echo #Connection 2 > +--connection node_2 > +STOP SLAVE; > +RESET SLAVE; > +reset master; > + > +--echo #Connection 3 > +--connection node_3 > +reset master; > + > +--echo #Connection 1 > +--connection node_1 > +reset master; > + > diff --git a/mysql-test/suite/galera/t/galera_gtid_slave_sst_rsync.cnf > b/mysql-test/suite/galera/t/galera_gtid_slave_sst_rsync.cnf > new file mode 100644 > index 0000000..bb9c8e8 > --- /dev/null > +++ b/mysql-test/suite/galera/t/galera_gtid_slave_sst_rsync.cnf > @@ -0,0 +1,18 @@ > +!include ../galera_2nodes_as_slave.cnf > + > +[mysqld] > +log-bin=mysqld-bin > +log-slave-updates > +binlog-format=ROW > +wsrep_sst_method=rsync > +[mysqld.1] > +gtid-domain-id=1 > +[mysqld.2] > +gtid-domain-id=2 > +wsrep_gtid_mode=1 > +wsrep_gtid_domain_id=2 > +[mysqld.3] > +gtid-domain-id=2 > +wsrep_gtid_mode=1 > +wsrep_gtid_domain_id=2 > + > diff --git a/mysql-test/suite/galera/t/galera_gtid_slave_sst_rsync.test > b/mysql-test/suite/galera/t/galera_gtid_slave_sst_rsync.test > new file mode 100644 > index 0000000..3fe94ad > --- /dev/null > +++ b/mysql-test/suite/galera/t/galera_gtid_slave_sst_rsync.test > @@ -0,0 +1,207 @@ > +# > +# Test Galera as a slave to a MariaDB master using GTIDs > +# > +# suite/galera/galera_2nodes_as_slave.cnf describes the setup of the > nodes > +# > +# In addition to performing DDL and DML, we check that the gtid of the > master is preserved inside the cluster > +# > + > +--source include/big_test.inc > +--source include/have_innodb.inc > +# As node #1 is not a Galera node, we connect to node #2 in order to run > include/galera_cluster.inc > +--connect node_2a, 127.0.0.1, root, , test, $NODE_MYPORT_2 > +--source include/galera_cluster.inc > + > +--echo #Connection 2 > +--connection node_2 > +--disable_query_log > +--eval CHANGE MASTER TO MASTER_HOST='127.0.0.1', MASTER_USER='root', > MASTER_PORT=$NODE_MYPORT_1,master_use_gtid=slave_pos; > +--enable_query_log > +START SLAVE; > +--sleep 1 > + > + > +--echo #Connection 1 > +--connection node_1 > +CREATE TABLE t2 (f1 INTEGER PRIMARY KEY, f2 int unique) ENGINE=InnoDB; > +INSERT INTO t2 VALUES(1,11); > +INSERT INTO t2 VALUES(2,22); > +INSERT INTO t2 VALUES(3,33); > + > +SELECT @@global.gtid_binlog_state; > +--source include/save_master_gtid.inc > + > +--echo #Connection 2 > +--connection node_2 > +--source include/sync_with_master_gtid.inc > +SELECT @@global.gtid_binlog_state; > + > +INSERT INTO t2 VALUES(4,44); > +INSERT INTO t2 VALUES(5,55); > +INSERT INTO t2 VALUES(6,66); > +SELECT @@global.gtid_binlog_state; > + > +--echo #Connection 3 > +--connect node_3, 127.0.0.1, root, , test, $NODE_MYPORT_3 > +--let $wait_condition = SELECT COUNT(*) = 1 FROM > INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME= 't2'; > +--source include/wait_condition.inc > +--let $wait_condition = SELECT COUNT(*) = 6 FROM t2; > +--source include/wait_condition.inc > + > +INSERT INTO t2 VALUES(7,77); > +INSERT INTO t2 VALUES(8,88); > +SELECT @@global.gtid_binlog_state; > + > +#Perform SST > +--echo #Connection 1 > +--connection node_1 > +CREATE TABLE t1 (f1 CHAR(255)) ENGINE=InnoDB; > +SET AUTOCOMMIT=OFF; > +START TRANSACTION; > +INSERT INTO t1 VALUES ('node1_committed_before'); > +INSERT INTO t1 VALUES ('node1_committed_before'); > +COMMIT; > +--source include/save_master_gtid.inc > + > +--echo #Connection 2 > +--connection node_2 > +--source include/sync_with_master_gtid.inc > +SET AUTOCOMMIT=OFF; > +START TRANSACTION; > +INSERT INTO t1 VALUES ('node2_committed_before'); > +INSERT INTO t1 VALUES ('node2_committed_before'); > +COMMIT; > + > +--echo #Connection 3 > +--connection node_3 > +--let $wait_condition = SELECT COUNT(*) = 1 FROM > INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME= 't1'; > +--source include/wait_condition.inc > +--let $wait_condition = SELECT COUNT(*) = 4 FROM t1; > +--source include/wait_condition.inc > +--let $node_1= node_2 > +--let $node_2= node_3 > +--source include/auto_increment_offset_save.inc > +--echo Shutting down server ... > +--source include/shutdown_mysqld.inc > + > + > +--echo #Connection 2 > +--connection node_2 > +--let $wait_condition = SELECT VARIABLE_VALUE = 1 FROM > INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = > 'wsrep_cluster_size' > +--source include/wait_condition.inc > +SET AUTOCOMMIT=OFF; > +START TRANSACTION; > +INSERT INTO t1 VALUES ('node1_committed_during'); > +INSERT INTO t1 VALUES ('node1_committed_during'); > +COMMIT; > + > +--echo #Connection 3 > +--connection node_3 > +--echo Starting server ... > +--source include/start_mysqld.inc > +--source include/wait_until_ready.inc > +--source include/auto_increment_offset_restore.inc > + > +SET AUTOCOMMIT=OFF; > +START TRANSACTION; > +INSERT INTO t1 VALUES ('node3_committed_after'); > +INSERT INTO t1 VALUES ('node3_committed_after'); > +COMMIT; > + > +--echo #Connection 2 > +--connection node_2 > +--let $wait_condition = SELECT COUNT(*) = 8 FROM t1; > +--source include/wait_condition.inc > +Select * from t1 order by f1; > + > +--echo #Connection 3 > +--connection node_3 > +Select * from t1 order by f1; > + > +#SST Done > +--sleep 1 > +--echo #Connection 2 > +--connection node_2 > +SELECT @@global.gtid_binlog_state; > + > +--echo #Connection 3 > +--connection node_3 > +SELECT @@global.gtid_binlog_state; > + > +--echo #Connection 1 > +--connection node_1 > +SET AUTOCOMMIT=ON; > +#drop table t1; > +#CREATE TABLE t1 (f1 CHAR(255)) ENGINE=InnoDB; > + > +--echo #Connection 2 > +--connection node_2 > +SET AUTOCOMMIT=ON; > +--echo #Connection 3 > +--connection node_3 > +SET AUTOCOMMIT=ON; > + > +# > +#stop slave on node 2 > +--echo #Connection 2 > +--connection node_2 > +STOP slave; > +--sleep 1 > +INSERT INTO t1 VALUES ('node2_slave_stoped'); > + > +--echo #Connection 1 > +--connection node_1 > +INSERT INTO t1 VALUES ('node1_normal_entry'); > +--source include/save_master_gtid.inc > + > +#start slave > +--echo #Connection 2 > +--connection node_2 > +INSERT INTO t1 VALUES ('node2_slave_stoped_inserted'); > +start slave; > +--source include/sync_with_master_gtid.inc > +INSERT INTO t1 VALUES ('node2_slave_started'); > +SELECT count(*) from t1; > +SELECT @@global.gtid_binlog_state; > + > +--echo #Connection 3 > +--connection node_3 > +--let $wait_condition = SELECT COUNT(*) = 12 FROM t1; > +--source include/wait_condition.inc > +SELECT count(*) from t1; > +SELECT @@global.gtid_binlog_state; > + > +--echo #Connection 1 > +--connection node_1 > +DROP TABLE t2,t1; > + > +# Unfortunately without the sleep below the following statement fails > with "query returned no rows", which > +# is difficult to understand given that it is an aggregate query. A > "query execution was interrupted" > +# warning is also reported by MTR, which is also weird. > +# > + > +--sleep 3 > + > +--echo #Connection 2 > +--connection node_2 > +--let $wait_condition = SELECT COUNT(*) = 0 FROM > INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME = 't2'; > +--source include/wait_condition.inc > + > +--echo #Connection 3 > +--connection node_3 > +--let $wait_condition = SELECT COUNT(*) = 0 FROM > INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME = 't1'; > +--source include/wait_condition.inc > + > +--echo #Connection 2 > +--connection node_2 > +STOP SLAVE; > +RESET SLAVE ALL; > +reset master; > + > +--echo #Connection 3 > +--connection node_3 > +reset master; > + > +--echo #Connection 1 > +--connection node_1 > +reset master; > diff --git a/mysql-test/suite/galera_3nodes/galera_2x3nodes.cnf > b/mysql-test/suite/galera_3nodes/galera_2x3nodes.cnf > new file mode 100644 > index 0000000..3f39b82 > --- /dev/null > +++ b/mysql-test/suite/galera_3nodes/galera_2x3nodes.cnf > @@ -0,0 +1,122 @@ > +# Use default setting for mysqld processes > +!include include/default_mysqld.cnf > + > +[mysqld] > +log-bin > +log-slave-updates > +binlog-format=row > +innodb-autoinc-lock-mode=2 > +default-storage-engine=innodb > +wsrep_gtid_mode=1 > +gtid_ignore_duplicates > + > +wsrep-on=1 > +wsrep-provider=(a)ENV.WSREP_PROVIDER > +wsrep_node_address=127.0.0.1 > +# enforce read-committed characteristics across the cluster > +# wsrep-causal-reads=ON > +# wsrep-sync-wait=15 > + > +[mysqld.1] > +#galera_port=(a)OPT.port > +#ist_port=(a)OPT.port > +#sst_port=(a)OPT.port > +wsrep-cluster-address='gcomm://' > +wsrep_provider_options='base_port=(a)mysqld.1.#galera_port; > evs.suspect_timeout=PT300S;evs.inactive_timeout=PT1000M; > evs.install_timeout=PT155S;evs.keepalive_period = PT100S' > + > +wsrep_sst_receive_address=127.0.0.2:@mysqld.1.#sst_port > +wsrep_node_incoming_address=127.0.0.1:@mysqld.1.port > +wsrep_sst_receive_address='127.0.0.1:@mysqld.1.#sst_port' > + > +[mysqld.2] > +#galera_port=(a)OPT.port > +#ist_port=(a)OPT.port > +#sst_port=(a)OPT.port > +wsrep_cluster_address='gcomm://127.0.0.1:@mysqld.1.#galera_port' > +wsrep_provider_options='base_port=(a)mysqld.2.#galera_port; > evs.suspect_timeout=PT300S;evs.inactive_timeout=PT1000M; > evs.install_timeout=PT155S;evs.keepalive_period = PT100S' > + > +wsrep_sst_receive_address=127.0.0.2:@mysqld.2.#sst_port > +wsrep_node_incoming_address=127.0.0.1:@mysqld.2.port > +wsrep_sst_receive_address='127.0.0.1:@mysqld.2.#sst_port' > + > +[mysqld.3] > +#galera_port=(a)OPT.port > +#ist_port=(a)OPT.port > +#sst_port=(a)OPT.port > +wsrep_cluster_address='gcomm://127.0.0.1:@mysqld.1.#galera_port' > +wsrep_provider_options='base_port=(a)mysqld.3.#galera_port; > evs.suspect_timeout=PT300S;evs.inactive_timeout=PT1000M; > evs.install_timeout=PT155S;evs.keepalive_period = PT100S' > + > +wsrep_sst_receive_address=127.0.0.2:@mysqld.3.#sst_port > +wsrep_node_incoming_address=127.0.0.1:@mysqld.3.port > +wsrep_sst_receive_address='127.0.0.1:@mysqld.3.#sst_port' > + > + > +[mysqld.4] > +wsrep_cluster_name=cluster2 > +#galera_port=(a)OPT.port > +#ist_port=(a)OPT.port > +#sst_port=(a)OPT.port > + > +wsrep-cluster-address='gcomm://' > +wsrep_provider_options='base_port=(a)mysqld.4.#galera_port; > evs.suspect_timeout=PT300S;evs.inactive_timeout=PT1000M; > evs.install_timeout=PT155S;evs.keepalive_period = PT100S' > + > +wsrep_sst_receive_address=127.0.0.2:@mysqld.4.#sst_port > +wsrep_node_incoming_address=127.0.0.1:@mysqld.4.port > +wsrep_sst_receive_address='127.0.0.1:@mysqld.4.#sst_port' > + > +[mysqld.5] > +wsrep_cluster_name=cluster2 > +#galera_port=(a)OPT.port > +#ist_port=(a)OPT.port > +#sst_port=(a)OPT.port > +wsrep_cluster_address='gcomm://127.0.0.1:@mysqld.4.#galera_port' > +wsrep_provider_options='base_port=(a)mysqld.5.#galera_port; > evs.suspect_timeout=PT300S;evs.inactive_timeout=PT1000M; > evs.install_timeout=PT155S;evs.keepalive_period = PT100S' > + > +wsrep_sst_receive_address=127.0.0.2:@mysqld.5.#sst_port > +wsrep_node_incoming_address=127.0.0.1:@mysqld.5.port > +wsrep_sst_receive_address='127.0.0.1:@mysqld.5.#sst_port' > + > +[mysqld.6] > +wsrep_cluster_name=cluster2 > +#galera_port=(a)OPT.port > +#ist_port=(a)OPT.port > +#sst_port=(a)OPT.port > +wsrep_cluster_address='gcomm://127.0.0.1:@mysqld.4.#galera_port' > +wsrep_provider_options='base_port=(a)mysqld.6.#galera_port; > evs.suspect_timeout=PT300S;evs.inactive_timeout=PT1000M; > evs.install_timeout=PT155S;evs.keepalive_period = PT100S' > + > +wsrep_sst_receive_address=127.0.0.2:@mysqld.6.#sst_port > +wsrep_node_incoming_address=127.0.0.1:@mysqld.6.port > +wsrep_sst_receive_address='127.0.0.1:@mysqld.6.#sst_port' > + > +[ENV] > +NODE_MYPORT_1= @mysqld.1.port > +NODE_MYSOCK_1= @mysqld.1.socket > + > +NODE_MYPORT_2= @mysqld.2.port > +NODE_MYSOCK_2= @mysqld.2.socket > + > +NODE_MYPORT_3= @mysqld.3.port > +NODE_MYSOCK_3= @mysqld.3.socket > + > +NODE_MYPORT_4= @mysqld.4.port > +NODE_MYSOCK_4= @mysqld.4.socket > + > +NODE_MYPORT_5= @mysqld.5.port > +NODE_MYSOCK_5= @mysqld.5.socket > + > +NODE_MYPORT_6= @mysqld.6.port > +NODE_MYSOCK_6= @mysqld.6.socket > + > +NODE_GALERAPORT_1= @mysqld.1.#galera_port > +NODE_GALERAPORT_2= @mysqld.2.#galera_port > +NODE_GALERAPORT_3= @mysqld.3.#galera_port > +NODE_GALERAPORT_4= @mysqld.4.#galera_port > +NODE_GALERAPORT_5= @mysqld.5.#galera_port > +NODE_GALERAPORT_6= @mysqld.6.#galera_port > + > +NODE_SSTPORT_1= @mysqld.1.#sst_port > +NODE_SSTPORT_2= @mysqld.2.#sst_port > +NODE_SSTPORT_3= @mysqld.3.#sst_port > +NODE_SSTPORT_4= @mysqld.4.#sst_port > +NODE_SSTPORT_5= @mysqld.5.#sst_port > +NODE_SSTPORT_6= @mysqld.6.#sst_port > diff --git a/mysql-test/suite/galera_3nodes/r/galera_gtid_2_cluster.result > b/mysql-test/suite/galera_3nodes/r/galera_gtid_2_cluster.result > new file mode 100644 > index 0000000..52c82ab > --- /dev/null > +++ b/mysql-test/suite/galera_3nodes/r/galera_gtid_2_cluster.result > @@ -0,0 +1,117 @@ > +cluster 1 node 1 > +SHOW STATUS LIKE 'wsrep_cluster_size'; > +Variable_name Value > +wsrep_cluster_size 3 > +cluster 1 node 2 > +SHOW STATUS LIKE 'wsrep_cluster_size'; > +Variable_name Value > +wsrep_cluster_size 3 > +cluster 1 node 3 > +SHOW STATUS LIKE 'wsrep_cluster_size'; > +Variable_name Value > +wsrep_cluster_size 3 > +cluster 2 node 1 > +SHOW STATUS LIKE 'wsrep_cluster_size'; > +Variable_name Value > +wsrep_cluster_size 3 > +cluster 2 node 2 > +SHOW STATUS LIKE 'wsrep_cluster_size'; > +Variable_name Value > +wsrep_cluster_size 3 > +cluster 2 node 3 > +SHOW STATUS LIKE 'wsrep_cluster_size'; > +Variable_name Value > +wsrep_cluster_size 3 > +change master to master_host='127.0.0.1', master_user='root', > master_port=16003, master_use_gtid=current_pos, ignore_server_ids=(12,13);; > +start slave; > +include/wait_for_slave_to_start.inc > +select @@gtid_binlog_state; > +@@gtid_binlog_state > + > +select @@gtid_slave_pos; > +@@gtid_slave_pos > + > +change master to master_host='127.0.0.1', master_user='root', > master_port=16000, master_use_gtid=current_pos, ignore_server_ids=(22,23);; > +start slave; > +include/wait_for_slave_to_start.inc > +select @@gtid_binlog_state; > +@@gtid_binlog_state > + > +select @@gtid_slave_pos; > +@@gtid_slave_pos > + > +cluster 1 node 1 > +create table t1 (cluster_domain_id int ,node_server_id int, seq_no int); > +insert into t1 values (1, 11, 2); > +select @@gtid_binlog_state; > +@@gtid_binlog_state > +1-11-2 > +#wait for sync cluster 1 and 2 > +include/save_master_gtid.inc > +include/sync_with_master_gtid.inc > +cluster 2 node 1 > +insert into t1 values (2, 21, 1); > +select @@gtid_binlog_state; > +@@gtid_binlog_state > +1-11-2,2-21-1 > +select * from t1; > +cluster_domain_id node_server_id seq_no > +1 11 2 > +2 21 1 > +#wait for sync cluster 1 and 2 > +include/save_master_gtid.inc > +include/sync_with_master_gtid.inc > +cluster 1 node 2 > +select @@gtid_binlog_state; > +@@gtid_binlog_state > +1-11-2,2-21-1 > +insert into t1 values (1, 12, 3); > +select @@gtid_binlog_state; > +@@gtid_binlog_state > +1-11-2,1-12-3,2-21-1 > +#wait for sync cluster 1 and 2 > +include/save_master_gtid.inc > +include/sync_with_master_gtid.inc > +cluster 1 node 3 > +select @@gtid_binlog_state; > +@@gtid_binlog_state > +1-11-2,1-12-3,2-21-1 > +insert into t1 values (1, 13, 4); > +select @@gtid_binlog_state; > +@@gtid_binlog_state > +1-12-3,1-11-2,1-13-4,2-21-1 > +#wait for sync cluster 1 and 2 > +include/save_master_gtid.inc > +include/sync_with_master_gtid.inc > +cluster 2 node 2 > +select @@gtid_binlog_state; > +@@gtid_binlog_state > +1-12-3,1-11-2,1-13-4,2-21-1 > +insert into t1 values (2, 22, 2); > +select @@gtid_binlog_state; > +@@gtid_binlog_state > +1-12-3,1-11-2,1-13-4,2-21-1,2-22-2 > +#wait for sync cluster 2 and 1 > +include/save_master_gtid.inc > +include/sync_with_master_gtid.inc > +cluster 2 node 3 > +select @@gtid_binlog_state; > +@@gtid_binlog_state > +1-12-3,1-11-2,1-13-4,2-21-1,2-22-2 > +insert into t1 values (2, 23, 3); > +select @@gtid_binlog_state; > +@@gtid_binlog_state > +1-12-3,1-11-2,1-13-4,2-21-1,2-22-2,2-23-3 > +#wait for sync cluster 2 and 1 > +include/save_master_gtid.inc > +include/sync_with_master_gtid.inc > +cluster 1 node 1 > +select @@gtid_binlog_state; > +@@gtid_binlog_state > +1-12-3,1-11-2,1-13-4,2-21-1,2-22-2,2-23-3 > +drop table t1; > +stop slave; > +change master to master_use_gtid=no, ignore_server_ids=(); > +cluster 2 node 1 > +stop slave; > +change master to master_use_gtid=no, ignore_server_ids=(); > diff --git a/mysql-test/suite/galera_3nodes/t/galera_gtid_2_cluster.cnf > b/mysql-test/suite/galera_3nodes/t/galera_gtid_2_cluster.cnf > new file mode 100644 > index 0000000..dc5535e > --- /dev/null > +++ b/mysql-test/suite/galera_3nodes/t/galera_gtid_2_cluster.cnf > @@ -0,0 +1,28 @@ > +# We need a dedicated .cnf file, even if empty, in order to force this > test to run > +# alone on a freshly started cluster. Otherwise there are adverse > interactions with > +# following tests such as galera_3nodes.galera_var_dirty_reads2 > + > +!include ../galera_2x3nodes.cnf > +[mysqld.1] > +wsrep_gtid_domain_id=1 > +server-id=11 > + > +[mysqld.2] > +wsrep_gtid_domain_id=1 > +server-id=12 > + > +[mysqld.3] > +wsrep_gtid_domain_id=1 > +server-id=13 > + > +[mysqld.4] > +wsrep_gtid_domain_id=2 > +server-id=21 > + > +[mysqld.5] > +wsrep_gtid_domain_id=2 > +server-id=22 > + > +[mysqld.6] > +wsrep_gtid_domain_id=2 > +server-id=23 > diff --git a/mysql-test/suite/galera_3nodes/t/galera_gtid_2_cluster.test > b/mysql-test/suite/galera_3nodes/t/galera_gtid_2_cluster.test > new file mode 100644 > index 0000000..4c9e611 > --- /dev/null > +++ b/mysql-test/suite/galera_3nodes/t/galera_gtid_2_cluster.test > @@ -0,0 +1,143 @@ > +# > +# This test creates 2x 3 nodes galera cluster. > +# The whole test case > +# A <-> B <-> C {Galera cluster 1} > +# | {Circular Async replication} > +# D <-> E <-> F {Galera cluster 2} > +# We will write on any random node to see if gtid is consitent or not > +# Then we will kill node D and set up the replication between A and E > +# To see whether fail over works or not. > + > +--source include/big_test.inc > +--source include/galera_cluster.inc > +--source include/have_innodb.inc > + > +--connection node_1 > +--echo cluster 1 node 1 > +SHOW STATUS LIKE 'wsrep_cluster_size'; > + > +--connection node_2 > +--echo cluster 1 node 2 > +SHOW STATUS LIKE 'wsrep_cluster_size'; > + > +--connect node_3, 127.0.0.1, root, , test, $NODE_MYPORT_3 > +--connection node_3 > +--echo cluster 1 node 3 > +SHOW STATUS LIKE 'wsrep_cluster_size'; > + > +--connect node_4, 127.0.0.1, root, , test, $NODE_MYPORT_4 > +--connection node_4 > +--echo cluster 2 node 1 > +SHOW STATUS LIKE 'wsrep_cluster_size'; > + > +--connect node_5, 127.0.0.1, root, , test, $NODE_MYPORT_5 > +--connection node_5 > +--echo cluster 2 node 2 > +SHOW STATUS LIKE 'wsrep_cluster_size'; > + > +--connect node_6, 127.0.0.1, root, , test, $NODE_MYPORT_6 > +--connection node_6 > +--echo cluster 2 node 3 > +SHOW STATUS LIKE 'wsrep_cluster_size'; > + > +--connection node_1 > +--eval change master to master_host='127.0.0.1', master_user='root', > master_port=$NODE_MYPORT_4, master_use_gtid=current_pos, > ignore_server_ids=(12,13); > +start slave; > +--source include/wait_for_slave_to_start.inc > +select @@gtid_binlog_state; > +select @@gtid_slave_pos; > +#--query_vertical SHOW SLAVE STATUS; > + > +--connection node_4 > +--eval change master to master_host='127.0.0.1', master_user='root', > master_port=$NODE_MYPORT_1, master_use_gtid=current_pos, > ignore_server_ids=(22,23); > +start slave; > +--source include/wait_for_slave_to_start.inc > +select @@gtid_binlog_state; > +select @@gtid_slave_pos; > +#--query_vertical SHOW SLAVE STATUS; > + > +--echo cluster 1 node 1 > +--connection node_1 > +create table t1 (cluster_domain_id int ,node_server_id int, seq_no int); > +insert into t1 values (1, 11, 2); > +select @@gtid_binlog_state; > + > +--echo #wait for sync cluster 1 and 2 > +--connection node_1 > +--source include/save_master_gtid.inc > +--connection node_4 > +--source include/sync_with_master_gtid.inc > + > +--echo cluster 2 node 1 > +--connection node_4 > +insert into t1 values (2, 21, 1); > +select @@gtid_binlog_state; > +select * from t1; > + > +--echo #wait for sync cluster 1 and 2 > +--connection node_1 > +--source include/save_master_gtid.inc > +--connection node_4 > +--source include/sync_with_master_gtid.inc > + > + > +--echo cluster 1 node 2 > +--connection node_2 > +select @@gtid_binlog_state; > +insert into t1 values (1, 12, 3); > +select @@gtid_binlog_state; > + > +--echo #wait for sync cluster 1 and 2 > +--connection node_1 > +--source include/save_master_gtid.inc > +--connection node_4 > +--source include/sync_with_master_gtid.inc > + > +--echo cluster 1 node 3 > +--connection node_3 > +select @@gtid_binlog_state; > +insert into t1 values (1, 13, 4); > +select @@gtid_binlog_state; > + > +--echo #wait for sync cluster 1 and 2 > +--connection node_1 > +--source include/save_master_gtid.inc > +--connection node_4 > +--source include/sync_with_master_gtid.inc > + > +--echo cluster 2 node 2 > +--connection node_5 > +select @@gtid_binlog_state; > +insert into t1 values (2, 22, 2); > +select @@gtid_binlog_state; > + > +--echo #wait for sync cluster 2 and 1 > +--connection node_4 > +--source include/save_master_gtid.inc > +--connection node_1 > +--source include/sync_with_master_gtid.inc > + > +--echo cluster 2 node 3 > +--connection node_6 > +select @@gtid_binlog_state; > +insert into t1 values (2, 23, 3); > +select @@gtid_binlog_state; > + > +--echo #wait for sync cluster 2 and 1 > +--connection node_4 > +--source include/save_master_gtid.inc > +--connection node_1 > +--source include/sync_with_master_gtid.inc > + > + > +--echo cluster 1 node 1 > +--connection node_1 > +select @@gtid_binlog_state; > +drop table t1; > +stop slave; > +change master to master_use_gtid=no, ignore_server_ids=(); > + > +--echo cluster 2 node 1 > +--connection node_4 > +stop slave; > +change master to master_use_gtid=no, ignore_server_ids=(); > diff --git a/sql/log.cc b/sql/log.cc > index 07d4074..bb8b1d3 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -5544,7 +5544,36 @@ THD::binlog_start_trans_and_stmt() > cache_mngr->trx_cache.get_prev_position() == MY_OFF_T_UNDEF) > { > this->binlog_set_stmt_begin(); > - if (in_multi_stmt_transaction_mode()) > + bool mstmt_mode= in_multi_stmt_transaction_mode(); > +#ifdef WITH_WSREP > + /* Write Gtid > + Get domain id only when gtid mode is set > + If this event is replicate through a master then , > + we will forward the same gtid another nodes > + We have to do this only one time in mysql transaction. > + Since this function is called multiple times , We will check for > + ha_info->is_started() > + */ > + Ha_trx_info *ha_info; > + ha_info= this->ha_data[binlog_hton->slot].ha_info + (mstmt_mode ? 1 > : 0); > + > + if (!ha_info->is_started() && wsrep_gtid_mode && > this->variables.gtid_seq_no) > + { > + binlog_cache_mngr *const cache_mngr= > + (binlog_cache_mngr*) thd_get_ha_data(this, binlog_hton); > + > + IO_CACHE *file= > + cache_mngr->get_binlog_cache_log(use_trans_cache(this, true)); > + Log_event_writer writer(file); > + Gtid_log_event gtid_event(this, this->variables.gtid_seq_no, > + this->variables.gtid_domain_id, > + true, LOG_EVENT_SUPPRESS_USE_F, > + true, 0); > + gtid_event.server_id= this->variables.server_id; > + writer.write(&gtid_event); > + } > +#endif > + if (mstmt_mode) > trans_register_ha(this, TRUE, binlog_hton); > trans_register_ha(this, FALSE, binlog_hton); > /* > @@ -5826,7 +5855,7 @@ MYSQL_BIN_LOG::write_gtid_event(THD *thd, bool > standalone, > DBUG_PRINT("enter", ("standalone: %d", standalone)); > > #ifdef WITH_WSREP > - if (WSREP(thd) && thd->wsrep_trx_meta.gtid.seqno != -1 && > wsrep_gtid_mode) > + if (WSREP(thd) && thd->wsrep_trx_meta.gtid.seqno != -1 && > wsrep_gtid_mode && !thd->variables.gtid_seq_no) > { > domain_id= wsrep_gtid_domain_id; > } else { > @@ -6840,6 +6869,24 @@ int MYSQL_BIN_LOG::write_cache(THD *thd, IO_CACHE > *cache) > mysql_mutex_assert_owner(&LOCK_log); > if (reinit_io_cache(cache, READ_CACHE, 0, 0, 0)) > DBUG_RETURN(ER_ERROR_ON_WRITE); > + uchar *read_pos= cache->read_pos; > + > +#ifdef WITH_WSREP > + /* > + In the cache we have gtid event if , below condition is true, > + then we will simply increment the read position of cache by > + the length of Gtid_log_event. > + */ > + if (wsrep_gtid_mode && thd->variables.gtid_seq_no) > + { > + uchar *head= read_pos; > + uint data_len= uint4korr(head + EVENT_LEN_OFFSET); > + uint event_type= (uchar)head[EVENT_TYPE_OFFSET]; > + if (event_type == GTID_LOG_EVENT) > + cache->read_pos+= data_len; > + } > +#endif > + > uint length= my_b_bytes_in_cache(cache), group, carry, hdr_offs; > long val; > ulong end_log_pos_inc= 0; // each event processed adds > BINLOG_CHECKSUM_LEN 2 t > diff --git a/sql/wsrep_mysqld.cc b/sql/wsrep_mysqld.cc > index bdc8491..285bb52 100644 > --- a/sql/wsrep_mysqld.cc > +++ b/sql/wsrep_mysqld.cc > @@ -1227,6 +1227,16 @@ int wsrep_to_buf_helper( > if (!ret && writer.write(&gtid_ev)) ret= 1; > } > #endif /* GTID_SUPPORT */ > + if (wsrep_gtid_mode && thd->variables.gtid_seq_no) > + { > + Gtid_log_event gtid_event(thd, thd->variables.gtid_seq_no, > + thd->variables.gtid_domain_id, > + true, LOG_EVENT_SUPPRESS_USE_F, > + true, 0); > + gtid_event.server_id= thd->variables.server_id; > + if (!gtid_event.is_valid()) ret= 0; > + ret= writer.write(&gtid_event); > + } > > /* if there is prepare query, add event for it */ > if (!ret && thd->wsrep_TOI_pre_query) > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

[Maria-developers] Issue with commit 0f43279c / MDEV-13936
by jerome brauge 12 Dec '17

12 Dec '17

Hi All, Since some time ago, my app crashed the server (10.2.-ext) in some case (repeatable) with the following stack trace : On Linux Centos 7: ./sql/mysqld(my_print_stacktrace+0x38)[0x7f463f49a093] ./sql/mysqld(handle_fatal_signal+0x3a3)[0x7f463ed2747a] /lib64/libpthread.so.0(+0xf370)[0x7f463dffd370] ./sql/mysqld(_ZN13Explain_query10get_selectEj+0x40)[0x7f463ebf4d06] ./sql/mysqld(_ZN13Explain_query8add_nodeEP12Explain_node+0x203)[0x7f463ebf4f15] ./sql/mysqld(_ZN4JOIN24save_explain_data_internEP13Explain_querybbbPKc+0x959)[0x7f463eaf3db9] ./sql/mysqld(_ZN4JOIN17save_explain_dataEP13Explain_querybbbb+0x14b)[0x7f463eabb963] ./sql/mysqld(_ZN4JOIN13build_explainEv+0xe2)[0x7f463eab3b70] ./sql/mysqld(_ZN4JOIN8optimizeEv+0xb6)[0x7f463eab3d2e] ./sql/mysqld(_ZN13st_select_lex31optimize_unflattened_subqueriesEb+0x263)[0x7f463ea57b5d] ./sql/mysqld(_ZN4JOIN31optimize_unflattened_subqueriesEv+0x24)[0x7f463ec1b746] ./sql/mysqld(_ZN4JOIN15optimize_stage2Ev+0x1cdb)[0x7f463eab7171] ./sql/mysqld(_ZN4JOIN14optimize_innerEv+0x14dd)[0x7f463eab542d] ./sql/mysqld(_ZN4JOIN8optimizeEv+0x7a)[0x7f463eab3cf2] ./sql/mysqld(_Z12mysql_selectP3THDP10TABLE_LISTjR4ListI4ItemEPS4_jP8st_orderS9_S7_S9_yP13select_resultP18st_select_lex_unitP13st_select_lex+0x263)[0x7f463eabccd2] ./sql/mysqld(_Z13handle_selectP3THDP3LEXP13select_resultm+0x16c)[0x7f463eab13e6] ./sql/mysqld(_Z21mysql_execute_commandP3THD+0x4b5b)[0x7f463ea76fb4] ./sql/mysqld(_ZN13sp_instr_stmt9exec_coreEP3THDPj+0x80)[0x7f463eeaafb0] ./sql/mysqld(_ZN13sp_lex_keeper23reset_lex_and_exec_coreEP3THDPjbP8sp_instr+0x1d1)[0x7f463eeaa549] ./sql/mysqld(_ZN13sp_instr_stmt7executeEP3THDPj+0x1b5)[0x7f463eeaabf9] ./sql/mysqld(_ZN7sp_head7executeEP3THDb+0x754)[0x7f463eea4ff6] ./sql/mysqld(_ZN7sp_head17execute_procedureEP3THDP4ListI4ItemE+0x971)[0x7f463eea74df] ./sql/mysqld(+0x645959)[0x7f463ea71959] ./sql/mysqld(_ZN12Sql_cmd_call7executeEP3THD+0x270)[0x7f463ea7240c] ./sql/mysqld(_Z21mysql_execute_commandP3THD+0x9f95)[0x7f463ea7c3ee] ./sql/mysqld(_ZN18Prepared_statement7executeEP6Stringb+0x508)[0x7f463ea9e468] ./sql/mysqld(_ZN18Prepared_statement12execute_loopEP6StringbPhS2_+0x200)[0x7f463ea9c6fa] ./sql/mysqld(+0x66e113)[0x7f463ea9a113] ./sql/mysqld(_Z19mysqld_stmt_executeP3THDPcj+0x97)[0x7f463ea99cd7] ./sql/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcjbb+0xc9e)[0x7f463ea6e68d] ./sql/mysqld(_Z10do_commandP3THD+0x700)[0x7f463ea6d315] ./sql/mysqld(_Z24do_handle_one_connectionP7CONNECT+0x1ca)[0x7f463ebbe3d2] ./sql/mysqld(handle_one_connection+0x30)[0x7f463ebbe125] ./sql/mysqld(+0x1012852)[0x7f463f43e852] /lib64/libpthread.so.0(+0x7dc5)[0x7f463dff5dc5] /lib64/libc.so.6(clone+0x6d)[0x7f463c01673d] Or on Windows : Unhandled exception thrown: read access violation. Dynamic_array<Explain_select *>::at(...) returned 0xDB925000. > mysqld.exe!Explain_query::~Explain_query() Line 71 C++ mysqld.exe!Explain_query::`scalar deleting destructor'(unsigned int) C++ mysqld.exe!delete_explain_query(LEX * lex) Line 2397 C++ mysqld.exe!sp_lex_keeper::reset_lex_and_exec_core(THD * thd, unsigned int * nextp, bool open_tables, sp_instr * instr) Line 3130 C++ mysqld.exe!sp_instr_stmt::execute(THD * thd, unsigned int * nextp) Line 3266 C++ mysqld.exe!sp_head::execute(THD * thd, bool merge_da_on_success) Line 1213 C++ mysqld.exe!sp_head::execute_procedure(THD * thd, List<Item> * args) Line 2072 C++ mysqld.exe!do_execute_sp(THD * thd, sp_head * sp) Line 2907 C++ mysqld.exe!Sql_cmd_call::execute(THD * thd) Line 3147 C++ mysqld.exe!mysql_execute_command(THD * thd) Line 6223 C++ mysqld.exe!Prepared_statement::execute(String * expanded_query, bool open_cursor) Line 4682 C++ mysqld.exe!Prepared_statement::execute_loop(String * expanded_query, bool open_cursor, unsigned char * packet, unsigned char * packet_end) Line 4111 C++ mysqld.exe!mysql_stmt_execute_common(THD * thd, unsigned long stmt_id, unsigned char * packet, unsigned char * packet_end, unsigned long cursor_flags, bool bulk_op, bool read_types) Line 3132 C++ mysqld.exe!mysqld_stmt_execute(THD * thd, char * packet_arg, unsigned int packet_length) Line 3028 C++ mysqld.exe!dispatch_command(enum_server_command command, THD * thd, char * packet, unsigned int packet_length, bool is_com_multi, bool is_next_command) Line 1763 C++ mysqld.exe!do_command(THD * thd) Line 1369 C++ mysqld.exe!threadpool_process_request(THD * thd) Line 366 C++ mysqld.exe!tp_callback(TP_connection * c) Line 192 C++ mysqld.exe!tp_callback(_TP_CALLBACK_INSTANCE * instance, void * context) Line 377 C++ mysqld.exe!work_callback(_TP_CALLBACK_INSTANCE * instance, void * context, _TP_WORK * work) Line 451 C++ The sql statement involved (in a stored procedure): Query (0x7f457df65bb8): INSERT INTO qnwtc( enrqnwtc, priqnwtc, lprqnwtc, cprqnwtc, nuiqnwtc) SELECT 'PRI' ,occgtpar ,intgtpar ,ch1gtpar , NAME_CONST('wpio_ident',110325)FROM gtpara WHERE padgtpar = 'PRIQNTAC' AND numgtets = NAME_CONST('wsysets',_utf8'IFR' COLLATE 'utf8_bin')AND etagtpar = 'A' Where gtpara is a view. If the stored procedure is called alone, it works fine. I've revert commit 0f43279c (MDEV-13936) and now my app works fine again. Regards.

1 0

Re: [Maria-developers] Mdev-10715 Initial Patch
by Sachin Setiya 11 Dec '17

11 Dec '17

Hi Andrei! Thanks for the review. On Thu, Oct 12, 2017 at 11:48 PM, <andrei.elkin(a)pp.inet.fi> wrote: > Howdy, Sachin! > > I've checked the patch mostly from the high-level design pov to gather > few suggestions. Find them on the top of diff hunks (hopefully the diff > remains applicable) below indented with '*review*>'. > > > > revision-id: daf9a67ea13c5c7309f85a104e7c07ec68f65e88 (mariadb-10.1.28-10-gdaf9a67) > parent(s): 5eb666ad37ff80e5f8dc714bb68fcb1e269b797a > author: Sachin Setiya > committer: Sachin Setiya > timestamp: 2017-10-10 17:45:21 +0530 > message: > > Mdev-10715 Initial Patch > *review*> The initial patch is a very good time to write down the work > intent and its key features. Done > Could you please do that and highlight how WSREP has really > become multi-source compatible. > > > --- > mysql-test/suite/galera/r/galera_gtid_slave.result | 18 +++ > .../r/galera_gtid_slave_sst_mysqldump.result | 92 ++++++++++++ > .../galera/r/galera_gtid_slave_sst_rsync.result | 92 ++++++++++++ > mysql-test/suite/galera/t/galera_gtid_slave.cnf | 18 +++ > mysql-test/suite/galera/t/galera_gtid_slave.test | 67 +++++++++ > .../galera/t/galera_gtid_slave_sst_mysqldump.cnf | 18 +++ > .../galera/t/galera_gtid_slave_sst_mysqldump.test | 156 +++++++++++++++++++++ > .../suite/galera/t/galera_gtid_slave_sst_rsync.cnf | 19 +++ > .../galera/t/galera_gtid_slave_sst_rsync.test | 156 +++++++++++++++++++++ > sql/log.cc | 37 ++++- > sql/wsrep_mysqld.cc | 9 ++ > 11 files changed, 681 insertions(+), 1 deletion(-) > > diff --git a/mysql-test/suite/galera/r/galera_gtid_slave.result b/mysql-test/suite/galera/r/galera_gtid_slave.result > new file mode 100644 > index 0000000..7ab10c7 > --- /dev/null > +++ b/mysql-test/suite/galera/r/galera_gtid_slave.result > @@ -0,0 +1,18 @@ > +START SLAVE; > +CREATE TABLE t1 (f1 INTEGER PRIMARY KEY) ENGINE=InnoDB; > +INSERT INTO t1 VALUES(1); > +SELECT @@global.gtid_binlog_state; > +@@global.gtid_binlog_state > +1-1-2 > +INSERT INTO t1 VALUES(2); > +INSERT INTO t1 VALUES(3); > +SELECT @@global.gtid_binlog_state; > +@@global.gtid_binlog_state > +1-1-2,2-2-2 > +INSERT INTO t1 VALUES(4); > +SELECT @@global.gtid_binlog_state; > +@@global.gtid_binlog_state > +1-1-2,2-2-2,2-3-3 > +DROP TABLE t1; > +STOP SLAVE; > +RESET SLAVE ALL; > diff --git a/mysql-test/suite/galera/r/galera_gtid_slave_sst_mysqldump.result b/mysql-test/suite/galera/r/galera_gtid_slave_sst_mysqldump.result > new file mode 100644 > index 0000000..8e8e2e0 > --- /dev/null > +++ b/mysql-test/suite/galera/r/galera_gtid_slave_sst_mysqldump.result > @@ -0,0 +1,92 @@ > +START SLAVE; > +CREATE TABLE t2 (f1 INTEGER PRIMARY KEY, f2 int unique) ENGINE=InnoDB; > +INSERT INTO t2 VALUES(1,11); > +INSERT INTO t2 VALUES(2,22); > +INSERT INTO t2 VALUES(3,33); > +SELECT @@global.gtid_binlog_state; > +@@global.gtid_binlog_state > +1-1-4 > +INSERT INTO t2 VALUES(4,44); > +INSERT INTO t2 VALUES(5,55); > +INSERT INTO t2 VALUES(6,66); > +SELECT @@global.gtid_binlog_state; > +@@global.gtid_binlog_state > +1-1-4,2-2-3 > +INSERT INTO t2 VALUES(7,77); > +INSERT INTO t2 VALUES(8,88); > +SELECT @@global.gtid_binlog_state; > +@@global.gtid_binlog_state > +1-1-4,2-2-3,2-3-5 > +CREATE TABLE t1 (f1 CHAR(255)) ENGINE=InnoDB; > +SET AUTOCOMMIT=OFF; > +START TRANSACTION; > +INSERT INTO t1 VALUES ('node1_committed_before'); > +INSERT INTO t1 VALUES ('node1_committed_before'); > +COMMIT; > +SET AUTOCOMMIT=OFF; > +START TRANSACTION; > +INSERT INTO t1 VALUES ('node2_committed_before'); > +INSERT INTO t1 VALUES ('node2_committed_before'); > +COMMIT; > +Shutting down server ... > +SET AUTOCOMMIT=OFF; > +START TRANSACTION; > +INSERT INTO t1 VALUES ('node1_committed_during'); > +INSERT INTO t1 VALUES ('node1_committed_during'); > +COMMIT; > +Starting server ... > +SET AUTOCOMMIT=OFF; > +START TRANSACTION; > +INSERT INTO t1 VALUES ('node3_committed_after'); > +INSERT INTO t1 VALUES ('node3_committed_after'); > +COMMIT; > +Select * from t1 order by f1; > +f1 > +node1_committed_before > +node1_committed_before > +node1_committed_during > +node1_committed_during > +node2_committed_before > +node2_committed_before > +node3_committed_after > +node3_committed_after > +Select * from t1 order by f1; > +f1 > +node1_committed_before > +node1_committed_before > +node1_committed_during > +node1_committed_during > +node2_committed_before > +node2_committed_before > +node3_committed_after > +node3_committed_after > +SELECT @@global.gtid_binlog_state; > +@@global.gtid_binlog_state > +1-1-6,2-2-7,2-3-8 > +SELECT @@global.gtid_binlog_state; > +@@global.gtid_binlog_state > +1-1-6,2-2-7,2-3-8 > +SET AUTOCOMMIT=ON; > +SET AUTOCOMMIT=ON; > +SET AUTOCOMMIT=ON; > +STOP slave; > +INSERT INTO t1 VALUES ('node2_slave_stoped'); > +INSERT INTO t1 VALUES ('node1_slave_stoped'); > +INSERT INTO t1 VALUES ('node2_slave_stoped_inserted'); > +start slave; > +INSERT INTO t1 VALUES ('node2_slave_started'); > +SELECT COUNT(*) from t1; > +COUNT(*) > +11 > +SELECT @@global.gtid_binlog_state; > +@@global.gtid_binlog_state > +1-1-6,2-3-8,2-2-11 > +SELECT COUNT(*) from t1; > +COUNT(*) > +11 > +SELECT @@global.gtid_binlog_state; > +@@global.gtid_binlog_state > +1-1-6,2-3-8,2-2-11 > +DROP TABLE t2,t1; > +STOP SLAVE; > +RESET SLAVE ALL; > diff --git a/mysql-test/suite/galera/r/galera_gtid_slave_sst_rsync.result b/mysql-test/suite/galera/r/galera_gtid_slave_sst_rsync.result > new file mode 100644 > index 0000000..9ab4601 > --- /dev/null > +++ b/mysql-test/suite/galera/r/galera_gtid_slave_sst_rsync.result > @@ -0,0 +1,92 @@ > +START SLAVE; > +CREATE TABLE t2 (f1 INTEGER PRIMARY KEY, f2 int unique) ENGINE=InnoDB; > +INSERT INTO t2 VALUES(1,11); > +INSERT INTO t2 VALUES(2,22); > +INSERT INTO t2 VALUES(3,33); > +SELECT @@global.gtid_binlog_state; > +@@global.gtid_binlog_state > +1-1-4 > +INSERT INTO t2 VALUES(4,44); > +INSERT INTO t2 VALUES(5,55); > +INSERT INTO t2 VALUES(6,66); > +SELECT @@global.gtid_binlog_state; > +@@global.gtid_binlog_state > +1-1-4,2-2-3 > +INSERT INTO t2 VALUES(7,77); > +INSERT INTO t2 VALUES(8,88); > +SELECT @@global.gtid_binlog_state; > +@@global.gtid_binlog_state > +1-1-4,2-2-3,2-3-5 > +CREATE TABLE t1 (f1 CHAR(255)) ENGINE=InnoDB; > +SET AUTOCOMMIT=OFF; > +START TRANSACTION; > +INSERT INTO t1 VALUES ('node1_committed_before'); > +INSERT INTO t1 VALUES ('node1_committed_before'); > +COMMIT; > +SET AUTOCOMMIT=OFF; > +START TRANSACTION; > +INSERT INTO t1 VALUES ('node2_committed_before'); > +INSERT INTO t1 VALUES ('node2_committed_before'); > +COMMIT; > +Shutting down server ... > +SET AUTOCOMMIT=OFF; > +START TRANSACTION; > +INSERT INTO t1 VALUES ('node1_committed_during'); > +INSERT INTO t1 VALUES ('node1_committed_during'); > +COMMIT; > +Starting server ... > +SET AUTOCOMMIT=OFF; > +START TRANSACTION; > +INSERT INTO t1 VALUES ('node3_committed_after'); > +INSERT INTO t1 VALUES ('node3_committed_after'); > +COMMIT; > +Select * from t1 order by f1; > +f1 > +node1_committed_before > +node1_committed_before > +node1_committed_during > +node1_committed_during > +node2_committed_before > +node2_committed_before > +node3_committed_after > +node3_committed_after > +Select * from t1 order by f1; > +f1 > +node1_committed_before > +node1_committed_before > +node1_committed_during > +node1_committed_during > +node2_committed_before > +node2_committed_before > +node3_committed_after > +node3_committed_after > +SELECT @@global.gtid_binlog_state; > +@@global.gtid_binlog_state > +1-1-6,2-2-7,2-3-8 > +SELECT @@global.gtid_binlog_state; > +@@global.gtid_binlog_state > +1-1-6,2-2-7,2-3-8 > +SET AUTOCOMMIT=ON; > +SET AUTOCOMMIT=ON; > +SET AUTOCOMMIT=ON; > +STOP slave; > +INSERT INTO t1 VALUES ('node2_slave_stoped'); > +INSERT INTO t1 VALUES ('node1_slave_stoped'); > +INSERT INTO t1 VALUES ('node2_slave_stoped_inserted'); > +start slave; > +INSERT INTO t1 VALUES ('node2_slave_started'); > +SELECT COUNT(*) from t1; > +COUNT(*) > +11 > +SELECT @@global.gtid_binlog_state; > +@@global.gtid_binlog_state > +1-1-7,2-3-8,2-2-11 > +SELECT COUNT(*) from t1; > +COUNT(*) > +12 > +SELECT @@global.gtid_binlog_state; > +@@global.gtid_binlog_state > +1-1-7,2-3-8,2-2-11 > +DROP TABLE t2,t1; > +STOP SLAVE; > +RESET SLAVE ALL; > diff --git a/mysql-test/suite/galera/t/galera_gtid_slave.cnf b/mysql-test/suite/galera/t/galera_gtid_slave.cnf > new file mode 100644 > index 0000000..409d0d1 > --- /dev/null > +++ b/mysql-test/suite/galera/t/galera_gtid_slave.cnf > @@ -0,0 +1,18 @@ > +!include ../galera_2nodes_as_slave.cnf > + > +[mysqld] > +log-bin=mysqld-bin > +log-slave-updates > +binlog-format=ROW > + > +[mysqld.1] > +gtid-domain-id=1 > +[mysqld.2] > +gtid-domain-id=2 > +wsrep_gtid_mode=1 > +wsrep_gtid_domain_id=2 > +[mysqld.3] > +gtid-domain-id=2 > +wsrep_gtid_mode=1 > +wsrep_gtid_domain_id=2 > + > diff --git a/mysql-test/suite/galera/t/galera_gtid_slave.test b/mysql-test/suite/galera/t/galera_gtid_slave.test > new file mode 100644 > index 0000000..3b3f6f3 > --- /dev/null > +++ b/mysql-test/suite/galera/t/galera_gtid_slave.test > @@ -0,0 +1,67 @@ > +# > +# Test Galera as a slave to a MariaDB master using GTIDs > +# > +# suite/galera/galera_2nodes_as_slave.cnf describes the setup of the nodes > +# suite/galera/t/galera_as_slave_gtid.cnf has the GTID options > +# > +# In addition to performing DDL and DML, we check that the gtid of the master is preserved inside the cluster > +# > + > +--source include/have_innodb.inc > + > +# As node #1 is not a Galera node, we connect to node #2 in order to run include/galera_cluster.inc > +--connect node_2a, 127.0.0.1, root, , test, $NODE_MYPORT_2 > +--source include/galera_cluster.inc > + > +--connection node_2 > +--disable_query_log > +--eval CHANGE MASTER TO MASTER_HOST='127.0.0.1', MASTER_USER='root', MASTER_PORT=$NODE_MYPORT_1; > +--enable_query_log > +START SLAVE; > + > +--connection node_1 > +CREATE TABLE t1 (f1 INTEGER PRIMARY KEY) ENGINE=InnoDB; > +INSERT INTO t1 VALUES(1); > + > +SELECT @@global.gtid_binlog_state; > + > +--connection node_2 > +--let $wait_condition = SELECT COUNT(*) = 1 FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME = 't1'; > +--source include/wait_condition.inc > + > +--let $wait_condition = SELECT COUNT(*) = 1 FROM t1; > +--source include/wait_condition.inc > +INSERT INTO t1 VALUES(2); > +INSERT INTO t1 VALUES(3); > +SELECT @@global.gtid_binlog_state; > + > +--connect node_3, 127.0.0.1, root, , test, $NODE_MYPORT_3 > +--sleep 1 > +--let $wait_condition = SELECT COUNT(*) = 3 FROM t1; > +--source include/wait_condition.inc > + > +INSERT INTO t1 VALUES(4); > +SELECT @@global.gtid_binlog_state; > + > +--connection node_1 > +DROP TABLE t1; > + > +# > +# Unfortunately without the sleep below the following statement fails with "query returned no rows", which > +# is difficult to understand given that it is an aggregate query. A "query execution was interrupted" > +# warning is also reported by MTR, which is also weird. > +# > + > +--sleep 1 > + > +--connection node_2 > +--let $wait_condition = SELECT COUNT(*) = 0 FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME = 't1'; > +--source include/wait_condition.inc > + > +--connection node_3 > +--let $wait_condition = SELECT COUNT(*) = 0 FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME = 't1'; > +--source include/wait_condition.inc > + > +--connection node_2 > +STOP SLAVE; > +RESET SLAVE ALL; > diff --git a/mysql-test/suite/galera/t/galera_gtid_slave_sst_mysqldump.cnf b/mysql-test/suite/galera/t/galera_gtid_slave_sst_mysqldump.cnf > new file mode 100644 > index 0000000..409d0d1 > --- /dev/null > +++ b/mysql-test/suite/galera/t/galera_gtid_slave_sst_mysqldump.cnf > @@ -0,0 +1,18 @@ > +!include ../galera_2nodes_as_slave.cnf > + > +[mysqld] > +log-bin=mysqld-bin > +log-slave-updates > +binlog-format=ROW > + > +[mysqld.1] > +gtid-domain-id=1 > +[mysqld.2] > +gtid-domain-id=2 > +wsrep_gtid_mode=1 > +wsrep_gtid_domain_id=2 > +[mysqld.3] > +gtid-domain-id=2 > +wsrep_gtid_mode=1 > +wsrep_gtid_domain_id=2 > + > diff --git a/mysql-test/suite/galera/t/galera_gtid_slave_sst_mysqldump.test b/mysql-test/suite/galera/t/galera_gtid_slave_sst_mysqldump.test > new file mode 100644 > index 0000000..67aad21 > --- /dev/null > +++ b/mysql-test/suite/galera/t/galera_gtid_slave_sst_mysqldump.test > @@ -0,0 +1,156 @@ > +# > +# Test Galera as a slave to a MariaDB master using GTIDs > +# > +# suite/galera/galera_2nodes_as_slave.cnf describes the setup of the nodes > +# suite/galera/t/galera_as_slave_gtid.cnf has the GTID options > +# > +# In addition to performing DDL and DML, we check that the gtid of the master is preserved inside the cluster > +# > + > +--source include/big_test.inc > +--source include/have_innodb.inc > +# As node #1 is not a Galera node, we connect to node #2 in order to run include/galera_cluster.inc > +--connect node_2a, 127.0.0.1, root, , test, $NODE_MYPORT_2 > +--source include/galera_cluster.inc > + > +--connection node_2 > +--disable_query_log > +--eval CHANGE MASTER TO MASTER_HOST='127.0.0.1', MASTER_USER='root', MASTER_PORT=$NODE_MYPORT_1; > +--enable_query_log > +START SLAVE; > + > +--connection node_1 > +CREATE TABLE t2 (f1 INTEGER PRIMARY KEY, f2 int unique) ENGINE=InnoDB; > +INSERT INTO t2 VALUES(1,11); > +INSERT INTO t2 VALUES(2,22); > +INSERT INTO t2 VALUES(3,33); > + > +SELECT @@global.gtid_binlog_state; > + > +--connection node_2 > +--let $wait_condition = SELECT COUNT(*) = 1 FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME = 't2'; > +--source include/wait_condition.inc > + > +--let $wait_condition = SELECT COUNT(*) = 3 FROM t2; > +--source include/wait_condition.inc > +INSERT INTO t2 VALUES(4,44); > +INSERT INTO t2 VALUES(5,55); > +INSERT INTO t2 VALUES(6,66); > +SELECT @@global.gtid_binlog_state; > + > +--connect node_3, 127.0.0.1, root, , test, $NODE_MYPORT_3 > +--sleep 1 > +--let $wait_condition = SELECT COUNT(*) = 6 FROM t2; > +--source include/wait_condition.inc > + > +INSERT INTO t2 VALUES(7,77); > +INSERT INTO t2 VALUES(8,88); > +SELECT @@global.gtid_binlog_state; > + > +#Perform SST > +--connection node_1 > +CREATE TABLE t1 (f1 CHAR(255)) ENGINE=InnoDB; > +SET AUTOCOMMIT=OFF; > +START TRANSACTION; > +INSERT INTO t1 VALUES ('node1_committed_before'); > +INSERT INTO t1 VALUES ('node1_committed_before'); > +COMMIT; > + > +--connection node_2 > +SET AUTOCOMMIT=OFF; > +START TRANSACTION; > +INSERT INTO t1 VALUES ('node2_committed_before'); > +INSERT INTO t1 VALUES ('node2_committed_before'); > +COMMIT; > + > +--connection node_3 > +--echo Shutting down server ... > +--source include/shutdown_mysqld.inc > + > + > +--connection node_2 > +--let $wait_condition = SELECT VARIABLE_VALUE = 1 FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_cluster_size' > +--source include/wait_condition.inc > +SET AUTOCOMMIT=OFF; > +START TRANSACTION; > +INSERT INTO t1 VALUES ('node1_committed_during'); > +INSERT INTO t1 VALUES ('node1_committed_during'); > +COMMIT; > + > +--connection node_3 > +--echo Starting server ... > +--source include/start_mysqld.inc > +--source include/wait_until_ready.inc > + > +SET AUTOCOMMIT=OFF; > +START TRANSACTION; > +INSERT INTO t1 VALUES ('node3_committed_after'); > +INSERT INTO t1 VALUES ('node3_committed_after'); > +COMMIT; > + > +--connection node_2 > +Select * from t1 order by f1; > + > +--connection node_3 > +Select * from t1 order by f1; > + > +#SST Done > +--sleep 1 > +--connection node_2 > +SELECT @@global.gtid_binlog_state; > + > +--connection node_3 > +SELECT @@global.gtid_binlog_state; > + > +# > +#stop slave on node 2 > +--connection node_1 > +SET AUTOCOMMIT=ON; > +#drop table t1; > +#CREATE TABLE t1 (f1 CHAR(255)) ENGINE=InnoDB; > + > +--connection node_2 > +SET AUTOCOMMIT=ON; > +--connection node_3 > +SET AUTOCOMMIT=ON; > + > +--connection node_2 > +STOP slave; > +--sleep 1 > +INSERT INTO t1 VALUES ('node2_slave_stoped'); > + > +--connection node_1 > +INSERT INTO t1 VALUES ('node1_slave_stoped'); > +#start slave > +--connection node_2 > +INSERT INTO t1 VALUES ('node2_slave_stoped_inserted'); > +start slave; > +INSERT INTO t1 VALUES ('node2_slave_started'); > +SELECT COUNT(*) from t1; > +SELECT @@global.gtid_binlog_state; > + > +--connection node_3 > +SELECT COUNT(*) from t1; > +SELECT @@global.gtid_binlog_state; > + > +--connection node_1 > +DROP TABLE t2,t1; > + > +# Unfortunately without the sleep below the following statement fails with "query returned no rows", which > +# is difficult to understand given that it is an aggregate query. A "query execution was interrupted" > +# warning is also reported by MTR, which is also weird. > +# > + > +--sleep 3 > + > +--connection node_2 > +--let $wait_condition = SELECT COUNT(*) = 0 FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME = 't2'; > +--source include/wait_condition.inc > + > +--connection node_3 > +--let $wait_condition = SELECT COUNT(*) = 0 FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME = 't1'; > +--source include/wait_condition.inc > + > +--connection node_2 > +STOP SLAVE; > +RESET SLAVE ALL; > diff --git a/mysql-test/suite/galera/t/galera_gtid_slave_sst_rsync.cnf b/mysql-test/suite/galera/t/galera_gtid_slave_sst_rsync.cnf > new file mode 100644 > index 0000000..2c82531 > --- /dev/null > +++ b/mysql-test/suite/galera/t/galera_gtid_slave_sst_rsync.cnf > @@ -0,0 +1,19 @@ > +!include ../galera_2nodes_as_slave.cnf > + > +[mysqld] > +log-bin=mysqld-bin > +log-slave-updates > +binlog-format=ROW > +wsrep_sst_method=rsync > + > +[mysqld.1] > +gtid-domain-id=1 > +[mysqld.2] > +gtid-domain-id=2 > +wsrep_gtid_mode=1 > +wsrep_gtid_domain_id=2 > +[mysqld.3] > +gtid-domain-id=2 > +wsrep_gtid_mode=1 > +wsrep_gtid_domain_id=2 > + > diff --git a/mysql-test/suite/galera/t/galera_gtid_slave_sst_rsync.test b/mysql-test/suite/galera/t/galera_gtid_slave_sst_rsync.test > new file mode 100644 > index 0000000..67aad21 > --- /dev/null > +++ b/mysql-test/suite/galera/t/galera_gtid_slave_sst_rsync.test > @@ -0,0 +1,156 @@ > +# > +# Test Galera as a slave to a MariaDB master using GTIDs > +# > +# suite/galera/galera_2nodes_as_slave.cnf describes the setup of the nodes > +# suite/galera/t/galera_as_slave_gtid.cnf has the GTID options > +# > +# In addition to performing DDL and DML, we check that the gtid of the master is preserved inside the cluster > +# > + > +--source include/big_test.inc > +--source include/have_innodb.inc > +# As node #1 is not a Galera node, we connect to node #2 in order to run include/galera_cluster.inc > +--connect node_2a, 127.0.0.1, root, , test, $NODE_MYPORT_2 > +--source include/galera_cluster.inc > + > +--connection node_2 > +--disable_query_log > +--eval CHANGE MASTER TO MASTER_HOST='127.0.0.1', MASTER_USER='root', MASTER_PORT=$NODE_MYPORT_1; > +--enable_query_log > +START SLAVE; > + > +--connection node_1 > +CREATE TABLE t2 (f1 INTEGER PRIMARY KEY, f2 int unique) ENGINE=InnoDB; > +INSERT INTO t2 VALUES(1,11); > +INSERT INTO t2 VALUES(2,22); > +INSERT INTO t2 VALUES(3,33); > + > +SELECT @@global.gtid_binlog_state; > + > +--connection node_2 > +--let $wait_condition = SELECT COUNT(*) = 1 FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME = 't2'; > +--source include/wait_condition.inc > + > +--let $wait_condition = SELECT COUNT(*) = 3 FROM t2; > +--source include/wait_condition.inc > +INSERT INTO t2 VALUES(4,44); > +INSERT INTO t2 VALUES(5,55); > +INSERT INTO t2 VALUES(6,66); > +SELECT @@global.gtid_binlog_state; > + > +--connect node_3, 127.0.0.1, root, , test, $NODE_MYPORT_3 > +--sleep 1 > +--let $wait_condition = SELECT COUNT(*) = 6 FROM t2; > +--source include/wait_condition.inc > + > +INSERT INTO t2 VALUES(7,77); > +INSERT INTO t2 VALUES(8,88); > +SELECT @@global.gtid_binlog_state; > + > +#Perform SST > +--connection node_1 > +CREATE TABLE t1 (f1 CHAR(255)) ENGINE=InnoDB; > +SET AUTOCOMMIT=OFF; > +START TRANSACTION; > +INSERT INTO t1 VALUES ('node1_committed_before'); > +INSERT INTO t1 VALUES ('node1_committed_before'); > +COMMIT; > + > +--connection node_2 > +SET AUTOCOMMIT=OFF; > +START TRANSACTION; > +INSERT INTO t1 VALUES ('node2_committed_before'); > +INSERT INTO t1 VALUES ('node2_committed_before'); > +COMMIT; > + > +--connection node_3 > +--echo Shutting down server ... > +--source include/shutdown_mysqld.inc > + > + > +--connection node_2 > +--let $wait_condition = SELECT VARIABLE_VALUE = 1 FROM INFORMATION_SCHEMA.GLOBAL_STATUS WHERE VARIABLE_NAME = 'wsrep_cluster_size' > +--source include/wait_condition.inc > +SET AUTOCOMMIT=OFF; > +START TRANSACTION; > +INSERT INTO t1 VALUES ('node1_committed_during'); > +INSERT INTO t1 VALUES ('node1_committed_during'); > +COMMIT; > + > +--connection node_3 > +--echo Starting server ... > +--source include/start_mysqld.inc > +--source include/wait_until_ready.inc > + > +SET AUTOCOMMIT=OFF; > +START TRANSACTION; > +INSERT INTO t1 VALUES ('node3_committed_after'); > +INSERT INTO t1 VALUES ('node3_committed_after'); > +COMMIT; > + > +--connection node_2 > +Select * from t1 order by f1; > + > +--connection node_3 > +Select * from t1 order by f1; > + > +#SST Done > +--sleep 1 > +--connection node_2 > +SELECT @@global.gtid_binlog_state; > + > +--connection node_3 > +SELECT @@global.gtid_binlog_state; > + > +# > +#stop slave on node 2 > +--connection node_1 > +SET AUTOCOMMIT=ON; > +#drop table t1; > +#CREATE TABLE t1 (f1 CHAR(255)) ENGINE=InnoDB; > + > +--connection node_2 > +SET AUTOCOMMIT=ON; > +--connection node_3 > +SET AUTOCOMMIT=ON; > + > +--connection node_2 > +STOP slave; > +--sleep 1 > +INSERT INTO t1 VALUES ('node2_slave_stoped'); > + > +--connection node_1 > +INSERT INTO t1 VALUES ('node1_slave_stoped'); > +#start slave > +--connection node_2 > +INSERT INTO t1 VALUES ('node2_slave_stoped_inserted'); > +start slave; > +INSERT INTO t1 VALUES ('node2_slave_started'); > +SELECT COUNT(*) from t1; > +SELECT @@global.gtid_binlog_state; > + > +--connection node_3 > +SELECT COUNT(*) from t1; > +SELECT @@global.gtid_binlog_state; > + > +--connection node_1 > +DROP TABLE t2,t1; > + > +# Unfortunately without the sleep below the following statement fails with "query returned no rows", which > +# is difficult to understand given that it is an aggregate query. A "query execution was interrupted" > +# warning is also reported by MTR, which is also weird. > +# > + > +--sleep 3 > + > +--connection node_2 > +--let $wait_condition = SELECT COUNT(*) = 0 FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME = 't2'; > +--source include/wait_condition.inc > + > +--connection node_3 > +--let $wait_condition = SELECT COUNT(*) = 0 FROM INFORMATION_SCHEMA.TABLES WHERE TABLE_NAME = 't1'; > +--source include/wait_condition.inc > + > +--connection node_2 > +STOP SLAVE; > +RESET SLAVE ALL; > > *review*> The better timing for the non-zero seq_no Gtid write deems to > inside THD::binlog_start_trans_and_stmt() invoked few instructions > earlier. This method is invoked once per *transaction* which > guarantees no duplication which the current chosen point seems to > concede. Perhaps you change its singature to pass the writer. Done as we discussed. this is added into binlog_statrt_trans_and_stmt() And I am checking for ha_info->is_started(). > Also I think we may write the non-zero seq_no Gtid generally, regardless > of WITH_WSREP (naturally to block doing that later at binlog commit > phase, also commented below). This commit will go into 10.1 so as we discussed , we can do this later in 10.3. > > diff --git a/sql/log.cc b/sql/log.cc > index a9f486d..53eeadf 100644 > --- a/sql/log.cc > +++ b/sql/log.cc > @@ -5655,6 +5655,23 @@ int THD::binlog_write_table_map(TABLE *table, bool is_transactional, > DBUG_RETURN(error); > } > } > +#ifdef WITH_WSREP > + /* Get domain id only when gtid mode is set > + If this event is replicate through a master then , > + we will forward the same gtid another nodes > + */ > + if (wsrep_gtid_mode && this->variables.gtid_seq_no) > + { > + Gtid_log_event gtid_event(this, this->variables.gtid_seq_no, > + this->variables.gtid_domain_id, > + true, LOG_EVENT_SUPPRESS_USE_F, > + true, 0); > + gtid_event.server_id= this->variables.server_id; > + > + if ((error= writer.write(&gtid_event))) > + DBUG_RETURN(error); > + } > +#endif > if ((error= writer.write(&the_event))) > DBUG_RETURN(error); > > @@ -5826,7 +5843,7 @@ MYSQL_BIN_LOG::write_gtid_event(THD *thd, bool standalone, > DBUG_PRINT("enter", ("standalone: %d", standalone)); > > #ifdef WITH_WSREP > - if (WSREP(thd) && thd->wsrep_trx_meta.gtid.seqno != -1 && wsrep_gtid_mode) > + if (WSREP(thd) && thd->wsrep_trx_meta.gtid.seqno != -1 && wsrep_gtid_mode && !thd->variables.gtid_seq_no) > { > domain_id= wsrep_gtid_domain_id; > } else { > > *review*> I'm suggest we check the non-zero seq_no status in > MYSQL_BIN_LOG::write_gtid_event() > > *review*>diff --git a/sql/log.cc b/sql/log.cc > *review*>index 2fcb6f6dbf6..8229426a5bd 100644 > *review*>--- a/sql/log.cc > *review*>+++ b/sql/log.cc > *review*>@@ -5867,7 +5867,6 @@ MYSQL_BIN_LOG::write_gtid_event(THD *thd, bool standalone, > *review*> err= rpl_global_gtid_binlog_state.update_with_next_gtid(domain_id, > *review*> local_server_id, &gtid); > *review*> seq_no= gtid.seq_no; > *review*>- } > *review*> if (err) > *review*> DBUG_RETURN(true); > *review*> thd->last_commit_gtid= gtid; > *review*>@@ -5880,6 +5879,7 @@ MYSQL_BIN_LOG::write_gtid_event(THD *thd, bool standalone, > *review*> DBUG_ASSERT(this == &mysql_bin_log); > *review*> if (write_event(&gtid_event)) > *review*> DBUG_RETURN(true); > *review*>+ } > *review*> status_var_add(thd->status_var.binlog_bytes_written, gtid_event.data_written); > *review*> > *review*> DBUG_RETURN(false); > > *review*> so only the zero auto-evaluated seq_no Gtid would make to the > method you're trying to patch. And the hunk is unneeded in this > case. Like above we generalize WITH_WSREP away. > ^^^^^^^ I am not sure about this hunk , I think this was mistake. I have reverted this change. > @@ -6718,6 +6735,24 @@ int MYSQL_BIN_LOG::write_cache(THD *thd, IO_CACHE *cache) > mysql_mutex_assert_owner(&LOCK_log); > if (reinit_io_cache(cache, READ_CACHE, 0, 0, 0)) > DBUG_RETURN(ER_ERROR_ON_WRITE); > + uchar *read_pos= cache->read_pos; > + > +#ifdef WITH_WSREP > + /* > + In the cache we have gtid event if , below condition is true, > + then we will simply increment the read position of cache by > + the length of Gtid_log_event. > + */ > + if (wsrep_gtid_mode && thd->variables.gtid_seq_no) > + { > + uchar *head= read_pos; > + uint data_len= uint4korr(head + EVENT_LEN_OFFSET); > + uint event_type= (uchar)head[EVENT_TYPE_OFFSET]; > + if (event_type == GTID_LOG_EVENT) > + cache->read_pos+= data_len; > + } > +#endif > + > uint length= my_b_bytes_in_cache(cache), group, carry, hdr_offs; > long val; > ulong end_log_pos_inc= 0; // each event processed adds BINLOG_CHECKSUM_LEN 2 t > > *review*> The following block is fine *but* we shall explain > what is GTID_SUPPORT, whether it actually for us. > Well , there is no such thing as GTID_SUPPORT :( , we are using ifdef without even defining gtid_support ╭─sachin@sachin-desk ~/10715/server ‹bb-10.1-10715*› ╰─$ grep -Rnie 'GTID_SUPPORT' sql/wsrep_sst.cc:290:#ifdef GTID_SUPPORT sql/wsrep_sst.cc:292:#endif /* GTID_SUPPORT */ sql/wsrep_applier.cc:130:#ifdef GTID_SUPPORT sql/wsrep_applier.cc:141:#endif /* GTID_SUPPORT */ sql/wsrep_applier.cc:309:#ifdef GTID_SUPPORT sql/wsrep_applier.cc:311:#endif /* GTID_SUPPORT */ sql/wsrep_mysqld.cc:50:#ifdef GTID_SUPPORT sql/wsrep_mysqld.cc:53:#endif /* GTID_SUPPORT */ sql/wsrep_mysqld.cc:256:#ifdef GTID_SUPPORT sql/wsrep_mysqld.cc:274:#endif /* GTID_SUPPORT */ sql/wsrep_mysqld.cc:413:#ifdef GTID_SUPPORT sql/wsrep_mysqld.cc:415:#endif /* GTID_SUPPORT */ sql/wsrep_mysqld.cc:1222:#ifdef GTID_SUPPORT sql/wsrep_mysqld.cc:1229:#endif /* GTID_SUPPORT */ sql/sql_class.h:4085:#ifdef GTID_SUPPORT sql/sql_class.h:4087:#endif /* GTID_SUPPORT */ sql/wsrep_mysqld.h:261:#ifdef GTID_SUPPORT sql/wsrep_mysqld.h:263:#endif /* GTID_SUPPORT */ > diff --git a/sql/wsrep_mysqld.cc b/sql/wsrep_mysqld.cc > index 33689a9..65751f1 100644 > --- a/sql/wsrep_mysqld.cc > +++ b/sql/wsrep_mysqld.cc > @@ -1227,6 +1227,15 @@ int wsrep_to_buf_helper( > if (!ret && writer.write(&gtid_ev)) ret= 1; > } > #endif /* GTID_SUPPORT */ > + if (wsrep_gtid_mode && thd->variables.gtid_seq_no) > + { > + Gtid_log_event gtid_event(thd, thd->variables.gtid_seq_no, > + thd->variables.gtid_domain_id, > + true, LOG_EVENT_SUPPRESS_USE_F, > + true, 0); > + gtid_event.server_id= thd->variables.server_id; > + ret= writer.write(&gtid_event); > + } > > /* if there is prepare query, add event for it */ > if (!ret && thd->wsrep_TOI_pre_query) -- Regards Sachin Setiya Software Engineer at MariaDB

1 0

[Maria-developers] Mysterious files after build
by Michal Schorm 09 Dec '17

09 Dec '17

Hello, I'd like to know more about some files, that are left after MariaDB build in Fedora. I do not ship them, however I haven't found any explanation from previous maintainers. I gathered some clues and thoughts about them. Questions are: What are those files and what they does?; Is it safe to ship them?; Should I start to ship them? /usr/bin/mysql_embedded - you (MariaDB) ship this file in client package %{_datadir}/mariadb/binary-configure - this script creates the MySQL system tables and starts the server. %{_datadir}/mariadb/magic - FS files first-bytes recoginiton? %{_datadir}/mariadb/mysql.server %{_datadir}/%{pkg_name}/mysqld_multi.server - those shouldn't be needed on systemd OSs ? /usr/bin/mytop - you (mariaDB) do not ship this file -- Michal Schorm Associate Software Engineer Core Services - Databases Team Red Hat

2 1

Re: [Maria-developers] b610986: MDEV-14593 human-readable XA RECOVER
by Sergei Golubchik 08 Dec '17

08 Dec '17

Hi, Alexey! Please, see few comments below. On Dec 07, Alexey Botchkov wrote: > revision-id: b610986c9c226e9b9cbe7c80ceaead1de0a55535 (mariadb-10.2.11-26-gb610986) > parent(s): 4d016e6ed2d32298977a66e125cbfcae39e23847 > committer: Alexey Botchkov > timestamp: 2017-12-07 23:49:12 +0400 > message: > > MDEV-14593 human-readable XA RECOVER. > > diff --git a/sql/handler.cc b/sql/handler.cc > index 7eed722..128721c 100644 > --- a/sql/handler.cc > +++ b/sql/handler.cc > @@ -1978,6 +1978,97 @@ int ha_recover(HASH *commit_list) > } > > /** > + return the XID as it appears in the SQL function's arguments. > + So this string can be passed to XA START, XA PREPARE etc... > + > + @note > + the 'buf' has to have space for at least SQL_XIDSIZE bytes. > +*/ > + > + > +/* > + 'a'..'z' 'A'..'Z', '0'..'9' > + and '-' '_' ' ' symbols don't have to be > + converted. I'd say it's overcomplication, why not to convert everything to hex? > +*/ > + > +static const char xid_needs_conv[128]= > +{ > + 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, > + 1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1, > + 0,1,1,1,1,1,1,1,1,1,1,1,1,0,1,1, > + 0,0,0,0,0,0,0,0,0,0,1,1,1,1,1,1, > + 1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, > + 0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,0, > + 1,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, > + 0,0,0,0,0,0,0,0,0,0,0,1,1,1,1,1 > +}; > + > +uint get_sql_xid(XID *xid, char *buf) > +{ > + int tot_len= xid->gtrid_length + xid->bqual_length; > + int i; > + const char *orig_buf= buf; > + > + for (i=0; i<tot_len; i++) > + { > + uchar c= ((uchar *) xid->data)[i] > + if (c >= 128 || xid_needs_conv[c]) > + break; > + } > + > + if (i >= tot_len) > + { > + /* No need to convert characters to hexadecimals. */ > + *buf++= '\''; > + memcpy(buf, xid->data, xid->gtrid_length); > + buf+= xid->gtrid_length; > + *buf++= '\''; > + if (xid->bqual_length > 0 || xid->formatID != 1) > + { > + *buf++= ','; > + *buf++= '\''; > + memcpy(buf, xid->data+xid->gtrid_length, xid->bqual_length); > + buf+= xid->bqual_length; > + *buf++= '\''; > + } > + } > + else > + { > + *buf++= 'X'; > + *buf++= '\''; > + for (i= 0; i < xid->gtrid_length; i++) > + { > + *buf++=_dig_vec_lower[((uchar*) xid->data)[i] >> 4]; > + *buf++=_dig_vec_lower[((uchar*) xid->data)[i] & 0x0f]; > + } > + *buf++= '\''; > + if (xid->bqual_length > 0 || xid->formatID != 1) > + { > + *buf++= ','; > + *buf++= 'X'; > + *buf++= '\''; > + for (; i < tot_len; i++) > + { > + *buf++=_dig_vec_lower[((uchar*) xid->data)[i] >> 4]; > + *buf++=_dig_vec_lower[((uchar*) xid->data)[i] & 0x0f]; > + } > + *buf++= '\''; > + } > + } > + > + if (xid->formatID != 1) > + { > + *buf++= ','; > + buf+= my_longlong10_to_str_8bit(&my_charset_bin, buf, > + MY_INT64_NUM_DECIMAL_DIGITS, -10, xid->formatID); > + } > + > + return buf - orig_buf; > +} > + > + > +/** > return the list of XID's to a client, the same way SHOW commands do. > > @note > diff --git a/sql/handler.h b/sql/handler.h > index 486ba56..91e3168 100644 > --- a/sql/handler.h > +++ b/sql/handler.h > @@ -628,6 +628,16 @@ struct xid_t { > }; > typedef struct xid_t XID; > > +/* > + The size of XID string representation in the form > + 'gtrid', 'bqual', formatID > + see xid_t::get_sql_string() for details. > +*/ > +#define SQL_XIDSIZE (XIDDATASIZE * 2 + 8 + MY_INT64_NUM_DECIMAL_DIGITS) > +/* The 'buf' has to have space for at least SQL_XIDSIZE bytes. */ > +uint get_sql_xid(XID *xid, char *buf); Not sure you need this as an extern function in the header. Why not to keep it static in handler.cc? > /* for recover() handlerton call */ > #define MIN_XID_LIST_SIZE 128 > #define MAX_XID_LIST_SIZE (1024*128) > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index 17e6904..d0fa642 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -1705,6 +1705,7 @@ bool my_yyoverflow(short **a, YYSTYPE **b, ulong *yystacksize); > %token WRITE_SYM /* SQL-2003-N */ > %token X509_SYM > %token XA_SYM > +%token XID_SYM > %token XML_SYM > %token XOR > %token YEAR_MONTH_SYM You forgot to add XID_SYM to the keyword_sp rule. Also I don't see why we should have CONVERT XID syntax if the output is incompatible. Perhaps it'd be better to have a different syntax? FORMAT=SQL? Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] Crash on view with union
by jerome brauge 08 Dec '17

08 Dec '17

Hi all, The following script crashs the server (built with last bb-10.2-ext sources). use test; CREATE or replace TABLE `gtfmt` ( `numgtfmt` char(10) COLLATE utf8_bin NOT NULL ) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin; create or replace view gtfmtl(numgtfmt) as select 'x' from gtfmt union select 'x' from gtfmt ; SELECT * FROM gtfmtl WHERE numgtfmt = NAME_CONST('wnumgtfmt',_utf8'QEDITIONS' COLLATE 'utf8_bin'); Stack : > mysqld.exe!my_sigabrt_handler(int sig) Line 488 C mysqld.exe!raise(int signum) Line 516 C++ mysqld.exe!abort() Line 64 C++ mysqld.exe!common_assert_to_stderr_direct(const wchar_t * const expression, const wchar_t * const file_name, const unsigned int line_number) Line 123 C++ mysqld.exe!common_assert_to_stderr<wchar_t>(const wchar_t * const expression, const wchar_t * const file_name, const unsigned int line_number) Line 142 C++ mysqld.exe!_wassert(const wchar_t * expression, const wchar_t * file_name, unsigned int line_number) Line 404 C++ mysqld.exe!Item_func::fix_fields(THD * thd, Item * * ref) Line 330 C++ mysqld.exe!Item_str_func::fix_fields(THD * thd, Item * * ref) Line 106 C++ mysqld.exe!Item_name_const::fix_fields(THD * thd, Item * * ref) Line 2186 C++ mysqld.exe!Item_func::fix_fields(THD * thd, Item * * ref) Line 361 C++ mysqld.exe!JOIN::optimize_inner() Line 1311 C++ mysqld.exe!JOIN::optimize() Line 1121 C++ mysqld.exe!st_select_lex_unit::optimize() Line 1344 C++ mysqld.exe!mysql_derived_optimize(THD * thd, LEX * lex, TABLE_LIST * derived) Line 887 C++ mysqld.exe!mysql_handle_single_derived(LEX * lex, TABLE_LIST * derived, unsigned int phases) Line 197 C++ mysqld.exe!JOIN::optimize_inner() Line 1356 C++ mysqld.exe!JOIN::optimize() Line 1121 C++ mysqld.exe!mysql_select(THD * thd, TABLE_LIST * tables, unsigned int wild_num, List<Item> & fields, Item * conds, unsigned int og_num, st_order * order, st_order * group, Item * having, st_order * proc_param, unsigned __int64 select_options, select_result * result, st_select_lex_unit * unit, st_select_lex * select_lex) Line 3758 C++ mysqld.exe!handle_select(THD * thd, LEX * lex, select_result * result, unsigned long setup_tables_done_option) Line 364 C++ mysqld.exe!execute_sqlcom_select(THD * thd, TABLE_LIST * all_tables) Line 6473 C++ mysqld.exe!mysql_execute_command(THD * thd) Line 3717 C++ mysqld.exe!mysql_parse(THD * thd, char * rawbuf, unsigned int length, Parser_state * parser_state, bool is_com_multi, bool is_next_command) Line 7935 C++ mysqld.exe!dispatch_command(enum_server_command command, THD * thd, char * packet, unsigned int packet_length, bool is_com_multi, bool is_next_command) Line 1820 C++ mysqld.exe!do_command(THD * thd) Line 1369 C++ mysqld.exe!threadpool_process_request(THD * thd) Line 366 C++ mysqld.exe!tp_callback(TP_connection * c) Line 192 C++ mysqld.exe!tp_callback(_TP_CALLBACK_INSTANCE * instance, void * context) Line 377 C++ mysqld.exe!work_callback(_TP_CALLBACK_INSTANCE * instance, void * context, _TP_WORK * work) Line 451 C++ Best regard, Jérôme.

1 0

[Maria-developers] -Wnon-virtual-dtor and scripts in
by Alexander Barkov 08 Dec '17

08 Dec '17

Hello Vlad, Are we still supposed to use these scripts in the BUILD directory: compile-pentium64-debug compile-pentium64-debug-all compile-pentium64-debug-max compile-pentium64-gcov compile-pentium64-gprof compile-pentium64-max compile-pentium64-valgrind-max I stopped using them long time ago. But these scripts set some extra warnings for gcc, so now it's easy to overlook some warnings. Monty is still using these scripts and gets those extra warnings. In particular, -Wnon-virtual-dtor is often a subject of warnings. Is there a way to enable the -Wnon-virtual-dtor for gcc in the default build, i.e. when just using "cmake" instead of BUILD scripts? Thanks!

2 2

Re: [Maria-developers] 78a1779: mysqldump fix for invisible column
by Sergei Golubchik 07 Dec '17

07 Dec '17

Hi, Sachin! Looks fine. Two comments: 1. optimization idea, see below 2. add a test case that actually restores from the dump (and checks that it was restored correctly). mysqldump.test has examples, search for $MYSQL. After that - please push! On Dec 07, sachin wrote: > revision-id: 78a1779f29ed8d2a96658583065270de8bdd14b4 (mariadb-10.3.2-61-g78a1779) > parent(s): e8c8300a1c7e7e2b99e98c5c6986df69943cb661 > author: Sachin Setiya > committer: Sachin Setiya > timestamp: 2017-12-07 16:09:16 +0530 > message: > > mysqldump fix for invisible column > > Actually there are 2 issues in the case of invisible columns > > 1st `select fields from t1` will have more fields then `select * from t1`. > So instead of `select * from t1` we are using `select a,b,invisible from t1` > these fields are supplied from `select fields from t1`. > > 2nd We are using --complete-insert when we detect that this table is using > invisible columns. > > diff --git a/client/mysqldump.c b/client/mysqldump.c > index a260065..818bc07 100644 > --- a/client/mysqldump.c > +++ b/client/mysqldump.c > @@ -2971,6 +2979,12 @@ static uint get_table_structure(char *table, char *db, char *table_type, > DBUG_RETURN(0); > } > > + while ((row= mysql_fetch_row(result))) > + { > + if (strlen(row[SHOW_EXTRA]) && strstr(row[SHOW_EXTRA],"INVISIBLE")) > + complete_insert= 1; > + } > + mysql_data_seek(result, 0); > /* > If write_data is true, then we build up insert statements for > the table's data. Note: in subsequent lines of code, this test > @@ -3006,10 +3020,16 @@ static uint get_table_structure(char *table, char *db, char *table_type, > { > dynstr_append_checked(&insert_pat, ", "); > } > - init=1; > dynstr_append_checked(&insert_pat, > quote_name(row[SHOW_FIELDNAME], name_buff, 0)); > } > + if (init) > + { > + dynstr_append_checked(&select_field_names, ", "); > + } > + dynstr_append_checked(&select_field_names, > + quote_name(row[SHOW_FIELDNAME], name_buff, 0)); > + init=1; I think, you can do it with just one loop: while ((row= mysql_fetch_row(result))) { if (strlen(row[SHOW_EXTRA]) && strstr(row[SHOW_EXTRA],"INVISIBLE")) complete_insert= 1; if (init) dynstr_append_checked(&select_field_names, ", "); dynstr_append_checked(&select_field_names, quote_name(row[SHOW_FIELDNAME], name_buff, 0)); init=1; } if (complete_insert) dynstr_append_checked(&insert_pat, select_field_names->str); > } > num_fields= mysql_num_rows(result); > mysql_free_result(result); Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] 339dd3d: Invisible Column mysqldump bug fix
by Sergei Golubchik 07 Dec '17

07 Dec '17

Hi, Sachin! Summary: 1. Add a test case, it should include a mix of tables with and without invisible columns, and columns with strange names (reserved words, spaces). 2. Always use full SELECT syntax, even if the table has no invisible columns. On Dec 05, sachin wrote: > revision-id: 339dd3d7508a1e5c7c014b5a59ed99ff0ad96542 (mariadb-10.3.2-59-g339dd3d) > parent(s): 6ed3374c8a67ae64448783ab107d1a11cb87f40c > author: Sachin Setiya > committer: Sachin Setiya > timestamp: 2017-12-05 05:35:28 +0530 > message: > > Invisible Column mysqldump bug fix > > --- > client/mysqldump.c | 46 +++++++++++++++++++++++++++++++++++++++++----- > 1 file changed, 41 insertions(+), 5 deletions(-) > > diff --git a/client/mysqldump.c b/client/mysqldump.c > index a260065..e59b203 100644 > --- a/client/mysqldump.c > +++ b/client/mysqldump.c > @@ -115,7 +115,7 @@ static my_bool verbose= 0, opt_no_create_info= 0, opt_no_data= 0, opt_no_data_m > opt_events= 0, opt_comments_used= 0, > opt_alltspcs=0, opt_notspcs= 0, opt_logging, > opt_drop_trigger= 0 ; > -static my_bool insert_pat_inited= 0, debug_info_flag= 0, debug_check_flag= 0; > +static my_bool insert_pat_inited= 0, debug_info_flag= 0, debug_check_flag= 0, invisble_table= 0; typo, *invisible* > static ulong opt_max_allowed_packet, opt_net_buffer_length; > static MYSQL mysql_connection,*mysql=0; > static DYNAMIC_STRING insert_pat; > @@ -2726,7 +2726,7 @@ static uint get_table_structure(char *table, char *db, char *table_type, > complete_insert= 0; > if ((write_data= !(*ignore_flag & IGNORE_DATA))) > { > - complete_insert= opt_complete_insert; > + invisble_table= complete_insert= opt_complete_insert; > if (!insert_pat_inited) > { > insert_pat_inited= 1; > @@ -2971,6 +2971,12 @@ static uint get_table_structure(char *table, char *db, char *table_type, > DBUG_RETURN(0); > } > > + while ((row= mysql_fetch_row(result))) > + { > + if (strlen(row[SHOW_EXTRA]) && strstr(row[SHOW_EXTRA],"INVISIBLE")) > + invisble_table= complete_insert= opt_complete_insert= 1; I think, you need to set invisible_table flag per table, just as you do now, but you should not change complete_insert or opt_complete_insert. That is, if a database has many tables, some with invisible columns and some have only normal columns - tables with invisible columns should use complete_insert, while other tables should not. Basically, INSERT should contain column names if (invisible_table || opt_complete_insert) > + } > + mysql_data_seek(result, 0); > /* > If write_data is true, then we build up insert statements for > the table's data. Note: in subsequent lines of code, this test > @@ -3617,7 +3623,7 @@ static void dump_table(char *table, char *db) > { > char ignore_flag; > char buf[200], table_buff[NAME_LEN+3]; > - DYNAMIC_STRING query_string; > + DYNAMIC_STRING query_string, select_field_names; > char table_type[NAME_LEN]; > char *result_table, table_buff2[NAME_LEN*2+3], *opt_quoted_table; > int error= 0; > @@ -3687,7 +3693,25 @@ static void dump_table(char *table, char *db) > verbose_msg("-- Sending SELECT query...\n"); > > init_dynamic_string_checked(&query_string, "", 1024, 1024); > + init_dynamic_string_checked(&select_field_names, "", 1024, 1024); > > + if (invisble_table) > + { > + /* > + Copy field name from insert_pat into select_field_names because > + select * from table will leave hidden columns. > + We will search for "(" in insert_pat and will copy the whole data > + into select_field_names , and then we will search for ")" in > + select_field_names and decrese the lenght accordingly. > + */ > + char *ptr= insert_pat.str; > + while ( *ptr++ != '(' ){} > + dynstr_append_checked(&select_field_names, ptr); > + ptr= &select_field_names.str[select_field_names.length - 1]; > + while ( *ptr-- != ')') > + select_field_names.length--; > + select_field_names.str[select_field_names.length- 1]= 0; please, check how INSERT code is doing it, you need to quote column names. > + } > if (path) > { > char filename[FN_REFLEN], tmp_path[FN_REFLEN]; > @@ -3708,7 +3732,13 @@ static void dump_table(char *table, char *db) > > /* now build the query string */ > > - dynstr_append_checked(&query_string, "SELECT /*!40001 SQL_NO_CACHE */ * INTO OUTFILE '"); > + dynstr_append_checked(&query_string, "SELECT /*!40001 SQL_NO_CACHE */ "); > + if (invisble_table) > + dynstr_append_checked(&query_string, select_field_names.str); > + else > + dynstr_append_checked(&query_string, "*"); I don't think you need to bother, you can always have a complete column list here. It should not break anything. > + dynstr_append_checked(&query_string, " INTO OUTFILE '"); > + dynstr_free(&select_field_names); > dynstr_append_checked(&query_string, filename); > dynstr_append_checked(&query_string, "'"); > > @@ -3757,7 +3787,13 @@ static void dump_table(char *table, char *db) > "\n--\n-- Dumping data for table %s\n--\n", > fix_for_comment(result_table)); > > - dynstr_append_checked(&query_string, "SELECT /*!40001 SQL_NO_CACHE */ * FROM "); > + dynstr_append_checked(&query_string, "SELECT /*!40001 SQL_NO_CACHE */ "); > + if (invisble_table) > + dynstr_append_checked(&query_string, select_field_names.str); > + else > + dynstr_append_checked(&query_string, "*"); same > + dynstr_free(&select_field_names); > + dynstr_append_checked(&query_string, " FROM "); > dynstr_append_checked(&query_string, result_table); > > if (where) > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 1

Re: [Maria-developers] 0f47171: MDEV-14005 Remove need for Partition Key to be part of Primary Key.
by Sergei Golubchik 06 Dec '17

06 Dec '17

Hi, Alexey! On Dec 04, Alexey Botchkov wrote: > revision-id: 0f4717118e135c178fc1b12065e33ac8a5e21d0b (mariadb-10.3.2-82-g0f47171) > parent(s): ddac2d7a1ef725361c7e83b819cb0bec72db024a > committer: Alexey Botchkov > timestamp: 2017-12-04 17:45:13 +0400 > message: > > MDEV-14005 Remove need for Partition Key to be part of Primary Key. > > Now we check all partitions for the unique keys if necessary > in ha_partition::write_row and ha_partition::update_row. > > diff --git a/sql/ha_partition.cc b/sql/ha_partition.cc > index 6bebf5d..12a6287 100644 > --- a/sql/ha_partition.cc > +++ b/sql/ha_partition.cc > @@ -372,6 +372,7 @@ void ha_partition::init_handler_variables() > part_share= NULL; > m_new_partitions_share_refs.empty(); > m_part_ids_sorted_by_num_of_records= NULL; > + m_cu_errkey= (uint) -1; > > #ifdef DONT_HAVE_TO_BE_INITALIZED > m_start_key.flag= 0; > @@ -4088,6 +4089,79 @@ void ha_partition::try_semi_consistent_read(bool yes) > } > > > +int ha_partition::check_files_for_key(uchar *key, int n_key, > + int partition_to_skip, int *res) > +{ > +// uint num_subparts= part_info->num_subparts; > + for (uint n=0; n < m_part_info->num_parts; n++) > + { > + handler *f= m_file[n]; > + int ires; > + > + if ((int) n == partition_to_skip) > + continue; > + > + f->active_index= n_key; > + if ((*res= f->extra(HA_EXTRA_KEYREAD)) || > + (*res= f->ha_index_init(n_key, FALSE))) > + return 1; > + > + *res= f->index_read_map(m_part_info->table->record[0], > + key, HA_WHOLE_KEY, HA_READ_KEY_EXACT); use index_read_idx_map here > + ires= f->ha_index_end(); > + > + if (*res == HA_ERR_KEY_NOT_FOUND) > + continue; > + > + if (*res == 0) > + *res= ires ? ires : HA_ERR_FOUND_DUPP_KEY; > + > + m_last_part= n; > + m_cu_errkey= n_key; > + return 1; > + } > + > + *res= 0; > + return 0; > +} > + > + > +int ha_partition::check_uniques_insert(uchar *buf, > + int partition_to_skip, int *res) > +{ > + uint n_key; > + > + for (n_key=0; n_key < m_part_info->n_uniques_to_check; n_key++) > + { can you use key_copy() here? > + uchar *cbuf= m_part_info->unique_key_buf[0]; > + KEY *ckey= m_part_info->uniques_to_check[n_key]; > + uint n; > + > + for (n=0; n < ckey->user_defined_key_parts; n++) > + { > + const KEY_PART_INFO *cpart= ckey->key_part + n; > + uint maybe_null= MY_TEST(cpart->null_bit); > + Field *f= cpart->field; > + my_ptrdiff_t ofs= buf-table->record[0]; > + > + f->move_field_offset(ofs); > + if (maybe_null) > + cbuf[0]= f->is_null(); > + f->get_key_image(cbuf+maybe_null, cpart->length, Field::itRAW); > + cbuf+= cpart->store_length; > + f->move_field_offset(-ofs); > + } > + > + if (check_files_for_key(m_part_info->unique_key_buf[0], n_key, > + partition_to_skip, res)) > + return 1; > + } > + > + *res= 0; > + return 0; > +} > + > + > /**************************************************************************** > MODULE change record > ****************************************************************************/ > @@ -4199,6 +4273,11 @@ int ha_partition::write_row(uchar * buf) > error= HA_ERR_NOT_IN_LOCK_PARTITIONS; > goto exit; > } > + > + if (table->part_info->n_uniques_to_check && > + check_uniques_insert(buf, part_id, &error)) > + goto exit; This would _probably_ introduce the race condition I've mentioned in a comment to MDEV-14005. Say, you have a table with two partitions and a unique key. Now, two threads try to insert rows, the first thread inserts 1 into the first partition (there are more columns in a table and a partitioning key on those columns, but I omit these details for simplicity). And a second thread inserts 1 into the second partition. So, the first thread checks for 1 in the second partition, finds nothing. Second thread checks for 1 in the first partition, finds nothing. Then the first thread inserts 1 into the first partition, second thread inserts 1 into the second partition. Monty's idea was to insert optimistically - first you insert, then check for duplicates. Not ideal, in the above example you can have both threads getting a duplicate key error. There could be other, e.g. locking, solutions too. > m_last_part= part_id; > DBUG_PRINT("info", ("Insert in partition %d", part_id)); > start_part_bulk_insert(thd, part_id); Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] MDEV-13995 MAX(timestamp) returns a wrong result near DST change
by Alexander Barkov 04 Dec '17

04 Dec '17

Hi Sergei, Please review a patch for MDEV-13995. Thanks!

1 0

[Maria-developers] Review for MDEV-8320 Allow index usage for DATE(datetime_column) = const
by Alexander Barkov 30 Nov '17

30 Nov '17

Hello Sergey, I generally like your patch for MDEV-8320: http://lists.askmonty.org/pipermail/commits/2017-November/011680.html However, there is one problem with it. We're changing the code to have data type plugins. Every data type plugin will provide its own list of SQL functions. Those data type specific functions should be able to use the same optimization approach without having to change the server code. Therefore, everything related to YEAR(field) transformation should reside inside Item_func_year, everything related to DATE(field) transformation should reside inside Item_date_typecast. I reorganized the code to implement this idea. Note, also I changed the logic behind DATE(indexed_date_field) transformation slightly. The DATE() just gets removed. DATE(indexed_date_field) = const is changed to: indexed_date_field = const instead of: indexed_date_field BETWEEN ... AND ... Please have a look: sargable_date_cond.result is slightly different in my version. Greetings.

1 1

[Maria-developers] Delete vs multi_delete performance issue
by jerome brauge 29 Nov '17

29 Nov '17

Hi All, I've a question on kind of delete on whose we have performance issues. Consider two tables: CREATE TABLE `gapfl` ( `pgegapfl` varchar(150) COLLATE utf8_bin NOT NULL, `blkgapfl` varchar(30) COLLATE utf8_bin NOT NULL, `fldgapfl` varchar(30) COLLATE utf8_bin NOT NULL, `ordgapfl` int(11) NOT NULL, ... `stlgapfl` varchar(30) COLLATE utf8_bin DEFAULT NULL, UNIQUE KEY `gapfl_fld` (`pgegapfl`,`blkgapfl`,`fldgapfl`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin and CREATE TABLE `gapge` ( `nuugapge` varchar(150) COLLATE utf8_bin NOT NULL, `nuigapge` int(11) NOT NULL, `frmgapge` varchar(30) COLLATE utf8_bin NOT NULL, `prsgapge` varchar(60) COLLATE utf8_bin NOT NULL, `typgapge` varchar(2) COLLATE utf8_bin NOT NULL, `catgapge` varchar(1) COLLATE utf8_bin NOT NULL, `numgapge` int(11) NOT NULL, `aplgapge` varchar(8) COLLATE utf8_bin DEFAULT NULL, `tragapge` varchar(8) COLLATE utf8_bin DEFAULT NULL, ... `mafgapge` varchar(1) COLLATE utf8_bin NOT NULL, UNIQUE KEY `gapge_nuu` (`nuugapge`), UNIQUE KEY `gapge_frm` (`frmgapge`,`prsgapge`,`typgapge`,`numgapge`,`tragapge`) ) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin Table "gapfl " contains 364740 rows Table "gapge " contains 25423 rows The following delete tooks more than 38 seconds to delete 0 rows . DELETE FROM gapfl WHERE pgegapfl IN ( SELECT nuugapge FROM gapge WHERE frmgapge = 'AMCEVA' AND prsgapge = 'QUALIAC.*.*') It explain plan is : +------+--------------------+-------+-----------------+---------------------+-----------+---------+------+--------+-------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +------+--------------------+-------+-----------------+---------------------+-----------+---------+------+--------+-------------+ | 1 | PRIMARY | gapfl | ALL | NULL | NULL | NULL | NULL | 356417 | Using where | | 2 | DEPENDENT SUBQUERY | gapge | unique_subquery | gapge_nuu,gapge_frm | gapge_nuu | 452 | func | 1 | Using where | +------+--------------------+-------+-----------------+---------------------+-----------+---------+------+--------+-------------+ It's not really good and I don't understand why the subselect is consider as a "DEPENDENT SUBQUERY" (there is no correlation columns). Now, I convert this delete into a multi_delete : DELETE gapfl FROM gapfl WHERE pgegapfl IN ( SELECT nuugapge FROM gapge WHERE frmgapge = 'AMCEVA' AND prsgapge = 'QUALIAC.*.*') It tooks 0.00 second and it explain plan is : +------+-------------+-------+------+---------------------+-----------+---------+--------------------+------+--------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +------+-------------+-------+------+---------------------+-----------+---------+--------------------+------+--------------------------+ | 1 | PRIMARY | gapge | ref | gapge_nuu,gapge_frm | gapge_frm | 274 | const,const | 1 | Using where; Using index | | 1 | PRIMARY | gapfl | ref | gapfl_fld | gapfl_fld | 452 | dvt.gapge.nuugapge | 7 | | +------+-------------+-------+------+---------------------+-----------+---------+--------------------+------+--------------------------+ as expected. Is there any way to have the right plan with single delete ? Or do we must convert all our complex update to multi_update ? Sorry if it's not the right place to post this query. Regards.

1 0

[Maria-developers] Error 1422 - Explicit or implicit commit is not allowed in stored function or trigger
by jerome brauge 29 Nov '17

29 Nov '17

Hello Alexander, I'm facing to a new issue with my stored procedures. Some of them are called directly or from trigger. When these procedures are called directly, they manage the transaction and when they are called from trigger, they doesn't manage the transaction. This works fine on Oracle, SQLServer , ... but fail on Mariadb because the check is done in is_not_allowed_in_function() before the call. use test; set sql_mode=oracle; create or replace table t1 (c1 int); insert into t1 values (1); delimiter / create or replace procedure p1(manage_transactionnal int) is begin if manage_transactionnal = 1 then commit; end if; end; / create or replace trigger tg1 after delete on t1 for each row begin call p1(0); end; / delimiter ; start transaction; delete from t1; rollback; What do you think about this ? Regards, Jérôme.

2 2

[Maria-developers] MariaDB 10.2 now available on CentOS 6 &7 via SCL !!
by Michal Schorm 29 Nov '17

29 Nov '17

Hello everybody, In case you missed it, I just want to share an announce of availability of MariaDB 10.2 on CentOS 6 & 7 as a Software collections. Here are the announce emails from CentOS mailing list, which contains most basic howto: https://lists.centos.org/pipermail/centos-announce/2017-November/022640.html https://lists.centos.org/pipermail/centos-announce/2017-November/022642.html -- This made MariaDB 10.2 a big step closer to a new mass of users, because the standard MDB version in CentOS & RHEL is 5.5. It took quite an effort to fine-tune and perk-up the MDB 10.2 since its first attemps in Fedora, however it is always pleasure to cooperate with developers like you :) -- Michal Schorm Associate Software Engineer Core Services - Databases Team Red Hat

1 0

[Maria-developers] Interaction between rpl_slave_state and rpl_binlog_state
by Sachin Setiya 28 Nov '17

28 Nov '17

Hi Kristian, Andrei I have some question related to rpl_slave_state. Suppose A circular async replication between A < -- > B (gtid_ignore_duplicates on) Now, we set some temp server_id on server A , lets say `X`. Now the problem is each event group which originates from A is executed 2 times. For example we insert into table t1 and gtid is 0-X-2. The event goes to slave B B applies it, And send it back to A, Since its server_is different from A global server_id , this event is not filtered in queue_event(). Now event goes to handle_sql thread , And check_duplicate_gtid function is called. Since rpl_slave_state::hash does not have a element whose sequence_no is >= 2, So this event is applied again. Andrei suggested a solution of checking rpl_binlog_state in check_duplicate_gtid, This solution solves some problem but creates some problem. (multi_source.gtid_ignore_duplicates fails in master_sync, test for mdev_10715 also fails ). Reason for this is sync_with_master calls master_gtid_wait , which internally calls gtid_waiting::wait_for_gtid , and this checks rpl_salve_state::hash. Although this method can also be patched for to look for rpl_gtid_global_binlog_state. But I am not sure whether this will solve all problem or create new problem. Another solution might be somehow update rpl_slave_state::hash when we write gtid_event in log. But this does not make sense. rpl_slave_state should be used for slave replication usage. I think we need a more better solution for this. -- Regards sachin

3 4

[Maria-developers] Add hidden level 2/3 (PSEDUO_HIDDEN_COLUMN / COMPLETELY_HIDDEN_COLUMN) to mariadb
by Sachin Setiya 27 Nov '17

27 Nov '17

Hello Everyone! Currently I am implementing 10177. But I have one doubt I am not sure how to implement a function which will create hidden level 2/3 field for server. Suppose we create a function (say add_hidden_field(.....)) which will add the field to table T. But the issue is where should we call this function in mysql_prepare_create_table / mysql_prepare_alter_table or init_from_binary_frm_image Doing it from init_from_binary_frm_image make sense because this way server and slave frm of table T will be same (We have decided that we wont log hidden level 2/3 field to binlog) , But the issue is optimizer wont be ready to decide whether we require a hidden level 3 field at that time. We can also do it from mysql_prepare_create_table , but I dont think this wont be good idea because at this time optimizer wont be ready plus this will complicate binlog handling. Or may be we should have a function which can modify table object any time. But I am not sure if it is possible/good idea ? What you guys think ? -- Regards Sachin Setiya Software Engineer at MariaDB

2 4

Re: [Maria-developers] [Commits] b2571e5: Removed ut_crc32_byte_by_byte: never used
by Marko Mäkelä 23 Nov '17

23 Nov '17

Indeed, the functions and function pointers were never used. On Thu, Nov 23, 2017 at 12:36 PM, Sergey Vojtovich <svoj(a)mariadb.org> wrote: > revision-id: b2571e534f9eed873cc9b1ce5f48b109c1ee0fcc > (mariadb-10.3.2-65-gb2571e5) > parent(s): 0bfe5e349210c01404d20dfc8660d5224bd9c658 > committer: Sergey Vojtovich > timestamp: 2017-11-23 14:35:50 +0400 > message: > > Removed ut_crc32_byte_by_byte: never used > > --- > storage/innobase/include/ut0crc32.h | 4 ---- > storage/innobase/ut/ut0crc32.cc | 48 ------------------------------ > ------- > 2 files changed, 52 deletions(-) > > diff --git a/storage/innobase/include/ut0crc32.h > b/storage/innobase/include/ut0crc32.h > index 36b389b..509ac09 100644 > --- a/storage/innobase/include/ut0crc32.h > +++ b/storage/innobase/include/ut0crc32.h > @@ -51,10 +51,6 @@ extern ut_crc32_func_t ut_crc32; > when converting byte strings to integers internally. */ > extern ut_crc32_func_t ut_crc32_legacy_big_endian; > > -/** Pointer to CRC32-byte-by-byte calculation function (byte order > agnostic, > -but very slow). */ > -extern ut_crc32_func_t ut_crc32_byte_by_byte; > - > extern const char* ut_crc32_implementation; > > #endif /* ut0crc32_h */ > diff --git a/storage/innobase/ut/ut0crc32.cc b/storage/innobase/ut/ > ut0crc32.cc > index 386d32f..6383e6d 100644 > --- a/storage/innobase/ut/ut0crc32.cc > +++ b/storage/innobase/ut/ut0crc32.cc > @@ -93,10 +93,6 @@ ut_crc32_func_t ut_crc32; > when converting byte strings to integers internally. */ > ut_crc32_func_t ut_crc32_legacy_big_endian; > > -/** Pointer to CRC32-byte-by-byte calculation function (byte order > agnostic, > -but very slow). */ > -ut_crc32_func_t ut_crc32_byte_by_byte; > - > /** Text description of CRC32 implementation */ > const char* ut_crc32_implementation; > > @@ -409,26 +405,6 @@ ut_crc32_legacy_big_endian_hw( > > return(~crc); > } > - > -/** Calculates CRC32 using hardware/CPU instructions. > -This function processes one byte at a time (very slow) and thus it does > -not depend on the byte order of the machine. > -@param[in] buf data over which to calculate CRC32 > -@param[in] len data length > -@return CRC-32C (polynomial 0x11EDC6F41) */ > -uint32_t > -ut_crc32_byte_by_byte_hw( > - const byte* buf, > - ulint len) > -{ > - uint32_t crc = 0xFFFFFFFFU; > - > - while (len > 0) { > - ut_crc32_8_hw(&crc, &buf, &len); > - } > - > - return(~crc); > -} > #endif /* defined(__GNUC__) && defined(__x86_64__) */ > > /* CRC32 software implementation. */ > @@ -669,28 +645,6 @@ ut_crc32_legacy_big_endian_sw( > return(~crc); > } > > -/** Calculates CRC32 in software, without using CPU instructions. > -This function processes one byte at a time (very slow) and thus it does > -not depend on the byte order of the machine. > -@param[in] buf data over which to calculate CRC32 > -@param[in] len data length > -@return CRC-32C (polynomial 0x11EDC6F41) */ > -uint32_t > -ut_crc32_byte_by_byte_sw( > - const byte* buf, > - ulint len) > -{ > - uint32_t crc = 0xFFFFFFFFU; > - > - ut_a(ut_crc32_slice8_table_initialized); > - > - while (len > 0) { > - ut_crc32_8_sw(&crc, &buf, &len); > - } > - > - return(~crc); > -} > - > /********************************************************************//** > Initializes the data structures used by ut_crc32*(). Does not do any > allocations, would not hurt if called twice, but would be pointless. */ > @@ -701,7 +655,6 @@ ut_crc32_init() > ut_crc32_slice8_table_init(); > ut_crc32 = ut_crc32_sw; > ut_crc32_legacy_big_endian = ut_crc32_legacy_big_endian_sw; > - ut_crc32_byte_by_byte = ut_crc32_byte_by_byte_sw; > ut_crc32_implementation = "Using generic crc32 instructions"; > > #if defined(__GNUC__) && defined(__x86_64__) > @@ -735,7 +688,6 @@ ut_crc32_init() > if (features_ecx & 1 << 20) { > ut_crc32 = ut_crc32_hw; > ut_crc32_legacy_big_endian = ut_crc32_legacy_big_endian_hw; > - ut_crc32_byte_by_byte = ut_crc32_byte_by_byte_hw; > ut_crc32_implementation = "Using SSE2 crc32 instructions"; > } > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- Marko Mäkelä, Lead Developer InnoDB MariaDB Corporation DON’T MISS M|18 MariaDB User Conference February 26 - 27, 2018 New York City https://m18.mariadb.com/

1 0

Re: [Maria-developers] [Commits] f28af02: Removed HW acceleration for big endian checksum
by Marko Mäkelä 23 Nov '17

23 Nov '17

Looks OK to me. This can only slightly punish the very few people who are upgrading from a buggy MySQL 5.6 or MariaDB 10.0 or 10.1 installation on a big-endian machine (such as SPARC) into a newer version. And those users could invoke innochecksum to rewrite the page checksums. On Thu, Nov 23, 2017 at 12:57 PM, Sergey Vojtovich <svoj(a)mariadb.org> wrote: > revision-id: f28af027a0439d760c6d26eb67a932e97eb665d7 > (mariadb-10.3.2-66-gf28af02) > parent(s): b2571e534f9eed873cc9b1ce5f48b109c1ee0fcc > committer: Sergey Vojtovich > timestamp: 2017-11-23 14:53:16 +0400 > message: > > Removed HW acceleration for big endian checksum > > Big endian checksum is needed to workaround 10+ years old bug, where > checksum > was calculated incorrectly on big endian hardware. We can still checksum > such > tablespaces using software implementation of CRC32. > > --- > storage/innobase/include/ut0crc32.h | 4 +- > storage/innobase/ut/ut0crc32.cc | 89 +----------------------------- > ------- > 2 files changed, 3 insertions(+), 90 deletions(-) > > diff --git a/storage/innobase/include/ut0crc32.h > b/storage/innobase/include/ut0crc32.h > index 509ac09..527a3e81 100644 > --- a/storage/innobase/include/ut0crc32.h > +++ b/storage/innobase/include/ut0crc32.h > @@ -47,9 +47,9 @@ typedef uint32_t (*ut_crc32_func_t)(const byte* > ptr, ulint len); > /** Pointer to CRC32 calculation function. */ > extern ut_crc32_func_t ut_crc32; > > -/** Pointer to CRC32 calculation function, which uses big-endian byte > order > +/** CRC32 calculation function, which uses big-endian byte order > when converting byte strings to integers internally. */ > -extern ut_crc32_func_t ut_crc32_legacy_big_endian; > +extern uint32_t ut_crc32_legacy_big_endian(const byte* buf, ulint len); > > extern const char* ut_crc32_implementation; > > diff --git a/storage/innobase/ut/ut0crc32.cc b/storage/innobase/ut/ > ut0crc32.cc > index 6383e6d..36c52b7 100644 > --- a/storage/innobase/ut/ut0crc32.cc > +++ b/storage/innobase/ut/ut0crc32.cc > @@ -89,10 +89,6 @@ mysys/my_perf.c, contributed by Facebook under the > following license. > /** Pointer to CRC32 calculation function. */ > ut_crc32_func_t ut_crc32; > > -/** Pointer to CRC32 calculation function, which uses big-endian byte > order > -when converting byte strings to integers internally. */ > -ut_crc32_func_t ut_crc32_legacy_big_endian; > - > /** Text description of CRC32 implementation */ > const char* ut_crc32_implementation; > > @@ -238,37 +234,6 @@ ut_crc32_64_hw( > *len -= 8; > } > > -/** Calculate CRC32 over 64-bit byte string using a hardware/CPU > instruction. > -The byte string is converted to a 64-bit integer using big endian byte > order. > -@param[in,out] crc crc32 checksum so far when this function is called, > -when the function ends it will contain the new checksum > -@param[in,out] data data to be checksummed, the pointer will be > advanced > -with 8 bytes > -@param[in,out] len remaining bytes, it will be decremented with 8 */ > -inline > -void > -ut_crc32_64_legacy_big_endian_hw( > - uint32_t* crc, > - const byte** data, > - ulint* len) > -{ > - uint64_t data_int = *reinterpret_cast<const > uint64_t*>(*data); > - > -#ifndef WORDS_BIGENDIAN > - data_int = ut_crc32_swap_byteorder(data_int); > -#else > - /* Currently we only support x86_64 (little endian) CPUs. In case > - some big endian CPU supports a CRC32 instruction, then maybe we > will > - NOT need a byte order swap here. */ > -#error Dont know how to handle big endian CPUs > -#endif /* WORDS_BIGENDIAN */ > - > - *crc = ut_crc32_64_low_hw(*crc, data_int); > - > - *data += 8; > - *len -= 8; > -} > - > /** Calculates CRC32 using hardware/CPU instructions. > @param[in] buf data over which to calculate CRC32 > @param[in] len data length > @@ -355,56 +320,6 @@ ut_crc32_hw( > > return(~crc); > } > - > -/** Calculates CRC32 using hardware/CPU instructions. > -This function uses big endian byte ordering when converting byte sequence > to > -integers. > -@param[in] buf data over which to calculate CRC32 > -@param[in] len data length > -@return CRC-32C (polynomial 0x11EDC6F41) */ > -uint32_t > -ut_crc32_legacy_big_endian_hw( > - const byte* buf, > - ulint len) > -{ > - uint32_t crc = 0xFFFFFFFFU; > - > - /* Calculate byte-by-byte up to an 8-byte aligned address. After > - this consume the input 8-bytes at a time. */ > - while (len > 0 && (reinterpret_cast<uintptr_t>(buf) & 7) != 0) { > - ut_crc32_8_hw(&crc, &buf, &len); > - } > - > - while (len >= 128) { > - /* This call is repeated 16 times. 16 * 8 = 128. */ > - ut_crc32_64_legacy_big_endian_hw(&crc, &buf, &len); > - ut_crc32_64_legacy_big_endian_hw(&crc, &buf, &len); > - ut_crc32_64_legacy_big_endian_hw(&crc, &buf, &len); > - ut_crc32_64_legacy_big_endian_hw(&crc, &buf, &len); > - ut_crc32_64_legacy_big_endian_hw(&crc, &buf, &len); > - ut_crc32_64_legacy_big_endian_hw(&crc, &buf, &len); > - ut_crc32_64_legacy_big_endian_hw(&crc, &buf, &len); > - ut_crc32_64_legacy_big_endian_hw(&crc, &buf, &len); > - ut_crc32_64_legacy_big_endian_hw(&crc, &buf, &len); > - ut_crc32_64_legacy_big_endian_hw(&crc, &buf, &len); > - ut_crc32_64_legacy_big_endian_hw(&crc, &buf, &len); > - ut_crc32_64_legacy_big_endian_hw(&crc, &buf, &len); > - ut_crc32_64_legacy_big_endian_hw(&crc, &buf, &len); > - ut_crc32_64_legacy_big_endian_hw(&crc, &buf, &len); > - ut_crc32_64_legacy_big_endian_hw(&crc, &buf, &len); > - ut_crc32_64_legacy_big_endian_hw(&crc, &buf, &len); > - } > - > - while (len >= 8) { > - ut_crc32_64_legacy_big_endian_hw(&crc, &buf, &len); > - } > - > - while (len > 0) { > - ut_crc32_8_hw(&crc, &buf, &len); > - } > - > - return(~crc); > -} > #endif /* defined(__GNUC__) && defined(__x86_64__) */ > > /* CRC32 software implementation. */ > @@ -600,7 +515,7 @@ integers. > @param[in] len data length > @return CRC-32C (polynomial 0x11EDC6F41) */ > uint32_t > -ut_crc32_legacy_big_endian_sw( > +ut_crc32_legacy_big_endian( > const byte* buf, > ulint len) > { > @@ -654,7 +569,6 @@ ut_crc32_init() > { > ut_crc32_slice8_table_init(); > ut_crc32 = ut_crc32_sw; > - ut_crc32_legacy_big_endian = ut_crc32_legacy_big_endian_sw; > ut_crc32_implementation = "Using generic crc32 instructions"; > > #if defined(__GNUC__) && defined(__x86_64__) > @@ -687,7 +601,6 @@ ut_crc32_init() > > if (features_ecx & 1 << 20) { > ut_crc32 = ut_crc32_hw; > - ut_crc32_legacy_big_endian = ut_crc32_legacy_big_endian_hw; > ut_crc32_implementation = "Using SSE2 crc32 instructions"; > } > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- Marko Mäkelä, Lead Developer InnoDB MariaDB Corporation DON’T MISS M|18 MariaDB User Conference February 26 - 27, 2018 New York City https://m18.mariadb.com/

1 0

Re: [Maria-developers] [Commits] 14bd406: Less code to depend on ut_crc32_init()
by Marko Mäkelä 23 Nov '17

23 Nov '17

Thanks, this looks OK to me. I guess the function ut_crc32_init() will be made optional separately. On Thu, Nov 23, 2017 at 1:22 PM, Sergey Vojtovich <svoj(a)mariadb.org> wrote: > revision-id: 14bd406970c9bb18c744a452359aa4505f325b2d > (mariadb-10.3.2-67-g14bd406) > parent(s): f28af027a0439d760c6d26eb67a932e97eb665d7 > committer: Sergey Vojtovich > timestamp: 2017-11-23 15:22:10 +0400 > message: > > Less code to depend on ut_crc32_init() > > --- > storage/innobase/include/ut0crc32.h | 1 + > storage/innobase/ut/ut0crc32.cc | 20 +++++++------------- > 2 files changed, 8 insertions(+), 13 deletions(-) > > diff --git a/storage/innobase/include/ut0crc32.h > b/storage/innobase/include/ut0crc32.h > index 527a3e81..32ad066 100644 > --- a/storage/innobase/include/ut0crc32.h > +++ b/storage/innobase/include/ut0crc32.h > @@ -51,6 +51,7 @@ extern ut_crc32_func_t ut_crc32; > when converting byte strings to integers internally. */ > extern uint32_t ut_crc32_legacy_big_endian(const byte* buf, ulint len); > > +/** Text description of CRC32 implementation */ > extern const char* ut_crc32_implementation; > > #endif /* ut0crc32_h */ > diff --git a/storage/innobase/ut/ut0crc32.cc b/storage/innobase/ut/ > ut0crc32.cc > index 36c52b7..48f6b82 100644 > --- a/storage/innobase/ut/ut0crc32.cc > +++ b/storage/innobase/ut/ut0crc32.cc > @@ -86,12 +86,6 @@ mysys/my_perf.c, contributed by Facebook under the > following license. > #include "univ.i" > #include "ut0crc32.h" > > -/** Pointer to CRC32 calculation function. */ > -ut_crc32_func_t ut_crc32; > - > -/** Text description of CRC32 implementation */ > -const char* ut_crc32_implementation; > - > /** Swap the byte order of an 8 byte integer. > @param[in] i 8-byte integer > @return 8-byte integer */ > @@ -125,6 +119,13 @@ ut_crc32_power8( > { > return crc32c_vpmsum(0, buf, len); > } > + > +ut_crc32_func_t ut_crc32 = ut_crc32_power8; > +const char* ut_crc32_implementation = "Using POWER8 crc32 > instructions"; > +#else > +uint32_t ut_crc32_sw(const byte* buf, ulint len); > +ut_crc32_func_t ut_crc32 = ut_crc32_sw; > +const char* ut_crc32_implementation = "Using generic crc32 > instructions"; > #endif > > #if defined(__GNUC__) && defined(__x86_64__) > @@ -568,8 +569,6 @@ ut_crc32_init() > /*===========*/ > { > ut_crc32_slice8_table_init(); > - ut_crc32 = ut_crc32_sw; > - ut_crc32_implementation = "Using generic crc32 instructions"; > > #if defined(__GNUC__) && defined(__x86_64__) > uint32_t vend[3]; > @@ -603,10 +602,5 @@ ut_crc32_init() > ut_crc32 = ut_crc32_hw; > ut_crc32_implementation = "Using SSE2 crc32 instructions"; > } > - > -#elif defined(HAVE_CRC32_VPMSUM) > - ut_crc32 = ut_crc32_power8; > - ut_crc32_implementation = "Using POWER8 crc32 instructions"; > #endif > - > } > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- Marko Mäkelä, Lead Developer InnoDB MariaDB Corporation DON’T MISS M|18 MariaDB User Conference February 26 - 27, 2018 New York City https://m18.mariadb.com/

1 0

[Maria-developers] IB: trx_t:: start_time and start_time_micro
by Aleksey Midenkov 21 Nov '17

21 Nov '17

Hello! Different semantics of these 2 members doesn't look good: time_t start_time; /*!< time the state last time became TRX_STATE_ACTIVE */ ib_uint64_t start_time_micro; /*!< start time of transaction in microseconds */ Either `start_time_micro` should be drawn to the base of `start_time` or it should be explicitly commented that it has nothing to do with `start_time`. I.e. there is nothing wrong on algorithmic level, but such data members may (and certainly would) lead developers to misuse of `start_time_micro`. -- All the best, Aleksey Midenkov @midenok

2 2

[Maria-developers] Please review MDEV-11292 Item_func_sp::fix_length_and_dec() does not set collation derivation properly
by Alexander Barkov 20 Nov '17

20 Nov '17

Hello Alexey, Please review a patch for MDEV-11292 (for 10.3). Thanks.

2 2

[Maria-developers] MariaDB GUI: BSL 1.1 - Debate
by Juan Telleria 19 Nov '17

19 Nov '17

Dear MariaDb Developers, I am writing because I would like to open a debate whether if MariaDB, now that has gained investment from Microsoft, Alibaba, and the European Union, shall create its own SQL Graphical User Interface, as good as (or better than) MySQL Workbench. I agree that MariaDB includes by default HeidiSQL in its Windows builds, but in my opinion, if MariaDB really wants to look Oracle's MySQL face to face, HeidiSQL is not nearly as good as MySQL Workbench. If this wants to be done: 1st) The brand new GUI shall have a Business Source Licence v1.1, which only companies which invest in MariaDB would be able to use. 2nd) In 3 years now, MariaDB would be better positioned and have a fully Open Source GUI which would allow MariaDB to look face to face to Oracle. 3rd) And in 4/5 years now, MariaDB Corporation shall be brave enough to to continue Developing this GUI as a truly open source product, without the BSL, and adopting a purely GPLv2 with incremental improvements, distributing it through the MariaDB Foundation Webpage, as individuals or other companies apart from MariaDB Corporation, will not support the product if it is not totally Open Source. So BSL v1.1 might prove useful to launch the project, but with time, it shall have a GPLv2 for gaining investment from third companies. ¿What do you thing on the topic? Thank you all, Juan Telleria

1 0

[Maria-developers] Please review MDEV-13788 Server crash when issuing bad SQL partition syntax
by Alexander Barkov 16 Nov '17

16 Nov '17

Hello Sergei, Can you please review a patch for MDEV-13788, for 5.5? Thanks!

2 3

Re: [Maria-developers] MDEV-10173.
by Sergei Golubchik 15 Nov '17

15 Nov '17

Hi, Alexey! On Nov 14, Alexey Botchkov wrote: > Hi, Serg. > > I'd like your opinion about this issue. > Georg complains about weird column metadata > https://jira.mariadb.org/browse/MDEV-10173 > > Personally i see slightly different result, but basically same thing. > -------------------------------- > MariaDB [(none)]> select from_unixtime("0"); > Field 1: `from_unixtime("0")` > Catalog: `def` > Database: `` > Table: `` > Org_table: `` > Type: DATETIME > Collation: binary (63) > Length: 26 > Max_length: 19 > Decimals: 39 > Flags: BINARY Right, I get used to 31 being NOT_FIXED_DEC. But recently Monty changed it to 39. > +---------------------+ > | from_unixtime("0") | > +---------------------+ > | 1970-01-01 04:00:00 | > +---------------------+ > 1 row in set (0.00 sec) > -------------------------------- > > From what i see it is not a bug. The 'Decimals: 39' is not a real > precision here, but just a hint to be flexible converting the data to > the string. The 'Length: 26' takes into account the possible > 'second_part' of 6 digits. Agree. > Georg said on this: "Well, the client "trusts" the metadata. If there > is decimal 31, it appends 31 decimal places and returned truncation > error the metadata (MYSQL_FIELD) describes the data which are sent > from server to client. If there are no decimal places, it should be > zero. " Yes, the cient trusts the metadata, but if decimals is NOT_FIXED_DEC it means "unknown", not 31 (or 39). If the number of decimals for a timestamp is unknown, the max length is 26: "YYYY-MM-DD hh:mm:ss.uuuuuu" > As an improvement i can see using '7' instead of '39' as an > 'NOT_FIXED_DEC' marker for the Item_temporal_func. Why? One wouldn't be able simply to compare with NOT_FIXED_DEC, it'll be type dependent. > Or we can just document that '39' value so clients interpret it right. Isn't it documented? > Serg, maybe you can suggest something? Close as "Not a bug" :) Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] 730a22c: Fixed bug mdev-13453 Executing a query via CTE requires more permissions
by Oleksandr Byelkin 14 Nov '17

14 Nov '17

Am 14.11.2017 um 01:06 schrieb IgorBabaev: > revision-id: 730a22c1ce1010e853e7a3e807d1d07409ec4dfc (mariadb-10.2.10-28-g730a22c) > parent(s): 1e2d4f677e08294691a7d218acb3b9f78727ae18 > author: Igor Babaev > committer: Igor Babaev > timestamp: 2017-11-13 16:06:04 -0800 > message: > > Fixed bug mdev-13453 Executing a query via CTE requires more permissions > than the query itself > > ACL checks were not properly supported for tables used in CTE > specifications. This patch fixes the problem. > > --- > mysql-test/r/cte_nonrecursive.result | 58 ++++++++++++++++++++++++++++++++++++ > mysql-test/t/cte_nonrecursive.test | 51 +++++++++++++++++++++++++++++++ > sql/sql_acl.cc | 4 +++ > sql/sql_cte.cc | 3 +- > sql/sql_parse.cc | 4 +++ > 5 files changed, 119 insertions(+), 1 deletion(-) > > diff --git a/mysql-test/r/cte_nonrecursive.result b/mysql-test/r/cte_nonrecursive.result > index ebe1aae..92f9151 100644 > --- a/mysql-test/r/cte_nonrecursive.result > +++ b/mysql-test/r/cte_nonrecursive.result > @@ -1147,3 +1147,61 @@ SELECT * FROM cte_test; > a > 1 > DROP VIEW cte_test; > +# > +# MDEV-13453: > +# [skip] > --- a/mysql-test/t/cte_nonrecursive.test > +++ b/mysql-test/t/cte_nonrecursive.test > @@ -790,3 +790,54 @@ SHOW CREATE VIEW cte_test; > SELECT * FROM cte_test; > > DROP VIEW cte_test; > + > +--echo # > +--echo # MDEV-13453: privileges checking for CTE > +--echo # > + This is for sure do not match results above, so please fix it and make sure that full test suite passed. [skip] > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -7557,6 +7557,10 @@ bool check_grant(THD *thd, ulong want_access, TABLE_LIST *tables, > tl->correspondent_table ? tl->correspondent_table : tl; > sctx= t_ref->security_ctx ? t_ref->security_ctx : thd->security_ctx; > > + if (tl->with || > + (tl->with= tl->select_lex->find_table_def_in_with_clauses(tl))) > + continue; > + > const ACL_internal_table_access *access= > get_cached_table_access(&t_ref->grant.m_internal, > t_ref->get_db_name(), > diff --git a/sql/sql_cte.cc b/sql/sql_cte.cc > index 6fe08e3..e1bd455 100644 > --- a/sql/sql_cte.cc > +++ b/sql/sql_cte.cc > @@ -823,9 +823,10 @@ st_select_lex_unit *With_element::clone_parsed_spec(THD *thd, > tbl; > tbl= tbl->next_global) > { > - tbl->grant.privilege= with_table->grant.privilege; > spec_tables_tail= tbl; > } > + if (check_table_access(thd, SELECT_ACL, spec_tables, FALSE, UINT_MAX, FALSE)) > + goto err; > if (spec_tables) > { > if (with_table->next_global) > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > index bf5144b..39a4da9 100644 > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -3443,6 +3443,10 @@ mysql_execute_command(THD *thd) > ulong privileges_requested= lex->exchange ? SELECT_ACL | FILE_ACL : > SELECT_ACL; > > + res= check_dependencies_in_with_clauses(thd->lex->with_clauses_list); > + if (res) > + break; Above works only for SELECT, do we support CTE in subselects which can be put everywhere? > + > if (all_tables) > res= check_table_access(thd, > privileges_requested, > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

Re: [Maria-developers] Handle failures from malloc
by Sergey Petrunia 14 Nov '17

14 Nov '17

Hi Monty, Please find my comments below. Anything not commented on seems to be ok to me. > commit 7d9c24f6790c66d2f457837661852bf5e870b605 > Author: Michael Widenius <monty(a)mariadb.org> > Date: Tue Nov 14 07:47:58 2017 +0200 > > Handle failures from malloc > > Most "new" failures fixed in the following files: > - sql_select.cc > - item.cc > - opt_subselect.cc > > diff --git a/sql/item.cc b/sql/item.cc > index 2285ae28041..baae2a9b6fb 100644 > --- a/sql/item.cc > +++ b/sql/item.cc > @@ -1261,7 +1261,7 @@ Item *Item::safe_charset_converter(THD *thd, CHARSET_INFO *tocs) > if (!needs_charset_converter(tocs)) > return this; > Item_func_conv_charset *conv= new (thd->mem_root) Item_func_conv_charset(thd, this, tocs, 1); > - return conv->safe ? conv : NULL; > + return conv && conv->safe ? conv : NULL; > } > I am not fully convinced that returning NULL from this function is called everywhere. It is of course better to return NULL and hope for the best than to crash right away. > @@ -3296,6 +3301,8 @@ void Item_field::fix_after_pullout(st_select_lex *new_parent, Item **ref, > } > > Name_resolution_context *ctx= new Name_resolution_context(); > + if (ctx) > + return; // Fatal error set > if (context->select_lex == new_parent) > { > /* I assume this is a typo it should be "if (!ctx)" > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index abbf2616537..7e42890143b 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -713,7 +713,7 @@ JOIN::prepare(TABLE_LIST *tables_init, > union_part= unit_arg->is_unit_op(); > > if (select_lex->handle_derived(thd->lex, DT_PREPARE)) > - DBUG_RETURN(1); > + DBUG_RETURN(-1); > > thd->lex->current_select->context_analysis_place= NO_MATTER; > thd->lex->current_select->is_item_list_lookup= 1; I assume this is just for the sake of consistency with other DBUG_RETURN statements in this function? > @@ -14422,7 +14446,7 @@ static COND* substitute_for_best_equal_field(THD *thd, JOIN_TAB *context_tab, > This works OK with PS/SP re-execution as changes are made to > the arguments of AND/OR items only > */ > - if (new_item != item) > + if (new_item && new_item != item) > li.replace(new_item); > } new_item was returned from a substitute_for_best_equal_field() call. That one doesn't ever return NULL. It will return without substitution if it encounters an error. So this change is not needed. > @@ -25044,8 +25103,11 @@ int JOIN::save_explain_data_intern(Explain_query *output, > > if (message) > { > - explain= new (output->mem_root) Explain_select(output->mem_root, > - thd->lex->analyze_stmt); > + if (!(explain= new (output->mem_root) > + Explain_select(output->mem_root, > + thd->lex->analyze_stmt))) > + DBUG_RETURN(1); > + > join->select_lex->set_explain_type(true); > > explain->select_id= join->select_lex->select_number; The return value of this function is not checked. I assume the above statement is there so that we dont try to continue to execute the function if we've got an OOM error? In this case, probably JOIN::save_explain_data_intern() should also have a similar check for its tab->save_explain_data(eta, used_tables, distinct_arg, first_top_tab); call? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] STRICT mode and Cast
by jerome brauge 14 Nov '17

14 Nov '17

Hello Alexander, To have a behavior the nearest possible to Oracle, I use STRICT_TRANS_TABLES sql_mode. But then I have a problem with cast to varchar that failed with "truncated incorrect value" . Ex: set sql_mode='ORACLE,STRICT_TRANS_TABLES'; delimiter / CREATE or replace procedure p3(b1 integer) AS strres VARCHAR(4); BEGIN strres:=cast('123456' as char(4)); END / call p3(-1) / -- Failed with : ERROR 1292 (22007): Truncated incorrect CHAR(4) value: '123456' But select cast('11111' as varchar(2)) from dual; just issue a warning, not an error. On Oracle, Sybase,SQLServer and DB2, cast a string to a shorter string never issue warnings nor error. What do you think about ? Regards, Jérôme.

2 4

[Maria-developers] Patch for MDEV-10574
by jerome brauge 13 Nov '17

13 Nov '17

Hello Alexander, I've begun to work on new sql_mode EMPTY_STRING_IS_NULL_FUNCTIONS. Can you take a glance on this patch to known if I'm on the right direction ? I've also found a different behavior between Oracle and Mariadb on cast of string in a numeric format. Mariadb send a warning or a note and return 0 when conversion failed while Oracle send an error. MariaDB [test]> select cast(' ' as integer) from dual; +----------------------+ | cast(' ' as integer) | +----------------------+ | 0 | +----------------------+ DVTORA> select cast(' ' as integer) from dual; select cast(' ' as integer) from dual * ERROR at line 1: ORA-01722: invalid number Are "cast" and aggregate functions to be handled by this mdev? Regards, Jérôme. > -----Message d'origine----- > De : Alexander Barkov [mailto:bar@mariadb.org] > Envoyé : samedi 7 octobre 2017 11:44 > À : jerome brauge > Objet : Re: Patch for MDEV-10574 > > Hello Jerome, > > > On 10/06/2017 01:41 PM, jerome brauge wrote: > > Hello Alexander, > > > >> -----Message d'origine----- > >> De : Alexander Barkov [mailto:bar@mariadb.org] Envoyé : jeudi 5 > >> octobre 2017 21:26 À : jerome brauge Objet : Re: Patch for MDEV-10574 > >> > >> Hello Jerome, > >> > >> > >> On 10/05/2017 12:45 PM, jerome brauge wrote: > >>> Hello Alexander, > >>> Have you gone ahead on your reflection about nulls and empty strings ? > >> > >> Yes, I think so. > >> > >> One addition: Monty suggested to rename flags to: > >> - EMPTY_STRING_IS_NULL for literals > >> - EMPTY_STRING_IS_NULL_RESULTS for functions. > >> > >> I'm fine with EMPTY_STRING_IS_NULL for literals, but think that > >> EMPTY_STRING_IS_NULL_FUNCTIONS should be more clear than > >> EMPTY_STRING_IS_NULL_RESULTS. > >> Which name do you prefer for functions? > > > > I think like you. > > For me, EMPTY_STRING_IS_NULL_RESULTS sounds like including > RESULTSET > > of select (which is perhaps included in point 4) > > Okey, let's go with _FUNCTIONS then. > > > > >>> > >>> Can I > >>> - redo my pull request for substring_oracle function > >>> - make a patch for first new sql_mode > >> (MODE_EMPTY_AS_NULL_LITERALS) > >> > >> > >> Yes, please. Let's move forward. > >> > >> Let's push the patch for substr_oracle first. > >> Let's keep it return empty strings for the cases when the > >> substring_length parameters is less than 1. > >> It will start automatically returning NULL when we add the new > >> sql_mode for functions. > > > > Pull request is done. > > I have merged it. Thanks! > > > > >> Also, let's go ahead with the part for literals. > > > > Ok, I will make a separate patch. > > Many thanks. > > > >> > >> > >> I have created MDEVs. Please use them in the commit comment and in > >> the > >> tests: > >> > >> MDEV-14012 sql_mode=Oracle: substr(): treat position 0 as position 1 > >> MDEV-14013 sql_mode=EMPTY_STRING_IS_NULL > >> > >> > >> Thanks! > >> > >>> > >>> Regards, > >>> Jérôme. > >>> > >>> > >>>> -----Message d'origine----- > >>>> De : Alexander Barkov [mailto:bar@mariadb.org] Envoyé : mercredi 27 > >>>> septembre 2017 08:50 À : jerome brauge Objet : Re: Patch for > >>>> MDEV-10574 > >>>> > >>>> Hello Jerome, > >>>> > >>>> I told to Monty yesterday. > >>>> > >>>> We were going to implement the following transformations: > >>>> > >>>>> We are considering to implement both of the following options > >>>>> (with a > >>>>> switch) > >>>>> > >>>>> Transform Insert > >>>>> - insert values ("") -> insert values (null) > >>>>> > >>>>> Transform Select > >>>>> > >>>>> where v=x => (v <> "" and V=X) > >>>>> where v is null => (v="" or v is null) > >>>>> > >>>> > >>>> See MDEV-10574 for details. > >>>> > >>>> Monty did not fully understand the idea of translating literals and > >> functions > >>>> from empty strings to null. He asked if you can explain use cases > >>>> for these transformations. > >>>> > >>>> But we agreed that: > >>>> > >>>> 1. It's OK to have multiple NULL handling flags in sql_mode. > >>>> > >>>> 2. New flags will not be enabled in 10.3, neither in > >>>> sql_mode=DEFAULT, > >> nor in > >>>> sql_mode=ORACLE. So one will have to specify them explicitly: > >>>> set > >>>> > >> > sql_mode='ORACLE,MODE_null_transfrormation_flag1,MODE_null_transfor > >>>> mation_flag2'; > >>>> > >>>> > >>>> > >>>> So I propose we introduce the following flags: > >>>> > >>>> > >>>> 1. MODE_EMPTY_AS_NULL_LITERALS - for literals and PS parameters, > as > >> in > >>>> your patch. > >>>> > >>>> 2. MODE_EMPTY_AS_NULL_FUNCTIONS - for functions. > >>>> This should be done as a separate patch. > >>>> Note, your patch currently implements empty-to-null translation > >>>> only for Item_str_func descendants that use make_empty_result(). > >>>> This patch should be extended to cover Item_str_func descendants > >>>> which do not use make_empty_result(), as well as > >>>> all other Item_func_or_sum descendants > >>>> (which are not Item_str_func descendants). > >>>> > >>>> > >>>> 3. MODE_EMPTY_AS_NULL_UPDATES > >>>> > >>>> For INSERT and UPDATE transformation, as in MDEV: > >>>> > >>>> insert values ("") -> insert values (null) > >>>> update set a='' -> update set a=null; > >>>> > >>>> > >>>> 4. MODE_EMPTY_AS_NULL_PREDICATES > >>>> > >>>> For predicate transformation, as in MDEV: > >>>> > >>>> where v=x => (v <> "" and V=X) > >>>> where v is null => (v="" or v is null) > >>>> > >>>> > >>>> What do you think about this proposal with multiple flags? > >>>> Note, we can do N1 and N2 now, and add N3 and N4 and later. > >>>> > >>>> Also, can you please write a description why you need N1 and N2. > >>>> Monty thinks that N3 and N4 should cover most use cases. > >>>> So we need a good rationale for Monty and Sergei explaining why we > >> adding > >>>> these flags N1 and N2. > >>>> > >>>> > >>>> Thanks! > >>>> > >>>> > >>>> On 09/26/2017 04:33 PM, jerome brauge wrote: > >>>>> Hi Alexander, > >>>>> I'm disappointed. > >>>>> Can we at least keep this part in sql_prepare : > >>>>> > >>>>> /* > >>>>> set_param_str_oracle : convert empty string to null value > >>>>> This allow to bind empty string in JDBC as a null value. > >>>>> Ex : > >>>>> String sel="select coalesce(?,'null param') from dual"; > >>>>> PreparedStatement ps_sel = m_cnx.prepareStatement(sel); > >>>>> ps_sel.setString(1, ""); > >>>>> ResultSet rs = ps_sel.executeQuery(); > >>>>> while (rs.next()) > >>>>> { > >>>>> System.out.println(rs.getString(1)); > >>>>> } > >>>>> Returns : 'null param' > >>>>> */ > >>>>> static void set_param_str_oracle(Item_param *param, uchar **pos, > >> ulong > >>>>> len) { > >>>>> ulong length= get_param_length(pos, len); > >>>>> if (length == 0) > >>>>> param->set_null(); > >>>>> else > >>>>> { > >>>>> if (length > len) > >>>>> length= len; > >>>>> param->set_str((const char *) *pos, length); > >>>>> *pos+= length; > >>>>> } > >>>>> } > >>>>> > >>>>> Regards, > >>>>> Jérôme. > >>>>> > >>>>> > >>>>>> -----Message d'origine----- > >>>>>> De : Alexander Barkov [mailto:bar@mariadb.org] Envoyé : mardi 26 > >>>>>> septembre 2017 13:34 À : jerome brauge Objet : Re: Patch for > >>>>>> MDEV-10574 > >>>>>> > >>>>>> Hello Jérôme, > >>>>>> > >>>>>> I'm afraid we cannot accept this change at this point of time. > >>>>>> > >>>>>> The problem is that all around the code we assume that NULL is > >>>>>> not equal to ''. > >>>>>> > >>>>>> Changing '' to NULL only in the visible part of the iceberg (such > >>>>>> as in the parser and in Item_str_func::make_empty_result()) > >>>>>> will likely introduce various anomalies in fundamental things > >>>>>> like joins execution, optimizer, equality propagation, etc. > >>>>>> > >>>>>> Sorry, we don't have resources to dive into this topic deeper now. > >>>>>> Moreover, Oracle's documentation suggests not to reply on the > >>>>>> fact > >> that '' > >>>>>> and NULL are treated as same, as this can change in the future. > >>>>>> > >>>>>> > >>>>>> > >>>>>> On 09/20/2017 03:52 PM, jerome brauge wrote: > >>>>>>> Hello Alexander, > >>>>>>> Here is a new version of this patch (minor optimizations) > >>>>>>> > >>>>>>> Regards, > >>>>>>> Jérôme. > >>>>>>> > >>>>>>> > >>>>>>>> -----Message d'origine----- > >>>>>>>> De : Maria-developers [mailto:maria-developers- > >>>>>>>> bounces+j.brauge=qualiac.com(a)lists.launchpad.net] De la part > de > >>>>>>>> bounces+jerome > >>>>>>>> brauge > >>>>>>>> Envoyé : mardi 19 septembre 2017 16:09 À : 'Alexander Barkov' > >>>>>>>> Cc : MariaDB Developers (maria-developers(a)lists.launchpad.net) > >>>>>>>> Objet : [Maria-developers] Patch for MDEV-10574 > >>>>>>>> > >>>>>>>> Alexander, > >>>>>>>> Here is the patch I was talking about. > >>>>>>>> It not address all aspects of MDEV-10574, especially, empty > >>>>>>>> strings in table column are not view as null. > >>>>>>>> But as in oracle mode, no empty string in columns should be > >>>>>>>> created, an fully oracle application will never fall in this case. > >>>>>>>> > >>>>>>>> Regards, > >>>>>>>> Jérôme. > >>>>>>>> > >>>>>>>> > >>>>>>>>> -----Message d'origine----- > >>>>>>>>> De : jerome brauge > >>>>>>>>> Envoyé : mardi 19 septembre 2017 14:13 À : 'Alexander Barkov' > >>>>>>>>> Objet : RE: Oracle strings functions compatibility : substr > >>>>>>>>> > >>>>>>>>> Hello Alexander, > >>>>>>>>> Yes I'm aware. > >>>>>>>>> This point will be treated by my next patch which will also > >>>>>>>>> affect the others string functions (a part of MDEV-10574). > >>>>>>>>> I will refresh this new patch with the current development > >>>>>>>>> branch, and I submit to you. > >>>>>>>>> Regards. > >>>>>>>>> Jérôme. > >>>>>>>>> > >>>>>>>>>> -----Message d'origine----- > >>>>>>>>>> De : Alexander Barkov [mailto:bar@mariadb.org] Envoyé : > mardi > >> 19 > >>>>>>>>>> septembre 2017 13:16 À : jerome brauge Objet : Re: Oracle > >>>>>>>>>> strings functions compatibility : substr > >>>>>>>>>> > >>>>>>>>>> Hi Jerome, > >>>>>>>>>> > >>>>>>>>>> I noticed one more difference in SUBSTR: > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> SELECT SUBSTR('aaa',1,-1) FROM DUAL; > >>>>>>>>>> > >>>>>>>>>> MariaDB returns an empty string. > >>>>>>>>>> Oracle returns NULL. > >>>>>>>>>> > >>>>>>>>>> Here's Oracle's documentation: > >>>>>>>>>> > >>>>>>>>>> > >>>>>> > >> https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions162. > >>>>>>>>>> ht > >>>>>>>>>> m > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> I think we should do this in the same patch, to avoid > >>>>>>>>>> behavior change in the future. > >>>>>>>>>> > >>>>>>>>>> Would you like to try doing this additional change? > >>>>>>>>>> Or would you like me to make an incremental patch on top of > >> yours? > >>>>>>>>>> > >>>>>>>>>> Thanks! > >>>>>>>>>> > >>>>>>>>>> On 09/19/2017 10:33 AM, jerome brauge wrote: > >>>>>>>>>>> Hello Alexander, > >>>>>>>>>>> It's done. > >>>>>>>>>>> > >>>>>>>>>>> Do you have news from Sergei for MDEV-12874, MDEV-13417 > >> and > >>>>>>>> MDEV- > >>>>>>>>>> 13418 ? > >>>>>>>>>> > >>>>>>>>>> I asked Sergei, awaiting for his answer. I'll let you know. > >>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> Thank you very much. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> -----Message d'origine----- De : Alexander Barkov > >>>>>>>>>>>> [mailto:bar@mariadb.org] Envoyé : lundi > >>>>>>>>>>>> 18 septembre 2017 18:12 À : jerome brauge Objet : Re: > >>>>>>>>>>>> Oracle strings functions compatibility : substr > >>>>>>>>>>>> > >>>>>>>>>>>> Hello Jerome, > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> On 09/18/2017 05:14 PM, jerome brauge wrote: > >>>>>>>>>>>>> Hello Alexander, > >>>>>>>>>>>>> I don't understand how I've missed such an obvious > solution ! > >>>>>>>>>>>>> Here is the new patch. > >>>>>>>>>>>> > >>>>>>>>>>>> Thanks. Now it looks simpler. > >>>>>>>>>>>> > >>>>>>>>>>>> Can you please do one small thing: > >>>>>>>>>>>> move get_position() from "public:" to "protected:". > >>>>>>>>>>>> > >>>>>>>>>>>> Ok to make a pull request after this change. > >>>>>>>>>>>> > >>>>>>>>>>>> Thank you very much! > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> Regards. > >>>>>>>>>>>>> Jérôme. > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>>> -----Message d'origine----- De : Alexander Barkov > >>>>>>>>>>>>>> [mailto:bar@mariadb.org] Envoyé : > >> lundi > >>>>>>>>>>>>>> 18 septembre 2017 11:30 À : jerome brauge Objet : Re: > >> Oracle > >>>>>>>>>>>>>> strings functions compatibility : substr > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Hello Jerome, > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On 08/17/2017 04:43 PM, jerome brauge wrote: > >>>>>>>>>>>>>>> Hello Alexander, > >>>>>>>>>>>>>>> Here is a patch for function SUBSTR in sql_mode=oracle > >>>>>>>>>>>>>>> (If position is 0, > >>>>>>>>>>>>>> then it is treated as 1). > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> I'm thinking of how to get a smaller patch. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Wouldn't it be possible to introduce: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> virtual longlong get_position(); > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> longlong Item_func_substr::get_position() { > >>>>>>>>>>>>>> return args[1]->val_int(); } > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> longlong Item_func_substr_oracle::get_position() > >>>>>>>>>>>>>> { > >>>>>>>>>>>>>> longlong pos= args[1]->val_int(); > >>>>>>>>>>>>>> return pos == 0 ? 1 : pos; } > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> and just replace all calls for args[1]->val_int() to > >> get_position() > >>>> ? > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Thanks! > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Regards, > >>>>>>>>>>>>>>> Jérôme. > >>>>>>>>>>>>>>>

2 3

Re: [Maria-developers] [Commits] f87dddaaa95: MDEV-7865: Server crashes in Item_equal_iterator<List_iterator_fast, Item>::get_curr_field on query
by Sergey Petrunia 09 Nov '17

09 Nov '17

On Sat, Oct 28, 2017 at 12:47:03AM +0530, Varun wrote: > revision-id: f87dddaaa95f9e7fc597c8839b539cc6f816c750 (mariadb-5.5.56-91-gf87dddaaa95) > parent(s): fb5fe497e526f0677a7e2f86f34237e7efd4b806 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2017-10-28 00:46:43 +0530 > message: > > MDEV-7865: Server crashes in Item_equal_iterator<List_iterator_fast, Item>::get_curr_field on query > with impossible condition and OR/AND expressions > > When multiple-equalites are merged into one in the function Item_equal::merge_with_check, then the fields that are merged from one > equality to the other still point to the previous mulitple_equality while the should be pointing to the new equality > after the merge is done .. > diff --git a/sql/item_cmpfunc.cc b/sql/item_cmpfunc.cc > index ebe088e5092..8c79359ce88 100644 > --- a/sql/item_cmpfunc.cc > +++ b/sql/item_cmpfunc.cc > @@ -5832,7 +5832,18 @@ bool Item_equal::merge_with_check(Item_equal *item, bool save_merged) > if (intersected) > { > if (!save_merged) > + { > + Item *item_it; > + fi.rewind(); > + while ((item_it= fi++)) > + { Wrong indentation > + if (!contains(fi.get_curr_field())) > + { > + item_it->set_item_equal(this); I am not sure if this can actually make a difference, but is there any reason we don't just call set_item_equal unconditionally here? My concern is that f.get_curr_field() checks for Field* objects, while set_item_equal() call applies to Items (either Item_field or Item_direct_view_ref) What if there are two different Item objects that refer to the same field? (this looks like it should not happen but I am not 100% certain about this). > + } > + } > merge(item); > + } > else > { > Item *c= item->get_const(); > @@ -5843,9 +5854,12 @@ bool Item_equal::merge_with_check(Item_equal *item, bool save_merged) > Item *item; > fi.rewind(); > while ((item= fi++)) > - { > + { > if (!contains(fi.get_curr_field())) > + { > add(item); > + item->set_item_equal(this); > + } here, we have save_merged=true, that is, the old Item_equal will still be used. Is this ok that we call set_item_equal($new_equality) for its members? If an object is a participant in two Item_equal, which one should it point to? the "most specific" one? I'll try to ask Igor about this today. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 1

[Maria-developers] MDEV-13418 Compatibility: The order of evaluation of SELECT..INTO assignments
by jerome brauge 09 Nov '17

09 Nov '17

Hello Alexander. I've begun to implement your proposal but now I'm not sure that it's a better solution than mine. Let me explain . - first : number of temporary variables can be significant because we don't know when they are really needed and their scope are local to the statement. declare b1 INTEGER; declare res INTEGER; ... if b1 = 0 then select 1,b1+1 into b1, res from dual; end if; if b1 = 1 then select 2,b1+2 into b1, res from dual; end if; will be transform in : declare b1 INTEGER; declare res INTEGER; ... if b1 = 0 then declare _b1 INTEGER default res; declare _res INTEGER default res; select 1,b1+1 into _b1, _res from dual; set b1=_b1; set res=_res; -- _res is not needed, but we don't know because the select statement is not parsed end if; if b1 = 1 then declare _b1 INTEGER default res; declare _res INTEGER default res; select 2,b1+2 into b1, res from dual; set b1=_b1; set res=_res; -- same thing here, and we have declare two variables for each target variables end if; Perhaps we could - declare these temporary variables only one time in the first frame of the stored procedure (may be tricky) - parse columns of each select to know what variables are really assigned and reused (heavy cost in cpu and time) - second : if we can't determine variables which must have a temporary variable, number of sp_instr_set and sp_instr_set_var will be too high and their cpu cost is not acceptable. My first solution has a fixed memory impact (and memory is not an issue nowadays), and especially a very light cpu cost. What do you think about ? Regard, Jérôme. > -----Message d'origine----- > De : Alexander Barkov [mailto:bar@mariadb.org] > Envoyé : lundi 30 octobre 2017 10:02 > À : jerome brauge > Objet : Re: MDEV-14139 Anchored data types for variables > > > > On 10/28/2017 07:29 PM, Alexander Barkov wrote: > > > > On 10/27/2017 10:27 PM, Alexander Barkov wrote: > >> Hello Jerome, > >> > >> I have implemented "MDEV-14139 Anchored data types for variables". > >> and pushed it to bb-10.2-ext: > >> > >> > https://github.com/MariaDB/server/commit/5dd5253f7e50c21fa758e2eb58f > 3 > >> aa9c9754e733 > >> > >> So now it should be easier to implement consistent SET by creating > >> backup variables. > >> > >> > >> LEX::sp_variable_declarations_vartype_finalize() implements the logic > >> which copies data type from another variable. > >> > >> > >> The idea is that for all variables, which are assignment targets in a > >> SET or a SELECT INTO statement, we create a backup variable. > >> > >> It will involve these calls for every such variable: > >> > >> LEX::sp_variable_declarations_init(thd, 1); > >> sp_pcontext::add_variable(thd, backup_variable_name); > >> LEX::sp_variable_declarations_vartype_finalize(thd, 1, > >> orig_variable_name, def); > >> > >> where "def" is Item_splocal created for the original variable. > > > > Just an idea: instead of creating sp_instr_set, it's easier to > > introduce a new class sp_instr_set_var, to copy the value from one > > variable to another variable. > > > > This operation will not need neither Item, nor sp_lex_keeper. It will > > only need two offsets: > > - the source variable offset and > > - the target variable offset. > > > > Using these offsets, we can access to > > spcont->m_var_table->field[source] and > > spcont->spcont->m_var_table->field[target] > > and copy the value between them using Field::store_field(). > > > > This won't however for the ROW variables at the moment, because ROW > > fields are stored in the Item_spvar_args::m_table member of > > Item_field_row. > > > > It seems we need a new class Field_row and move Virtual_tmp_table from > > Item_field_row to Field_row. > > > > I will try to do it. > > > I have implemented "MDEV-14212 Add Field_row for SP ROW variables" and > pushed to bb-10.2-ext. > > Also, added a comment: > > MDEV-13418 Compatibility: The order of evaluation of SELECT..INTO > assignments > > > Now, when MDEV-14212 is done, these declarations: > > DECLARE a_tmp TYPE OF a DEFAULT a; > DECLARE b_tmp TYPE OF b DEFAULT b; > > and these assignments: > > SET a=a_tmp; > SET b=b_tmp; > > can use a new sp_instr_setvar instead of sp_inst_set. > > The new command sp_instr_setvar can do direct copying between two > spcont->m_var_table->field[XXX], without a need to create Item and LEX. > > > > >> > >> Greetings. > >>

2 9

[Maria-developers] Summary of the InnoDB changes in the AliSQL 5.6.32 branch
by Marko Mäkelä 08 Nov '17

08 Nov '17

In preparation for the 2017-2 MariaDB Developer Unconference in Shenzhen <https://mariadb.org/2017-2-developers-unconference-and-related-events-shenz…> next week, I cloned the https://github.com/alibaba/AliSQL repository and reviewed the changes to storage/innobase. The repository is based on commit c3a8ca1ed0063d4bca761ac9b57718796242ccc1 (tag: mysql-5.6.32) in the https://github.com/mysql/mysql-server/ repository. I read the 13,587 lines of diff generated by the following command: git diff c3a8ca1ed0063d4bca761ac9b57718796242ccc1 storage/innobase Features that already exist in MariaDB in some form: * Split the btr_search_latch (already present in MySQL 5.7/MariaDB 10.2.2) * Buffer pool "hazard pointer" (5.7/10.2.2) * Faster read view creation (done differently in WL#6578 MySQL 5.7/MariaDB 10.2.2) * Persistent auto_increment (done differently in MDEV-6076 in MariaDB 10.2.4) * Instant ADD COLUMN (done differently in MDEV-11369 in MariaDB 10.3.2) * Sequence engine (done differently in MDEV-10139 in MariaDB 10.3.1) New features: * Counters for buffer pool LRU eviction: MONITOR_LRU_BATCH are split into MONITOR_LRU_BATCH_FLUSH and MONITOR_LRU_BATCH_EVICT (Sounds like a good idea to me; maybe MariaDB should export these via status variables.) * Issue#24 SPLIT LOG BUFFER TO ROTATE LOG WRITE (I’d go further and change the LSN to count mtr_t::commit() instead of bytes, and to use a global buffer for mtr_t::log. This would require a new redo log format.) * Issue#31 OPTIMIZE CHECK/GRANT OF INNODB TABLE LOCK (I’d rather replace InnoDB table locks and dict_operation_lock with MDL.) * Issue#29: ADD INFORMATION_SCHEMA.INNODB_RSEG TABLE (I’d rather split the undo log into tablename.ibu files, also to ease import/export/backup.) * Issue#9: PROVIDE ADAPTIVE ALGORITHM FOR INNODB CONCURRENCY TICKETS (I think that the whole ticket business should be moved to the thread pool plugin.) * Issue#34: Support force drop table (I’d rather remove the InnoDB dictionary and have transactional .frm files, instead of writing work-arounds.) Best regards, Marko -- Marko Mäkelä, Lead Developer InnoDB MariaDB Corporation

1 0

Re: [Maria-developers] d78273d: MDEV-14288 Add MTR option to run tests with real server defaults
by Sergei Golubchik 05 Nov '17

05 Nov '17

Hi, Elena! On Nov 05, elenst(a)montyprogram.com wrote: > revision-id: d78273d1e53308c8fe78b9e067a88c43bd05e540 > parent(s): 737a4da94ae9b25a7ebe828c5b35a1ce4acf03d6 > committer: Elena Stepanova > branch nick: bb-10.0-elenst > timestamp: 2017-11-05 01:30:40 +0200 > message: > > MDEV-14288 Add MTR option to run tests with real server defaults > > diff --git a/mysql-test/lib/My/ConfigFactory.pm b/mysql-test/lib/My/ConfigFactory.pm > index b8ce038..dcdb534 100644 > --- a/mysql-test/lib/My/ConfigFactory.pm > +++ b/mysql-test/lib/My/ConfigFactory.pm > @@ -37,7 +37,9 @@ sub add_opt_values { > > # add auto-options > $config->insert('OPT', 'port' => sub { fix_port($self, $config) }); > - $config->insert('mysqld', "loose-skip-plugin-$_" => undef) for (@::optional_plugins); > + if (!$::opt_default_server_options) { > + $config->insert('mysqld', "loose-skip-plugin-$_" => undef) for (@::optional_plugins); > + } Dunno. I'd prefer to remove magic from mtr, not to add it. Generating different configs feels like magic. I wonder whether this could work: - mtr always disables plugins, but in a separate .cnf file. And default_mysqld.cnf will !include it, but yours won't. or - all this disabling will go where, say, @DISABLE_ALL_PLUGINS line is. it'll be in default_mysqld.cnf but not in yours. > } > > my @pre_rules= > diff --git a/mysql-test/lib/mtr_cases.pm b/mysql-test/lib/mtr_cases.pm > index 38c52b7..26a4edd 100644 > --- a/mysql-test/lib/mtr_cases.pm > +++ b/mysql-test/lib/mtr_cases.pm > @@ -876,7 +876,10 @@ sub collect_one_test_case { > if (not $config) > { > # Suite has no config, autodetect which one to use > - if ($tinfo->{rpl_test}) { > + if ($::opt_default_server_options) { > + $config= "include/hard_default_my.cnf"; > + } > + elsif ($tinfo->{rpl_test}) { yuck. I never liked that rpl suite magically gets a different cnf file. may be it's time to fix it? e.g. by moving it into suite/rpl/my.cnf or into include/rpl_init.cnf ? > $config= "suite/rpl/my.cnf"; > } else { > $config= "include/default_my.cnf"; > diff --git a/mysql-test/mysql-test-run.pl b/mysql-test/mysql-test-run.pl > index 33d9c7b..997fb82 100755 > --- a/mysql-test/mysql-test-run.pl > +++ b/mysql-test/mysql-test-run.pl > @@ -131,6 +131,7 @@ our $plugindir; > my $path_vardir_trace; # unix formatted opt_vardir for trace files > my $opt_tmpdir; # Path to use for tmp/ dir > my $opt_tmpdir_pid; > +our $opt_default_server_options; > > my $opt_start; > my $opt_start_dirty; > @@ -1106,6 +1107,9 @@ sub command_line_setup { > # Extra config file to append to all generated configs > 'defaults-extra-file=s' => \&collect_option, > > + # Use hard defaults instead of heavily modified MTR configuration > + 'default-server-options' => \$opt_default_server_options, with the changes above, you could use --defaults-file=hard_default_my or something like that (even better with a short cnf file name) You could even commit a bunch of templates, like vanilla.cnf, rpm.cnf, debian.cnf and use them like ./mtr --defaults-file=debian > + > # Control what test suites or cases to run > 'force+' => \$opt_force, > 'with-ndbcluster-only' => \&collect_option, > @@ -3509,12 +3517,14 @@ sub mysql_install_db { > mtr_add_arg($args, "--basedir=%s", $install_basedir); > mtr_add_arg($args, "--datadir=%s", $install_datadir); > mtr_add_arg($args, "--plugin-dir=%s", $plugindir); > - mtr_add_arg($args, "--default-storage-engine=myisam"); > - mtr_add_arg($args, "--loose-skip-plugin-$_") for @optional_plugins; > - # starting from 10.0 bootstrap scripts require InnoDB > - mtr_add_arg($args, "--loose-innodb"); > - mtr_add_arg($args, "--loose-innodb-log-file-size=5M"); > - mtr_add_arg($args, "--disable-sync-frm"); > + if (!$opt_default_server_options) { > + mtr_add_arg($args, "--default-storage-engine=myisam"); > + mtr_add_arg($args, "--loose-skip-plugin-$_") for @optional_plugins; > + # starting from 10.0 bootstrap scripts require InnoDB > + mtr_add_arg($args, "--loose-innodb"); > + mtr_add_arg($args, "--loose-innodb-log-file-size=5M"); > + mtr_add_arg($args, "--disable-sync-frm"); > + } why is that? it's just mysql_install_db > mtr_add_arg($args, "--tmpdir=%s", "$opt_vardir/tmp/"); > mtr_add_arg($args, "--core-file"); > mtr_add_arg($args, "--console"); Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] 58c9df453f5: Remove usage of unknown parameter.
by Marko Mäkelä 02 Nov '17

02 Nov '17

This file should not have been re-added in the first place. I removed this badly working file for a good reason. Please try to read the git history before re-adding broken tests. We now have many useless and outright broken tests in --suite=innodb_gis that need some cleanup. I would have liked to review the test changes. Good tests are as valuable as good code. commit 86e0a73eaa166f752d62b31e96925e29c4fe0c8c Author: Marko Mäkelä <marko.makela(a)mariadb.com> Date: Mon Aug 7 23:38:27 2017 +0300 Remove wait_innodb_all_purged.inc The file wait_innodb_all_purged.inc waited for InnoDB purge in a way that only worked in debug builds. The file wait_all_purged.inc provides a better mechanism. The wait_all_purged.inc was added almost 6 months ago: file commit 0bfa3dff8b95ed8b8bd133e434b47189857121ac Author: Marko Mäkelä <marko.makela(a)mariadb.com> Date: Tue May 16 12:07:26 2017 +0300 MDEV-12698 innodb.innodb_stats_del_mark test failure On Tue, Oct 24, 2017 at 12:24 PM, jan <jan.lindstrom(a)mariadb.com> wrote: > revision-id: 58c9df453f5d8408dd189fb2e5ae4445446b6701 (mariadb-10.2.9-80-g58c9df453f5) > parent(s): 39f86dd27a4c772f0b0a65fbf9b125be9e2dc8ac > author: Jan Lindström > committer: Jan Lindström > timestamp: 2017-10-24 12:23:57 +0300 > message: > > Remove usage of unknown parameter. > > --- > mysql-test/include/wait_innodb_all_purged.inc | 1 - > 1 file changed, 1 deletion(-) > > diff --git a/mysql-test/include/wait_innodb_all_purged.inc b/mysql-test/include/wait_innodb_all_purged.inc > index 5bf816fd72f..c5edc5937da 100644 > --- a/mysql-test/include/wait_innodb_all_purged.inc > +++ b/mysql-test/include/wait_innodb_all_purged.inc > @@ -48,7 +48,6 @@ WHERE VARIABLE_NAME = 'INNODB_PURGE_TRX_ID_AGE';`; > } > if (!$success) > { > - set global innodb_purge_run_now=ON; > real_sleep 0.1; > dec $wait_counter; > } > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- Marko Mäkelä, Lead Developer InnoDB MariaDB Corporation

1 0

[Maria-developers] 10.2.10 compiler warnings Fedora 26 / GCC7
by Reindl Harald 01 Nov '17

01 Nov '17

maybe worth to take a look [ 8%] Building C object libmariadb/mariadb_config/CMakeFiles/mariadb_config.dir/mariadb_config.c.o /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/libmariadb/mariadb_config/mariadb_config.c:9:49: warning: backslash and newline separated by space #define LIBS "-L/usr/lib64/mysql/ -lmariadb "\ [ 20%] Building C object libmariadb/libmariadb/CMakeFiles/mariadb_obj.dir/ma_dtoa.c.o /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/libmariadb/libmariadb/ma_dtoa.c: In function 'dtoa': /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/libmariadb/libmariadb/ma_dtoa.c:1465:14: warning: this statement may fall through [-Wimplicit-fallthrough=] leftright= 0; ~~~~~~~~~^~~ /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/libmariadb/libmariadb/ma_dtoa.c:1467:3: note: here case 4: ^~~~ /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/libmariadb/libmariadb/ma_dtoa.c:1473:14: warning: this statement may fall through [-Wimplicit-fallthrough=] leftright= 0; ~~~~~~~~~^~~ /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/libmariadb/libmariadb/ma_dtoa.c:1475:3: note: here case 5: ^~~~ At top level: /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/libmariadb/libmariadb/ma_dtoa.c:1161:21: warning: 'tinytens' defined but not used [-Wunused-const-variable=] static const double tinytens[]= ^~~~~~~~ [ 24%] Building C object strings/CMakeFiles/strings.dir/dtoa.c.o /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/libmariadb/libmariadb/mariadb_dyncol.c: In function 'mariadb_dyncol_val_double': /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/libmariadb/libmariadb/mariadb_dyncol.c:4049:15: warning: argument 1 null where non-null expected [-Wnonnull] *dbl= strtod(str, &end); ^~~~~~~~~~~~~~~~~ In file included from /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/libmariadb/include/ma_global.h:168:0, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/libmariadb/libmariadb/mariadb_dyncol.c:64: /usr/include/stdlib.h:126:15: note: in a call to function 'strtod' declared here extern double strtod (const char *__restrict __nptr, [ 33%] Building C object mysys/CMakeFiles/mysys.dir/my_malloc.c.o /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/mysys/my_lib.c: In function 'my_dir': /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/mysys/my_lib.c:144:3: warning: 'readdir_r' is deprecated [-Wdeprecated-declarations] while (!(READDIR(dirp,(struct dirent*) dirent_tmp,dp))) ^~~~~ In file included from /usr/include/features.h:410:0, from /usr/include/pthread.h:21, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/include/my_global.h:278, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/mysys/mysys_priv.h:19, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/mysys/my_lib.c:19: /usr/include/dirent.h:189:12: note: declared here extern int __REDIRECT (readdir_r, [ 68%] Building C object storage/maria/CMakeFiles/aria.dir/ma_check.c.o /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/storage/myisam/myisamchk.c: In function 'myisamchk': /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/storage/myisam/myisamchk.c:1411:22: warning: '%ld' directive writing between 1 and 10 bytes into a region of size 8 [-Wformat-overflow=] sprintf(null_pos,"%ld",(long) keyseg->null_pos+1); ^~~ /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/storage/myisam/myisamchk.c:1411:21: note: directive argument in the range [1, 4294967296] sprintf(null_pos,"%ld",(long) keyseg->null_pos+1); ^~~~~ In file included from /usr/include/stdio.h:939:0, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/include/my_global.h:322, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/include/my_base.h:23, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/include/myisam.h:26, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/storage/myisam/myisamdef.h:20, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/storage/myisam/fulltext.h:21, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/storage/myisam/myisamchk.c:18: /usr/include/bits/stdio2.h:33:10: note: '__builtin___sprintf_chk' output between 2 and 11 bytes into a destination of size 8 return __builtin___sprintf_chk (__s, __USE_FORTIFY_LEVEL - 1, ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ __bos (__s), __fmt, __va_arg_pack ()); ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [ 68%] Building C object storage/maria/CMakeFiles/aria.dir/ma_preload.c.o /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/storage/myisam/mi_test3.c: In function 'test_write': /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/storage/myisam/mi_test3.c:367:30: warning: '%7ld' directive writing between 7 and 11 bytes into a region of size 8 [-Wformat-overflow=] sprintf((char*) record.id,"%7ld",(long) getpid()); ^~~~ /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/storage/myisam/mi_test3.c:367:29: note: directive argument in the range [-2147483648, 2147483647] sprintf((char*) record.id,"%7ld",(long) getpid()); ^~~~~~ In file included from /usr/include/stdio.h:939:0, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/include/my_global.h:322, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/include/my_base.h:23, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/include/myisam.h:26, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/storage/myisam/mi_test3.c:20: /usr/include/bits/stdio2.h:33:10: note: '__builtin___sprintf_chk' output between 8 and 12 bytes into a destination of size 8 return __builtin___sprintf_chk (__s, __USE_FORTIFY_LEVEL - 1, ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ __bos (__s), __fmt, __va_arg_pack ()); ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [ 71%] Building C object storage/maria/CMakeFiles/aria_chk.dir/maria_chk.c.o /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/storage/maria/maria_chk.c: In function 'maria_chk': /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/storage/maria/maria_chk.c:1703:22: warning: '%ld' directive writing between 1 and 10 bytes into a region of size 8 [-Wformat-overflow=] sprintf(null_pos,"%ld",(long) keyseg->null_pos+1); ^~~ /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/storage/maria/maria_chk.c:1703:21: note: directive argument in the range [1, 4294967296] sprintf(null_pos,"%ld",(long) keyseg->null_pos+1); ^~~~~ In file included from /usr/include/stdio.h:939:0, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/include/my_global.h:322, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/include/my_base.h:23, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/include/maria.h:22, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/storage/maria/maria_def.h:18, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/storage/maria/ma_fulltext.h:20, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/storage/maria/maria_chk.c:18: /usr/include/bits/stdio2.h:33:10: note: '__builtin___sprintf_chk' output between 2 and 11 bytes into a destination of size 8 return __builtin___sprintf_chk (__s, __USE_FORTIFY_LEVEL - 1, ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ __bos (__s), __fmt, __va_arg_pack ()); ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ [ 73%] Building C object storage/maria/CMakeFiles/ma_test3.dir/ma_test3.c.o /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/storage/maria/ma_test3.c: In function 'test_write': /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/storage/maria/ma_test3.c:365:30: warning: '%7ld' directive writing between 7 and 11 bytes into a region of size 8 [-Wformat-overflow=] sprintf((char*) record.id,"%7ld", (long) getpid()); ^~~~ /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/storage/maria/ma_test3.c:365:29: note: directive argument in the range [-2147483648, 2147483647] sprintf((char*) record.id,"%7ld", (long) getpid()); ^~~~~~ In file included from /usr/include/stdio.h:939:0, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/include/my_global.h:322, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/include/my_base.h:23, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/include/maria.h:22, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/storage/maria/maria_def.h:18, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.10/storage/maria/ma_test3.c:20: /usr/include/bits/stdio2.h:33:10: note: '__builtin___sprintf_chk' output between 8 and 12 bytes into a destination of size 8 return __builtin___sprintf_chk (__s, __USE_FORTIFY_LEVEL - 1, ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ __bos (__s), __fmt, __va_arg_pack ()); ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

1 0

Re: [Maria-developers] [Commits] f88fcf2: MDEV-12542 Add bind_address system variable.
by Sergei Golubchik 30 Oct '17

30 Oct '17

Hi, Alexey! On Oct 30, Alexey Botchkov wrote: > revision-id: f88fcf2514c64df8408e9df41dc129ab6f655c18 (mariadb-10.3.2-17-gf88fcf2) > parent(s): ecee3c71e1e74039e75089f1d76c0b6eea9e43ba > committer: Alexey Botchkov > timestamp: 2017-10-30 07:37:25 +0400 > message: > > MDEV-12542 Add bind_address system variable. > > bind_address variable added. > > --- > .../suite/sys_vars/r/bind_address_basic.result | 57 +++++++++++++ > .../suite/sys_vars/t/bind_address_basic.test | 95 ++++++++++++++++++++++ There's no need to add these files anymore (since 10.1) They were supposed to test basic properties like num/string/readonly, etc. But now sysvars_server_notembedded.result has that for all variables anyway. So these *_basic tests are not needed. > sql/mysqld.cc | 3 - > sql/mysqld.h | 1 + > sql/sys_vars.cc | 7 ++ > 6 files changed, 174 insertions(+), 3 deletions(-) > Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] MDEV-13997 Change Item_bool_rowready_func2 to cache const items at fix time rather than evaluation time
by Alexander Barkov 29 Oct '17

29 Oct '17

Hello Sergei, Please review a patch for MDEV-13997. It's needed for: MDEV-13995 MAX(timestamp) returns a wrong result near DST change and for: MDEV-4912 Add a plugin to field types (column types) Thanks!

2 3

[Maria-developers] Review Custom Aggregate Functions
by Vicențiu Ciorbaru 28 Oct '17

28 Oct '17

Hi Varun! Here's a final review for Custom Aggregate Functions * You haven't run the full test suite, there are a few failing tests. Please update those. rpl.rpl_sp funcs_1.is_columns_mysql funcs_1.storedproc * Push your code to a BB branch and rebase it on top of current 10.3. * Fix commits messages while rebasing. Use the reword flag to clean those up. The rule is this: 1. Have one line that doesn't exceed 80 Characters as a Title. 2. Leave one empty line 3. Add the rest of the commit message. Don't exceed 80 Characters per line. > diff --git a/sql/sp_head.h b/sql/sp_head.h > index bb516187a57..d447807bb3b 100644 > --- a/sql/sp_head.h > +++ b/sql/sp_head.h > @@ -334,6 +334,11 @@ class sp_head :private Query_arena, > execute_function(THD *thd, Item **args, uint argcount, Field *return_fld); > > bool Rename this to execute_function please and delete the now dead-code from the current execute_function. Update comments for this function appropriately. Please run all tests again after this and make sure you update results for all the failing ones. > + execute_aggregate_function(THD *thd, Item **args, uint argcount, > + Field *return_fld, sp_rcontext **nctx, > + MEM_ROOT *caller_mem_root); > + > + bool > execute_procedure(THD *thd, List<Item> *args); > > static void > @@ -813,6 +818,9 @@ class sp_head :private Query_arena, > bool > execute(THD *thd, bool merge_da_on_success); > This is no longer used please remove! > + bool > + execute_agg(THD *thd, bool merge_da_on_success); > + > /** > Perform a forward flow analysis in the generated code. > Mark reachable instructions, for the optimizer. > +++ b/mysql-test/t/sp_agg.test A lot of empty lines at the end of this test. Please delete those. Ping me once you have a green buildbot before pushing. Vicentiu

1 0

[Maria-developers] Review of Percentile Window Functions
by Vicențiu Ciorbaru 28 Oct '17

28 Oct '17

Hi Varun, Here's the final review for Percentile Functions. Just 2 minor coding style comments. Rebase on top of current 10.3 and push to a bb-* branch to double check nothing breaks. Make sure that no merge commit happens. Regards, Vicențiu > diff --git a/sql/item_windowfunc.h b/sql/item_windowfunc.h > index 77cbd556e60..589f7cf5d96 100644 > --- a/sql/item_windowfunc.h > +++ b/sql/item_windowfunc.h > @@ -678,6 +687,277 @@ class Item_sum_ntile : public Item_sum_window_with_row_count > ulong current_row_count_; > }; > > +class Item_sum_percentile_disc : public Item_sum_cume_dist, > + public Type_handler_hybrid_field_type > +{ > +public: > + Item_sum_percentile_disc(THD *thd, Item* arg) : Item_sum_cume_dist(thd, arg), > + Type_handler_hybrid_field_type(&type_handler_longlong), > + value(NULL), val_calculated(FALSE), first_call(TRUE), > + prev_value(0), order_item(NULL){} > + > + double val_real() > + { > + if (get_row_count() == 0 || get_arg(0)->is_null()) > + { > + null_value= true; > + return 0; > + } > + null_value= false; > + return value->val_real(); > + } > + > + longlong val_int() > + { > + if (get_row_count() == 0 || get_arg(0)->is_null()) > + { > + null_value= true; > + return 0; > + } > + null_value= false; > + return value->val_int(); > + } > + > + my_decimal* val_decimal(my_decimal* dec) > + { > + if (get_row_count() == 0 || get_arg(0)->is_null()) > + { > + null_value= true; > + return 0; > + } > + null_value= false; > + return value->val_decimal(dec); > + } > + > + String* val_str(String *str) > + { > + if (get_row_count() == 0 || get_arg(0)->is_null()) > + { > + null_value= true; > + return 0; > + } > + null_value= false; > + return value->val_str(str); > + } > + > + bool add() > + { > + Item *arg= get_arg(0); > + if (arg->is_null()) > + return false; > + > + if (first_call) > + { > + prev_value= arg->val_real(); > + if (prev_value > 1 || prev_value < 0) > + { > + my_error(ER_ARGUMENT_OUT_OF_RANGE, MYF(0), func_name()); > + return true; > + } > + first_call= false; > + } > + > + double arg_val= arg->val_real(); > + > + if (prev_value != arg_val) > + { > + my_error(ER_ARGUMENT_NOT_CONSTANT, MYF(0), func_name()); > + return true; > + } > + > + if (val_calculated) > + return false; > + > + value->store(order_item); > + value->cache_value(); > + if (value->null_value) > + return false; > + > + Item_sum_cume_dist::add(); > + double val= Item_sum_cume_dist::val_real(); > + > + if (val >= prev_value && !val_calculated) > + val_calculated= true; > + return false; > + } > + > + enum Sumfunctype sum_func() const > + { > + return PERCENTILE_DISC_FUNC; > + } > + > + void clear() > + { > + val_calculated= false; > + first_call= true; > + value->clear(); > + Item_sum_cume_dist::clear(); > + } > + > + const char*func_name() const > + { > + return "percentile_disc"; > + } > + > + void update_field() {} > + void set_type_handler(Window_spec *window_spec); > + const Type_handler *type_handler() const > + {return Type_handler_hybrid_field_type::type_handler();} > + > + void fix_length_and_dec() > + { > + decimals = 10; // TODO-cvicentiu find out how many decimals the standard > + // requires. > + } > + > + Item *get_copy(THD *thd, MEM_ROOT *mem_root) > + { return get_item_copy<Item_sum_percentile_disc>(thd, mem_root, this); } > + void setup_window_func(THD *thd, Window_spec *window_spec); > + void setup_hybrid(THD *thd, Item *item); > + bool fix_fields(THD *thd, Item **ref); > + > +private: > + Item_cache *value; > + bool val_calculated; > + bool first_call; > + double prev_value; > + Item *order_item; > +}; > + > +class Item_sum_percentile_cont : public Item_sum_cume_dist, > + public Type_handler_hybrid_field_type > +{ > +public: > + Item_sum_percentile_cont(THD *thd, Item* arg) : Item_sum_cume_dist(thd, arg), > + Type_handler_hybrid_field_type(&type_handler_double), > + floor_value(NULL), ceil_value(NULL), first_call(TRUE),prev_value(0), > + ceil_val_calculated(FALSE), floor_val_calculated(FALSE), order_item(NULL){} > + > + double val_real() > + { > + if (get_row_count() == 0 || get_arg(0)->is_null()) > + { > + null_value= true; > + return 0; > + } > + null_value= false; > + double val= 1 + prev_value * (get_row_count()-1); > + > + /* > + Applying the formula to get the value > + If (CRN = FRN = RN) then the result is (value of expression from row at RN) > + Otherwise the result is > + (CRN - RN) * (value of expression for row at FRN) + > + (RN - FRN) * (value of expression for row at CRN) > + */ > + > + if(ceil(val) == floor(val)) > + return floor_value->val_real(); > + > + double ret_val= ((val - floor(val)) * ceil_value->val_real()) + > + ((ceil(val) - val) * floor_value->val_real()); > + > + return ret_val; > + } > + > + bool add() > + { > + Item *arg= get_arg(0); > + if (arg->is_null()) > + return false; > + > + if (first_call) > + { > + first_call= false; > + prev_value= arg->val_real(); > + if (prev_value > 1 || prev_value < 0) > + { > + my_error(ER_ARGUMENT_OUT_OF_RANGE, MYF(0), func_name()); > + return true; > + } > + } > + > + double arg_val= arg->val_real(); > + if (prev_value != arg_val) > + { > + my_error(ER_ARGUMENT_NOT_CONSTANT, MYF(0), func_name()); > + return true; > + } > + > + if (!floor_val_calculated) > + { > + floor_value->store(order_item); > + floor_value->cache_value(); > + if (floor_value->null_value) > + return false; Add one extra space before return > + } > + if (floor_val_calculated && !ceil_val_calculated) > + { > + ceil_value->store(order_item); > + ceil_value->cache_value(); > + if (ceil_value->null_value) > + return false; Add one extra space before return > + } > + > + Item_sum_cume_dist::add(); > + double val= 1 + prev_value * (get_row_count()-1); > + > + if (!floor_val_calculated && get_row_number() == floor(val)) > + floor_val_calculated= true; too much indentation here. Delete 2 spaces > + > + if (!ceil_val_calculated && get_row_number() == ceil(val)) > + ceil_val_calculated= true; too much indentation here. Delete 2 spaces > + return false; > + } > + > + enum Sumfunctype sum_func() const > + { > + return PERCENTILE_CONT_FUNC; > + } > + > + void clear() > + { > + first_call= true; > + floor_value->clear(); > + ceil_value->clear(); > + floor_val_calculated= false; > + ceil_val_calculated= false; > + Item_sum_cume_dist::clear(); > + } > + > + const char*func_name() const > + { > + return "percentile_cont"; > + } > + void update_field() {} > + void set_type_handler(Window_spec *window_spec); > + const Type_handler *type_handler() const > + {return Type_handler_hybrid_field_type::type_handler();} > + > + void fix_length_and_dec() > + { > + decimals = 10; // TODO-cvicentiu find out how many decimals the standard > + // requires. > + } > + > + Item *get_copy(THD *thd, MEM_ROOT *mem_root) > + { return get_item_copy<Item_sum_percentile_cont>(thd, mem_root, this); } > + void setup_window_func(THD *thd, Window_spec *window_spec); > + void setup_hybrid(THD *thd, Item *item); > + bool fix_fields(THD *thd, Item **ref); > + > +private: > + Item_cache *floor_value; > + Item_cache *ceil_value; > + bool first_call; > + double prev_value; > + bool ceil_val_calculated; > + bool floor_val_calculated; > + Item *order_item; > +}; > + > + > + > > class Item_window_func : public Item_func_or_sum > { > diff --git a/sql/lex.h b/sql/lex.h > index 17dd45f9c0b..fcb44a0f836 100644 > --- a/sql/lex.h > +++ b/sql/lex.h > @@ -735,6 +736,7 @@ static SYMBOL sql_functions[] = { > { "LAG", SYM(LAG_SYM)}, > { "LEAD", SYM(LEAD_SYM)}, > { "MAX", SYM(MAX_SYM)}, > + { "MEDIAN", SYM(MEDIAN_SYM)}, Fix indentation here > { "MID", SYM(SUBSTRING)}, /* unireg function */ > { "MIN", SYM(MIN_SYM)}, > { "NOW", SYM(NOW_SYM)}, > \ No newline at end of file

1 0

Re: [Maria-developers] 9145b7d: MDEV-11153 - Introduce status variables for table cache monitoring and tuning
by Sergei Golubchik 27 Oct '17

27 Oct '17

Hi, Sergey! Looks ok, thanks. Just one question: On Aug 10, Sergey Vojtovich wrote: > revision-id: 9145b7dd11a0e95016304e53acf306daac323580 (mariadb-10.3.0-49-g9145b7d) > parent(s): 63ad4fe5bbe77fe17ab441fcce0106bdee9b618d > committer: Sergey Vojtovich > timestamp: 2017-08-10 15:45:03 +0400 > message: > > MDEV-11153 - Introduce status variables for table cache monitoring and tuning > > Status variables added: Table_open_cache_hits, Table_open_cache_misses, > Table_open_cache_overflows. > > System variables added: table_open_cache_active_instances (read-only). > > diff --git a/sql/sys_vars.cc b/sql/sys_vars.cc > index e89dd47..92721b6 100644 > --- a/sql/sys_vars.cc > +++ b/sql/sys_vars.cc > @@ -3219,6 +3219,12 @@ static Sys_var_ulong Sys_table_cache_size( > BLOCK_SIZE(1), NO_MUTEX_GUARD, NOT_IN_BINLOG, ON_CHECK(0), > ON_UPDATE(fix_table_open_cache)); > > +static Sys_var_uint Sys_table_cache_active_instances( > + "table_open_cache_active_instances", > + "Number of active table cache instances", > + READ_ONLY GLOBAL_VAR(tc_active_instances), NO_CMD_LINE, > + VALID_RANGE(1, 64), DEFAULT(1), BLOCK_SIZE(1)); Why did you do it as a sysvar, not as a status variable? Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 2

[Maria-developers] Code syntax: questions on pointers, etc.
by Aleksey Midenkov 27 Oct '17

27 Oct '17

Recent refactorings of replacing C strings with LEX_CSTRING which is no doubt a good thing raise some questions: 1. Is it still guaranteed that Field::field_name.str is NULL-terminated? 2. It is still passed as a pointer to functions. Why is that? The main feature of C++ references is that it cannot be NULL, so we get segfault on top of the stack (closer to a cause), not the bottom of it. I see that pointers are now widely used and mainly assumed to be always non-NULL (i.e. dereferenced without assertion). But placing such implicit contract on data is not evident and bug-prone. IMHO it's much better to use references whenever it is possible (and when there is no need in cosy NULL semantic). What do you think? But for such lightweight structs like LEX_CSTRING it is even better to pass by value, so we could have the conventience of type cast. 3. LEX_CSTRING and LEX_STRING are now non-convertible. Why not to make: template <typename char_t> struct st_mysql_lex_string { char_t *str; size_t length; }; typedef st_mysql_lex_string<char *> LEX_STRING; typedef st_mysql_lex_string<const char *> LEX_CSTRING; typedef st_mysql_lex_string<const unsigned char *> LEX_CUSTRING; ? 4. There are some duplicate types: MYSQL_LEX_STRING, MYSQL_CONST_LEX_STRING. Why? -- All the best, Aleksey Midenkov @midenok

2 3

[Maria-developers] Obsolete GTID domain delete on master (MDEV-12012, MDEV-11969)
by andrei.elkin＠pp.inet.fi 24 Oct '17

24 Oct '17

Hello. Let me propose methods to clean master off unused gtid domains. The issue is quite practical, as a couple of references on the subject line tell. Either of them narrates a scenario of two default domain-id masters serving to one slave initially by "legacy" non-gtid protocol. When later the masters have changed their common domain to different private ones *and* the slave turns gtid replication ON, it can't connect to a master due to the gtid protocol. And the reason is the prior-to-gtid-connect gained slave's gtid position consisting of gtids generated by the other master obviously (the two masters never replicated each other) does not fit to the current master binlog state. The past default domain-id is actually permanent past from the user perspective in these cases. Its events have been already replicated and none new will be generated and replicated. Therefore such domain conceptually may be cleaned away from either the masters and slave states. Once it's done, the gtid-enabled slave will successfully connect to any master. The slave state purge is simple SET @@global.gtid_slave_pos to a sequence free of the purged domain. The master side binlog state one requires a new "command" unless the user is happy with RESET MASTER. While setting the new gtid binlog state to be old-domain-free we would like for the new "command" to preserve the existing binlog files. This could be accomplished as Kristian suggests in MDEV-12012 (I could not find any earlier references to the idea) as a new option to FLUSH BINARY LOGS DELETE DOMAIN d1, d2 KN> This command would check that the current binlog state is equal to KN> the GTID_LIST_LOG_EVENT at the start of the first binary log file, KN> within the specified domains. If not, it is an error. But if so, KN> the new binary log file created by FLUSH will be written with the KN> specified domains omitted from GTID_LIST_LOG_EVENT (and the current KN> binlog state updated accordingly). The idea looks quite sane, I only could not grasp why presence of being deleted domains in the very first binlog's GTID_LIST_LOG_EVENT list is warrant for throwing error. Maybe we should leave it out to the user, Kristian? That is to decide what domain is garbage regardless of the binlog state history. While the FLUSH way looks sufficient and robust I could not help to think over an alternative. Consider a scenario when a domain's sequence number got run out of range. While deems unrealistic in practice we can simulate it with SET @@SESSION.gtid_domain_id=11; SET @@SESSION.server_id=1; SET @@SESSION.gtid_seq_no=18446744073709551615; /* Exec some dummy loggable query, e.g */ CREATE TABLE IF NOT EXISTS `table_dummy`; SHOW LOCAL VARIABLES LIKE '%gtid_binlog_pos%'; 11-1-18446744073709551615 SET @@SESSION.gtid_seq_no=0 DROP TABLE `table_dummy`; SHOW LOCAL VARIABLES LIKE '%gtid_binlog_pos%'; 11-1-0 I've used two gtids to show domain overflow because I also liked to read the zero of the last gtid as ... there's *nothing* in this domain. So it's actually a new "namesake" one, replacing the old that is wrapped around. The 1st group of events created in the new domain - 11-1-1 - could shadow the old domain's 11-1-1 as well as all the rest of the old domain from gtid replication protocol. And that means the old domain is actually deleted. So if my reading of zero is correct the binlog status would be empty instead. That's how we also can approach the master side old gtid domain purging: 1. Leave wrapping around an old domain to the user via running the queries like above; 2. The binary logger would be made to react on the fact of wrap-around with binary log rotation ("internal" FLUSH BINARY LOG). And the new binlog file won't contain the wrapped "away" domain (because there are no new event group in it of yet). I would be glad to hear your opinions, dear colleagues. Cheers, Andrei

4 27

[Maria-developers] Bitmap indexing with Mariadb
by chanaka rathnayaka 16 Oct '17

16 Oct '17

Hi, I am Chanaka who is an undergraduate engineering student. I am doing a research to check, how bitmap indexing improves the query performance and I hope to use MariaDB as database engine. So I need to know whether MariaDB has bitmap indexing already or not ? Thank you, Regards, Chanaka <http://www.justforlearning.info> Chanaka Rathnayaka / undergraduate, Department of Computer Science and Engineering, University of Moratuwa, Sri Lanka bmcrathnayaka(a)gmail.com / +94776608457 [image: Twitter] <https://htmlsig.com/t/0000001BX4MCF> [image: Facebook] <https://htmlsig.com/t/0000001BR3W18> [image: LinkedIn] <https://htmlsig.com/t/0000001BT4W90> [image: Github] <https://htmlsig.com/t/0000001BYB6WK>

2 1

[Maria-developers] MDEV-13550 is test case required?
by Aleksey Midenkov 14 Oct '17

14 Oct '17

Hello! Do we have to write test cases for such coding errors? Because this is developer factor: either he does such errors or not. What are conditions of getting PR merged?

2 2

Re: [Maria-developers] MDEV-14013
by Alexander Barkov 14 Oct '17

14 Oct '17

Hello Jerome, Thank you very much for you contribution! On 10/11/2017 02:32 PM, jerome brauge wrote: > Hello Alexander, > Here is the patch for MDEV-14013, can you review it ? The patch looks very good. I only found small things: 1. LEX::make_text_literal_concat() thd->variables.character_set_client was used instread of thd->variables.collation_connection 2. LEX::make_text_literal_concat() was somewhat hard to read. I propose to change the logic to do: if (item->type() == Item::NULL_ITEM) { } instead of if (item->type() != Item::NULL_ITEM || !(thd->variables.sql_mode & MODE_EMPTY_STRING_IS_NULL)) I think it looks clearer this way. Also, I added some DBUG_ASSERTs to make more sure that a non-Item_string item is not erroneously cast to Item_string. 3. Also, I found that it's easier to place the new methods into THD instead of LEX. Please find a patch with changes 1,2,3 attached. Do you agree with these changes? Also, can you please add some tests: - SHOW CREATE VIEW into "Test in a view" in empty_string_literal.result.diff - EXPLAIN EXTENDED for the affected grammar: EXPLAIN EXTENDED SELECT ''; EXPLAIN EXTENDED SELECT _latin1''; EXPLAIN EXTENDED SELECT N''; EXPLAIN EXTENDED SELECT '' ''; Thanks! > > Thanks. > > >> -----Message d'origine----- >> De : Alexander Barkov [mailto:bar@mariadb.org] >> Envoyé : mercredi 11 octobre 2017 04:15 >> À : jerome brauge >> Objet : Re: Question about MDEV-10574 >> >> Hello Jerome, >> >> >> On 10/10/2017 04:23 PM, jerome brauge wrote: >>> Hello Alexander, >>> For sql_mode EMPTY_STRING_IS_NULL, do you think that it must cover call >> of statement with use of user variable initialized with an empty string ? >>> >>> Ex : >>> SET sql_mode=default; >>> SET @param=''; >>> >>> SET sql_mode=EMPTY_STRING_IS_NULL; >>> PREPARE test FROM 'select 1,?'; >>> EXECUTE test USING @param; >>> >>> Or >>> >>> delimiter / >>> CREATE OR REPLACE PROCEDURE p1(IN a VARCHAR(50)) BEGIN >>> IF a is null THEN >>> SELECT 'IS NULL'; >>> ELSE >>> select 'IS NOT NULL'; >>> END IF; >>> END; >>> / >>> delimiter ; >>> SET sql_mode=default; >>> SET @a=''; >>> >>> SET sql_mode=EMPTY_STRING_IS_NULL; >>> EXECUTE IMMEDIATE 'CALL p1(?)' USING @a; CALL p1(@a); >>> >>> For me, it's a study case and we can forget it. >> >> I agree. Oracle doesn't have user variables. >> >>> >>> Regards, >>> Jérôme. >>> >>>> -----Message d'origine----- >>>> De : Alexander Barkov [mailto:bar@mariadb.org] Envoyé : samedi 7 >>>> octobre 2017 11:44 À : jerome brauge Objet : Re: Patch for MDEV-10574 >>>> >>>> Hello Jerome, >>>> >>>> >>>> On 10/06/2017 01:41 PM, jerome brauge wrote: >>>>> Hello Alexander, >>>>> >>>>>> -----Message d'origine----- >>>>>> De : Alexander Barkov [mailto:bar@mariadb.org] Envoyé : jeudi 5 >>>>>> octobre 2017 21:26 À : jerome brauge Objet : Re: Patch for >>>>>> MDEV-10574 >>>>>> >>>>>> Hello Jerome, >>>>>> >>>>>> >>>>>> On 10/05/2017 12:45 PM, jerome brauge wrote: >>>>>>> Hello Alexander, >>>>>>> Have you gone ahead on your reflection about nulls and empty strings >> ? >>>>>> >>>>>> Yes, I think so. >>>>>> >>>>>> One addition: Monty suggested to rename flags to: >>>>>> - EMPTY_STRING_IS_NULL for literals >>>>>> - EMPTY_STRING_IS_NULL_RESULTS for functions. >>>>>> >>>>>> I'm fine with EMPTY_STRING_IS_NULL for literals, but think that >>>>>> EMPTY_STRING_IS_NULL_FUNCTIONS should be more clear than >>>>>> EMPTY_STRING_IS_NULL_RESULTS. >>>>>> Which name do you prefer for functions? >>>>> >>>>> I think like you. >>>>> For me, EMPTY_STRING_IS_NULL_RESULTS sounds like including >>>> RESULTSET >>>>> of select (which is perhaps included in point 4) >>>> >>>> Okey, let's go with _FUNCTIONS then. >>>> >>>>> >>>>>>> >>>>>>> Can I >>>>>>> - redo my pull request for substring_oracle function >>>>>>> - make a patch for first new sql_mode >>>>>> (MODE_EMPTY_AS_NULL_LITERALS) >>>>>> >>>>>> >>>>>> Yes, please. Let's move forward. >>>>>> >>>>>> Let's push the patch for substr_oracle first. >>>>>> Let's keep it return empty strings for the cases when the >>>>>> substring_length parameters is less than 1. >>>>>> It will start automatically returning NULL when we add the new >>>>>> sql_mode for functions. >>>>> >>>>> Pull request is done. >>>> >>>> I have merged it. Thanks! >>>> >>>>> >>>>>> Also, let's go ahead with the part for literals. >>>>> >>>>> Ok, I will make a separate patch. >>>>> Many thanks. >>>>> >>>>>> >>>>>> >>>>>> I have created MDEVs. Please use them in the commit comment and in >>>>>> the >>>>>> tests: >>>>>> >>>>>> MDEV-14012 sql_mode=Oracle: substr(): treat position 0 as position >>>>>> 1 >>>>>> MDEV-14013 sql_mode=EMPTY_STRING_IS_NULL >>>>>> >>>>>> >>>>>> Thanks! >>>>>> >>>>>>> >>>>>>> Regards, >>>>>>> Jérôme. >>>>>>> >>>>>>> >>>>>>>> -----Message d'origine----- >>>>>>>> De : Alexander Barkov [mailto:bar@mariadb.org] Envoyé : mercredi >>>>>>>> 27 septembre 2017 08:50 À : jerome brauge Objet : Re: Patch for >>>>>>>> MDEV-10574 >>>>>>>> >>>>>>>> Hello Jerome, >>>>>>>> >>>>>>>> I told to Monty yesterday. >>>>>>>> >>>>>>>> We were going to implement the following transformations: >>>>>>>> >>>>>>>>> We are considering to implement both of the following options >>>>>>>>> (with a >>>>>>>>> switch) >>>>>>>>> >>>>>>>>> Transform Insert >>>>>>>>> - insert values ("") -> insert values (null) >>>>>>>>> >>>>>>>>> Transform Select >>>>>>>>> >>>>>>>>> where v=x => (v <> "" and V=X) >>>>>>>>> where v is null => (v="" or v is null) >>>>>>>>> >>>>>>>> >>>>>>>> See MDEV-10574 for details. >>>>>>>> >>>>>>>> Monty did not fully understand the idea of translating literals >>>>>>>> and >>>>>> functions >>>>>>>> from empty strings to null. He asked if you can explain use cases >>>>>>>> for these transformations. >>>>>>>> >>>>>>>> But we agreed that: >>>>>>>> >>>>>>>> 1. It's OK to have multiple NULL handling flags in sql_mode. >>>>>>>> >>>>>>>> 2. New flags will not be enabled in 10.3, neither in >>>>>>>> sql_mode=DEFAULT, >>>>>> nor in >>>>>>>> sql_mode=ORACLE. So one will have to specify them explicitly: >>>>>>>> set >>>>>>>> >>>>>> >>>> >> sql_mode='ORACLE,MODE_null_transfrormation_flag1,MODE_null_transfor >>>>>>>> mation_flag2'; >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> So I propose we introduce the following flags: >>>>>>>> >>>>>>>> >>>>>>>> 1. MODE_EMPTY_AS_NULL_LITERALS - for literals and PS >> parameters, >>>> as >>>>>> in >>>>>>>> your patch. >>>>>>>> >>>>>>>> 2. MODE_EMPTY_AS_NULL_FUNCTIONS - for functions. >>>>>>>> This should be done as a separate patch. >>>>>>>> Note, your patch currently implements empty-to-null translation >>>>>>>> only for Item_str_func descendants that use >> make_empty_result(). >>>>>>>> This patch should be extended to cover Item_str_func descendants >>>>>>>> which do not use make_empty_result(), as well as >>>>>>>> all other Item_func_or_sum descendants >>>>>>>> (which are not Item_str_func descendants). >>>>>>>> >>>>>>>> >>>>>>>> 3. MODE_EMPTY_AS_NULL_UPDATES >>>>>>>> >>>>>>>> For INSERT and UPDATE transformation, as in MDEV: >>>>>>>> >>>>>>>> insert values ("") -> insert values (null) >>>>>>>> update set a='' -> update set a=null; >>>>>>>> >>>>>>>> >>>>>>>> 4. MODE_EMPTY_AS_NULL_PREDICATES >>>>>>>> >>>>>>>> For predicate transformation, as in MDEV: >>>>>>>> >>>>>>>> where v=x => (v <> "" and V=X) >>>>>>>> where v is null => (v="" or v is null) >>>>>>>> >>>>>>>> >>>>>>>> What do you think about this proposal with multiple flags? >>>>>>>> Note, we can do N1 and N2 now, and add N3 and N4 and later. >>>>>>>> >>>>>>>> Also, can you please write a description why you need N1 and N2. >>>>>>>> Monty thinks that N3 and N4 should cover most use cases. >>>>>>>> So we need a good rationale for Monty and Sergei explaining why >>>>>>>> we >>>>>> adding >>>>>>>> these flags N1 and N2. >>>>>>>> >>>>>>>> >>>>>>>> Thanks! >>>>>>>> >>>>>>>> >>>>>>>> On 09/26/2017 04:33 PM, jerome brauge wrote: >>>>>>>>> Hi Alexander, >>>>>>>>> I'm disappointed. >>>>>>>>> Can we at least keep this part in sql_prepare : >>>>>>>>> >>>>>>>>> /* >>>>>>>>> set_param_str_oracle : convert empty string to null value >>>>>>>>> This allow to bind empty string in JDBC as a null value. >>>>>>>>> Ex : >>>>>>>>> String sel="select coalesce(?,'null param') from dual"; >>>>>>>>> PreparedStatement ps_sel = m_cnx.prepareStatement(sel); >>>>>>>>> ps_sel.setString(1, ""); >>>>>>>>> ResultSet rs = ps_sel.executeQuery(); >>>>>>>>> while (rs.next()) >>>>>>>>> { >>>>>>>>> System.out.println(rs.getString(1)); >>>>>>>>> } >>>>>>>>> Returns : 'null param' >>>>>>>>> */ >>>>>>>>> static void set_param_str_oracle(Item_param *param, uchar >> **pos, >>>>>> ulong >>>>>>>>> len) { >>>>>>>>> ulong length= get_param_length(pos, len); >>>>>>>>> if (length == 0) >>>>>>>>> param->set_null(); >>>>>>>>> else >>>>>>>>> { >>>>>>>>> if (length > len) >>>>>>>>> length= len; >>>>>>>>> param->set_str((const char *) *pos, length); >>>>>>>>> *pos+= length; >>>>>>>>> } >>>>>>>>> } >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> Jérôme. >>>>>>>>> >>>>>>>>> >>>>>>>>>> -----Message d'origine----- >>>>>>>>>> De : Alexander Barkov [mailto:bar@mariadb.org] Envoyé : mardi >>>>>>>>>> 26 septembre 2017 13:34 À : jerome brauge Objet : Re: Patch for >>>>>>>>>> MDEV-10574 >>>>>>>>>> >>>>>>>>>> Hello Jérôme, >>>>>>>>>> >>>>>>>>>> I'm afraid we cannot accept this change at this point of time. >>>>>>>>>> >>>>>>>>>> The problem is that all around the code we assume that NULL is >>>>>>>>>> not equal to ''. >>>>>>>>>> >>>>>>>>>> Changing '' to NULL only in the visible part of the iceberg >>>>>>>>>> (such as in the parser and in >>>>>>>>>> Item_str_func::make_empty_result()) >>>>>>>>>> will likely introduce various anomalies in fundamental things >>>>>>>>>> like joins execution, optimizer, equality propagation, etc. >>>>>>>>>> >>>>>>>>>> Sorry, we don't have resources to dive into this topic deeper now. >>>>>>>>>> Moreover, Oracle's documentation suggests not to reply on the >>>>>>>>>> fact >>>>>> that '' >>>>>>>>>> and NULL are treated as same, as this can change in the future. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 09/20/2017 03:52 PM, jerome brauge wrote: >>>>>>>>>>> Hello Alexander, >>>>>>>>>>> Here is a new version of this patch (minor optimizations) >>>>>>>>>>> >>>>>>>>>>> Regards, >>>>>>>>>>> Jérôme. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> -----Message d'origine----- >>>>>>>>>>>> De : Maria-developers [mailto:maria-developers- >>>>>>>>>>>> bounces+j.brauge=qualiac.com(a)lists.launchpad.net] De la part >>>> de >>>>>>>>>>>> bounces+jerome >>>>>>>>>>>> brauge >>>>>>>>>>>> Envoyé : mardi 19 septembre 2017 16:09 À : 'Alexander Barkov' >>>>>>>>>>>> Cc : MariaDB Developers >>>>>>>>>>>> (maria-developers(a)lists.launchpad.net) >>>>>>>>>>>> Objet : [Maria-developers] Patch for MDEV-10574 >>>>>>>>>>>> >>>>>>>>>>>> Alexander, >>>>>>>>>>>> Here is the patch I was talking about. >>>>>>>>>>>> It not address all aspects of MDEV-10574, especially, empty >>>>>>>>>>>> strings in table column are not view as null. >>>>>>>>>>>> But as in oracle mode, no empty string in columns should be >>>>>>>>>>>> created, an fully oracle application will never fall in this case. >>>>>>>>>>>> >>>>>>>>>>>> Regards, >>>>>>>>>>>> Jérôme. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> -----Message d'origine----- >>>>>>>>>>>>> De : jerome brauge >>>>>>>>>>>>> Envoyé : mardi 19 septembre 2017 14:13 À : 'Alexander >> Barkov' >>>>>>>>>>>>> Objet : RE: Oracle strings functions compatibility : substr >>>>>>>>>>>>> >>>>>>>>>>>>> Hello Alexander, >>>>>>>>>>>>> Yes I'm aware. >>>>>>>>>>>>> This point will be treated by my next patch which will also >>>>>>>>>>>>> affect the others string functions (a part of MDEV-10574). >>>>>>>>>>>>> I will refresh this new patch with the current development >>>>>>>>>>>>> branch, and I submit to you. >>>>>>>>>>>>> Regards. >>>>>>>>>>>>> Jérôme. >>>>>>>>>>>>> >>>>>>>>>>>>>> -----Message d'origine----- De : Alexander Barkov >>>>>>>>>>>>>> [mailto:bar@mariadb.org] Envoyé : >>>> mardi >>>>>> 19 >>>>>>>>>>>>>> septembre 2017 13:16 À : jerome brauge Objet : Re: Oracle >>>>>>>>>>>>>> strings functions compatibility : substr >>>>>>>>>>>>>> >>>>>>>>>>>>>> Hi Jerome, >>>>>>>>>>>>>> >>>>>>>>>>>>>> I noticed one more difference in SUBSTR: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> SELECT SUBSTR('aaa',1,-1) FROM DUAL; >>>>>>>>>>>>>> >>>>>>>>>>>>>> MariaDB returns an empty string. >>>>>>>>>>>>>> Oracle returns NULL. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Here's Oracle's documentation: >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>> >>>>>> >> https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions162. >>>>>>>>>>>>>> ht >>>>>>>>>>>>>> m >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> I think we should do this in the same patch, to avoid >>>>>>>>>>>>>> behavior change in the future. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Would you like to try doing this additional change? >>>>>>>>>>>>>> Or would you like me to make an incremental patch on top >> of >>>>>> yours? >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks! >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 09/19/2017 10:33 AM, jerome brauge wrote: >>>>>>>>>>>>>>> Hello Alexander, >>>>>>>>>>>>>>> It's done. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Do you have news from Sergei for MDEV-12874, MDEV- >> 13417 >>>>>> and >>>>>>>>>>>> MDEV- >>>>>>>>>>>>>> 13418 ? >>>>>>>>>>>>>> >>>>>>>>>>>>>> I asked Sergei, awaiting for his answer. I'll let you know. >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thank you very much. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -----Message d'origine----- De : Alexander Barkov >>>>>>>>>>>>>>>> [mailto:bar@mariadb.org] Envoyé : lundi >>>>>>>>>>>>>>>> 18 septembre 2017 18:12 À : jerome brauge Objet : Re: >>>>>>>>>>>>>>>> Oracle strings functions compatibility : substr >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hello Jerome, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On 09/18/2017 05:14 PM, jerome brauge wrote: >>>>>>>>>>>>>>>>> Hello Alexander, >>>>>>>>>>>>>>>>> I don't understand how I've missed such an obvious >>>> solution ! >>>>>>>>>>>>>>>>> Here is the new patch. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks. Now it looks simpler. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Can you please do one small thing: >>>>>>>>>>>>>>>> move get_position() from "public:" to "protected:". >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Ok to make a pull request after this change. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thank you very much! >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Regards. >>>>>>>>>>>>>>>>> Jérôme. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> -----Message d'origine----- De : Alexander Barkov >>>>>>>>>>>>>>>>>> [mailto:bar@mariadb.org] Envoyé : >>>>>> lundi >>>>>>>>>>>>>>>>>> 18 septembre 2017 11:30 À : jerome brauge Objet : Re: >>>>>> Oracle >>>>>>>>>>>>>>>>>> strings functions compatibility : substr >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hello Jerome, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On 08/17/2017 04:43 PM, jerome brauge wrote: >>>>>>>>>>>>>>>>>>> Hello Alexander, >>>>>>>>>>>>>>>>>>> Here is a patch for function SUBSTR in >> sql_mode=oracle >>>>>>>>>>>>>>>>>>> (If position is 0, >>>>>>>>>>>>>>>>>> then it is treated as 1). >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> I'm thinking of how to get a smaller patch. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Wouldn't it be possible to introduce: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> virtual longlong get_position(); >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> longlong Item_func_substr::get_position() { >>>>>>>>>>>>>>>>>> return args[1]->val_int(); } >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> longlong Item_func_substr_oracle::get_position() >>>>>>>>>>>>>>>>>> { >>>>>>>>>>>>>>>>>> longlong pos= args[1]->val_int(); >>>>>>>>>>>>>>>>>> return pos == 0 ? 1 : pos; } >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> and just replace all calls for args[1]->val_int() to >>>>>> get_position() >>>>>>>> ? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks! >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>>>> Jérôme. >>>>>>>>>>>>>>>>>>>

2 2

Re: [Maria-developers] [Commits] e0a1e63: MDEV-9619: Assertion `null_ref_table' failed in virtual table_map Item_direct_view_ref::used_tables() const on 2nd execution of PS
by Sergei Golubchik 13 Oct '17

13 Oct '17

Hi, Oleksandr! On Feb 18, Oleksandr Byelkin wrote: > revision-id: e0a1e632e4e6084cd989f217943618f4e9ac9179 (mariadb-5.5.54-25-ge0a1e63) > parent(s): 29d78dbb44ee9890b6bc28873344f20fc9157928 > committer: Oleksandr Byelkin > timestamp: 2017-02-18 17:47:31 +0100 > message: > > MDEV-9619: Assertion `null_ref_table' failed in virtual table_map > Item_direct_view_ref::used_tables() const on 2nd execution of PS > > Refer left expression indirectly in case it changes from execution to > execution. Looks reasonable. Ok to push with these changes: 1. replace - DBUG_ASSERT(null_ref_table); + DBUG_ASSERT(fixed); as discussed. 2. Remove bool Item_direct_view_ref::fix_fields(THD *thd, Item **reference) { - DBUG_ASSERT(1); these two changes should, probably, be done in a separate cleanup commit. 3. > diff --git a/sql/item.h b/sql/item.h > index 89155ac..dff7bc4 100644 > --- a/sql/item.h > +++ b/sql/item.h > @@ -1543,6 +1543,8 @@ class Item { > virtual void mark_as_condition_AND_part(TABLE_LIST *embedding) {}; > }; > > +typedef Item (*Item_ptr); > + remove this, your patch compiles just fine without it. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] Please review MDEV-10802 TIMESTAMP NOT NULL field with explicit_defaults_for_timestamp and NO_ZERO_DATE shouldn't throw error
by Alexander Barkov 13 Oct '17

13 Oct '17

Hi Sergei, Please review a patch for MDEV-10802. Thanks!

2 2

Re: [Maria-developers] [Commits] 06bb7ee: MDEV-10892 - rpl.rpl_semi_sync_uninstall_plugin fails with Assertion `0' failure
by Sergei Golubchik 13 Oct '17

13 Oct '17

Hi, Sergey! On Oct 12, Sergey Vojtovich wrote: > revision-id: 06bb7eef9882536fc1d99b4462a65b53ec9f137e (mariadb-5.5.57-14-g06bb7ee) > parent(s): 93aadda513d8b5c2b49001514e235c4fdd73e08a > committer: Sergey Vojtovich > timestamp: 2017-10-12 15:37:45 +0400 > message: > > MDEV-10892 - rpl.rpl_semi_sync_uninstall_plugin fails with Assertion `0' failure > in buildbot > > Removed plugin_array_version: it is being checked without mutex protection and > thus is prone to data race and race conditions. In effect plugins are not > protected from concurrent destruction. > > Removed state_mask inversion: doesn't seem to make any sense. > > When collecting local plugins list, only add plugins that match state_mask. > > Use plugin ref counting to protect against concurrent plugin destruction. > diff --git a/sql/sql_show.cc b/sql/sql_show.cc > index 6e04564..512548b 100644 > --- a/sql/sql_show.cc > +++ b/sql/sql_show.cc > @@ -262,7 +262,7 @@ int fill_plugins(THD *thd, TABLE_LIST *tables, COND *cond) > TABLE *table= tables->table; > > if (plugin_foreach_with_mask(thd, show_plugins, MYSQL_ANY_PLUGIN, > - ~PLUGIN_IS_FREED, table)) > + ~(PLUGIN_IS_FREED | PLUGIN_IS_DYING), table)) > DBUG_RETURN(1); > > DBUG_RETURN(0); > @@ -5294,7 +5294,8 @@ int fill_schema_engines(THD *thd, TABLE_LIST *tables, COND *cond) > DBUG_ENTER("fill_schema_engines"); > if (plugin_foreach_with_mask(thd, iter_schema_engines, > MYSQL_STORAGE_ENGINE_PLUGIN, > - ~PLUGIN_IS_FREED, tables->table)) > + ~(PLUGIN_IS_FREED | PLUGIN_IS_DYING), > + tables->table)) > DBUG_RETURN(1); > DBUG_RETURN(0); > } agree, ok to push Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] 96e982b: MDEV-7140 - Sporadic buildbot failures in rpl.rpl_semi_sync_uninstall_plugin
by Sergei Golubchik 13 Oct '17

13 Oct '17

Hi, Sergey! On Oct 12, Sergey Vojtovich wrote: > revision-id: 96e982bb1a72db47a41b97bb632b595b347d2956 (mariadb-5.5.57-14-g96e982b) > parent(s): 93aadda513d8b5c2b49001514e235c4fdd73e08a > committer: Sergey Vojtovich > timestamp: 2017-10-12 14:42:17 +0400 > message: > > MDEV-7140 - Sporadic buildbot failures in rpl.rpl_semi_sync_uninstall_plugin > > Removed plugin_array_version: it is being checked without mutex protection and > thus is prone to data race and race conditions. In effect plugins are not > protected from concurrent destruction. > > Removed state_mask inversion: doesn't seem to make any sense. > > When collecting local plugins list, only add plugins that match state_mask. > > Use plugin ref counting to protect against concurrent plugin destruction. ok to push Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] MDEV-13530 VARBINARY doesn't convert to to BLOB for sizes 65533, 65534 and 65535
by Alexander Barkov 12 Oct '17

12 Oct '17

Hi Sergei, Please review a patch for MDEV-13530. Thanks!

1 0

Re: [Maria-developers] 46b93fc: MDEV-13676: Field "create Procedure" is NULL, even if the the user has role which is the definer. (SHOW CREATE PROCEDURE)
by Sergei Golubchik 11 Oct '17

11 Oct '17

Hi, Vicentiu! Looks good! A couple of comments, see below. Ok to push after fixing, the second review is not needed. On Oct 09, Vicentiu Ciorbaru wrote: > revision-id: 46b93fc2a2fa198d721813d9b93bbe09d2e36eb3 (mariadb-10.0.32-50-g46b93fc) > parent(s): dbeffabc83ed01112e09d7e782d44f044cfcb691 > author: Vicențiu Ciorbaru > committer: Vicențiu Ciorbaru > timestamp: 2017-10-09 13:32:40 +0300 > message: > > MDEV-13676: Field "create Procedure" is NULL, even if the the user has role which is the definer. (SHOW CREATE PROCEDURE) > > During show create procedure we ommited to check the current role, if it > is the actual definer of the procedure. In addition, we should support > indirectly granted roles to the current role. Implemented a recursive > lookup to search the tree of grants if the rolename is present. We both had to check the SQL standard to see what the behavior should be. Please add here (in the commit comment) a reference to the exact part of the standard that specifies this behavior. Like SQL Standard 2016, Part ... Section ... View I_S.ROUTINES selects ROUTINE_BODY and its WHERE clause says that the GRANTEE must be either PUBLIC, or CURRENT_USER or in the ENABLED_ROLES. > diff --git a/sql/sp_head.cc b/sql/sp_head.cc > index ea9e1c1..3dd1a65 100644 > --- a/sql/sp_head.cc > +++ b/sql/sp_head.cc > @@ -2588,10 +2588,18 @@ bool check_show_routine_access(THD *thd, sp_head *sp, bool *full_access) > *full_access= ((!check_table_access(thd, SELECT_ACL, &tables, FALSE, > 1, TRUE) && > (tables.grant.privilege & SELECT_ACL) != 0) || > + /* Check if user owns the routine. */ > (!strcmp(sp->m_definer_user.str, > thd->security_ctx->priv_user) && > !strcmp(sp->m_definer_host.str, > - thd->security_ctx->priv_host))); > + thd->security_ctx->priv_host)) || > + /* Check if current role or any of the sub-granted roles > + own the routine. */ > + (sp->m_definer_host.length == 0 && > + (!strcmp(sp->m_definer_user.str, > + thd->security_ctx->priv_role) || > + check_role_is_granted(thd->security_ctx->priv_role, NULL, > + sp->m_definer_user.str)))); Perhaps you could simplify the condition above a little bit, by replacing it with /* Check if current role or any of the sub-granted roles own the routine. */ (sp->m_definer_host.length == 0 && check_role_is_granted(thd->security_ctx->priv_user, thd->security_ctx->priv_host, sp->m_definer_user.str))); Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 1

[Maria-developers] MDEV-14038 ALTER TABLE does not exit on error with InnoDB + bad default function
by Alexander Barkov 11 Oct '17

11 Oct '17

Hello Marko, please review a patch for MDEV-14038. Thanks!

2 1

[Maria-developers] MDEV-13972 crash in Item_func_sec_to_time::get_date
by Alexander Barkov 09 Oct '17

09 Oct '17

Hello Sergei, Please review a patch for MDEV-13972. The patch is for 5.5, as we agreed in slack. Thanks!

2 3

Re: [Maria-developers] [Commits] 0240b866abc: MDEV-13838: Wrong result after altering a partitioned table
by Marko Mäkelä 09 Oct '17

09 Oct '17

On Mon, Oct 9, 2017 at 3:00 PM, jan <jan.lindstrom(a)mariadb.com> wrote: > revision-id: 109463fdb86d (mariadb-10.0.32-54-g0240b866abc) > parent(s): 172cc70bf8c0aea3d2d0c73bcf94f36c172b769a > author: Jan Lindström > committer: Jan Lindström > timestamp: 2017-10-09 14:52:51 +0300 > message: > > MDEV-13838: Wrong result after altering a partitioned table > > Reverted incorrect changes done on MDEV-7367 and MDEV-9469. Fixes properly > also related bugs: > > MDEV-13668: InnoDB unnecessarily rebuilds table when renaming a column and > adding index > MDEV-9469: 'Incorrect key file' on ALTER TABLE > MDEV-9548: Alter table (renaming and adding index) fails with "Incorrect > key file for table" > MDEV-10535: ALTER TABLE causes standalone/wsrep cluster crash > MDEV-13640: ALTER TABLE CHANGE and ADD INDEX on auto_increment column > fails with "Incorrect key file for table..." > This is a very welcome change that looks correct to me. I only have a few comments on the formatting and tests. The patch does not cleanly apply to 10.0 due to some difference in the comments of removed functions. Later, you informed me that this is https://github.com/MariaDB/server/commit/a8db085c51ccfd6df737d9261abac9d2b0… on bb-10.0-MDEV-13838 <https://github.com/MariaDB/server/compare/bb-10.0-MDEV-13838>. +Warnings: > +Note 1831 Duplicate index `c4_2`. This is deprecated and will be > disallowed in a future release. > The intention of the test was not to create duplicated indexes. The "rename column" and "add index" operations were split into two ALTER TABLE in order to work around the limitation that was introduced in a previous incorrect commit. You should have removed the added ALTER TABLE statements instead of adding SHOW CREATE TABLE statements. Please refer to https://github.com/MariaDB/server/commit/f56bd70f51020c67a30415ae381ae93b90… and revert the relevant parts of the test changes. The other tests seem to be adjusted fine. > @@ -171,3 +172,39 @@ ALTER TABLE ticket > SHOW CREATE TABLE ticket; > > DROP TABLE ticket; > + > +# > +# MDEV-13838: Wrong result after altering a partitioned table > +# > + > +CREATE TABLE `t` ( > +`id` bigint(20) unsigned NOT NULL auto_increment, > +`d` date NOT NULL, > +`a` bigint(20) unsigned NOT NULL, > +`b` smallint(5) unsigned DEFAULT NULL, > +PRIMARY KEY (`id`,`d`) > +) ENGINE=InnoDB DEFAULT CHARSET=latin1 COLLATE=latin1_general_cs > STATS_SAMPLE_PAGES=200 > +/*!50500 PARTITION BY RANGE COLUMNS(d) > +(PARTITION p20170913 VALUES LESS THAN ('2017-09-14') ENGINE = InnoDB, > +PARTITION p20170914 VALUES LESS THAN ('2017-09-15') ENGINE = InnoDB, > +PARTITION p20170915 VALUES LESS THAN ('2017-09-16') ENGINE = InnoDB, > +PARTITION p20170916 VALUES LESS THAN ('2017-09-17') ENGINE = InnoDB, > +PARTITION p20170917 VALUES LESS THAN ('2017-09-18') ENGINE = InnoDB, > +PARTITION p99991231 VALUES LESS THAN (MAXVALUE) ENGINE = InnoDB) */; > Please clean up the CREATE TABLE statement. Remove trailing space and unnecessary quotes and comments. > +insert into t(d,a,b) values ('2017-09-15',rand()*10000,rand()*10); > +insert into t(d,a,b) values ('2017-09-15',rand()*10000,rand()*10); > + > +replace into t(d,a,b) select '2017-09-15',rand()*10000,rand()*10 from t > t1, t t2, t t3, t t4, t t5, t t6, t t7, t t8, t t9, t t10, t t11, t t12, t > t13, t t14; > Does the table really have to be this big for the bug to repeat? Can you please reduce the test? Shouldn’t it work with 1 or 2 partitions and a few records in each? > + > +select count(*) from t where d ='2017-09-15'; > + > +ALTER TABLE t CHANGE b c smallint(5) unsigned , ADD KEY idx_d_a (d, a); > +SHOW CREATE TABLE t; > +analyze table t; > + > +select count(*) from t where d ='2017-09-15'; > +select count(*) from t force index(primary) where d ='2017-09-15'; > + > +DROP TABLE t; > + > Please do not end files in empty lines. > @@ -1625,13 +1606,9 @@ innobase_create_index_def( > index->ind_type |= DICT_FTS; > } > > - if (!new_clustered) { > - altered_table = NULL; > - } > - > for (i = 0; i < n_fields; i++) { > - innobase_create_index_field_def( > - altered_table, &key->key_part[i], > &index->fields[i], fields); > + innobase_create_index_field_def(new_clustered, > + altered_table, &key->key_part[i], > &index->fields[i]); > } > > DBUG_VOID_RETURN; > Do not add the first parameter to the same line with the function name. Instead, add it on a line on its own. In this way, it follows the indentation rules set by (defun innodb-cc-mode (&optional map) "InnoDB adjustments to cc-mode. In MAP, binds C-c C-c to 'compile." (if (and (buffer-file-name) (string-match "/\$innobase\\|xtradb\$.*/.*[ihc]\\'" (buffer-file-name))) (progn (setq mode-name (concat "Ib" mode-name) c-basic-offset 8 c-label-minimum-indentation 0) (add-to-list 'c-offsets-alist '(c . 0)) (add-to-list 'c-offsets-alist '(arglist-intro . +)) (add-to-list 'c-offsets-alist '(label . [0])))) (if map (define-key map "\C-c\C-c" 'compile))) > @@ -2121,7 +2097,7 @@ struct ha_innobase_inplace_ctx : public > inplace_alter_handler_ctx > /** mapping of old column numbers to new ones, or NULL */ > const ulint* col_map; > /** new column names, or NULL if nothing was renamed */ > - const char** col_names; > + const char** col_names; > /** added AUTO_INCREMENT column position, or ULINT_UNDEFINED */ > const ulint add_autoinc; > /** default values of ADD COLUMN, or NULL */ > Please do not change TAB to spaces. > @@ -3067,7 +3043,7 @@ prepare_inplace_alter_table_dict( > > ctx->add_index[a] = row_merge_create_index( > ctx->trx, ctx->new_table, > - &index_defs[a], ctx->col_names); > + &index_defs[a]); > > add_key_nums[a] = index_defs[a].key_number; > Consider joining the line after new_table. Best regards, Marko

1 0

[Maria-developers] MDEV-11819 NO_ZERO_IN_DATE: Incorrect generated column value
by Alexander Barkov 06 Oct '17

06 Oct '17

Hello Sergei, Please review a patch for MDEV-11819. Thanks.

2 1

[Maria-developers] Add RMariaDB CRAN Link to MariaDB Foundation Download Webpage
by Juan Telleria 06 Oct '17

06 Oct '17

Dear R Developers, A new package has been released for the "R Statistical Programming Language" in which the R community has put a great effort into it: RMariaDB. It enables a robust data transfer between R and MariaDB. The link on CRAN (R package repository) can be accessed from: https://cran.r-project.org/web/packages/RMariaDB/index.html And can be installed in R by using the following statement: install.packages("RMariaDB") library(RMariaDB) So, just as PHP community developed connector was distributed in the past within MySQL webpage, I do think that this connector could benefit from being distributed (or having a link to CRAN) from the MariaDB Foundation Download Webpage. I believe that many people has putted a great effort on this connector, and it would benefit from greater visibility for better support and maintenance. Thank you, Juan

2 1

Re: [Maria-developers] [Commits] 3f7e619647f: MDEV-13928: Missing symbols building RocksDB on macOS 10.12.6
by Sergei Golubchik 05 Oct '17

05 Oct '17

Hi, Varun! Why did it fail only on MacOS X? On Oct 06, Varun wrote: > revision-id: 3f7e619647f7bc1ec31a6118b403da805c48ae51 (mariadb-10.2.5-653-g3f7e619647f) > parent(s): 08c493c62afae0e81f9da6bb9b86efe99a9a763d > author: Varun Gupta > committer: Varun Gupta > timestamp: 2017-10-06 00:20:08 +0530 > message: > > MDEV-13928: Missing symbols building RocksDB on macOS 10.12.6 > > rdb_sst_info.cc must be compiled with RTTI > > --- > storage/rocksdb/CMakeLists.txt | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/storage/rocksdb/CMakeLists.txt b/storage/rocksdb/CMakeLists.txt > index 6db82119b45..80a6614b757 100644 > --- a/storage/rocksdb/CMakeLists.txt > +++ b/storage/rocksdb/CMakeLists.txt > @@ -170,6 +170,7 @@ IF(CMAKE_CXX_COMPILER_ID MATCHES "GNU" OR CMAKE_CXX_COMPILER_ID MATCHES "Clang") > # includes requires it. So, now everything in MariaRocks is compiled with > # -frtti) > set_source_files_properties(event_listener.cc rdb_cf_options.cc > + rdb_sst_info.cc > PROPERTIES COMPILE_FLAGS -frtti) > ENDIF() Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] MDEV-13417 / MDEV - 13418 The order of evaluation of SELECT..INTO assignments
by jerome brauge 03 Oct '17

03 Oct '17

Hello Sergei, Here is a patch for MDEV-13417 and MDEV-13418. Until you choose the right name for this new sql_mode, I temporarily chosen "SET_CONSISTENCY". It is set only for sql_mode=oracle for now. Can you review it ? Thank you. Regard.

2 6

[Maria-developers] MDEV-10596
by jerome brauge 27 Sep '17

27 Sep '17

Hi Alexander, In sql_mode=oracle, when a stored procedure varchar parameter is defined without size, you have chosen to translate it by VARCHAR(4000) (or CHAR(2000) for CHAR parameter). Oracle does not work like this. Size is inherited from the size of argument at runtime. Example: CREATE OR REPLACE PROCEDURE p1(p OUT VARCHAR) AS BEGIN p:='0123456789'; END; / declare w varchar(10); begin p1(w); end; / --> work fine declare w varchar(8); begin p1(w); end; / --> failed with : ERROR at line 1: ORA-06502: PL/SQL: numeric or value error: character string buffer too small ORA-06512: at "OPS$DBO.P1", line 4 ORA-06512: at line 3 Furthermore, since Oracle 9, VARCHAR datatype in PL/SQL is not limited to 4000 char but to 32k. It's the size of varchar column in a table that is limited to 4000 (until Oracle 12C which allow 32k when MAX_STRING_SIZE=EXTENDED). I've done the attached patch to resize these parameters at runtime. What do you think about it ? Another question : Oracle does not allow to change the value of an IN parameter. The following procedure cannot be compile : CREATE OR REPLACE PROCEDURE p1(p IN VARCHAR) AS BEGIN p:='0123456789'; END; / Warning: Procedure created with compilation errors. //sun10:1521/CS> select * from user_errors; NAME TYPE SEQUENCE LINE POSITION ------------------------------ ------------ ---------- ---------- ---------- TEXT ------------------------------------------------------------------------------------------------------------------------------------ ATTRIBUTE MESSAGE_NUMBER --------- -------------- P1 PROCEDURE 1 4 3 PLS-00363: expression 'P' cannot be used as an assignment target ERROR 363 Mariadb allow this. It's fine for us but do you have plan to change this behavior ? (or add a new STRICT_xxx mode ?) Regard, Jérôme.

2 3

[Maria-developers] GCC7 warnings
by Reindl Harald 24 Sep '17

24 Sep '17

hard to say how serious they are but maybe a good idea to take a look /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/libmariadb/mariadb_config/mariadb_config.c:9:49: warning: backslash and newline separated by space #define LIBS "-L/usr/lib64/mysql/ -lmariadb "\ /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/libmariadb/libmariadb/mariadb_lib.c: In function 'mariadb_get_infov': /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/libmariadb/libmariadb/mariadb_lib.c:3872:30: warning: this statement may fall through [-Wimplicit-fallthrough=] *((unsigned long *)arg)= mysql->client_flag; ~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~ /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/libmariadb/libmariadb/mariadb_lib.c:3875:3: note: here default: ^~~~~~~ In file included from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/plugin/handler_socket/libhsclient/hstcpcli.cpp:13:0: /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/plugin/handler_socket/libhsclient/hstcpcli.hpp:33:14: warning: 'template<class> class std::auto_ptr' is deprecated [-Wdeprecated-declarations] typedef std::auto_ptr<hstcpcli_i> hstcpcli_ptr; ^~~~~~~~ In file included from /usr/include/c++/7/memory:80:0, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/plugin/handler_socket/libhsclient/hstcpcli.hpp:15, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/plugin/handler_socket/libhsclient/hstcpcli.cpp:13: /usr/include/c++/7/bits/unique_ptr.h:51:28: note: declared here template<typename> class auto_ptr; /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/libmariadb/libmariadb/ma_dtoa.c: In function 'dtoa': /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/libmariadb/libmariadb/ma_dtoa.c:1465:14: warning: this statement may fall through [-Wimplicit-fallthrough=] leftright= 0; ~~~~~~~~~^~~ /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/libmariadb/libmariadb/ma_dtoa.c:1467:3: note: here case 4: ^~~~ /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/libmariadb/libmariadb/ma_dtoa.c:1473:14: warning: this statement may fall through [-Wimplicit-fallthrough=] leftright= 0; ~~~~~~~~~^~~ /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/libmariadb/libmariadb/ma_dtoa.c:1475:3: note: here case 5: ^~~~ At top level: /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/libmariadb/libmariadb/ma_dtoa.c:1161:21: warning: 'tinytens' defined but not used [-Wunused-const-variable=] static const double tinytens[]= ^~~~~~~~ /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/libmariadb/libmariadb/mariadb_dyncol.c: In function 'mariadb_dyncol_val_double': /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/libmariadb/libmariadb/mariadb_dyncol.c:4044:7: warning: this statement may fall through [-Wimplicit-fallthrough=] { ^ /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/libmariadb/libmariadb/mariadb_dyncol.c:4060:5: note: here case DYN_COL_DATETIME: ^~~~ /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/libmariadb/libmariadb/mariadb_dyncol.c:4050:15: warning: argument 1 null where non-null expected [-Wnonnull] *dbl= strtod(str, &end); ^~~~~~~~~~~~~~~~~ In file included from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/libmariadb/include/ma_global.h:168:0, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/libmariadb/libmariadb/mariadb_dyncol.c:64: /usr/include/stdlib.h:126:15: note: in a call to function 'strtod' declared here extern double strtod (const char *__restrict __nptr, ^~~~~~ /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/libmariadb/unittest/libmariadb/t_aurora.c: In function 'test_reconnect': /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/libmariadb/unittest/libmariadb/t_aurora.c:100:3: warning: ignoring return value of 'system', declared with attribute warn_unused_result [-Wunused-result] system("/usr/local/aws/bin/aws rds failover-db-cluster --db-cluster-identifier instance-1-cluster"); ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/mysys/my_lib.c: In function 'my_dir': /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/mysys/my_lib.c:144:3: warning: 'readdir_r' is deprecated [-Wdeprecated-declarations] while (!(READDIR(dirp,(struct dirent*) dirent_tmp,dp))) ^~~~~ In file included from /usr/include/features.h:410:0, from /usr/include/pthread.h:21, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/include/my_global.h:278, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/mysys/mysys_priv.h:19, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/mysys/my_lib.c:19: /usr/include/dirent.h:189:12: note: declared here extern int __REDIRECT (readdir_r, /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/storage/myisam/myisamchk.c: In function 'myisamchk': /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/storage/myisam/myisamchk.c:1411:22: warning: '%ld' directive writing between 1 and 10 bytes into a region of size 8 [-Wformat-overflow=] sprintf(null_pos,"%ld",(long) keyseg->null_pos+1); ^~~ /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/storage/myisam/myisamchk.c:1411:21: note: directive argument in the range [1, 4294967296] sprintf(null_pos,"%ld",(long) keyseg->null_pos+1); ^~~~~ In file included from /usr/include/stdio.h:939:0, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/include/my_global.h:322, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/include/my_base.h:23, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/include/myisam.h:26, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/storage/myisam/myisamdef.h:20, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/storage/myisam/fulltext.h:21, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/storage/myisam/myisamchk.c:18: /usr/include/bits/stdio2.h:33:10: note: '__builtin___sprintf_chk' output between 2 and 11 bytes into a destination of size 8 return __builtin___sprintf_chk (__s, __USE_FORTIFY_LEVEL - 1, ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ __bos (__s), __fmt, __va_arg_pack ()); ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/storage/myisam/mi_test3.c: In function 'test_write': /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/storage/myisam/mi_test3.c:367:30: warning: '%7ld' directive writing between 7 and 11 bytes into a region of size 8 [-Wformat-overflow=] sprintf((char*) record.id,"%7ld",(long) getpid()); ^~~~ /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/storage/myisam/mi_test3.c:367:29: note: directive argument in the range [-2147483648, 2147483647] sprintf((char*) record.id,"%7ld",(long) getpid()); ^~~~~~ In file included from /usr/include/stdio.h:939:0, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/include/my_global.h:322, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/include/my_base.h:23, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/include/myisam.h:26, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/storage/myisam/mi_test3.c:20: /usr/include/bits/stdio2.h:33:10: note: '__builtin___sprintf_chk' output between 8 and 12 bytes into a destination of size 8 return __builtin___sprintf_chk (__s, __USE_FORTIFY_LEVEL - 1, ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ __bos (__s), __fmt, __va_arg_pack ()); /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/storage/maria/maria_chk.c: In function 'maria_chk': /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/storage/maria/maria_chk.c:1703:22: warning: '%ld' directive writing between 1 and 10 bytes into a region of size 8 [-Wformat-overflow=] sprintf(null_pos,"%ld",(long) keyseg->null_pos+1); ^~~ /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/storage/maria/maria_chk.c:1703:21: note: directive argument in the range [1, 4294967296] sprintf(null_pos,"%ld",(long) keyseg->null_pos+1); ^~~~~ In file included from /usr/include/stdio.h:939:0, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/include/my_global.h:322, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/include/my_base.h:23, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/include/maria.h:22, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/storage/maria/maria_def.h:18, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/storage/maria/ma_fulltext.h:20, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/storage/maria/maria_chk.c:18: /usr/include/bits/stdio2.h:33:10: note: '__builtin___sprintf_chk' output between 2 and 11 bytes into a destination of size 8 return __builtin___sprintf_chk (__s, __USE_FORTIFY_LEVEL - 1, ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ __bos (__s), __fmt, __va_arg_pack ()); ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/storage/maria/ma_test3.c: In function 'test_write': /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/storage/maria/ma_test3.c:365:30: warning: '%7ld' directive writing between 7 and 11 bytes into a region of size 8 [-Wformat-overflow=] sprintf((char*) record.id,"%7ld", (long) getpid()); ^~~~ /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/storage/maria/ma_test3.c:365:29: note: directive argument in the range [-2147483648, 2147483647] sprintf((char*) record.id,"%7ld", (long) getpid()); ^~~~~~ In file included from /usr/include/stdio.h:939:0, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/include/my_global.h:322, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/include/my_base.h:23, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/include/maria.h:22, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/storage/maria/maria_def.h:18, from /home/builduser/rpmbuild/BUILD/mariadb-10.2.8/storage/maria/ma_test3.c:20: /usr/include/bits/stdio2.h:33:10: note: '__builtin___sprintf_chk' output between 8 and 12 bytes into a destination of size 8 return __builtin___sprintf_chk (__s, __USE_FORTIFY_LEVEL - 1, ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ __bos (__s), __fmt, __va_arg_pack ()); ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

1 0

Re: [Maria-developers] [Commits] 2846574452e: MDEV-11553 Can't restore a PERSISTENT column that uses DATE_FORMAT()
by Alexander Barkov 23 Sep '17

23 Sep '17

Hello Sergei, Ok to push. One small suggestion. Instead of having a standalone function, please move get_locale as method in Item. Something like this: const MY_LOCALE *Item::get_locale() { StringBuffer<MAX_FIELD_WIDTH> tmp; String *locale_name= val_str_ascii(&tmp); const MY_LOCALE *lc; if (!locale_name || !(lc= my_locale_by_name(locale_name->c_ptr_safe()))) { THD *thd= current_thd; push_warning_printf(thd, Sql_condition::WARN_LEVEL_WARN, ER_UNKNOWN_LOCALE, ER_THD(thd, ER_UNKNOWN_LOCALE), locale_name ? locale_name->c_ptr_safe() : "NULL"); lc= &my_locale_en_US; } return lc; } Thanks! On 09/22/2017 10:30 PM, serg(a)mariadb.org wrote: > revision-id: 2846574452e82b9f391ca5d41d5021ac94730dec (base-0.6-627-g2846574452e) > parent(s): 5a8b6e38967f1e4aa7fe4885e54ee8ea44097e8e > author: Sergei Golubchik > committer: Sergei Golubchik > timestamp: 2017-09-22 20:24:53 +0200 > message: > > MDEV-11553 Can't restore a PERSISTENT column that uses DATE_FORMAT() > > 3-argument form of DATE_FORMAT > > --- > mysql-test/r/locale.result | 71 +++++++++++++++++++++- > .../suite/vcol/r/vcol_blocked_sql_funcs.result | 2 + > .../suite/vcol/r/vcol_supported_sql_funcs.result | 15 +++++ > .../suite/vcol/t/vcol_blocked_sql_funcs_main.inc | 3 + > .../suite/vcol/t/vcol_supported_sql_funcs_main.inc | 6 ++ > mysql-test/t/locale.test | 28 +++++++++ > sql/item_create.cc | 27 +------- > sql/item_strfunc.cc | 10 +-- > sql/item_strfunc.h | 4 +- > sql/item_timefunc.cc | 25 ++++++-- > sql/item_timefunc.h | 27 +++++--- > sql/lex.h | 1 + > sql/sql_yacc.yy | 16 ++++- > sql/sql_yacc_ora.yy | 16 ++++- > 14 files changed, 202 insertions(+), 49 deletions(-) > > diff --git a/mysql-test/r/locale.result b/mysql-test/r/locale.result > index a02e80ed21e..b750d038205 100644 > --- a/mysql-test/r/locale.result > +++ b/mysql-test/r/locale.result > @@ -229,15 +229,80 @@ SET lc_time_names=@old_50915_lc_time_names; > # End of 10.1 tests > # > create view v1 as select > +date_format('2001-10-02', '%c %b %M') as a, > +date_format('2001-10-02', '%c %b %M', 'ru_RU') as a1, > format(123456789,2) as b, > format(123456789,2,'rm_CH') as b1; > select * from v1; > -b b1 > -123,456,789.00 123'456'789,00 > +a a1 b b1 > +10 Oct October 10 Окт Октября 123,456,789.00 123'456'789,00 > show create view v1; > View Create View character_set_client collation_connection > -v1 CREATE ALGORITHM=UNDEFINED DEFINER=`root`@`localhost` SQL SECURITY DEFINER VIEW `v1` AS select format(123456789,2) AS `b`,format(123456789,2,'rm_CH') AS `b1` utf8 utf8_general_ci > +v1 CREATE ALGORITHM=UNDEFINED DEFINER=`root`@`localhost` SQL SECURITY DEFINER VIEW `v1` AS select date_format('2001-10-02','%c %b %M') AS `a`,date_format('2001-10-02','%c %b %M','ru_RU') AS `a1`,format(123456789,2) AS `b`,format(123456789,2,'rm_CH') AS `b1` utf8 utf8_general_ci > drop view v1; > # > # End of 10.2 tests > # > +select date_format('2001-01-01', '%w %a %W', 'ro_RO'); > +date_format('2001-01-01', '%w %a %W', 'ro_RO') > +1 Lu Luni > +select date_format('2001-01-03', '%w %a %W', 'ro_RO'); > +date_format('2001-01-03', '%w %a %W', 'ro_RO') > +3 Mi Miercuri > +select date_format('2001-01-05', '%w %a %W', 'ro_RO'); > +date_format('2001-01-05', '%w %a %W', 'ro_RO') > +5 Vi Vineri > +select date_format('2001-01-07', '%w %a %W', 'ro_RO'); > +date_format('2001-01-07', '%w %a %W', 'ro_RO') > +0 Du Duminică > +select date_format('2001-01-01', '%w %a %W', 'de_AT'); > +date_format('2001-01-01', '%w %a %W', 'de_AT') > +1 Mon Montag > +select date_format('2001-02-01', '%w %a %W', 'de_AT'); > +date_format('2001-02-01', '%w %a %W', 'de_AT') > +4 Don Donnerstag > +select date_format('2001-03-01', '%w %a %W', 'de_AT'); > +date_format('2001-03-01', '%w %a %W', 'de_AT') > +4 Don Donnerstag > +select date_format('2001-01-01', '%w %a %W', 'en_US'); > +date_format('2001-01-01', '%w %a %W', 'en_US') > +1 Mon Monday > +select date_format('2001-03-01', '%c %b %M', 'en_US'); > +date_format('2001-03-01', '%c %b %M', 'en_US') > +3 Mar March > +select date_format('2001-01-01', '%w %a %W', 'rm_CH'); > +date_format('2001-01-01', '%w %a %W', 'rm_CH') > +1 gli glindesdi > +select date_format('2001-01-03', '%w %a %W', 'rm_CH'); > +date_format('2001-01-03', '%w %a %W', 'rm_CH') > +3 me mesemna > +select date_format('2001-01-05', '%w %a %W', 'rm_CH'); > +date_format('2001-01-05', '%w %a %W', 'rm_CH') > +5 ve venderdi > +select date_format('2001-01-07', '%w %a %W', 'rm_CH'); > +date_format('2001-01-07', '%w %a %W', 'rm_CH') > +0 du dumengia > +select date_format('2001-02-01', '%c %b %M', 'rm_CH'); > +date_format('2001-02-01', '%c %b %M', 'rm_CH') > +2 favr favrer > +select date_format('2001-04-01', '%c %b %M', 'rm_CH'); > +date_format('2001-04-01', '%c %b %M', 'rm_CH') > +4 avr avrigl > +select date_format('2001-06-01', '%c %b %M', 'rm_CH'); > +date_format('2001-06-01', '%c %b %M', 'rm_CH') > +6 zercl zercladur > +select date_format('2001-08-01', '%c %b %M', 'rm_CH'); > +date_format('2001-08-01', '%c %b %M', 'rm_CH') > +8 avust avust > +select date_format('2001-10-01', '%c %b %M', 'rm_CH'); > +date_format('2001-10-01', '%c %b %M', 'rm_CH') > +10 oct october > +select date_format('2001-12-01', '%c %b %M', 'rm_CH'); > +date_format('2001-12-01', '%c %b %M', 'rm_CH') > +12 dec december > +select date_format('2001-01-06', '%w %a %W', 'de_CH'); > +date_format('2001-01-06', '%w %a %W', 'de_CH') > +6 Sa Samstag > +select date_format('2001-09-01', '%c %b %M', 'de_CH'); > +date_format('2001-09-01', '%c %b %M', 'de_CH') > +9 Sep September > diff --git a/mysql-test/suite/vcol/r/vcol_blocked_sql_funcs.result b/mysql-test/suite/vcol/r/vcol_blocked_sql_funcs.result > index 6605f7c99b4..2d5bcfe553d 100644 > --- a/mysql-test/suite/vcol/r/vcol_blocked_sql_funcs.result > +++ b/mysql-test/suite/vcol/r/vcol_blocked_sql_funcs.result > @@ -247,3 +247,5 @@ ERROR HY000: Expression in the GENERATED ALWAYS AS clause is too big > # Constant expression > create or replace table t1 (a int as (PI()) PERSISTENT); > drop table if exists t1; > +create table t1 (a timestamp, b varchar(255) as (date_format(a, '%w %a %m %b')) stored); > +ERROR HY000: Function or expression 'date_format()' cannot be used in the GENERATED ALWAYS AS clause of `b` > diff --git a/mysql-test/suite/vcol/r/vcol_supported_sql_funcs.result b/mysql-test/suite/vcol/r/vcol_supported_sql_funcs.result > index 3fa4f6e1431..10ab32f9486 100644 > --- a/mysql-test/suite/vcol/r/vcol_supported_sql_funcs.result > +++ b/mysql-test/suite/vcol/r/vcol_supported_sql_funcs.result > @@ -2894,6 +2894,21 @@ a b > 2012-12-12 Wednesday Wed December Dec > drop table t1; > set sql_warnings = 0; > +# DATE_FORMAT() STORED > +set sql_warnings = 1; > +create table t1 (a date, b varchar(100) as (date_format(a, '%W %a %M %b', 'de_DE')) STORED); > +show create table t1; > +Table Create Table > +t1 CREATE TABLE `t1` ( > + `a` date DEFAULT NULL, > + `b` varchar(100) GENERATED ALWAYS AS (date_format(`a`,'%W %a %M %b','de_DE')) STORED > +) ENGINE=MyISAM DEFAULT CHARSET=latin1 > +insert into t1 values ('2012-12-12',default); > +select * from t1; > +a b > +2012-12-12 Mittwoch Mi Dezember Dez > +drop table t1; > +set sql_warnings = 0; > # CURRENT_USER() > set sql_warnings = 1; > create table t1 (a char, b varchar(32) as (current_user())); > diff --git a/mysql-test/suite/vcol/t/vcol_blocked_sql_funcs_main.inc b/mysql-test/suite/vcol/t/vcol_blocked_sql_funcs_main.inc > index 766d0c7410c..37de02a2b29 100644 > --- a/mysql-test/suite/vcol/t/vcol_blocked_sql_funcs_main.inc > +++ b/mysql-test/suite/vcol/t/vcol_blocked_sql_funcs_main.inc > @@ -354,3 +354,6 @@ eval create or replace table t1 (a int, b varchar(16384) as (concat(a,'$tmp_long > create or replace table t1 (a int as (PI()) PERSISTENT); > > drop table if exists t1; > + > +--error ER_GENERATED_COLUMN_FUNCTION_IS_NOT_ALLOWED > +create table t1 (a timestamp, b varchar(255) as (date_format(a, '%w %a %m %b')) stored); > diff --git a/mysql-test/suite/vcol/t/vcol_supported_sql_funcs_main.inc b/mysql-test/suite/vcol/t/vcol_supported_sql_funcs_main.inc > index dafc42098dd..fa2162e5981 100644 > --- a/mysql-test/suite/vcol/t/vcol_supported_sql_funcs_main.inc > +++ b/mysql-test/suite/vcol/t/vcol_supported_sql_funcs_main.inc > @@ -1197,6 +1197,12 @@ let $values1 = '2012-12-12',default; > let $rows = 1; > --source suite/vcol/inc/vcol_supported_sql_funcs.inc > > +--echo # DATE_FORMAT() STORED > +let $cols = a date, b varchar(100) as (date_format(a, '%W %a %M %b', 'de_DE')) STORED; > +let $values1 = '2012-12-12',default; > +let $rows = 1; > +--source suite/vcol/inc/vcol_supported_sql_funcs.inc > + > --echo # CURRENT_USER() > let $cols = a char, b varchar(32) as (current_user()); > let $values1 = 'a', default; > diff --git a/mysql-test/t/locale.test b/mysql-test/t/locale.test > index 4570b968ecf..a9a507bc387 100644 > --- a/mysql-test/t/locale.test > +++ b/mysql-test/t/locale.test > @@ -144,6 +144,8 @@ SET lc_time_names=@old_50915_lc_time_names; > > # Item::print > create view v1 as select > + date_format('2001-10-02', '%c %b %M') as a, > + date_format('2001-10-02', '%c %b %M', 'ru_RU') as a1, > format(123456789,2) as b, > format(123456789,2,'rm_CH') as b1; > select * from v1; > @@ -153,3 +155,29 @@ drop view v1; > --echo # > --echo # End of 10.2 tests > --echo # > + > +# > +# MDEV-11553 Can't restore a PERSISTENT column that uses DATE_FORMAT() > +# > +# 3-argument syntax for DATE_FORMAT() > +select date_format('2001-01-01', '%w %a %W', 'ro_RO'); > +select date_format('2001-01-03', '%w %a %W', 'ro_RO'); > +select date_format('2001-01-05', '%w %a %W', 'ro_RO'); > +select date_format('2001-01-07', '%w %a %W', 'ro_RO'); > +select date_format('2001-01-01', '%w %a %W', 'de_AT'); > +select date_format('2001-02-01', '%w %a %W', 'de_AT'); > +select date_format('2001-03-01', '%w %a %W', 'de_AT'); > +select date_format('2001-01-01', '%w %a %W', 'en_US'); > +select date_format('2001-03-01', '%c %b %M', 'en_US'); > +select date_format('2001-01-01', '%w %a %W', 'rm_CH'); > +select date_format('2001-01-03', '%w %a %W', 'rm_CH'); > +select date_format('2001-01-05', '%w %a %W', 'rm_CH'); > +select date_format('2001-01-07', '%w %a %W', 'rm_CH'); > +select date_format('2001-02-01', '%c %b %M', 'rm_CH'); > +select date_format('2001-04-01', '%c %b %M', 'rm_CH'); > +select date_format('2001-06-01', '%c %b %M', 'rm_CH'); > +select date_format('2001-08-01', '%c %b %M', 'rm_CH'); > +select date_format('2001-10-01', '%c %b %M', 'rm_CH'); > +select date_format('2001-12-01', '%c %b %M', 'rm_CH'); > +select date_format('2001-01-06', '%w %a %W', 'de_CH'); > +select date_format('2001-09-01', '%c %b %M', 'de_CH'); > diff --git a/sql/item_create.cc b/sql/item_create.cc > index e0bfe7a4402..88c1ac1f64f 100644 > --- a/sql/item_create.cc > +++ b/sql/item_create.cc > @@ -770,19 +770,6 @@ class Create_func_crosses : public Create_func_arg2 > #endif > > > -class Create_func_date_format : public Create_func_arg2 > -{ > -public: > - virtual Item *create_2_arg(THD *thd, Item *arg1, Item *arg2); > - > - static Create_func_date_format s_singleton; > - > -protected: > - Create_func_date_format() {} > - virtual ~Create_func_date_format() {} > -}; > - > - > class Create_func_datediff : public Create_func_arg2 > { > public: > @@ -4023,15 +4010,6 @@ Create_func_crosses::create_2_arg(THD *thd, Item *arg1, Item *arg2) > #endif > > > -Create_func_date_format Create_func_date_format::s_singleton; > - > -Item* > -Create_func_date_format::create_2_arg(THD *thd, Item *arg1, Item *arg2) > -{ > - return new (thd->mem_root) Item_func_date_format(thd, arg1, arg2, 0); > -} > - > - > Create_func_datediff Create_func_datediff::s_singleton; > > Item* > @@ -4530,7 +4508,7 @@ Create_func_from_unixtime::create_native(THD *thd, LEX_CSTRING *name, > Item *param_1= item_list->pop(); > Item *param_2= item_list->pop(); > Item *ut= new (thd->mem_root) Item_func_from_unixtime(thd, param_1); > - func= new (thd->mem_root) Item_func_date_format(thd, ut, param_2, 0); > + func= new (thd->mem_root) Item_func_date_format(thd, ut, param_2); > break; > } > default: > @@ -6483,7 +6461,7 @@ Create_func_time_format Create_func_time_format::s_singleton; > Item* > Create_func_time_format::create_2_arg(THD *thd, Item *arg1, Item *arg2) > { > - return new (thd->mem_root) Item_func_date_format(thd, arg1, arg2, 1); > + return new (thd->mem_root) Item_func_time_format(thd, arg1, arg2); > } > > > @@ -6844,7 +6822,6 @@ static Native_func_registry func_array[] = > { { C_STRING_WITH_LEN("CRC32") }, BUILDER(Create_func_crc32)}, > { { C_STRING_WITH_LEN("CROSSES") }, GEOM_BUILDER(Create_func_crosses)}, > { { C_STRING_WITH_LEN("DATEDIFF") }, BUILDER(Create_func_datediff)}, > - { { C_STRING_WITH_LEN("DATE_FORMAT") }, BUILDER(Create_func_date_format)}, > { { C_STRING_WITH_LEN("DAYNAME") }, BUILDER(Create_func_dayname)}, > { { C_STRING_WITH_LEN("DAYOFMONTH") }, BUILDER(Create_func_dayofmonth)}, > { { C_STRING_WITH_LEN("DAYOFWEEK") }, BUILDER(Create_func_dayofweek)}, > diff --git a/sql/item_strfunc.cc b/sql/item_strfunc.cc > index 83432f95612..3e12a545c97 100644 > --- a/sql/item_strfunc.cc > +++ b/sql/item_strfunc.cc > @@ -2668,11 +2668,11 @@ String *Item_func_soundex::val_str(String *str) > const int FORMAT_MAX_DECIMALS= 30; > > > -MY_LOCALE *Item_func_format::get_locale(Item *item) > +const MY_LOCALE *get_locale(Item *item) > { > - DBUG_ASSERT(arg_count == 3); > - String tmp, *locale_name= args[2]->val_str_ascii(&tmp); > - MY_LOCALE *lc; > + StringBuffer<MAX_FIELD_WIDTH> tmp; > + String *locale_name= item->val_str_ascii(&tmp); > + const MY_LOCALE *lc; > if (!locale_name || > !(lc= my_locale_by_name(locale_name->c_ptr_safe()))) > { > @@ -2712,7 +2712,7 @@ String *Item_func_format::val_str_ascii(String *str) > int dec; > /* Number of characters used to represent the decimals, including '.' */ > uint32 dec_length; > - MY_LOCALE *lc; > + const MY_LOCALE *lc; > DBUG_ASSERT(fixed == 1); > > dec= (int) args[1]->val_int(); > diff --git a/sql/item_strfunc.h b/sql/item_strfunc.h > index 685a64816be..542af13df5c 100644 > --- a/sql/item_strfunc.h > +++ b/sql/item_strfunc.h > @@ -28,6 +28,7 @@ > extern size_t username_char_length; > > class MY_LOCALE; > +const MY_LOCALE *get_locale(Item *item); > > class Item_str_func :public Item_func > { > @@ -932,14 +933,13 @@ class Item_func_make_set :public Item_str_func > > class Item_func_format :public Item_str_ascii_func > { > - MY_LOCALE *locale; > + const MY_LOCALE *locale; > public: > Item_func_format(THD *thd, Item *org, Item *dec): > Item_str_ascii_func(thd, org, dec) {} > Item_func_format(THD *thd, Item *org, Item *dec, Item *lang): > Item_str_ascii_func(thd, org, dec, lang) {} > > - MY_LOCALE *get_locale(Item *item); > String *val_str_ascii(String *); > void fix_length_and_dec(); > const char *func_name() const { return "format"; } > diff --git a/sql/item_timefunc.cc b/sql/item_timefunc.cc > index 86bde00cced..5e660c58583 100644 > --- a/sql/item_timefunc.cc > +++ b/sql/item_timefunc.cc > @@ -455,7 +455,8 @@ static bool extract_date_time(DATE_TIME_FORMAT *format, > */ > > static bool make_date_time(DATE_TIME_FORMAT *format, MYSQL_TIME *l_time, > - timestamp_type type, MY_LOCALE *locale, String *str) > + timestamp_type type, const MY_LOCALE *locale, > + String *str) > { > char intbuff[15]; > uint hours_i; > @@ -1833,7 +1834,14 @@ bool Item_func_sec_to_time::get_date(MYSQL_TIME *ltime, ulonglong fuzzy_date) > void Item_func_date_format::fix_length_and_dec() > { > THD* thd= current_thd; > - locale= thd->variables.lc_time_names; > + if (!is_time_format) > + { > + if (arg_count < 3) > + locale= thd->variables.lc_time_names; > + else > + if (args[2]->basic_const_item()) > + locale= get_locale(args[2]); > + } > > /* > Must use this_item() in case it's a local SP variable > @@ -1875,6 +1883,8 @@ bool Item_func_date_format::eq(const Item *item, bool binary_cmp) const > if (this == item) > return 1; > item_func= (Item_func_date_format*) item; > + if (arg_count != item_func->arg_count) > + return 0; > if (!args[0]->eq(item_func->args[0], binary_cmp)) > return 0; > /* > @@ -1884,6 +1894,8 @@ bool Item_func_date_format::eq(const Item *item, bool binary_cmp) const > */ > if (!args[1]->eq(item_func->args[1], 1)) > return 0; > + if (arg_count > 2 && !args[2]->eq(item_func->args[2], 1)) > + return 0; > return 1; > } > > @@ -1967,15 +1979,18 @@ String *Item_func_date_format::val_str(String *str) > String *format; > MYSQL_TIME l_time; > uint size; > - int is_time_flag = is_time_format ? TIME_TIME_ONLY : 0; > + const MY_LOCALE *lc= 0; > DBUG_ASSERT(fixed == 1); > > - if (get_arg0_date(&l_time, is_time_flag)) > + if (get_arg0_date(&l_time, is_time_format ? TIME_TIME_ONLY : 0)) > return 0; > > if (!(format = args[1]->val_str(str)) || !format->length()) > goto null_date; > > + if (!is_time_format && !(lc= locale) && !(lc= get_locale(args[2]))) > + goto null_date; // invalid locale > + > if (fixed_length) > size=max_length; > else > @@ -1998,7 +2013,7 @@ String *Item_func_date_format::val_str(String *str) > if (!make_date_time(&date_time_format, &l_time, > is_time_format ? MYSQL_TIMESTAMP_TIME : > MYSQL_TIMESTAMP_DATE, > - locale, str)) > + lc, str)) > return str; > > null_date: > diff --git a/sql/item_timefunc.h b/sql/item_timefunc.h > index 7d2fe46b644..20ecb4774b3 100644 > --- a/sql/item_timefunc.h > +++ b/sql/item_timefunc.h > @@ -797,22 +797,24 @@ class Item_func_from_days :public Item_datefunc > > class Item_func_date_format :public Item_str_func > { > - MY_LOCALE *locale; > + const MY_LOCALE *locale; > int fixed_length; > - const bool is_time_format; > String value; > +protected: > + bool is_time_format; > public: > - Item_func_date_format(THD *thd, Item *a, Item *b, bool is_time_format_arg): > - Item_str_func(thd, a, b), is_time_format(is_time_format_arg) {} > + Item_func_date_format(THD *thd, Item *a, Item *b): > + Item_str_func(thd, a, b), locale(0), is_time_format(false) {} > + Item_func_date_format(THD *thd, Item *a, Item *b, Item *c): > + Item_str_func(thd, a, b, c), locale(0), is_time_format(false) {} > String *val_str(String *str); > - const char *func_name() const > - { return is_time_format ? "time_format" : "date_format"; } > + const char *func_name() const { return "date_format"; } > void fix_length_and_dec(); > uint format_length(const String *format); > bool eq(const Item *item, bool binary_cmp) const; > bool check_vcol_func_processor(void *arg) > { > - if (is_time_format) > + if (arg_count > 2) > return false; > return mark_unsupported_function(func_name(), "()", arg, VCOL_SESSION_FUNC); > } > @@ -820,6 +822,17 @@ class Item_func_date_format :public Item_str_func > { return get_item_copy<Item_func_date_format>(thd, mem_root, this); } > }; > > +class Item_func_time_format: public Item_func_date_format > +{ > +public: > + Item_func_time_format(THD *thd, Item *a, Item *b): > + Item_func_date_format(thd, a, b) { is_time_format= true; } > + const char *func_name() const { return "time_format"; } > + bool check_vcol_func_processor(void *arg) { return false; } > + Item *get_copy(THD *thd, MEM_ROOT *mem_root) > + { return get_item_copy<Item_func_time_format>(thd, mem_root, this); } > +}; > + > > class Item_func_from_unixtime :public Item_datetimefunc > { > diff --git a/sql/lex.h b/sql/lex.h > index f2dac95fce5..67c3bc8620d 100644 > --- a/sql/lex.h > +++ b/sql/lex.h > @@ -726,6 +726,7 @@ static SYMBOL sql_functions[] = { > { "CURTIME", SYM(CURTIME)}, > { "DATE_ADD", SYM(DATE_ADD_INTERVAL)}, > { "DATE_SUB", SYM(DATE_SUB_INTERVAL)}, > + { "DATE_FORMAT", SYM(DATE_FORMAT_SYM)}, > { "DECODE", SYM(DECODE_SYM)}, > { "DENSE_RANK", SYM(DENSE_RANK_SYM)}, > { "EXTRACT", SYM(EXTRACT_SYM)}, > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index 7e89ad91a17..17d6f016b2c 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -875,7 +875,7 @@ bool my_yyoverflow(short **a, YYSTYPE **b, ulong *yystacksize); > Comments for TOKENS. > For each token, please include in the same line a comment that contains > the following tags: > - SQL-2011-N : Non Reserved keywird as per SQL-2011 > + SQL-2011-N : Non Reserved keyword as per SQL-2011 > SQL-2003-R : Reserved keyword as per SQL-2003 > SQL-2003-N : Non Reserved keyword as per SQL-2003 > SQL-1999-R : Reserved keyword as per SQL-1999 > @@ -1010,6 +1010,7 @@ bool my_yyoverflow(short **a, YYSTYPE **b, ulong *yystacksize); > %token DATA_SYM /* SQL-2003-N */ > %token DATETIME > %token DATE_ADD_INTERVAL /* MYSQL-FUNC */ > +%token DATE_FORMAT_SYM /* MYSQL-FUNC */ > %token DATE_SUB_INTERVAL /* MYSQL-FUNC */ > %token DATE_SYM /* SQL-2003-R */ > %token DAY_HOUR_SYM > @@ -9806,6 +9807,18 @@ function_call_nonkeyword: > if ($$ == NULL) > MYSQL_YYABORT; > } > + | DATE_FORMAT_SYM '(' expr ',' expr ')' > + { > + $$= new (thd->mem_root) Item_func_date_format(thd, $3, $5); > + if ($$ == NULL) > + MYSQL_YYABORT; > + } > + | DATE_FORMAT_SYM '(' expr ',' expr ',' expr ')' > + { > + $$= new (thd->mem_root) Item_func_date_format(thd, $3, $5, $7); > + if ($$ == NULL) > + MYSQL_YYABORT; > + } > | DECODE_SYM '(' expr ',' expr ')' > { > $$= new (thd->mem_root) Item_func_decode(thd, $3, $5); > @@ -14719,6 +14732,7 @@ keyword_sp_not_data_type: > | CYCLE_SYM {} > | DATA_SYM {} > | DATAFILE_SYM {} > + | DATE_FORMAT_SYM {} > | DAY_SYM {} > | DECODE_SYM {} > | DEFINER_SYM {} > diff --git a/sql/sql_yacc_ora.yy b/sql/sql_yacc_ora.yy > index a475914d369..54ddd3ae922 100644 > --- a/sql/sql_yacc_ora.yy > +++ b/sql/sql_yacc_ora.yy > @@ -284,7 +284,7 @@ bool my_yyoverflow(short **a, YYSTYPE **b, ulong *yystacksize); > Comments for TOKENS. > For each token, please include in the same line a comment that contains > the following tags: > - SQL-2011-N : Non Reserved keywird as per SQL-2011 > + SQL-2011-N : Non Reserved keyword as per SQL-2011 > SQL-2003-R : Reserved keyword as per SQL-2003 > SQL-2003-N : Non Reserved keyword as per SQL-2003 > SQL-1999-R : Reserved keyword as per SQL-1999 > @@ -419,6 +419,7 @@ bool my_yyoverflow(short **a, YYSTYPE **b, ulong *yystacksize); > %token DATA_SYM /* SQL-2003-N */ > %token DATETIME > %token DATE_ADD_INTERVAL /* MYSQL-FUNC */ > +%token DATE_FORMAT_SYM /* MYSQL-FUNC */ > %token DATE_SUB_INTERVAL /* MYSQL-FUNC */ > %token DATE_SYM /* SQL-2003-R */ > %token DAY_HOUR_SYM > @@ -9462,6 +9463,18 @@ column_default_non_parenthesized_expr: > if ($$ == NULL) > MYSQL_YYABORT; > } > + | DATE_FORMAT_SYM '(' expr ',' expr ')' > + { > + $$= new (thd->mem_root) Item_func_date_format(thd, $3, $5); > + if ($$ == NULL) > + MYSQL_YYABORT; > + } > + | DATE_FORMAT_SYM '(' expr ',' expr ',' expr ')' > + { > + $$= new (thd->mem_root) Item_func_date_format(thd, $3, $5, $7); > + if ($$ == NULL) > + MYSQL_YYABORT; > + } > | DECODE_SYM '(' expr ',' decode_when_list ')' > { > $5->push_front($3, thd->mem_root); > @@ -14893,6 +14906,7 @@ keyword_sp_not_data_type: > | CYCLE_SYM {} > | DATA_SYM {} > | DATAFILE_SYM {} > + | DATE_FORMAT_SYM {} > | DAY_SYM {} > | DECODE_SYM {} > | DEFINER_SYM {} > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits >

1 0

Re: [Maria-developers] [Commits] c132349bc79: MDEV-11846: ERROR 1114 (HY000) table full when performing GROUP BY
by Sergey Petrunia 21 Sep '17

21 Sep '17

Hi Varun, Ok to push. On Fri, Sep 15, 2017 at 01:28:53PM +0530, Varun wrote: > revision-id: c132349bc79d987947c8659e71601914cfec71e8 (mariadb-10.1.23-156-gc132349bc79) > parent(s): fa2701c6f7b028782cf231565f578b2fc0f10d51 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2017-09-15 13:27:11 +0530 > message: > > MDEV-11846: ERROR 1114 (HY000) table full when performing GROUP BY > > The problem is there is an overflow for the key_file_length. > Added the maximum limit for the key_file_length > > --- > storage/maria/ma_create.c | 41 +++++++++++++++++++++++++++++++++-------- > 1 file changed, 33 insertions(+), 8 deletions(-) > > diff --git a/storage/maria/ma_create.c b/storage/maria/ma_create.c > index 0ddd8b226e2..79e60a76abc 100644 > --- a/storage/maria/ma_create.c > +++ b/storage/maria/ma_create.c > @@ -660,11 +660,24 @@ int maria_create(const char *name, enum data_file_type datafile_type, > > if (length > max_key_length) > max_key_length= length; > - tot_length+= ((max_rows/(ulong) (((uint) maria_block_size - > + > + if (tot_length == ULLONG_MAX) > + continue; > + > + ulonglong tot_length_part= (max_rows/(ulong) (((uint) maria_block_size - > MAX_KEYPAGE_HEADER_SIZE - > KEYPAGE_CHECKSUM_SIZE)/ > - (length*2))) * > - maria_block_size); > + (length*2))); > + if (tot_length_part >= (ULLONG_MAX / maria_block_size + > + ULLONG_MAX % maria_block_size)) > + tot_length= ULLONG_MAX; > + else > + { > + if (tot_length > ULLONG_MAX - tot_length_part * maria_block_size) > + tot_length= ULLONG_MAX; > + else > + tot_length+= tot_length_part * maria_block_size; > + } > } > > unique_key_parts=0; > @@ -673,11 +686,24 @@ int maria_create(const char *name, enum data_file_type datafile_type, > uniquedef->key=keys+i; > unique_key_parts+=uniquedef->keysegs; > share.state.key_root[keys+i]= HA_OFFSET_ERROR; > - tot_length+= (max_rows/(ulong) (((uint) maria_block_size - > + > + if (tot_length == ULLONG_MAX) > + continue; > + ulonglong tot_length_part= (max_rows/(ulong) (((uint) maria_block_size - > MAX_KEYPAGE_HEADER_SIZE - > KEYPAGE_CHECKSUM_SIZE) / > - ((MARIA_UNIQUE_HASH_LENGTH + pointer)*2)))* > - (ulong) maria_block_size; > + ((MARIA_UNIQUE_HASH_LENGTH + pointer)*2))); > + > + if (tot_length_part >= (ULLONG_MAX / maria_block_size + > + ULLONG_MAX % maria_block_size)) > + tot_length= ULLONG_MAX; > + else > + { > + if (tot_length > ULLONG_MAX - tot_length_part * maria_block_size) > + tot_length= ULLONG_MAX; > + else > + tot_length+= tot_length_part * maria_block_size; > + } > } > keys+=uniques; /* Each unique has 1 key */ > key_segs+=uniques; /* Each unique has 1 key seg */ > @@ -746,8 +772,7 @@ int maria_create(const char *name, enum data_file_type datafile_type, > Get estimate for index file length (this may be wrong for FT keys) > This is used for pointers to other key pages. > */ > - tmp= (tot_length + maria_block_size * keys * > - MARIA_INDEX_BLOCK_MARGIN) / maria_block_size; > + tmp= (tot_length / maria_block_size + keys * MARIA_INDEX_BLOCK_MARGIN); > > /* > use maximum of key_file_length we calculated and key_file_length value we > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] [Commits] 0319ee58fbb: MDEV-13290: Assertion Assertion `!is_set() || (m_status == DA_OK_BULK && is_bulk_op())' or `! is_set()' failed
by Alexander Barkov 19 Sep '17

19 Sep '17

Hi Sanja, The patch looks OK for me. Please find one suggestion below: On 09/19/2017 05:56 PM, Oleksandr Byelkin wrote: > revision-id: 0319ee58fbb6372ccc8533613e0c55666abc948a (mariadb-10.0.32-32-g0319ee58fbb) > parent(s): 389f7cdf3ccd11a6b4fb9b6346436790254e615c > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2017-09-19 15:55:59 +0200 > message: > > MDEV-13290: Assertion Assertion `!is_set() || (m_status == DA_OK_BULK && is_bulk_op())' or `! is_set()' failed > > Check error status which can be set by conversion procedures. > > --- > mysql-test/r/insert.result | 22 ++++++++++++++++++++++ > mysql-test/t/insert.test | 26 ++++++++++++++++++++++++++ > sql/table.cc | 3 +++ > 3 files changed, 51 insertions(+) > > diff --git a/mysql-test/r/insert.result b/mysql-test/r/insert.result > index 82f3977e231..0efee36e22d 100644 > --- a/mysql-test/r/insert.result > +++ b/mysql-test/r/insert.result > @@ -717,3 +717,25 @@ insert ignore into t1 values (1,12); > Warnings: > Warning 1062 Duplicate entry '1' for key 'f1' > DROP TABLE t1; > +# > +# MDEV-13290 Assertion Assertion `!is_set() || (m_status == DA_OK_BULK > +# && is_bulk_op())' or `! is_set()' failed > +# > +SET @save_mode= @@sql_mode; > +SET sql_mode= 'STRICT_ALL_TABLES'; > +CREATE TABLE t1 (f1 INT DEFAULT 0, f2 INT); > +CREATE ALGORITHM = MERGE VIEW v1 AS SELECT f1, f2 FROM t1 WHERE f1 = 'x' WITH CHECK OPTION; > +REPLACE INTO v1 SET f2 = 1; > +ERROR 22007: Truncated incorrect DOUBLE value: 'x' > +drop view v1; > +CREATE ALGORITHM = MERGE VIEW v1 AS SELECT f1, f2 FROM t1 WHERE f1 = cast('' as decimal) WITH CHECK OPTION; > +REPLACE INTO v1 SET f2 = 1; > +ERROR 22007: Truncated incorrect DECIMAL value: '' > +drop view v1; > +SELECT 0,0 INTO OUTFILE 't1.txt'; > +CREATE ALGORITHM = MERGE VIEW v1 AS SELECT f1, f2 FROM t1 WHERE f1 = 'x' WITH CHECK OPTION; > +LOAD DATA INFILE 't1.txt' INTO TABLE v1; > +ERROR 22007: Truncated incorrect DOUBLE value: 'x' > +drop view v1; > +drop table t1; > +SET @@sql_mode= @save_mode; > diff --git a/mysql-test/t/insert.test b/mysql-test/t/insert.test > index ff8396fd7fd..d84c0cee90d 100644 > --- a/mysql-test/t/insert.test > +++ b/mysql-test/t/insert.test > @@ -573,3 +573,29 @@ insert ignore into t1 values (1,12) on duplicate key update f2=13; > set @@old_mode=""; > insert ignore into t1 values (1,12); > DROP TABLE t1; > + > +--echo # > +--echo # MDEV-13290 Assertion Assertion `!is_set() || (m_status == DA_OK_BULK > +--echo # && is_bulk_op())' or `! is_set()' failed > +--echo # > + > +SET @save_mode= @@sql_mode; > +SET sql_mode= 'STRICT_ALL_TABLES'; > +CREATE TABLE t1 (f1 INT DEFAULT 0, f2 INT); > +CREATE ALGORITHM = MERGE VIEW v1 AS SELECT f1, f2 FROM t1 WHERE f1 = 'x' WITH CHECK OPTION; > +--error ER_TRUNCATED_WRONG_VALUE > +REPLACE INTO v1 SET f2 = 1; Can you please add "SELECT * FROM t1" here and some other places (see below), to make sure that REPLACE did not actually insert new records. Please see my comments in MDEV. > +drop view v1; > +CREATE ALGORITHM = MERGE VIEW v1 AS SELECT f1, f2 FROM t1 WHERE f1 = cast('' as decimal) WITH CHECK OPTION; > +--error ER_TRUNCATED_WRONG_VALUE > +REPLACE INTO v1 SET f2 = 1; Please add "SELECT * FROM t1". > +drop view v1; > +SELECT 0,0 INTO OUTFILE 't1.txt'; > +CREATE ALGORITHM = MERGE VIEW v1 AS SELECT f1, f2 FROM t1 WHERE f1 = 'x' WITH CHECK OPTION; > +--error ER_TRUNCATED_WRONG_VALUE > +LOAD DATA INFILE 't1.txt' INTO TABLE v1; Please add "SELECT * FROM t1". > +let $MYSQLD_DATADIR= `select @@datadir`; > +remove_file $MYSQLD_DATADIR/test/t1.txt; > +drop view v1; > +drop table t1; > +SET @@sql_mode= @save_mode; > diff --git a/sql/table.cc b/sql/table.cc > index 975d9d53882..ff9c4217b7d 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -4642,6 +4642,9 @@ int TABLE_LIST::view_check_option(THD *thd, bool ignore_failure) > main_view->view_name.str); > return(VIEW_CHECK_ERROR); > } > + /* We check thd->error() because it can be set by conversion problem. */ > + if (thd->is_error()) > + return(VIEW_CHECK_ERROR); > return(VIEW_CHECK_OK); > } > > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits >

1 0

[Maria-developers] Patch for MDEV-10574
by jerome brauge 19 Sep '17

19 Sep '17

Alexander, Here is the patch I was talking about. It not address all aspects of MDEV-10574, especially, empty strings in table column are not view as null. But as in oracle mode, no empty string in columns should be created, an fully oracle application will never fall in this case. Regards, Jérôme. > -----Message d'origine----- > De : jerome brauge > Envoyé : mardi 19 septembre 2017 14:13 > À : 'Alexander Barkov' > Objet : RE: Oracle strings functions compatibility : substr > > Hello Alexander, > Yes I'm aware. > This point will be treated by my next patch which will also affect the others > string functions (a part of MDEV-10574). > I will refresh this new patch with the current development branch, and I > submit to you. > Regards. > Jérôme. > > > -----Message d'origine----- > > De : Alexander Barkov [mailto:bar@mariadb.org] Envoyé : mardi 19 > > septembre 2017 13:16 À : jerome brauge Objet : Re: Oracle strings > > functions compatibility : substr > > > > Hi Jerome, > > > > I noticed one more difference in SUBSTR: > > > > > > SELECT SUBSTR('aaa',1,-1) FROM DUAL; > > > > MariaDB returns an empty string. > > Oracle returns NULL. > > > > Here's Oracle's documentation: > > > > https://docs.oracle.com/cd/B19306_01/server.102/b14200/functions162.ht > > m > > > > > > I think we should do this in the same patch, to avoid behavior change > > in the future. > > > > Would you like to try doing this additional change? > > Or would you like me to make an incremental patch on top of yours? > > > > Thanks! > > > > On 09/19/2017 10:33 AM, jerome brauge wrote: > > > Hello Alexander, > > > It's done. > > > > > > Do you have news from Sergei for MDEV-12874, MDEV-13417 and MDEV- > > 13418 ? > > > > I asked Sergei, awaiting for his answer. I'll let you know. > > > > > > > > Thank you very much. > > > > > > > > > > > >> -----Message d'origine----- > > >> De : Alexander Barkov [mailto:bar@mariadb.org] Envoyé : lundi 18 > > >> septembre 2017 18:12 À : jerome brauge Objet : Re: Oracle strings > > >> functions compatibility : substr > > >> > > >> Hello Jerome, > > >> > > >> > > >> On 09/18/2017 05:14 PM, jerome brauge wrote: > > >>> Hello Alexander, > > >>> I don't understand how I've missed such an obvious solution ! > > >>> Here is the new patch. > > >> > > >> Thanks. Now it looks simpler. > > >> > > >> Can you please do one small thing: > > >> move get_position() from "public:" to "protected:". > > >> > > >> Ok to make a pull request after this change. > > >> > > >> Thank you very much! > > >> > > >> > > >>> > > >>> Regards. > > >>> Jérôme. > > >>> > > >>> > > >>>> -----Message d'origine----- > > >>>> De : Alexander Barkov [mailto:bar@mariadb.org] Envoyé : lundi 18 > > >>>> septembre 2017 11:30 À : jerome brauge Objet : Re: Oracle strings > > >>>> functions compatibility : substr > > >>>> > > >>>> Hello Jerome, > > >>>> > > >>>> > > >>>> On 08/17/2017 04:43 PM, jerome brauge wrote: > > >>>>> Hello Alexander, > > >>>>> Here is a patch for function SUBSTR in sql_mode=oracle (If > > >>>>> position is 0, > > >>>> then it is treated as 1). > > >>>> > > >>>> I'm thinking of how to get a smaller patch. > > >>>> > > >>>> Wouldn't it be possible to introduce: > > >>>> > > >>>> virtual longlong get_position(); > > >>>> > > >>>> > > >>>> > > >>>> longlong Item_func_substr::get_position() { > > >>>> return args[1]->val_int(); > > >>>> } > > >>>> > > >>>> longlong Item_func_substr_oracle::get_position() > > >>>> { > > >>>> longlong pos= args[1]->val_int(); > > >>>> return pos == 0 ? 1 : pos; > > >>>> } > > >>>> > > >>>> > > >>>> and just replace all calls for args[1]->val_int() to get_position() ? > > >>>> > > >>>> Thanks! > > >>>> > > >>>> > > >>>>> > > >>>>> Regards, > > >>>>> Jérôme. > > >>>>>

1 0

Re: [Maria-developers] NUMA Support in MariaDB
by serg＠mariadb.org 19 Sep '17

19 Sep '17

Hi, Sumit! On Sep 16, Sumit Lakra wrote: > The project 'NUMA Support in MariaDB' is still incomplete and like I > mentioned in my mails to my mentors, I am still willing to work on it > and see it through to the end. Great, thanks! > Most of the minor tasks along with some important tasks are already > complete. The important part that remains is to work on the task-queue > interface between SQL and InnoDB. I was unable to implement this for > the NUMA case and I won't be able to complete it without some help. > So, do you think you can connect me with someone from the > MariaDB/InnoDB/MySQL team who is an expert of this interface? I will > need someone who has a very good understanding of the present > structure to help me come up with a way to implement this for the NUMA > case. If you do find someone who can be of help, please see to it that > since GSoC is over (and otherwise as well), it shouldn't be a > Student-Mentor kind of thing, but more like two or more developers > working together trying to come up with a solution to this problem. Why wouldn't you ask the question on the maria-developers@ mailing list? I'll try to make sure that it'll be answered. > However, if you are unable to connect me with someone who has > expertise in the handler interface, I am afraid I won't be able to > complete this project on my own. In this case, would you like me to > create a pull request of the work done so far ( > https://github.com/theGodlessLakra/server/tree/NUMA_Support) Although > NUMA Support is incomplete, merging this code to the MariaDB code base > will allow other developers to continue working on this. Is it sufficiently complete to be useful as is? Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 1

Re: [Maria-developers] 6a72d15: MDEV-10767 /tmp/wsrep_recovery.${RANDOM} file created in unallowed SELinux context
by Sergei Golubchik 13 Sep '17

13 Sep '17

Hi, Sachin! On Aug 31, sachin wrote: > revision-id: 6a72d154d6ec231eeff861496e4d07101ac41553 (mariadb-10.1.26-16-g6a72d15) > parent(s): dda40b930498b70bb5546f857b27744039a5649d > author: sachin > committer: sachin > timestamp: 2017-08-31 12:17:10 +0530 > message: > > MDEV-10767 /tmp/wsrep_recovery.${RANDOM} file created in unallowed SELinux context > > Solution:- Allowed mysqld_t to open file with context initrc_tmp_t. Why does the server need it? I've only found scripts/galera_recovery.sh, that does: log_file=$(mktemp /tmp/wsrep_recovery.XXXXXX) /usr/sbin/mysqld $cmdline_args --user=$user --wsrep_recover --log-error="$log_file" For that use case, I wouldn't bother with a new selinux policy, it'd be safer to do log_file=$(mktemp /tmp/wsrep_recovery.XXXXXX) /usr/sbin/mysqld $cmdline_args --user=$user --wsrep_recover --disable-log-error 2>"$log_file" Note, that the similar piece of code is used in mysqld_safe.sh, only there it uses DATADIR/wsrep_recovery.XXXXXX, apparently somebody's attempt to work around selinux. Better to make it identical to galera_recovery.sh, I'd say. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 1

Re: [Maria-developers] [External] Obsolete GTID domain delete on master (MDEV-12012, MDEV-11969)
by andrei.elkin＠pp.inet.fi 13 Sep '17

13 Sep '17

Hello, Simon, Kristian. (The mail was meant to be sent out yesterday, but it got stuck in my outgoing box). > Simon Mudd <simon.mudd(a)booking.com> writes: > >> ids. Obviously once all appropriate bin logs have been purged >> (naturally by other means) then no special processing will be needed. > > Right. Hence my original idea (which was unfortunately never implemented so > far). If at some point a domain has been unused for so long that all GTIDs > in that domain are gone, it is relatively safe to pretend that the domain > never existed. > > I would like to understand if you can think of significant use cases where > the DBA needs to have active binlog files in the master containing some > domain, while simultaneously pretending that this domain never existed. > > Or if it is more of a general concern, and the inconvenience for users to > have to save old binlogs somewhere else than the master's data directory and > binlog index (SHOW BINARY LOGS). > >> removing old binary logs should _not_ IMO be done as a way of >> forgetting the past obsolete domains. >> BINLOGS are important so throwing them away is an issue. I think >> that somehow the code needs >> to be aware of the cut-off point and when the “stale domain ids” are removed.) Simon, initially I thought of masking out the problematic domain so that the most recent binlog file would not have it in its Gtid_list header. Yet I've given up that idea to have agreed the strict setup on master weighs much more. > > I understand the desire to not delete binlog files. And the mdev-12012 use case might even not require to conduct this purging/flush-delete-domain procedure if IGNORE_DOMAIN_IDS (for the zero id domain in question) would do. There seems to be MDEV-9108 in the way though, but conceptually the DBA may have a way to stay with the binlog files even having a problematic domain. > > The problem is: If you want to have GTIDs with some domain in your active > binlog files, _and_ you also want to pretend that this domain never existed, > what does it mean? What is the semantics? It creates a lot of complexities > for defining the semantics, for documenting it, for the users to understand > it, and for the code to implement it correctly. > > So basically, I do not understand what is the intended meaning of FLUSH > BINARY LOGS DELETE DOMAIN d _and_ at the same time keeping GTIDs with domain > d around in active binlog files? In what respects is the domain deleted, and > in what respects not? > > For the master, the binlog files are mainly used to stream to connecting > slaves. Deleting a domain means replacing the conceptual binlog history with > one in which that domain never existed. So that domain will be ignored in a > connecting slaves position, assuming it is served by another multi-source > master. If a new GTID in that domain appears later, it will be considered > the very first GTID ever in that domain. > > So consider what happens if there is anyway GTIDs in that domain deeper in > the binlog: > > 1. An already connected slave may be happily replicating those GTIDs. If > that slave reconnects (temporary network error for example), it will instead > fail with unknown GTID, or perhaps just start silently ignoring all further > GTIDs in that domain. This kind of unpredictable behaviour seems bad. > > 2. Suppose a slave connects with a position without the deleted domain. The > master starts reading the binlog from some point. What happens if a GTID is > encountered that contains the deleted domain? The slave will start > replicating that domain from some arbitrary point that depends on where it > happened to be in other domains at the last disconnect. This also seems > undesirable. > > There may be other scenarios that I did not think about. > >> DBAs do not like to remove bin logs “early" as unless you keep a copy >> somewhere you may lose valuable information, >> for recovery, for backups etc. Not everyone will be making automatic >> copies (as MySQL does not provide an automatic way to do this) > > Understood. Maybe what is needed is a PURGE BINARY LOGS that removes the > entries from the binlog index (SHOW BINARY LOGS), but leaves the files in > the file system for the convenience of the sysadmin? (Well, you can just > hand-edit binlog.index, but that requires master restart I think). Like I said above, a filtering solution could be helpful. > >> The other comment I see mentioned here was “make sure all slaves are >> up to date”. That’s going to be hard. The master can only be >> aware of “connected slaves” and if you have intermediate masters, or a > > Indeed, the master cannot ensure this. The idea is that the DBA, who decides > to delete a domain, must understand that this should not be done if any > slave still needs GTIDs from that domain. This is similar to configuring > normal binlog purge, where the DBA needs to ensure that binlogs are kept > long enough for the needs of the slowest slave. > >> FWIW expiring old domains is good to do. There’s a similar FR for > >> completely different the problem space is the same. Coming up with a >> solution which is simple to use and understand and also >> avoids where that’s possible making mistakes which may break >> replication is good. So thanks for looking at this. > > Indeed. And the input from people like you with strong operational > experience is very valuable to end up with a good solution, hence my request > for additional input. > > - Kristian. I hope clarifications given by Kristian back up the new feature idea. On the other hand it would be good to have a way for DBA to cope with the likes of mdev-12012 (there are few reports on this matter) without purging. I rely on a solution to MDEV-9108 in that regard. Cheers, Andrei

1 0

[Maria-developers] A problem with implementing Group Commit with Binlog with MyRocks
by Sergey Petrunia 12 Sep '17

12 Sep '17

Hello, This is about https://jira.mariadb.org/browse/MDEV-11934. I've encountered an insteresting issue here, so I thought I would consult on both MyRocks and MariaDB lists. == Some background == "group commit with binlog" feature needs to accomplish two goals: 1. Keep the binlog and the storage engine in sync. This is done by employing XA between the binlog and the storage engine. It works by making these calls: /* Make the transaction's changes to be ready to be committed (no conflicts with other transactions, etc) but do not commit them yet. The effects of the prepare operation must be synced to disk, as the storage engine needs to be able to recover (i.e. commit) the prepared transaction after a crash */ storage_engine->prepare(sync=true); /* After this call, the transaction is considered committed. In case of a crash, the recovery process will use the contents of the binlog to determine which of the prepared transactions are to be committed and which are to be rolled back. */ binlog->write(sync=true); /* Commit the transaction in the storage engine. This makes its changes visible to other transactions (and also releases its locks and so forth) Note that most of the time(*) we don't need to sync there. In case of a crash we will be able to recover using the binlog. */ storage_engine->commit(); 2. The second goal is to make operation performant. We need two coordinated disk flushes per transaction, the idea is to do "Group Commit" where multiple transactions share disk flushes. So, we need to do group commit and keep the storage engine and the binlog in sync while doing that. == Group Commit with Binlog in MySQL == MySQL (and fb/mysql-5.6 in particular) does in the following phases: Phase #1: Call storage_engine->prepare() for all transactions in the group. The call itself is not persistent. Phase #2: Call storage->engine->flush_logs(). This makes the effect of all Prepare operations from Phase#1 persistent. Phase #3: Write and sync the binary log. Phase #4: Call storage_engine->commit(). This does not need to be persistent. MyRocks implements them. == Group Commit with Binlog in MariaDB == MariaDB does not have these phases described above: > Phase #1: > Call storage_engine->prepare() for all transactions in the group. > The call itself is not persistent. > > Phase #2: Call storage->engine->flush_logs(). > This makes the effect of all Prepare operations from Phase#1 persistent. A quote from Kristian's description at https://lists.launchpad.net/maria-developers/msg10832.html >> So the idea is to do group prepare with the same group of transactions that >> will later group commit to the binlog. In MariaDB, this concept does not >> exist. Storage engine prepares are allowed to run in parallel and in any >> order compared to binlog commit. Initially this looked like it could work for MyRocks. MyRocks has a group commit implementation, both Prepare() and Commit() operations participate in groups. However when I implemented a group commit implementation I found its performance to be close to what one would expect if there was no commit grouping, and commit() call flushed to disk https://jira.mariadb.org/browse/MDEV-11934 has the details. == The issue == (I'm 95% certain about this. It's not 100% yet but it is very likely) RocksDB's Group Write (see rocksdb/rocksdb/db/db_impl_write.cc, DBImpl::WriteImpl function) handles both Prepare() and Commit() commands and does the following: 1. Controls writing the commited data into the MemTable 2. Writes transactions to WAL 3. Syncs the WAL. All three steps are done for the whole group. This has a consequence: a Commit() operation that does not need to sync the WAL will still be delayed if another operation in the group needs the WAL to be synced. This delay has a disastrous effect, because SQL layer tries to have the same order of transactions in the storage engine and in the binlog. In order to do that, it calls rocksdb_commit_ordered() for each transaction sequentially. Delaying one transaction causes a delay of the entire SQL-level commit group. == Possible solutions == I am not sure what to do. - Make the SQL layer's Group Commit implementation invoke hton->flush_logs() explicitly, like MySQL does? - Modify RocksDB so that Transaction::Commit(sync=false) do not use Group Write? I am not sure if this is possible: Group Write is not about only performance, it's about preventing concurrent MemTable writes. AFAIU one cannot just tell a certain DBImpl::WriteImpl() call to not participate in write groups and work as if there was no other activity. - Modify RocksDB so that Transaction::Commit(sync=false) does not wait until its write group finishes WAL sync? This could be doable but is potentially complex. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

3 5

[Maria-developers] ha_innobase::info_low() n_rows hack
by Aleksey Midenkov 11 Sep '17

11 Sep '17

In ha_innobase::info_low() there is following dirty hack: /* The MySQL optimizer seems to assume in a left join that n_rows is an accurate estimate if it is zero. Of course, it is not, since we do not have any locks on the rows yet at this phase. Since SHOW TABLE STATUS seems to call this function with the HA_STATUS_TIME flag set, while the left join optimizer does not set that flag, we add one to a zero value if the flag is not set. That way SHOW TABLE STATUS will show the best estimate, while the optimizer never sees the table empty. */ if (n_rows == 0 && !(flag & HA_STATUS_TIME)) { n_rows++; } It is very old (from 5.0 or earlier) and bug-prone. Because in ha_innobase::open(): info(HA_STATUS_NO_LOCK | HA_STATUS_VARIABLE | HA_STATUS_CONST); every opened empty table will be non-empty! I don't know what is the problem with join optimizer, but having storage engine to handle it seems not the right thing to do. Moreover, relying on HA_STATUS_TIME in this is definitely wrong. We can make join optimizer to ignore "0 rows case" for all storage engines. Is it big win from "1 row case" anyway? Or we can make new flag HA_JOIN_STAT and use it in make_join_statistics().

2 3

Re: [Maria-developers] A problem with implementing Group Commit with Binlog with MyRocks
by Sergey Petrunia 11 Sep '17

11 Sep '17

Hi Kristian, On Mon, Sep 11, 2017 at 01:02:31PM +0200, Kristian Nielsen wrote: > Sergey Petrunia <sergey(a)mariadb.com> writes: > > > This is about https://jira.mariadb.org/browse/MDEV-11934. I've encountered > > an insteresting issue here, so I thought I would consult on both MyRocks and > > MariaDB lists. > > Is your current code available somewhere? > No, it wasn't. Just pushed the current code into bb-10.2-mdev11934 branch: https://github.com/MariaDB/server/tree/bb-10.2-mdev11934 All changes are in the last commit. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] 2nd 2017 Developers Unconference
by Ian Gilfillan 11 Sep '17

11 Sep '17

The dates for the Developers Unconference in Shenzhen, China, are coming up fast. This is our first official event in Asia, and will be held from 13 November - 17 November. 13 November - new contributor day 14-15 November - traditional developers unconference 16-17 November - patch days As always, it's free to attend, and open to anyone interested in MariaDB development. Details are at https://mariadb.org/2017-2-developers-unconference-and-related-events-shenz… and signup on the Meetup page: https://www.meetup.com/MariaDB-Developers/events/241873269/

1 0

[Maria-developers] GSoC NUMA Evaluation Student failed Mail 2
by Sumit Lakra 06 Sep '17

06 Sep '17

Hello, I am not aware of what information regarding this project has been exchanged between my mentor Daniel Black and others including serg, so I am sharing this here for everyone to see. My last mail contains the conversations I had with my mentors. Some quotes from the final evaluation - "Your questions in the last week have been exceptionally keen even though these seem to be the same questions from 1-2 months ago" Remind me when you (Daniel Black) answered them. "Sumit, overall we are happy with the quality of work. Unfortunately the quantity of work showed a lack of putting in hours and limited test cases fell short of the project plan and organisational expectations" What limited test cases ? I wrote all the test cases you (Daniel Black) and JAN Lindstrom asked me to write. Which test case did I miss ? I even ran a bench-marking test with all possible values and sent you the results. However, testing the actual performance difference is something that can only be done in a real NUMA Machine. Sorry, I don't have that. And have you ever even tested my code in a real NUMA Machine ? I too deserve to know if my work has made any real difference on the machine it is supposed to work on. And I have asked you to run it on a real machine more than once over the past months. You are happy with the quality of the work. Good. You are unhappy with the quantity. How do you expect me to increase the quantity when I already completed the tasks I promised in my GSoC proposal except for those that you removed from the list. What project plan are you referring to ? My GSoC proposal - https://docs.google.com/document/d/1UbOQysgOOzCM7z5FPC7gmuCpyxPmB7RGg7y9Ogq… Tasks spreadsheet - https://docs.google.com/spreadsheets/d/1nE-qFXhwwhF0hpOX3wUlczPjlPZTK_UzBu_… The tasks can be categorised as follows. [1] The tasks mentioned in my GSoC proposal which were not removed / changed later. - I completed all these. [2] The tasks mentioned in my GSoC proposal which were crossed out / (under consideration to be dropped / unnecessary) / or were implemented differently. - I either did not complete these for obvious reasons or implemented some of them differently. [3] The tasks you added (in the last month) to the spreadsheet originally under the title 'Out of Time' in the last month. - I completed some of these too. Then when these were done you moved a task (the queue per node task) from the 'Out of Time' to the third month group. We both agreed it was an important part for the completion of the project and I tried my best to implement it. I spent days trying to figure it out, asked you for help, didn't get any, tried again and yet failed because I am no expert of either InnoDB or SQl side of MariaDB. Which organisational expectations did you ask me to fulfil ? When ? I don't remember ever being told about any organisational expectations. Perhaps these were your personal expectations of having a complete NUMA Support in MariaDB by the end of GSoC. If so, then I am really sorry I wasn't able to fulfill those expectations because I got stuck at an important part of the project which also was a seemingly impossible task to do and *YOU FAILED TO HELP ME* with it even though I repeatedly asked (begged actually) you for help. You just kept reminding me that to do the task was my job. That's some real good mentoring right there. Another quote from the final evaluation "Passing to get your continued work isn't honoring the rules of the project or being fair to Google" Yeah, How about showing some good mentor ship by helping me to complete the project while we had the time to do it within Google's timelines in stead. Oh! wait a sec... you never went through the code.. until 3-4 days ago. I was determined to complete it if you had helped me. I have been mentioning it in all mails since the mail I wrote to you and Jan on : 22nd Aug - "Last but not least, if you think it can still be done and have a idea, I will be more than willing to attempt it. After all, GSoC has only got me started to contribute to open-source and this evaluation won't be the end of it." 25th Aug - "I am sorry I am unable to create an implementation. It's not like I don't want to. I just can't completely understand how it has been implemented currently. I have tried to follow the code before, and I did again the whole day today, but I can't come up with an idea at all." 25th Aug - "Like I said, I am not unwilling to complete this task. But I can't even think of where to start. Since, the two of you have greater experience with InnoDB, maybe you should give it a shot. You will definitely understand the present structure better than I do. And if you can come up with even a verbal solution that sounds like it could work, I will implement it asap." 25th Aug - "I spent all day trying to think of any way to implement a queue-per-node thing, and I confess I failed to come up with one. So, in stead of spending more time on that, I will work on migrating these commits to a new branch today. And I also urge you to give this task a try yourselves." 25th Aug - " I haven't been able to implement the queue per node structure for background threads, and I am afraid I won't be able to do it without your help. It is beyond me. I gave it a few attempts and I have failed. It's not the implementation part but the 'coming up with a way to implement it' part that I haven't been able to figure out yet. I need you to know that I am trying my best here." 25th Aug - "But anyway I kindly request you two to try it out yourselves now. You don't have to do the work. I just urge you to take some time out this weekend and go through the code. Let me know how you think it may be possible to implement it. I will do it. I have 4 days 2 hrs before the deadline to submit the evaluation, ends. If you can come up with an implementation plan within the next 2 or 2.5 days, I assure you I will code the implementation within a day of that and probably commit it before submitting the final evaluation as well. I am willing to take this risk." On Aug 28th you wrote a mail and asked me "Knowing what plans you had and how they failed would have been good to know." So, on the same day I wrote you a mail with a detailed explanation of the attempts I had made in trying to complete the task, asking you again to try it yourself and help me if you could come up with an idea. You never did. 31st Aug http://marialog.archivist.info/2017-08-31.txt Me - "did you have an attempt at the task ? come up with any ideas yet ?" dragonheart - been a bit busy with my own work so far. I know I'm going to have to look at a lot of stuff to finish the final GSoC evaluation 4th Sept http://marialog.archivist.info/2017-09-04.txt Me - "You must have seen my mails. Did you have an attempt at the queue per node thing ? Any ideas ?" dragonheart - well, implementing it was your task. but no, haven't found the hours to look though the code again. As per our conversation on IRC clearly Daniel Black hadn't even gone through the code by 4th Sept or attempted to solve the problem. How did he hope to complete the final evaluation. Even if he went through the code a day before submitting his evaluation, isn't this too late for a mentor to go through his/her student's code. Here is a list of some questions I would like MariaDB staff to ponder upon especially serg. [1] Was the task doable at all ? [A] Not unless you are an expert of both the SQL side and the InnoDB side, expert enough to re-write a major part of the interfaace between them. Jan himself acknowledged he wouldn't be able to help me with this, because he has no experience with the interface. I really appreciate that. [2] Even if the task was doable by a team of SQL and InnoDB experts, was it doable by an individual student working on MariaDB for the past four months ? [A] Nope. [3] Why didn't Daniel Black help me with the task even when I clearly mentioned I would need help with the task weeks ago ? [A] ??? [4] Did my mentor Daniel Black even try to help me with this particular task ? Ever ? [A] Nope. [5] Did Daniel Black evaluate me on the basis of the number of commits or the tasks accomplished by those commits ? [A] Based on the evaluation, he clearly has been unhappy with the decreased number of commits, not the content, as all the tasks mentioned in the original proposal have been achieved along with a few more. [6] Can Daniel Black himself complete this task, and thus the project ? [A] I bet not. No doubt the project is incomplete because it involves doing a task that is nearly impossible to do, at least for me, especially without any help. But does GSoC means that the student has to necessarily complete the entire project. I have gone through the code submitting guidelines plenty of times. It clearly states that we must state which tasks were done, and which tasks remain. Obviously not all projects are complete, some only complete the project partially, but mention everything and continue working on them. I did the same, and mentioned in my evaluation that almost all tasks were complete, in stead of all, and I have always been willing to complete this project but man, I can't do something impossible, especially when the people I am supposed to ask for help themselves fail to help their students. Thanks, Sumit

1 0

[Maria-developers] GSoC NUMA Evaluation Student failed Mail 1
by Sumit Lakra 06 Sep '17

06 Sep '17

Hello Serg, I am sorry to bug you with this mail but I am not sure what discussions you and my mentors have had before I was failed in GSoC. I completed all the tasks I had mentioned in my proposal but the project is incomplete. No doubts about that, but it is so because I got stuck with a major task (added in the last month) of the project for which I received no help from my mentors. These are the conversations I had with my mentors recently regarding the same task. Thanks, Sumit ---------- Forwarded message ---------- From: Sumit Lakra <sumitlakradev(a)gmail.com> Date: Tue, Aug 22, 2017 at 1:04 AM Subject: GSoC Final Evaluation To: Daniel Black <daniel.black(a)au1.ibm.com>, Jan Lindström < jan.lindstrom(a)mariadb.com> Hello, I went through the code that the InnoDB background threads use to pick tasks and execute them, in detail. This is how they seem to work. The reader/writer threads are passed an integer which acts as a segment. They then call fil_aio_wait(segment) which calls os_aio_handler(segment, &node, &message, &type). The control then goes to the os_aio_simulated_handler(segment, m1, m2, request) where the code gets more complicated with AIO arrays and slots. It gets harder to understand how they choose their tasks. It is definitely not a simple queue structure from which they pick their tasks. Also, which buffer pool the task is related to can only be figured out quite later.. based on the value of m2, which stores the address of a bpage. A simple queue could have been easily replaced with multiple queues, i.e. a queue per numa-node like we had once discussed on IRC. Lastly, all these procedures are common for log threads as well. Another thing, you mentioned more than once you wanted the reader threads to look for a bpage in their local nodes before looking them up in other nodes, but they use the bpage structure itself like I mentioned. Obviously neither of us had a proper understanding of how InnoDB worked in these aspects when we started the project. These threads seem to operate on bpages mostly rather than buf_pools, which makes numa mapping even harder (buf_pools to numa nodes would comparatively be easier), but is definitely more efficient than buf_pools and hence shouldn't be changed. Then again there were cases when the tasks assigned to the background threads were done by other server threads as well, especially in case of flushing, but I successfully restricted them to their nodes. If we were to create queues per node, which queues would these other threads work on. In other words the way in which InnoDB was initially written and expanded has made it significantly difficult to be adapted for explicit NUMA support. It wouldn't just require restructuring and changing the way these threads use parameters, pick tasks etc, but may also require re-ordering the order in which they are executed. For example, the background threads are created before the user threads, and trying to use the NUMA node from user THD later on would mean more system calls and probably thread migrations. When you added the task in the spreadsheet, you were right to anticipate that it could require large cleanup of the InnoDB structure, but I am beginning to think it will be way more complicated. Also, as you once mentioned that most architectures have 2 to 4 NUMA nodes. No doubt, if we could implement a support for NUMA architectuure such that InnoDB would make the best use out of it, there would be a performance difference but it would still be quite negligible in a very fast computer, and I really don't think making such big changes to InnoDB would be worth the effort and risk. Trying to bring a major change to a working and verified code (written by someone else) is more error-prone than adding some modular code and functions to support a new feature. I hope you agree. Last but not least, if you think it can still be done and have a idea, I will be more than willing to attempt it. After all, GSoC has only got me started to contribute to open-source and this evaluation won't be the end of it. And since there's no other database software out there which supports NUMA (let's ignore numa-interleave) and since I was the first to start working on this, I will be really proud to see it through to the end. Some Evaluation stuff : I see only one text field in the form for a link to the work done, and it takes a single url. Reserving this for the github link, where do I add a link to the spreadsheet or documentation ? Assuming you have already verified the work done so far, I will be creating a pull request shortly, so I can add whether it was merged or not, in the evaluation. Thanking You, Sumit ---------- Forwarded message ---------- From: Daniel Black <daniel.black(a)au1.ibm.com> Date: Thu, Aug 24, 2017 at 12:11 PM Subject: Re: GSoC Final Evaluation To: Sumit Lakra <sumitlakradev(a)gmail.com>, Jan Lindström < jan.lindstrom(a)mariadb.com> On 22/08/17 05:34, Sumit Lakra wrote: > Hello, > > I went through the code that the InnoDB background threads use to pick > tasks and execute them, in detail. This is how they seem to work. > > The reader/writer threads are passed an integer which acts as a segment. > They then call fil_aio_wait(segment) which calls os_aio_handler(segment, > &node, &message, &type). The control then goes to the > os_aio_simulated_handler(segment, m1, m2, request) where the code gets > more complicated with AIO arrays and slots. It gets harder to understand > how they choose their tasks. It is definitely not a simple queue > structure from which they pick their tasks. Also, which buffer pool the > task is related to can only be figured out quite later.. based on the > value of m2, which stores the address of a bpage. > A simple queue could > have been easily replaced with multiple queues, i.e. a queue per > numa-node like we had once discussed on IRC. Yes, you know where requests are created and where the job message ends up so you should be able to create an implementation. Even if it doesn't preserver the existing behaviour it should be enough to test it. > Lastly, all these > procedures are common for log threads as well. relevance? > Another thing, you mentioned more than once you wanted the reader SQL threads? Or is this the same thing? > threads to look for a bpage in their local nodes before looking them up > in other nodes, but they use the bpage structure itself like I > mentioned > Obviously neither of us had a proper understanding of how > InnoDB worked in these aspects when we started the project. No project knows all the details. > These > threads seem to operate on bpages mostly rather than buf_pools, which > makes numa mapping even harder You could add a node_id to the class buf_page_t? > (buf_pools to numa nodes would > comparatively be easier), but is definitely more efficient than > buf_pools and hence shouldn't be changed. > Then again there were cases when the tasks assigned to the background > threads were done by other server threads as well, especially in case of > flushing, but I successfully restricted them to their nodes. If we were > to create queues per node, which queues would these other threads work on. Yes, which is why this issue was raised with you months ago. > In other words the way in which InnoDB was initially written and > expanded has made it significantly difficult to be adapted for explicit > NUMA support. The split of buffer pool instances was in appreciation that there there are hot locks causes by a large number of threads. It just needed more thought than implementing a few thread bindings. I'm sorry you though it was that easy. It wouldn't just require restructuring and changing the > way these threads use parameters, pick tasks etc, You've changed very few parameters so far. https://github.com/MariaDB/server/compare/10.2...theGodlessLakra:numa_one but may also require > re-ordering the order in which they are executed. I can't think of one example of this. > For example, the > background threads are created before the user threads, and trying to > use the NUMA node from user THD later on would mean more system calls > and probably thread migrations. Huh, what? "may..require..(bad design)". Which is why a mapping of THD to known background threads per node was desired, to avoid syscall thread migrations. > When you added the task in the spreadsheet, you were right to anticipate > that it could require large cleanup of the InnoDB structure, but I am > beginning to think it will be way more complicated. I mentioned in on 2017-07-10 too (http://marialog.archivist.info/2017-07-10.txt) too and you said you'd give it a try. > Also, as you once > mentioned that most architectures have 2 to 4 NUMA nodes. No doubt, if > we could implement a support for NUMA architectuure such that InnoDB > would make the best use out of it, there would be a performance > difference but it would still be quite negligible in a very fast > computer, Cross node cache-coherency takes significant clock cycles compared to local access. What you'd end up with is a system that has idle time that can't be used. By finishing this task this could be measured. local/remote perf access can be measure (perhaps even with fake_numa). https://joemario.github.io/blog/2016/09/01/c2c-blog/ > and I really don't think making such big changes to InnoDB > would be worth the effort and risk. There is no gain without risk. Small changes with careful review, running full tests between them is they way large changes are made low risk. > Trying to bring a major change to a > working and verified code (written by someone else) is more error-prone > than adding some modular code and functions to support a new feature. I > hope you agree. When developing you are always developing for current and future use. As such architectural changes need to be commiserate to the functional changes. Jan and I have placed emphasis on writing test cases. These ensure that changes. > Last but not least, if you think it can still be done and have a idea, I > will be more than willing to attempt it. After all, GSoC has only got me > started to contribute to open-source and this evaluation won't be the > end of it. > And since there's no other database software out there which > supports NUMA (let's ignore numa-interleave) and since I was the first > to start working on this There is, found https://github.com/aerospike/aerospike-server/blob/master/cf/src/hardware.c the other day. Wouldn't have helped much as it has a different architecture and getting on top of one was hard enough. >, I will be really proud to see it through to > the end. This week looks like you've done one commit that is largely a revert that Jan asked for. You have slowed down considerably. > Some Evaluation stuff : > I see only one text field in the form for a link to the work done, and > it takes a single url. Reserving this for the github link, where do I > add a link to the spreadsheet or documentation ? > > Assuming you have already verified the work done so far, I will be > creating a pull request shortly, so I can add whether it was merged or > not, in the evaluation. Looks like the fake_numa stuff is incomplete. Pinning of background threads looks odd as they aren't big allocators of memory and aren't associated with SQL threads/buffer pool instance nodes. I have looked though it. As a pull request 60 commits with 26(?) of them fixing prior commits will look messy for anyone trying to look at the changes. If you are up for trying to clean it up please give it a go. Doing it in a copied repository is a good idea as you've seen before it can get a bit hard to keep track of. > Thanking You, > Sumit > ---------- Forwarded message ---------- From: Sumit Lakra <sumitlakradev(a)gmail.com> Date: Fri, Aug 25, 2017 at 3:03 AM Subject: Re: GSoC Final Evaluation To: Daniel Black <daniel.black(a)au1.ibm.com> Cc: Jan Lindström <jan.lindstrom(a)mariadb.com> On Thu, Aug 24, 2017 at 12:11 PM, Daniel Black <daniel.black(a)au1.ibm.com> wrote: > > > On 22/08/17 05:34, Sumit Lakra wrote: > > Hello, > > > > I went through the code that the InnoDB background threads use to pick > > tasks and execute them, in detail. This is how they seem to work. > > > > The reader/writer threads are passed an integer which acts as a segment. > > They then call fil_aio_wait(segment) which calls os_aio_handler(segment, > > &node, &message, &type). The control then goes to the > > os_aio_simulated_handler(segment, m1, m2, request) where the code gets > > more complicated with AIO arrays and slots. It gets harder to understand > > how they choose their tasks. It is definitely not a simple queue > > structure from which they pick their tasks. Also, which buffer pool the > > task is related to can only be figured out quite later.. based on the > > value of m2, which stores the address of a bpage. > > > A simple queue could > > have been easily replaced with multiple queues, i.e. a queue per > > numa-node like we had once discussed on IRC. > > Yes, you know where requests are created and where the job message ends > up so you should be able to create an implementation. Even if it doesn't > preserver the existing behaviour it should be enough to test it. > I am sorry I am unable to create an implementation. It's not like I don't want to. I just can't completely understand how it has been implemented currently. I have tried to follow the code before, and I did again the whole day today, but I can't come up with an idea at all. > > > Lastly, all these > > procedures are common for log threads as well. > > relevance? The reader/writer/log threads all start by executing fil_aio_wait(segment), in srv0start.cc > > > Another thing, you mentioned more than once you wanted the reader > > SQL threads? Or is this the same thing? > > > threads to look for a bpage in their local nodes before looking them up > > in other nodes, but they use the bpage structure itself like I > > mentioned > > Obviously neither of us had a proper understanding of how > > InnoDB worked in these aspects when we started the project. > > No project knows all the details. > > > These > > threads seem to operate on bpages mostly rather than buf_pools, which > > makes numa mapping even harder > > You could add a node_id to the class buf_page_t? > Okay, but how to use it ? where ? In which functions ? > > > (buf_pools to numa nodes would > > comparatively be easier), but is definitely more efficient than > > buf_pools and hence shouldn't be changed. > > > > Then again there were cases when the tasks assigned to the background > > threads were done by other server threads as well, especially in case of > > flushing, but I successfully restricted them to their nodes. If we were > > to create queues per node, which queues would these other threads work > on. > > Yes, which is why this issue was raised with you months ago. > > > In other words the way in which InnoDB was initially written and > > expanded has made it significantly difficult to be adapted for explicit > > NUMA support. > > The split of buffer pool instances was in appreciation that there there > are hot locks causes by a large number of threads. > > It just needed more thought than implementing a few thread bindings. I'm > sorry you though it was that easy. > The split of buffer pool is not problematic to NUMA implementation. Its the task queues for the background threads that's wrecking my brain. > > It wouldn't just require restructuring and changing the > > way these threads use parameters, pick tasks etc, > > > You've changed very few parameters so far. > > https://github.com/MariaDB/server/compare/10.2...theGodlessLakra:numa_one > > but may also require > > re-ordering the order in which they are executed. > > I can't think of one example of this. > > For example, the > > background threads are created before the user threads, and trying to > > use the NUMA node from user THD later on would mean more system calls > > and probably thread migrations. > > Huh, what? "may..require..(bad design)". > One way (a really bad design), would be to create background threads when a request is made by a SQL thread, and then bind it to the node depending on the request i.e. the buffer pool instance in which the associated bpage belongs, which would mean more system calls and migrations. Obviously a very bad design. > > Which is why a mapping of THD to known background threads per node was > desired, to avoid syscall thread migrations. > The SQL threads were expected to be bound to numa nodes, which I did, but I can't think of how to map them to background threads. Can you think of a way to do this ? Let me know how you think this can be done. I will implement it asap. > When you added the task in the spreadsheet, you were right to anticipate > > that it could require large cleanup of the InnoDB structure, but I am > > beginning to think it will be way more complicated. > > I mentioned in on 2017-07-10 too > (http://marialog.archivist.info/2017-07-10.txt) too and you said you'd > give it a try. > > > Also, as you once > > mentioned that most architectures have 2 to 4 NUMA nodes. No doubt, if > > we could implement a support for NUMA architectuure such that InnoDB > > would make the best use out of it, there would be a performance > > difference but it would still be quite negligible in a very fast > > computer, > > Cross node cache-coherency takes significant clock cycles compared to > local access. What you'd end up with is a system that has idle time that > can't be used. > > By finishing this task this could be measured. local/remote perf access > can be measure (perhaps even with fake_numa). > https://joemario.github.io/blog/2016/09/01/c2c-blog/ > > > and I really don't think making such big changes to InnoDB > > would be worth the effort and risk. > > There is no gain without risk. Small changes with careful review, > running full tests between them is they way large changes are made low > risk. > Agreed, but I honestly can't make out the heads and tails of the changes that will be required to replace the present task queue with a queue-per-node structure :( > > > Trying to bring a major change to a > > working and verified code (written by someone else) is more error-prone > > than adding some modular code and functions to support a new feature. I > > hope you agree. > > When developing you are always developing for current and future use. As > such architectural changes need to be commiserate to the functional > changes. Jan and I have placed emphasis on writing test cases. These > ensure that changes. > Like I said, I am not unwilling to complete this task. But I can't even think of where to start. Since, the two of you have greater experience with InnoDB, maybe you should give it a shot. You will definitely understand the present structure better than I do. And if you can come up with even a verbal solution that sounds like it could work, I will implement it asap. > > > Last but not least, if you think it can still be done and have a idea, I > > will be more than willing to attempt it. After all, GSoC has only got me > > started to contribute to open-source and this evaluation won't be the > > end of it. > > > > And since there's no other database software out there which > > supports NUMA (let's ignore numa-interleave) and since I was the first > > to start working on this > > There is, found > https://github.com/aerospike/aerospike-server/blob/master/cf > /src/hardware.c > the other day. Wouldn't have helped much as it has a different > architecture and getting on top of one was hard enough. > > >, I will be really proud to see it through to > > the end. > > This week looks like you've done one commit that is largely a revert > that Jan asked for. You have slowed down considerably. > You have a good reason to be upset at me and I am not complaining. But although it doesn't look from Github like much work due to no changes, I spent a good deal of time brainstorming on the implementation of the first task from the spreadsheet under the "should be done" title, unfortunately to no success. > > > Some Evaluation stuff : > > I see only one text field in the form for a link to the work done, and > > it takes a single url. Reserving this for the github link, where do I > > add a link to the spreadsheet or documentation ? > > > > Assuming you have already verified the work done so far, I will be > > creating a pull request shortly, so I can add whether it was merged or > > not, in the evaluation. > > Looks like the fake_numa stuff is incomplete. Pinning of background > threads looks odd as they aren't big allocators of memory and aren't > associated with SQL threads/buffer pool instance nodes. > Fixed the fake numa test which was failing in non-debug builds. Most of the memory allocation is done during the buffer pool creation, and these have been managed properly while creating buffer pools per numa node. What other allocators of memory do you wish to be managed ? and how ? (Non-pinned threads should probably be allowed to allocate memory from any node) > > I have looked though it. As a pull request 60 commits with 26(?) of them > fixing prior commits will look messy for anyone trying to look at the > changes. If you are up for trying to clean it up please give it a go. > Doing it in a copied repository is a good idea as you've seen before it > can get a bit hard to keep track of. > I spent all day trying to think of any way to implement a queue-per-node thing, and I confess I failed to come up with one. So, in stead of spending more time on that, I will work on migrating these commits to a new branch today. And I also urge you to give this task a try yourselves. Thanks, Sumit ---------- Forwarded message ---------- From: Sumit Lakra <sumitlakradev(a)gmail.com> Date: Fri, Aug 25, 2017 at 6:52 PM Subject: Re: GSoC Final Evaluation To: Daniel Black <daniel.black(a)au1.ibm.com> Cc: Jan Lindström <jan.lindstrom(a)mariadb.com> Hello, I am currently working on copying the changes from numa_one branch to a different branch in a more orderly fashion. I haven't been able to implement the queue per node structure for background threads, and I am afraid I won't be able to do it without your help. It is beyond me. I gave it a few attempts and I have failed. It's not the implementation part but the 'coming up with a way to implement it' part that I haven't been able to figure out yet. I need you to know that I am trying my best here. Back in http://marialog.archivist.info/2017-07-10.txt, you also said "warning the whole io read/write thread segements and scheduling is really messy and could do with a significant rework", and you also mentioned that you and Jan would have an attempt at it yourselves, if I am unable to do it and I move on. Well, I regret moving on to SQL threads back then without informing you that I wasn't successfull. A 'queue-per-numa-node' is easier said than done. But anyway I kindly request you two to try it out yourselves now. You don't have to do the work. I just urge you to take some time out this weekend and go through the code. Let me know how you think it may be possible to implement it. I will do it. I have 4 days 2 hrs before the deadline to submit the evaluation, ends. If you can come up with an implementation plan within the next 2 or 2.5 days, I assure you I will code the implementation within a day of that and probably commit it before submitting the final evaluation as well. I am willing to take this risk. However, if you are unable to come up with an implementation plan, we don't really have to hurry and stress ourselves. We can work together (I can't do this on my own) on this particular task later on and come up with something, assuming I am not failed in the final evaluation. Happy Weekend, Sumit ---------- Forwarded message ---------- From: Daniel Black <daniel.black(a)au1.ibm.com> Date: Mon, Aug 28, 2017 at 7:56 AM Subject: Re: GSoC Final Evaluation To: Sumit Lakra <sumitlakradev(a)gmail.com> Cc: Jan Lindström <jan.lindstrom(a)mariadb.com> On 25/08/17 23:22, Sumit Lakra wrote: > Hello, > > I am currently working on copying the changes from numa_one branch to a > different branch in a more orderly fashion. This is looking good: https://github.com/theGodlessLakra/server/tree/NUMA_Support Don't be afraid to be very descriptive in the commit message especially about the problem being solved and what functionality it implements/changes. Don't re-say what the code is doing, just cover its meaning at a high level. > I haven't been able to > implement the queue per node structure for background threads, and I am > afraid I won't be able to do it without your help. It is beyond me. > I > gave it a few attempts and I have failed. It's not the implementation > part but the 'coming up with a way to implement it' part that I haven't > been able to figure out yet. I need you to know that I am trying my best > here. Knowing what plans you had and how they failed would have been good to know. > Back in http://marialog.archivist.info/2017-07-10.txt, > you also said "warning the whole io read/write thread segements and > scheduling is really messy and could do with a significant rework", To do it properly yes. > and > you also mentioned that you and Jan would have an attempt at it > yourselves, if I am unable to do it and I move on. Well, I regret moving > on to SQL threads back then without informing you that I wasn't > successfull. I probably should of harassed you a bit too. Acknowledging failures is an important part to learning, even if you just acknowledge to yourself. How did SQL threads end up? > A 'queue-per-numa-node' is easier said than done. But > anyway I kindly request you two to try it out yourselves now. You don't > have to do the work. I just urge you to take some time out this weekend > and go through the code. Let me know how you think it may be possible to > implement it. I will do it. Having only just read this, this is a bit late notice. The aspect of understanding the interoperation between parts is also a fundamental aspect of getting a plan to implement it. > I have 4 days 2 hrs before the deadline to submit the evaluation, ends. > If you can come up with an implementation plan within the next 2 or 2.5 > days, I assure you I will code the implementation within a day of that > and probably commit it before submitting the final evaluation as well. I > am willing to take this risk. I thought your submission of a URL left it open to still committing on it. Please don't risk missing it. Making sure the existing code/comments/documentation is as clean as possible is better than tackling new work now. > However, if you are unable to come up with an implementation plan, we > don't really have to hurry and stress ourselves. We can work together (I > can't do this on my own) on this particular task later on and come up > with something, assuming I am not failed in the final evaluation. > > Happy Weekend, > Sumit ---------- Forwarded message ---------- From: Sumit Lakra <sumitlakradev(a)gmail.com> Date: Mon, Aug 28, 2017 at 9:50 PM Subject: Re: GSoC Final Evaluation To: Daniel Black <daniel.black(a)au1.ibm.com> Cc: Jan Lindström <jan.lindstrom(a)mariadb.com> Hello, The following is a detailed explanation of what I did, how and why, while trying to come up with a solution to the task queue problem. However, before you go through this I would still request you to have an attempt at it yourselves, so your perspectives won't be affected by my ideas and you may come up with something different. I had once tried this before in https://github.com/MariaDB/ server/commit/4b8436d3f7c573668a0d37b54dab2d05e8e102be. I even pointed this commit out to once on IRC, (can't access logs of this time). Anyway I didn't put much focus on this as I was onto some other task. Before trying to replace the present queue structure with a queue-per-node, I first tried to let all the threads access this queue as before but only pick tasks dealing with their local nodes. Neither this queue nor these tasks seem to be easily distinguishable in code. The threads start by running fil_aio_wait(segment). This function calls os_aio_handler(segment, &node, &message, &type). This calls os_aio_simulated_handler(segment, m1, m2, request). This calls AIO::get_array_and_local_segment(&array, global_segment), and also creates SimulatedAIOHandler handler(array, segment), followed by calling handler.check_pending(global_segment, event), handler.init(n_slots), and slot = handler.check_completed(&n_reserved). Now, in order to skip a task to be done by other thread, what a thread must do is pick the task, check whether the task is related to the same node or not. If the task is concerned with the same node execute it or else leave it in the queue. This is where the problem occurs. In order to figure out whether a task is concerned with the same node or not, it must know which buffer pool instance the task is associated to. This can only be figured out after the task picks up the bpage it has to deal with. The bpage already has this information by now. Anyway, the bpage a thread has to deal with can only be found out by buf_page_t* bpage = static_cast<buf_page_t*>(*m2). Also, the task is not picked up at any particular place, but is picked up in parts in different functions. All these function calls seem to make some or other kind of changes to the segment/slot/array they are picking up the task from. So, when a thread finally gets a task and finds it concerned with a different node, then in order to safely return the task back to the queue, it must also undo the changes made by all these procedures so far. This is the difficult part. Compare this with pc_flush_slot() where I successfully implemented this method of skipping tasks if not related to local pool. In the above commit you will find I have a few lines like 'ib::info() << thread_name << " accessing buf_page on node : " << node_of_pool << " in fil_aio_wait()";' If you compile and run this code you can see in real time which threads are accessing which pools. I have given proper names to them. Also, when I tried to skip tasks which were not of the same queue by returning from the function os_aio_simulated_handler(), which partly worked but on trying to return from the funtion fil_aio_wait(), there seems to be either an infinite loop or a deadlock somewhere, as the server stops midway (doesn't abort) while starting up. Hence I have commented out those lines on lines 5463 and 5464 in fil0fil.cc in the above commit. Another thing is the role of segments here. Are these segments related to buffer pool instances somehow ? Because all these threads including the log threads start there work by executing fil_aio_wait(segment), but segment here seems to be just an incrementing integer. At one point assuming these segments could contain tasks related with different pools/nodes, I added a loop such that both the read and write threads loop around fil_aio_wait(segment) calling it with all valid segments. Didn't work out. These were attempted by me when I pushed this commit weeks ago in the temp branch. And even though I pointed it out to you, I think I should have given much more importance to this task. Over the past few days, when I tried to give this task another try, I couldn't come up with any idea either to replace this queue with a queue-per-node thing or even to complete this skip-if-diff-node thing.

1 0

Re: [Maria-developers] [Commits] dccebdf862d: Rdb_io_watchdog doesn't support osx so disabling it
by Vicențiu Ciorbaru 30 Aug '17

30 Aug '17

Hi Varun! Please change the CMakeLists.txt to be like so: diff --git a/storage/rocksdb/CMakeLists.txt b/storage/rocksdb/CMakeLists.txt index 5db6d88..6db8211 100644 --- a/storage/rocksdb/CMakeLists.txt +++ b/storage/rocksdb/CMakeLists.txt @@ -146,7 +146,10 @@ ADD_CONVENIENCE_LIBRARY(rocksdb_aux_lib ADD_DEPENDENCIES(rocksdb_aux_lib GenError) # MARIAROCKS-TODO: how to properly depend on -lrt ? -TARGET_LINK_LIBRARIES(rocksdb_aux_lib rocksdblib ${ZLIB_LIBRARY} -lrt) +TARGET_LINK_LIBRARIES(rocksdb_aux_lib rocksdblib ${ZLIB_LIBRARY}) +if (UNIX AND NOT APPLE) + TARGET_LINK_LIBRARIES(rocksdb_aux_lib -lrt) +endif() TARGET_LINK_LIBRARIES(rocksdb rocksdb_aux_lib) @@ -182,7 +185,9 @@ ENDIF() # ADD_SUBDIRECTORY(unittest) #ENDIF() -SET(rocksdb_static_libs ${rocksdb_static_libs} "-lrt") +if (UNIX AND NOT APPLE) + SET(rocksdb_static_libs ${rocksdb_static_libs} "-lrt") +endif() ADD_LIBRARY(rocksdb_tools STATIC rocksdb/tools/ldb_tool.cc Also, change how you define the __APPLE__ #ifdef like so: - #ifndef _WIN32 + #if not defined(_WIN32) && not defined(__APPLE__) This way, you don't have to add an extra #endif. The point of #ifndef is to avoid typing, but it doesn't work well with multiple conditions. We can make use #if then. Vicențiu On Wed, 30 Aug 2017 at 15:37 Varun <varunraiko1803(a)gmail.com> wrote: > revision-id: dccebdf862d66e65528e18562c4503f40b695eca > (mariadb-10.3.0-107-gdccebdf862d) > parent(s): e1051590b61b45532284071ac78c50e7a2f24c63 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2017-08-30 18:05:52 +0530 > message: > > Rdb_io_watchdog doesn't support osx so disabling it > > --- > storage/rocksdb/CMakeLists.txt | 4 ++++ > storage/rocksdb/ha_rocksdb.cc | 14 +++++++++++++- > storage/rocksdb/rdb_io_watchdog.cc | 2 ++ > storage/rocksdb/rdb_io_watchdog.h | 6 ++++-- > 4 files changed, 23 insertions(+), 3 deletions(-) > > diff --git a/storage/rocksdb/CMakeLists.txt > b/storage/rocksdb/CMakeLists.txt > index 5db6d888bb6..435fe784d6c 100644 > --- a/storage/rocksdb/CMakeLists.txt > +++ b/storage/rocksdb/CMakeLists.txt > @@ -146,7 +146,11 @@ ADD_CONVENIENCE_LIBRARY(rocksdb_aux_lib > ADD_DEPENDENCIES(rocksdb_aux_lib GenError) > > # MARIAROCKS-TODO: how to properly depend on -lrt ? > +if (APPLE) > +TARGET_LINK_LIBRARIES(rocksdb_aux_lib rocksdblib ${ZLIB_LIBRARY}) > +else() > TARGET_LINK_LIBRARIES(rocksdb_aux_lib rocksdblib ${ZLIB_LIBRARY} -lrt) > +endif() > > TARGET_LINK_LIBRARIES(rocksdb rocksdb_aux_lib) > > diff --git a/storage/rocksdb/ha_rocksdb.cc b/storage/rocksdb/ha_rocksdb.cc > index e4d9ce02f10..4ffe6caf70b 100644 > --- a/storage/rocksdb/ha_rocksdb.cc > +++ b/storage/rocksdb/ha_rocksdb.cc > @@ -173,9 +173,11 @@ Rdb_ddl_manager ddl_manager; > const char *m_mysql_gtid; > Rdb_binlog_manager binlog_manager; > > -#ifndef _WIN32 > +#ifndef _WIN32 > +#ifndef __APPLE__ > Rdb_io_watchdog *io_watchdog = nullptr; > #endif > +#endif > /** > MyRocks background thread control > N.B. This is besides RocksDB's own background threads > @@ -555,7 +557,9 @@ static void rocksdb_set_io_write_timeout( > DBUG_ASSERT(save != nullptr); > DBUG_ASSERT(rdb != nullptr); > #ifndef _WIN32 > +#ifndef __APPLE__ > DBUG_ASSERT(io_watchdog != nullptr); > +#endif > #endif > > RDB_MUTEX_LOCK_CHECK(rdb_sysvars_mutex); > @@ -564,7 +568,9 @@ static void rocksdb_set_io_write_timeout( > > rocksdb_io_write_timeout_secs = new_val; > #ifndef _WIN32 > +#ifndef __APPLE__ > io_watchdog->reset_timeout(rocksdb_io_write_timeout_secs); > +#endif > #endif > RDB_MUTEX_UNLOCK_CHECK(rdb_sysvars_mutex); > } > @@ -3746,7 +3752,9 @@ static int rocksdb_init_func(void *const p) { > */ > if (status.IsIOError() > #ifndef _WIN32 > +#ifndef __APPLE__ > && errno == ENOENT > +#endif > #endif > ) { > sql_print_information("RocksDB: Got ENOENT when listing column > families"); > @@ -3985,8 +3993,10 @@ static int rocksdb_init_func(void *const p) { > } > > #ifndef _WIN32 > +#ifndef __APPLE__ > io_watchdog = new Rdb_io_watchdog(directories); > io_watchdog->reset_timeout(rocksdb_io_write_timeout_secs); > +#endif > #endif > > // NO_LINT_DEBUG > @@ -4077,9 +4087,11 @@ static int rocksdb_done_func(void *const p) { > commit_latency_stats = nullptr; > > #ifndef _WIN32 > +#ifndef __APPLE__ > delete io_watchdog; > io_watchdog = nullptr; > #endif > +#endif > > // Disown the cache data since we're shutting down. > // This results in memory leaks but it improved the shutdown time. > diff --git a/storage/rocksdb/rdb_io_watchdog.cc > b/storage/rocksdb/rdb_io_watchdog.cc > index a599ba58aec..a9e79283382 100644 > --- a/storage/rocksdb/rdb_io_watchdog.cc > +++ b/storage/rocksdb/rdb_io_watchdog.cc > @@ -23,6 +23,7 @@ > > /* Rdb_io_watchdog doesn't work on Windows [yet] */ > #ifndef _WIN32 > +#ifndef __APPLE__ > > namespace myrocks { > > @@ -236,4 +237,5 @@ int Rdb_io_watchdog::reset_timeout(const uint32_t > &write_timeout) { > } // namespace myrocks > > #endif > +#endif > > diff --git a/storage/rocksdb/rdb_io_watchdog.h > b/storage/rocksdb/rdb_io_watchdog.h > index de8c1b9500e..12aec516f26 100644 > --- a/storage/rocksdb/rdb_io_watchdog.h > +++ b/storage/rocksdb/rdb_io_watchdog.h > @@ -35,7 +35,8 @@ > namespace myrocks { > > // Rdb_io_watchdog does not support Windows ATM. > -#ifndef _WIN32 > +#ifndef _WIN32 > +#ifndef __APPLE__ > > class Rdb_io_watchdog { > const int RDB_IO_WRITE_BUFFER_SIZE = 4096; > @@ -113,5 +114,6 @@ class Rdb_io_watchdog { > Rdb_io_watchdog &operator=(const Rdb_io_watchdog &) = delete; > }; > > -#endif > +#endif > +#endif > } // namespace myrocks > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

[Maria-developers] Review: MDEV-12985: Support percentile and media window functions
by Vicențiu Ciorbaru 30 Aug '17

30 Aug '17

Hi Varun! Here is the in-detail review for the percentile functions. Please add a grammar definition for median too and test it. All in all, not many changes, but please address my comments. If you disagree with some of them, let's discuss them. > diff --git a/mysql-test/r/win_percentile_cont.result b/mysql-test/r/win_percentile_cont.result > new file mode 100644 > index 00000000000..61f70892887 > --- /dev/null > +++ b/mysql-test/r/win_percentile_cont.result > diff --git a/mysql-test/t/win_percentile_cont.test b/mysql-test/t/win_percentile_cont.test > new file mode 100644 > index 00000000000..75fde963b2a > --- /dev/null > +++ b/mysql-test/t/win_percentile_cont.test > @@ -0,0 +1,55 @@ Please put more tests here, as we discussed. Also, to make test cases easier to follow, please add cume_dist as a function in the select to all these use cases. > +CREATE TABLE student (name CHAR(10), test double, score DECIMAL(19,4)); > +INSERT INTO student VALUES > +('Chun', 0, 3), ('Chun', 0, 7), > +('Kaolin', 0.5, 3), ('Kaolin', 0.6, 7), > +('Kaolin', 0.5, 4), > +('Tatiana', 0.8, 4), ('Tata', 0.8, 4); > + Make this --echo # No partition clause > +#no partition clause > +select name, percentile_disc(0.5) within group(order by score) over () from student; > +select name, percentile_cont(0.5) within group(order by score) over () from student; > + Same as before, use --echo # so we can see it in the result set too. > +# argument set to null > +select name, percentile_cont(null) within group(order by score) over (partition by name) from student; > +select name, percentile_disc(null) within group(order by score) over (partition by name) from student; > + Use --echo # > +# complete query with partition column > +select name, percentile_cont(0.5) within group(order by score) over (partition by name) as c from student; > +select name, percentile_disc(0.5) within group(order by score) over (partition by name) as c from student; > + Use --echo # > +#subqueries having percentile functions > + > +select * from ( select name , percentile_cont(0.5) within group ( order by score) over (partition by name ) from student ) as t; > +select * from ( select name , percentile_disc(0.5) within group ( order by score) over (partition by name ) from student ) as t; > +select name from student a where (select percentile_disc(0.5) within group (order by score) over (partition by name) from student b limit 1) >= 0.5; > + USe --echo > +# WITH STORED PROCEDURES > + > + USe --echo > +#DISALLOWED FIELDS IN ORDER BY CLAUSE > +--error ER_WRONG_TYPE_FOR_PERCENTILE_CONT > +select score, percentile_cont(0.5) within group(order by name) over (partition by score) from student; > +select score, percentile_disc(0.5) within group(order by name) over (partition by score) from student; > USe --echo > + > +# error with 2 order by elements > + > +--error ER_NOT_SINGLE_ELEMENT_ORDER_LIST > +select percentile_disc(0.5) within group(order by score,test) over (partition by name) from student; > +--error ER_NOT_SINGLE_ELEMENT_ORDER_LIST > +select percentile_cont(0.5) within group(order by score,test) over (partition by name) from student; > + Use --echo and fix that comment. We thought of a way for this right? > +#parameter value should be in the range of 0 to 1( NEED TO THINK A WAY FOR THIS) > +--error ER_ARGUMENT_OUT_OF_RANGE > +select percentile_disc(1.5) within group(order by score) over (partition by name) from student; > +--error ER_ARGUMENT_OUT_OF_RANGE > +select percentile_cont(1.5) within group(order by score) over (partition by name) from student; > + > +--error ER_ARGUMENT_NOT_CONSTANT > +select name,percentile_cont(test) within group(order by score) over (partition by name) from student; > +--error ER_ARGUMENT_NOT_CONSTANT > +select name, percentile_disc(test) within group(order by score) over (partition by name) from student; > + This is commented out. Not good, fix it please. And use --echo # > +#CHECK TYPE OF THE ARGUMENT, SHOULD BE ONLY NUMERICAL > +#select name, percentile_cont(name) within group(order by score) over (partition by name) from student; > + > > +drop table student; > diff --git a/sql/item.h b/sql/item.h > index 76ce4aa935f..108b0838dfc 100644 > --- a/sql/item.h > +++ b/sql/item.h > @@ -5245,6 +5246,7 @@ class Cached_item_int :public Cached_item_item > Cached_item_int(Item *item_par) :Cached_item_item(item_par),value(0) {} > bool cmp(void); > int cmp_read_only(); Delete this, it's not used. > + longlong get_value(){ return value;} > }; > > > @@ -5255,6 +5257,7 @@ class Cached_item_decimal :public Cached_item_item > Cached_item_decimal(Item *item_par); > bool cmp(void); > int cmp_read_only(); Delete this, it's not used. > + my_decimal *get_value(){ return &value;}; > }; > > class Cached_item_field :public Cached_item > diff --git a/sql/item_windowfunc.cc b/sql/item_windowfunc.cc > index 6ab903a81bb..916f4113682 100644 > --- a/sql/item_windowfunc.cc > +++ b/sql/item_windowfunc.cc > diff --git a/sql/item_windowfunc.h b/sql/item_windowfunc.h > index 77cbd556e60..ca2ed7136ec 100644 > --- a/sql/item_windowfunc.h > +++ b/sql/item_windowfunc.h > @@ -678,6 +687,275 @@ class Item_sum_ntile : public Item_sum_window_with_row_count > ulong current_row_count_; > }; > > +class Item_sum_percentile_disc : public Item_sum_cume_dist, > + public Type_handler_hybrid_field_type > +{ > +public: > + Item_sum_percentile_disc(THD *thd, Item* arg) : Item_sum_cume_dist(thd, arg), > + Type_handler_hybrid_field_type(&type_handler_longlong), > + value(NULL), val_calculated(FALSE), first_call(TRUE), > + prev_value(0), order_item(NULL){} > + > + double val_real() > + { > + if (get_row_count() == 0 || get_arg(0)->is_null()) > + { > + null_value= true; > + return 0; > + } > + null_value= false; > + return value->val_real(); > + } > + > + longlong val_int() > + { > + if (get_row_count() == 0 || get_arg(0)->is_null()) > + { > + null_value= true; > + return 0; > + } > + null_value= false; > + return value->val_int(); > + } > + > + my_decimal* val_decimal(my_decimal* dec) > + { > + if (get_row_count() == 0 || get_arg(0)->is_null()) > + { > + null_value= true; > + return 0; > + } > + null_value= false; > + return value->val_decimal(dec); > + } > + > + String* val_str(String *str) > + { > + if (get_row_count() == 0 || get_arg(0)->is_null()) > + { > + null_value= true; > + return 0; > + } > + null_value= false; > + return value->val_str(str); > + } > + > + bool add() > + { An extra space here before = > + Item *arg = get_arg(0); > + if (arg->is_null()) > + return false; > + > + if (first_call) > + { > + prev_value= arg->val_real(); We are missing a space after >. > + if (prev_value >1 || prev_value < 0) > + { > + my_error(ER_ARGUMENT_OUT_OF_RANGE, MYF(0)); > + return true; > + } > + first_call= false; > + } > + > + double arg_val= arg->val_real(); > + There is one extra space here. > + if (prev_value != arg_val) > + { > + my_error(ER_ARGUMENT_NOT_CONSTANT, MYF(0)); > + return true; > + } > + > + if (val_calculated) > + return false; > + > + value->store(order_item); > + value->cache_value(); > + if (value->null_value) > + return false; > + > + Item_sum_cume_dist::add(); > + double val= Item_sum_cume_dist::val_real(); > + > + if (val >= prev_value && !val_calculated) > + val_calculated= true; > + return false; > + } > + > + enum Sumfunctype sum_func() const > + { > + return PERCENTILE_DISC_FUNC; > + } > + > + void clear() > + { > + val_calculated= false; > + first_call= true; > + value->clear(); > + Item_sum_cume_dist::clear(); > + } > + > + const char*func_name() const > + { > + return "percentile_disc"; > + } > + > + void update_field() {} > + void set_type_handler(Window_spec *window_spec); > + const Type_handler *type_handler() const > + {return Type_handler_hybrid_field_type::type_handler();} > + > + void fix_length_and_dec() > + { > + decimals = 10; // TODO-cvicentiu find out how many decimals the standard > + // requires. > + } > + > + Item *get_copy(THD *thd, MEM_ROOT *mem_root) > + { return get_item_copy<Item_sum_percentile_disc>(thd, mem_root, this); } > + void setup_window_func(THD *thd, Window_spec *window_spec); > + void setup_hybrid(THD *thd, Item *item); > + > +private: > + Item_cache *value; > + bool val_calculated; > + bool first_call; > + double prev_value; > + Item *order_item; > +}; > + > +class Item_sum_percentile_cont : public Item_sum_cume_dist, > + public Type_handler_hybrid_field_type > +{ > +public: > + Item_sum_percentile_cont(THD *thd, Item* arg) : Item_sum_cume_dist(thd, arg), > + Type_handler_hybrid_field_type(&type_handler_double), > + floor_value(NULL), ceil_value(NULL), first_call(TRUE),prev_value(0), > + ceil_val_calculated(FALSE), floor_val_calculated(FALSE), order_item(NULL){} > + > + double val_real() > + { > + if (get_row_count() == 0 || get_arg(0)->is_null()) > + { > + null_value= true; > + return 0; > + } > + null_value= false; > + double val= 1 + prev_value * (get_row_count()-1); > + > + /* > + Applying the formula to get the value > + If (CRN = FRN = RN) then the result is (value of expression from row at RN) > + Otherwise the result is > + (CRN - RN) * (value of expression for row at FRN) + > + (RN - FRN) * (value of expression for row at CRN) > + */ > + > + if(ceil(val) == floor(val)) > + return floor_value->val_real(); > + > + double ret_val= ((val - floor(val)) * ceil_value->val_real()) + > + ((ceil(val) - val) * floor_value->val_real()); > + > + return ret_val; > + } > + > + bool add() > + { > + Item *arg = get_arg(0); > + if (arg->is_null()) > + return false; > + > + if (first_call) > + { > + first_call= false; > + prev_value= arg->val_real(); > + if (prev_value >1 || prev_value < 0) > + { > + my_error(ER_ARGUMENT_OUT_OF_RANGE, MYF(0)); > + return true; > + } > + } > + > + double arg_val= arg->val_real(); > + if (prev_value != arg_val) > + { > + my_error(ER_ARGUMENT_NOT_CONSTANT, MYF(0)); > + return true; > + } > + > + if (!floor_val_calculated) > + { > + floor_value->store(order_item); > + floor_value->cache_value(); > + if (floor_value->null_value) > + return false; > + } > + if (floor_val_calculated && !ceil_val_calculated) > + { > + ceil_value->store(order_item); > + ceil_value->cache_value(); > + if (ceil_value->null_value) > + return false; > + } > + > + Item_sum_cume_dist::add(); > + double val= 1 + prev_value * (get_row_count()-1); > + > + if (!floor_val_calculated && get_row_number() == floor(val)) > + floor_val_calculated= true; > + > + if (!ceil_val_calculated && get_row_number() == ceil(val)) > + ceil_val_calculated= true; > + return false; > + } > + > + enum Sumfunctype sum_func() const > + { > + return PERCENTILE_CONT_FUNC; > + } > + > + void clear() > + { > + first_call= true; > + floor_value->clear(); > + ceil_value->clear(); > + floor_val_calculated= false; > + ceil_val_calculated= false; > + Item_sum_cume_dist::clear(); > + } > + > + const char*func_name() const > + { > + return "percentile_cont"; > + } > + void update_field() {} > + void set_type_handler(Window_spec *window_spec); > + const Type_handler *type_handler() const > + {return Type_handler_hybrid_field_type::type_handler();} > + > + void fix_length_and_dec() > + { > + decimals = 10; // TODO-cvicentiu find out how many decimals the standard > + // requires. > + } > + > + Item *get_copy(THD *thd, MEM_ROOT *mem_root) > + { return get_item_copy<Item_sum_percentile_cont>(thd, mem_root, this); } > + void setup_window_func(THD *thd, Window_spec *window_spec); > + void setup_hybrid(THD *thd, Item *item); > + > +private: > + Item_cache *floor_value; > + Item_cache *ceil_value; > + bool first_call; > + double prev_value; > + bool ceil_val_calculated; > + bool floor_val_calculated; > + Item *order_item; > +}; > + > + > + > > class Item_window_func : public Item_func_or_sum > { > @@ -781,12 +1063,40 @@ class Item_window_func : public Item_func_or_sum > case Item_sum::DENSE_RANK_FUNC: > case Item_sum::PERCENT_RANK_FUNC: > case Item_sum::CUME_DIST_FUNC: > + case Item_sum::PERCENTILE_CONT_FUNC: > + case Item_sum::PERCENTILE_DISC_FUNC: > return true; > default: > return false; > } > } Let's make the grammar enforce this. This is silly. Is there a big limitation why we can't define a special rule for these functions? There already is order_by_single_element_list in sql_yacc.yy, it just isn't defined properly. We can get rid of these checks altogether then. > + bool only_single_element_order_list() const > + { > + switch (window_func()->sum_func()){ > + case Item_sum::PERCENTILE_CONT_FUNC: > + case Item_sum::PERCENTILE_DISC_FUNC: > + return true; > + default: > + return false; > + } > + } > + This is not needed. Put this in Item_sum_percentile_disc's fix_fields. > + void setting_handler_for_percentile_functions(Item_result rtype) const > + { > + switch (window_func()->sum_func()){ > + case Item_sum::PERCENTILE_DISC_FUNC: > + ((Item_sum_percentile_disc* ) window_func())->set_handler_by_cmp_type(rtype); > + break; > + default: > + return; > + } > + } > + Get rid of this function here. Put the logic of checking this result type in fix_fields specific to Item_sum_percentile_cont / disc. > + bool check_result_type_of_order_item(); > + > + > + > /* > Computation functions. > TODO: consoder merging these with class Group_bound_tracker. > diff --git a/sql/share/errmsg-utf8.txt b/sql/share/errmsg-utf8.txt > index 1f282e6aee5..d8d6975d92c 100644 > --- a/sql/share/errmsg-utf8.txt > +++ b/sql/share/errmsg-utf8.txt > @@ -7490,3 +7490,11 @@ ER_WRONG_INSERT_INTO_SEQUENCE > eng "Wrong INSERT into a SEQUENCE. One can only do single table INSERT into a squence object (like with mysqldump). If you want to change the SEQUENCE, use ALTER SEQUENCE instead." > ER_SP_STACK_TRACE > eng "At line %u in %s" This one will go away when we enforce the limitation in the grammar, make sure to remove it. > +ER_NOT_SINGLE_ELEMENT_ORDER_LIST > + eng "Incorrect number of elements in the order list for '%s'" > +ER_WRONG_TYPE_FOR_PERCENTILE_CONT > + eng "Numeric datatype is required for Percentile_CONT function" As discussed have these two errors print the function name to which the error is for, just like ER_NOT_SINGLE_ELEMENT_ORDER_LIST uses now. > +ER_ARGUMENT_NOT_CONSTANT > + eng "Argument to the percentile functions is not a constant" > +ER_ARGUMENT_OUT_OF_RANGE > + eng "Argument to the percentile functions does not belong to the range [0,1]" > diff --git a/sql/sql_window.cc b/sql/sql_window.cc > index 74e1ad25183..127d6d73ee6 100644 > --- a/sql/sql_window.cc > +++ b/sql/sql_window.cc > @@ -304,6 +304,12 @@ setup_windows(THD *thd, Ref_ptr_array ref_pointer_array, TABLE_LIST *tables, > win_func_item->update_used_tables(); > } > 2 problems with this code. 1. We can probably move this logic in Item_sum_percentile_[cont|disc]::fix_fields. See if that works. 2. There already is an identical while loop before this. USe that one, don't have an extra loop here for no reason. > + li.rewind(); > + while ((win_func_item= li++)) > + { > + if (win_func_item->check_result_type_of_order_item()) > + DBUG_RETURN(1); > + } > DBUG_RETURN(0); > } > > @@ -1658,6 +1666,49 @@ class Frame_unbounded_following_set_count : public Frame_unbounded_following > } > }; > This cursor counts all rows in the partition that don't have the order_item set to null. Let's rewrite this like so: Extract this logic: """ List_iterator_fast<Item_sum> it(sum_functions); Item_sum* item; while ((item= it++)) { Item_sum_window_with_row_count* item_with_row_count = static_cast<Item_sum_window_with_row_count *>(item); item_with_row_count->set_row_count(num_rows_in_partition); } """ 1. Nake it a protected method for Frame_unbounded_following_set_count. Lets call it set_win_funcs_row_count. 2. Rewrite Frame_unbounded_following_set_count to use that function. 3. Make a subclass Frame_unbounded_following_set_count_no_nulls 4. Make use of the protected method in the Frame_unbounded_following_set_count_no_nulls > > +class Frame_unbounded_following_set_count_special: public Frame_unbounded_following_set_count > +{ > + > +public: > + Frame_unbounded_following_set_count_special(THD *thd, > + SQL_I_List<ORDER> *partition_list, > + SQL_I_List<ORDER> *order_list, Item* arg) : > + Frame_unbounded_following_set_count(thd,partition_list, order_list) > + { > + order_item= order_list->first->item[0]; > + } > + void next_partition(ha_rows rownum) > + { > + ha_rows num_rows_in_partition= 0; > + if (cursor.fetch()) > + return; > + > + /* Walk to the end of the partition, find how many rows there are. */ > + do > + { > + if (!order_item->is_null()) > + num_rows_in_partition++; You are missing a space after } > + }while (!cursor.next()); > + > + List_iterator_fast<Item_sum> it(sum_functions); > + Item_sum* item; > + while ((item= it++)) > + { > + Item_sum_window_with_row_count* item_with_row_count = > + static_cast<Item_sum_window_with_row_count *>(item); > + item_with_row_count->set_row_count(num_rows_in_partition); > + } > + } > + > + ha_rows get_curr_rownum() const > + { > + return cursor.get_rownum(); > + } > + > +private: > + Item* order_item; > +}; > + > ///////////////////////////////////////////////////////////////////////////// > // ROWS-type frame bounds > ///////////////////////////////////////////////////////////////////////////// > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index a0bbf39b138..b3b34fce3a4 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -10210,6 +10215,11 @@ geometry_function: > Geometry::wkb_polygon, > Geometry::wkb_linestring)); > } I think this is something that should not be here right? What is this? > + | WITHIN '(' expr ',' expr ')' > + { > + $$= GEOM_NEW(thd, Item_func_spatial_precise_rel(thd, $3, $5, > + Item_func::SP_WITHIN_FUNC)); > + } > ; > > /* > @@ -10671,6 +10681,47 @@ simple_window_func: > } > ; > > +inverse_distribution_function: > + inverse_distribution_function_def WITHIN GROUP_SYM > + '(' > + { Select->prepare_add_window_spec(thd); } > + order_by_single_element_list ')' OVER_SYM > + '(' opt_window_ref opt_window_partition_clause ')' > + { > + LEX *lex= Lex; > + if (Select->add_window_spec(thd, lex->win_ref, > + Select->group_list, > + Select->order_list, > + NULL)) > + MYSQL_YYABORT; > + $$= new (thd->mem_root) Item_window_func(thd, (Item_sum *) $1, > + thd->lex->win_spec); > + if ($$ == NULL) > + MYSQL_YYABORT; > + if (Select->add_window_func((Item_window_func *) $$)) > + MYSQL_YYABORT; > + } > + ; > + > +inverse_distribution_function_def: > + PERCENTILE_CONT_SYM '(' expr ')' > + { > + $$= new (thd->mem_root) Item_sum_percentile_cont(thd, $3); > + if ($$ == NULL) > + MYSQL_YYABORT; > + } > + | PERCENTILE_DISC_SYM '(' expr ')' > + { > + $$= new (thd->mem_root) Item_sum_percentile_disc(thd, $3); > + if ($$ == NULL) > + MYSQL_YYABORT; > + } > + ; > + > See if you can make this accept only one order element. > +order_by_single_element_list: > + ORDER_SYM BY order_list > + ; > + > window_name: > ident > {

1 0

[Maria-developers] Patch for MDEV-12874 - UPDATE statements with the same source and target
by jerome brauge 29 Aug '17

29 Aug '17

Hello, Here is a patch for MDEV-12874 (on branch 10.2-ext). Can anyone review it ? Alexander, during my work on this MDEV, I've found a new issue with update statement and Oracle compatibility. create table t1 (c1 integer, c2 integer, c3 integer); insert into t1(c1,c2,c3) values (1,1,1); update t1 set c1 = c1+1, c2 = c1+1, c3 = c2+1; select * from t1; Mariadb : +------+------+------+ | c1 | c2 | c3 | +------+------+------+ | 2 | 3 | 4 | +------+------+------+ 1 row in set (0.00 sec) Oracle : DVTORA> C1 C2 C3 ---------- ---------- ---------- 2 2 2 Can you open a new MDEV for this ? Best regards, Jérôme.

3 8

[Maria-developers] Oracle strings functions compatibility : substr
by jerome brauge 17 Aug '17

17 Aug '17

Hello Alexander, Here is a patch for function SUBSTR in sql_mode=oracle (If position is 0, then it is treated as 1). Regards, Jérôme.

1 0

[Maria-developers] sys_vars.sysvars_server_embedded test failure
by Alexander Barkov 15 Aug '17

15 Aug '17

Hello Igor, sys_vars.sysvars_server_embedded is failing. It looks like it's related to your recent optimizer change. -SESSION_VALUE index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on -GLOBAL_VALUE index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on +SESSION_VALUE index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=off,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_grouping_derived=on +GLOBAL_VALUE index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown= Can you please fix this? Thanks!

1 0

[Maria-developers] Valgrind reports uninitialized value on "if (!with_two_phase_optimization..."
by Alexander Barkov 15 Aug '17

15 Aug '17

Hello Igor, Please check the output of "mtr --valgrind main.sp" I'm getting these warnings: line ==6490== Thread 6: ==6490== Conditional jump or move depends on uninitialised value(s) ==6490== at 0x7762A1: JOIN::optimize() (sql_select.cc:1123) ==6490== by 0x77F2A7: mysql_select(THD*, TABLE_LIST*, unsigned int, List<Item>&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsigned long long, select_result*, st_select_lex_unit*, st_select_lex*) (sql_select.cc:3721) ==6490== by 0x773994: handle_select(THD*, LEX*, select_result*, unsigned long) (sql_select.cc:376) ==6490== by 0x73F055: execute_sqlcom_select(THD*, TABLE_LIST*) (sql_parse.cc:6431) ==6490== by 0x7354C9: mysql_execute_command(THD*) (sql_parse.cc:3631) ==6490== by 0x742A48: mysql_parse(THD*, char*, unsigned int, Parser_state*, bool, bool) (sql_parse.cc:7886) ==6490== by 0x730684: dispatch_command(enum_server_command, THD*, char*, unsigned int, bool, bool) (sql_parse.cc:1814) ==6490== by 0x72EFF7: do_command(THD*) (sql_parse.cc:1372) ==6490== by 0x883380: do_handle_one_connection(CONNECT*) (sql_connect.cc:1354) ==6490== by 0x883100: handle_one_connection (sql_connect.cc:1260) ==6490== by 0x10D6329: pfs_spawn_thread (pfs.cc:1862) ==6490== by 0x4E3C619: start_thread (in /usr/lib64/libpthread-2.22.so) ==6490== by 0x6DB75FC: clone (in /usr/lib64/libc-2.22.so) ^ Found warnings in /home/bar/maria-git/server.10.2-ext/mysql-test/var/6/log/mysqld.1.err Thanks!

1 0

[Maria-developers] history of lf_hash
by MARK CALLAGHAN 14 Aug '17

14 Aug '17

Comments for lf_hash are sparse. Does it implement a well-known algorithm? https://github.com/mysql/mysql-server/blob/8.0/mysys/lf_hash.cc https://github.com/mysql/mysql-server/blob/8.0/include/lf.h -- Mark Callaghan mdcallag(a)gmail.com

2 1

[Maria-developers] On Recursive CTE and CONNECT BY
by Sergey Petrunia 12 Aug '17

12 Aug '17

Hi Vicentiu, This is a followup to the Wednesday discussion. Background: see attached file for two queries. One uses a Recursive-CTE, another uses CONNECT-BY syntax. They produce different results for some reason. Initial guess about the cause was depth-first vs breadth-first traversal. Short answer: No it's not. The reason is that Recursive-CTE query is not equivalent to the CONNECT-BY query. Results from the recursive-CTE query: ID N P CYCLE PATH 1 A 0 /1A 2 B A 0 /1A/2B 3 C A 0 /1A/3C 4 D B 0 /1A/2B/4D 5 D C 0 /1A/3C/5D 6 E D 0 /1A/3C/5D/6E 6 E D 0 /1A/2B/4D/6E 7 F E 0 /1A/2B/4D/6E/7F 7 F E 0 /1A/3C/5D/6E/7F 8 C F 0 /1A/3C/5D/6E/7F/8C 8 C F 0 /1A/2B/4D/6E/7F/8C 5 D C 0 /1A/2B/4D/6E/7F/8C/5D 12 rows selected. Results from CONNECT-BY query: ID N P CYCLE Path 1 A 0 /1A 2 B A 0 /1A/2B 4 D B 0 /1A/2B/4D 6 E D 0 /1A/2B/4D/6E 7 F E 0 /1A/2B/4D/6E/7F 8 C F 1 /1A/2B/4D/6E/7F/8C 3 C A 0 /1A/3C 5 D C 0 /1A/3C/5D 6 E D 0 /1A/3C/5D/6E 7 F E 1 /1A/3C/5D/6E/7F 10 rows selected. The rows that are missing are: ID N P CYCLE PATH 8 C F 0 /1A/3C/5D/6E/7F/8C 5 D C 0 /1A/2B/4D/6E/7F/8C/5D (denote this as MISSING-ROWS) Another thing to note is that the CTE query has returned CYCLE=0 for all rows, while the CONNECT BY one has returned two rows with CYCLE=1: 8 C F 1 /1A/2B/4D/6E/7F/8C 7 F E 1 /1A/3C/5D/6E/7F Note that these rows are parents of rows in MISSING-ROWS. I don't think the issue is caused by breadth-first vs depth-first traversal differences. The issue is caused by us not understanding how Oracle's NOCYCLE clause works. Take a look at the original-dataset.jpg, and PATH columns of MISSING-ROWS We see that the path forms the entire loop (or "cycle"). Oracle's NOCYCLE seems to be about *not* constructing the entire loop. To rule out breadth-first vs depth-first differences, let's make an example where there's just one loop: create table g3(a int, b int); insert into g3 values (1, 2); insert into g3 values (2, 3); insert into g3 values (3, 4); insert into g3 values (4, 5); insert into g3 values (5, 6); insert into g3 values (6, 7); insert into g3 values (7, 4); select a, CONNECT_BY_ISCYCLE as cycle, SYS_CONNECT_BY_PATH(a, '/') "Path" from g3 start with a=1 connect by NOCYCLE prior b= a; A CYCLE Path 1 0 /1 2 0 /1/2 3 0 /1/2/3 4 0 /1/2/3/4 5 0 /1/2/3/4/5 6 1 /1/2/3/4/5/6 The row with a=7 is not listed. If it did, the path would have had a complete cycle. It is a bit counter intuitive that /1/2/3/4/5/6 has CYCLE=1, while actually it does not form a cycle. It's one step short of doing so. The documentation is vague about this, but here's somebody thinking the same on the internet: http://www.sql.ru/forum/255418/zadachka-anagrammy-na-sql?hl=%e0%ed%e0%e3%f0… : While posting messages on MetaLink, I think I finally got the logic of NOCYCLE. Clue was right there in the name itself. NOCYCLE means do not cycle. So if you have a hierarchical query without NOCYCLE and a cycle is detected Oracle does not know how to handle it and throws ORA-01436. If NOCYCLE is used Oracle simply cuts of part of a branch right at loop point without raising an error (another word it simply does not follow the loop). And that explains why CONNECT_BY_ISCYCLE is set to 1 for the last row in first iteration of the loop. Otherwise, we would not get 1 at all. Now next logical step would be to implement CYCLE. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Win32 - Sysvar 'optimizer_switch' failed 'def_val < my_set_bits(typelib.count)'
by jerome brauge 11 Aug '17

11 Aug '17

Hi All, Build of last 10.2-ext fail on Win32 with message : Sysvar 'optimizer_switch' failed 'def_val < my_set_bits(typelib.count)' The problem seems to be in sys_vars.ic (Sys_var_set) : > > { > option.var_type|= GET_SET; > global_var(ulonglong)= def_val; > SYSVAR_ASSERT(typelib.count > 0); > SYSVAR_ASSERT(typelib.count <= 64); > SYSVAR_ASSERT(def_val <= my_set_bits(typelib.count)); Patch b14e2b04 introduce a new optimizer_switch and now 'optimizer_switch' has 32 values. my_set_bits works with unit32. A new function like this static inline uint64 my_set_bits64(int n) { return (((1ULL << (n - 1)) - 1) << 1) | 1; } might solve this issue. Regards.

1 0

[Maria-developers] Virtual columns in sql_mode=ORACLE
by jerome brauge 11 Aug '17

11 Aug '17

Hello Alexander, Something seems to be wrong in sql_yacc_ora.yy. (10.2-ext branch) We can't create a table with a virtual column because the grammar try to reduce the rule sp_cursor_name_and_offset. CREATE TABLE t1 (c1 INTEGER, c2 INTEGER AS (c1 MOD 10) VIRTUAL) ENGINE=InnoDb; ERROR 1324 (42000): Undefined CURSOR: c1 Regards, Jérôme.

2 1

[Maria-developers] Default value on column in sql_mode=ORACLE
by jerome brauge 11 Aug '17

11 Aug '17

Hello Alexander, Do you know why STRICT_TRANS_TABLES is not set in sql_mode=ORACLE ? The following script should not work: set sql_mode=oracle; create table t1 (c1 integer not null,c2 integer not null); insert into t1 (c1) values(1); Query OK, 1 row affected, 1 warning Regards, Jérôme.

2 1

Re: [Maria-developers] MDEV-13439: Database permissions are not enough to run a subquery with GROUP BY within a view
by Sergei Golubchik 09 Aug '17

09 Aug '17

Hi, Oleksandr! On Aug 07, Oleksandr Byelkin wrote: > revision-id: cf66c30ce592ad88d447568272bdced2046f2906 (mariadb-10.2.7-20-gcf66c30ce59) > parent(s): bcc10a5a447805ce64aa13ee6a037c1618219616 > committer: Oleksandr Byelkin > timestamp: 2017-08-07 13:42:35 +0200 > message: > > MDEV-13439: Database permissions are not enough to run a subquery with GROUP BY within a view > > The bug is result adding ability to have derived tables inside views. > Fixed checks should be a switch between view/derived or select derived and information schema. > > diff --git a/sql/sql_acl.cc b/sql/sql_acl.cc > index 77f502d3a7b..9bffcbe6a9e 100644 > --- a/sql/sql_acl.cc > +++ b/sql/sql_acl.cc > @@ -7594,8 +7594,11 @@ bool check_grant(THD *thd, ulong want_access, TABLE_LIST *tables, > /* > It is subquery in the FROM clause. VIEW set t_ref->derived after > table opening, but this function always called before table opening. > + > + NOTE: is_derived() can't be used here bacause subquery in this case typo "bacause" > + the FROM clase (derived tables) can be not be marked yet. > */ > - if (!t_ref->referencing_view) > + if (t_ref->is_anonymous_derived_table() || t_ref->schema_table) > { > /* > If it's a temporary table created for a subquery in the FROM > diff --git a/sql/sql_derived.cc b/sql/sql_derived.cc > index 649f745fdc4..8f8d051a976 100644 > --- a/sql/sql_derived.cc > +++ b/sql/sql_derived.cc > @@ -822,10 +822,12 @@ bool mysql_derived_prepare(THD *thd, LEX *lex, TABLE_LIST *derived) > table->derived_select_number= first_select->select_number; > table->s->tmp_table= INTERNAL_TMP_TABLE; > #ifndef NO_EMBEDDED_ACCESS_CHECKS > - if (derived->referencing_view) > + if (derived->is_view()) > table->grant= derived->grant; > else > { > + DBUG_ASSERT(derived->is_derived()); > + DBUG_ASSERT(derived->is_anonymous_derived_table()); Hmm, if derived->is_derived() is always true here, why there's an if() below? If it's redundant - please, remove it. > table->grant.privilege= SELECT_ACL; > if (derived->is_derived()) > derived->grant.privilege= SELECT_ACL; Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] commit_checkpoint_request() vs. thd_get_durability_property() (in relation to MDEV-11937)
by Kristian Nielsen 08 Aug '17

08 Aug '17

Monty asked me to fix MDEV-11937. This particular one is a performance regression in InnoDB commit. But there is a wider problem that I thought I should explain, so it can be perhaps avoided in the future. MariaDB and MySQL use different mechanisms for storage engines to avoid having to fsync during commit when binlog is enabled. In MariaDB, storage engines implement the commit_checkpoint_request() handlerton method. In MySQL, storage engines call thd_get_durability_property() to check if they can avoid fsync. The problem is that somehow the thd_get_durability_property() function was introduced into MariaDB code, but it is completely non-functional. So now there is code in InnoDB, TokuDB and RocksDB that calls this function and does not work correctly. This lead to performance regression due to extra fsync() calls. This seems to me a serious problem. Now new code can be merged and compile fine, where in reality it is wrong. There really should not be two separate and different mechanisms for the same thing, and certainly not with one of them non-functional. The "expected" approach would be to remove thd_get_durability_property() and update storage engines to use the corresponding MariaDB APIs (commit_ordered() and commit_checkpoint_request()). This should not be hard. A simple commit_checkpoint_request() implementation can just fsync all transactions immediately (similar to what MySQL does). A more detailed implementation can avoid any extra fsyncs, and just asynchroneously notify the upper layer with commit_checkpoint_notify_ha() later when such fsync happens normally (this is what InnoDB does). See comments in handler.h for details. Or was the intention to eventually replace the whole MariaDB binlog group commit implementation with the MySQL one, to make MariaDB less divergent? This would require a number of changes to MariaDB binlog and replication. The binlog recovery code should be replaced (MySQL does not have the ability to recover from more than one binlog). The binlog group commit code must be replaced as well, and the commit_ordered() mechanism removed. I think this would also require a re-design of the MariaDB in-order parallel replication. MySQL has some optional mechanism for in-order, but my understanding is that it is not sufficient to do optimistic parallel replication. I am not intimately familiar with the MySQL code though. I hope this helps. The wider problem behind MDEV-11937 is one of policy more than one of code bugs, so not too much else I can do to address it. ----------------------------------------------------------------------- Incidentally, I noticed some code in InnoDB trx0trx.cc: /* We set the HA_IGNORE_DURABILITY during prepare phase of binlog group commit to not flush redo log for every transaction here. So that we can flush prepared records of transactions to redo log in a group right before writing them to binary log during flush stage of binlog group commit. */ So the idea is to do group prepare with the same group of transactions that will later group commit to the binlog. In MariaDB, this concept does not exist. Storage engine prepares are allowed to run in parallel and in any order compared to binlog commit. So the InnoDB group prepare can include more transactions than participate in the binlog commit (but also less, of course). IIRC, the important thing is to ensure that all transactions are durably prepared in storage engines before being written to the binlog. In MariaDB, there is MYSQL_LOG_BIN::group_commit_queue that holds the list of transactions to be group committed to the binlog. A similar mechanism in MariaDB might use the group_commit_queue as the set of transactions to send to storage engines for group prepare. But if we wait for a prepare fsync() after building this list, some transactions that could prepare during this fsync may be unnecesarily delayed to the next binlog commit. The MySQL 5.7 code grabs the list of transactions to binlog group commit, and then flushes the log in _all_ storage engines, unconditionally. That just seems horrybly wrong, but I can't see any other way to read the code (from MYSQL_BIN_LOG::process_flush_stage_queue()): ha_flush_logs(NULL, true); It even does so while holding LOCK_log :-( I guess the MySQL idea is that there is only one storage engine anyway, InnoDB. - Kristian.

3 3

Re: [Maria-developers] 4c26b71c77d: MDEV-8960: Can't refer the same column twice in one ALTER TABLE
by Sergei Golubchik 04 Aug '17

04 Aug '17

Hi, Jan! A question and a suggestion, see below On Jul 27, jan wrote: > revision-id: 4c26b71c77d58fbeb3192e2142ef66efe5f122dd (mariadb-10.0.31-43-g4c26b71c77d) > parent(s): a8c1817846eb9f3fd8738e80db87cff642b9a78b > author: Jan Lindström > committer: Jan Lindström > timestamp: 2017-07-27 13:17:13 +0300 > message: > > MDEV-8960: Can't refer the same column twice in one ALTER TABLE > > Problem was that if column was created in alter table when > it was refered again it was not tried to find from list > of current columns. > > mysql_prepare_alter_table: > There is two cases > (1) If alter table adds a new column and then later alter > changes the field definition, there was no check from > list of new columns, instead an incorrect error was given. > (2) If alter table adds a new column and then later alter > changes the default, there was no check from list of > new columns, instead an incorrect error was given. > > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index 2eff8fd5e2f..3ba3ec6847e 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -7503,9 +7503,25 @@ mysql_prepare_alter_table(THD *thd, TABLE *table, > { > if (def->change && ! def->field) > { > - my_error(ER_BAD_FIELD_ERROR, MYF(0), def->change, > - table->s->table_name.str); > - goto err; > + /* > + Check if there is modify for newly added field. > + */ > + Create_field *find; > + find_it.rewind(); > + while((find=find_it++)) > + { > + if (!my_strcasecmp(system_charset_info,find->field_name, def->field_name)) > + break; > + } > + > + if (find && !find->field) Can it be here that find->field != NULL? I don't see how. But if yes, the ER_BAD_FIELD_ERROR looks rather weird in that case. If not - simple if (find) would be less confusing. > + find_it.remove(); > + else > + { > + my_error(ER_BAD_FIELD_ERROR, MYF(0), def->change, > + table->s->table_name.str); > + goto err; > + } > } > /* > Check that the DATE/DATETIME not null field we are going to add is > @@ -7571,6 +7587,29 @@ mysql_prepare_alter_table(THD *thd, TABLE *table, > find_it.after(def); // Put column after this > } > } > + /* > + Check if there is alter for newly added field. > + */ > + alter_it.rewind(); > + Alter_column *alter; > + while ((alter=alter_it++)) > + { > + if (!my_strcasecmp(system_charset_info,def->field_name, alter->name)) > + break; > + } > + if (alter) > + { > + if (def->sql_type == MYSQL_TYPE_BLOB) > + { > + my_error(ER_BLOB_CANT_HAVE_DEFAULT, MYF(0), def->change); > + goto err; > + } > + if ((def->def=alter->def)) // Use new default > + def->flags&= ~NO_DEFAULT_VALUE_FLAG; > + else > + def->flags|= NO_DEFAULT_VALUE_FLAG; > + alter_it.remove(); > + } Please, move this into a function. Don't just copy 20 lines around. For example if (apply_alter_column(alter_it, def, field->field_name)) goto err; > } > if (alter_info->alter_list.elements) > { Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] Review 2 of compressed columns
by Michael Widenius 04 Aug '17

04 Aug '17

Hi! Here is a review for the patches marked with fixup! MDEV-11371 - column compression > commit f1a3cd40ecc4c144ac32d43ab5c1e88c8e4fae7e > Author: Sergey Vojtovich <svoj(a)mariadb.org> > Date: Mon Jul 31 16:07:39 2017 +0400 > > fixup! MDEV-11371 - column compression > > In worst case we have to store data length + 1 for compressed data, > we should increase field length by 1 internally so that data is not > truncated. > > diff --git a/mysql-test/t/column_compression.test b/mysql-test/t/column_compression.test > index 256e96c..d3f8481 100644 > --- a/mysql-test/t/column_compression.test > +++ b/mysql-test/t/column_compression.test > @@ -54,8 +54,8 @@ SHOW CREATE TABLE t1; > --cat_file $MYSQLD_DATADIR/test/t1.CSV > DROP TABLE t1; > > ---echo # Test fields with length equal to data length > -CREATE TABLE t1(a VARCHAR(10) COMPRESSED); > +--echo # Test fields that don't fit data > +CREATE TABLE t1(a VARCHAR(9) COMPRESSED); > --error ER_DATA_TOO_LONG > INSERT INTO t1 VALUES(REPEAT('a', 10)); > INSERT INTO t1 VALUES(REPEAT(' ', 10)); Please also insert a correct row, and not only edge cases. (You may do it already, but I can't see that in the diff) > @@ -70,3 +70,12 @@ INSERT INTO t1 VALUES(REPEAT(' ', 255)); > SET column_compression_threshold=DEFAULT; > SELECT a, LENGTH(a) FROM t1; > DROP TABLE t1; > + > +--echo # Corner case: VARCHAR(255) COMPRESSED must have 2 bytes pack length > +CREATE TABLE t1(a VARCHAR(255) COMPRESSED); > +SHOW CREATE TABLE t1; > +SET column_compression_threshold=300; > +INSERT INTO t1 VALUES(REPEAT('a', 255)); > +SET column_compression_threshold=DEFAULT; > +SELECT a, LENGTH(a) FROM t1; > +DROP TABLE t1; Please insert also a row without column_compression_threshold, just to cover all cases in the test case. > diff --git a/sql/field.cc b/sql/field.cc > index 216fb95..e1dc193 100644 > --- a/sql/field.cc > +++ b/sql/field.cc > @@ -7555,7 +7555,8 @@ void Field_varstring::sql_type(String &res) const > length= cs->cset->snprintf(cs,(char*) res.ptr(), > res.alloced_length(), "%s(%d)", > (has_charset() ? "varchar" : "varbinary"), > - (int) field_length / charset()->mbmaxlen); > + (int) field_length / charset()->mbmaxlen - > + MY_TEST(compression_method())); > res.length(length); > if ((thd->variables.sql_mode & (MODE_MYSQL323 | MODE_MYSQL40)) && > has_charset() && (charset()->state & MY_CS_BINSORT)) > @@ -10015,6 +10016,8 @@ void Column_definition::create_length_to_internal_length(void) > case MYSQL_TYPE_STRING: > case MYSQL_TYPE_VARCHAR: > length*= charset->mbmaxlen; > + if (sql_type == MYSQL_TYPE_VARCHAR && compression_method()) > + length++; Add a comment: /* Compressed columns needs one extra byte to store the compression method. This byte is invisible to the end user, but not for the storage engine. */ > key_length= length; > pack_length= calc_pack_length(sql_type, length); > break; <cut> > commit 274ab73a89573554d07a6de1deb846ad47eb1004 > Author: Sergey Vojtovich <svoj(a)mariadb.org> > Date: Thu Jul 27 13:06:33 2017 +0400 > > fixup! MDEV-11371 - column compression > > Fixed BLOB COMPRESSED -> BLOB COMPRESSED replication. > > The problem was that Field_blob::unpack gets compressed data and calls > Field_blob_compressed::store(), which compressed it again. Fixed by calling > Field_blob::set_ptr() instead of ::store(). > > Note that now no data copy is performed and blob is pointing to data allocated > by replication routines. It used to work this way before 06fb8c2d10, but still > should be reviewed carefully to make sure no copy is actually needed. > patch ok. > commit 93e00970100503c3f365063d9261119f096bee48 > Author: Sergey Vojtovich <svoj(a)mariadb.org> > Date: Thu Jul 27 11:44:20 2017 +0400 > > Get rid of Field::do_save_field_metadata() > > It doesn't serve any purpose, but generates extra virtual function call. > ok > commit e1247d4991bb4ebd3f98788c7bfe31a34faf7b69 > Author: Sergey Vojtovich <svoj(a)mariadb.org> > Date: Thu Jul 27 10:58:47 2017 +0400 > > fixup! MDEV-11371 - column compression > > This is an addition to "Fixed compressed columns to work with partitioning". > > Mroonga calls key_copy() on write/delete, when column is not in read_set. > Adjust read_set for the duration of val_str(). > > diff --git a/sql/field.cc b/sql/field.cc > index ec82984..ab96ee9 100644 > --- a/sql/field.cc > +++ b/sql/field.cc > @@ -7659,8 +7659,12 @@ uint Field_varstring::get_key_image(uchar *buff, uint length, > { > String val; > uint local_char_length; > + my_bitmap_map *old_map; > > + old_map= dbug_tmp_use_all_columns(table, table->read_set); > val_str(&val, &val); > + dbug_tmp_restore_column_map(table->read_set, old_map); > + Shouldn't this be in mronga code and not in MariaDB? Looking at the original code, we didn't check for read_set before either for get_key_image, so I assume this is ok. > local_char_length= val.charpos(length / field_charset->mbmaxlen); > if (local_char_length < val.length()) > val.length(local_char_length); > commit 94040156d8007663beb6785ca653d41b26758301 > Author: Sergey Vojtovich <svoj(a)mariadb.org> > Date: Wed Jul 26 17:28:57 2017 +0400 > > fixup! MDEV-11371 - column compression > > Get rid of hardcoded "8" (zlib compression method id) by moving towards > support for multiple compression methods. > commit 6c8b1e563c7456eb0b8ca11a621333237de37ec4 > Author: Sergey Vojtovich <svoj(a)mariadb.org> > Date: Tue Jul 25 13:27:07 2017 +0400 > > fixup! MDEV-11371 - column compression > > Fixed compressed columns to work with partitioning. > > Reviewer note: this patch reuses get_key_image()/set_key_image() of > Field_varstring to avoid code duplication. Cons: this adds virtual function > calls val_str() and store(). > ok > commit 9c85ad8d105e74baae0b6aba0db01b4d6aac7ba6 > Author: Sergey Vojtovich <svoj(a)mariadb.org> > Date: Thu Jul 20 16:57:46 2017 +0400 > > fixup! MDEV-11371 - column compression ok. The only thing left is now to fix replication to handle compressed columns. We also have agreed to handle the case where the master and slave have different COMPRESSED column attributes for VARCHAR and BLOB. Regards, Monty

1 0

[Maria-developers] Debian 9 support for MaxScale
by Martin Toth 03 Aug '17

03 Aug '17

Hi all, I would like to ask if there is some release date for fixing https://jira.mariadb.org/browse/MXS-1292 . Debian 9 should be included as a build target ASAP. I am installing only D9 systems on new servers and I am unable to use builds for MaxScale, so its unusable in D9. Thanks. Martin

3 7

Re: [Maria-developers] 1 Transaction and several threads
by Jan Lindström 03 Aug '17

03 Aug '17

Hi, If we talk about bulk load I would write directly .ibd file format pages, now as InnoDB has always some order between pages doing this in parallel has challenges, we can read and parse input, sort it, reformat to InnoDB pages, write these pages (maybe you could do this also parallel). To speed this up, I would remove secondary keys that are not needed from table and rebuild them after bulk load. R: Jan On Thu, Aug 3, 2017 at 12:09 PM, Oleksandr Byelkin <sanja(a)mariadb.com> wrote: > Hi! > > I am thinking about efficient inserting in a table using several threads > (LOAD), is it possible (and how difficult if it does) to make inserts with > help of several threads abut inside one InnoDB transaction (maybe one XA > transaction)? >

3 2

[Maria-developers] Git jira integration plugin
by Daniel Black 03 Aug '17

03 Aug '17

For those that haven't seen it. https://www.npmjs.com/package/jira-git

1 0

[Maria-developers] Patch for MDEV-13417 - UPDATE produces wrong values if an updated column is later used as an update source
by jerome brauge 02 Aug '17

02 Aug '17

Hello, Here is a patch for MDEV-13417 (on branch 10.2-ext). Can anyone review it ? Best regards, Jérôme.

1 0

[Maria-developers] HAVE_LONG_LONG check
by Aleksey Midenkov 20 Jul '17

20 Jul '17

Do we need HAVE_LONG_LONG in item.cc: #ifdef HAVE_LONG_LONG case MYSQL_TYPE_LONGLONG: field= new (mem_root) Field_longlong((uchar*) 0, max_length, null_ptr, 0, Field::NONE, name, 0, unsigned_flag); break; #endif ? There is no such check in field.cc: case MYSQL_TYPE_LONGLONG: return new (mem_root) Field_longlong(ptr,field_length,null_pos,null_bit, unireg_check, field_name, f_is_zerofill(pack_flag) != 0, f_is_dec(pack_flag) == 0);

2 1

Re: [Maria-developers] 0aa6217: MDEV-13065 rpl.rpl_mdev-11092 fails sporadically in buildbot
by Sergei Golubchik 19 Jul '17

19 Jul '17

Hi, Sachin! On Jul 19, sachin wrote: > revision-id: 0aa621722d74d317f39b4c463800959ee84a6097 (mariadb-10.0.31-33-g0aa6217) As discussed, it should probably go into 5.5 > parent(s): 2f07709aed863ea1852a78a0d6ab718e49df8cca > author: Sachin Setiya > committer: Sachin Setiya > timestamp: 2017-07-19 11:59:43 +0530 > message: > > MDEV-13065 rpl.rpl_mdev-11092 fails sporadically in buildbot > > Problem > rpl.rpl_mdev-11092 fails in buildbot because after starting slave in > wait_for_slave_sql_error_and_skip.inc slave is started but there may be > chances that we have not skipped the last error and Last_SQL_Errno is > still not zero untill the end of rpl_end.inc , which will compare > Last_SQL_Errno to 0. So in this this case rpl_mdev-11092 fails. > > Solution > After starting slave in wait_for_slave_sql_error_and_skip.inc we will wait for > Last_SQL_Errno to become 0. > > --- > mysql-test/include/wait_for_slave_sql_error_and_skip.inc | 15 +++++++++++++++ > 1 file changed, 15 insertions(+) > > diff --git a/mysql-test/include/wait_for_slave_sql_error_and_skip.inc b/mysql-test/include/wait_for_slave_sql_error_and_skip.inc > index 9246c18..60c8319 100644 > --- a/mysql-test/include/wait_for_slave_sql_error_and_skip.inc > +++ b/mysql-test/include/wait_for_slave_sql_error_and_skip.inc > @@ -66,6 +66,21 @@ if (!$slave_skip_counter) { > } > source include/start_slave.inc; > > +# Wait for 5 seconds Last_SQL_Errno to become zero > +--let $wait_counter=50 > +while ($wait_counter) > +{ > + let $wait_condition= query_get_value("SHOW SLAVE STATUS", Last_SQL_Errno, 1); > + if ($wait_condition == $slave_sql_errno) > + { > + real_sleep 0.1; > + dec $wait_counter; > + } > + if (!$wait_condition) > + { > + --let $wait_counter = 0; > + } > +} I'd suggest to use include/wait_for_slave_param.inc here, like let $slave_param= Last_SQL_Errno; let $slave_param_value= 0; source include/wait_for_slave_param.inc; and put a comment before it, like # start_slave.inc returns when Slave_SQL_Running=Yes. But the slave # thread sets it before clearing Last_SQL_Errno. So we have to wait # for Last_SQL_Errno=0 separately. > --let $include_filename= wait_for_slave_sql_error_and_skip.inc [errno=$slave_sql_errno] > --source include/end_include_file.inc Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] 05b678bc8c1: MDEV-12489 The select stmt may fail due to "having clause is ambiguous" unexpected
by Oleksandr Byelkin 12 Jul '17

12 Jul '17

Hi, Sergei! Am 12.07.2017 um 18:13 schrieb serg(a)mariadb.org: > revision-id: 05b678bc8c166c9015ca3dc3ccd9e9c89094f3ca (mariadb-5.5.56-42-g05b678bc8c1) > parent(s): c83d6ff881dcfd1cc68124a9663c2a31ca7e2dce > author: Sergei Golubchik > committer: Sergei Golubchik > timestamp: 2017-07-12 17:46:19 +0200 > message: > > MDEV-12489 The select stmt may fail due to "having clause is ambiguous" unexpected > > only compare field references with an alias (from the SELECT clause) > when this reference doesn't specify an explicit table name part. > OK to push! [skip]

1 0

Re: [Maria-developers] [Commits] be55bbc2b20: MDEV-7826 Server crashes in Item_subselect::enumerate_field_refs_processor
by Oleksandr Byelkin 12 Jul '17

12 Jul '17

Hi, Sergei! Am 12.07.2017 um 12:49 schrieb serg(a)mariadb.org: > revision-id: be55bbc2b206f2fe295ad4179b23791c3119fe02 (mariadb-5.5.56-39-gbe55bbc2b20) > parent(s): c5975eaea174460e774e77717e972a8d32e6c8de > author: Sergei Golubchik > committer: Sergei Golubchik > timestamp: 2017-07-12 12:49:29 +0200 > message: > > MDEV-7826 Server crashes in Item_subselect::enumerate_field_refs_processor > > upper->item can be NULL if we're referring to an aggregate function OK to push! [skip]

1 0

Re: [Maria-developers] [Commits] c5975eaea17: MDEV-7339 Server crashes in Item_func_trig_cond::val_int
by Oleksandr Byelkin 12 Jul '17

12 Jul '17

Hi! Am 12.07.2017 um 10:39 schrieb serg(a)mariadb.org: > revision-id: c5975eaea174460e774e77717e972a8d32e6c8de (mariadb-5.5.56-38-gc5975eaea17) > parent(s): f305a7ce4bccbd56520d874e1d81a4f29bc17a96 > author: Sergei Golubchik > committer: Sergei Golubchik > timestamp: 2017-07-12 10:39:02 +0200 > message: > > MDEV-7339 Server crashes in Item_func_trig_cond::val_int > > Item_in_subselect::pushed_cond_guards[] array is allocated only when > left_expr->maybe_null. And it is used (for row expressions) when > left_expr->element_index(i)->maybe_null. > > For left_expr being a multi-column subquery, its maybe_null is > always false when the subquery doesn't use tables (see > Item_singlerow_subselect::fix_length_and_dec() > and subselect_single_select_engine::fix_length_and_dec()), > otherwise it's always true. > > But row elements can be NULL regardless, so let's always allocate > pushed_cond_guards for multi-column subqueries, no matter whether > its maybe_null was forced to true or false. OK to push! [skip]

1 0

Re: [Maria-developers] [Commits] 7801756: MDEV-13826: Floating point exception in Filesort_tracker::print_json_members(Json_writer*)
by Sergey Petrunia 12 Jul '17

12 Jul '17

Hi Varun, On Wed, Jul 12, 2017 at 12:43:02AM +0530, Varun wrote: > revision-id: 780175646bb9312184d8a676e86b125f663b5156 (mariadb-10.2.2-591-g7801756) > parent(s): 3904014ed37cc2b58844e8d24a5d6aaa69d80f59 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2017-07-12 00:41:52 +0530 > message: > > MDEV-13826: Floating point exception in Filesort_tracker::print_json_members(Json_writer*) > > Whenever Filesort_tracker has r_loops=0, r_ouptut_rows would be 0, so we should add the value zero to the member "r_output_rows" expliciytly > > --- > mysql-test/r/analyze_format_json.result | 60 +++++++++++++++++++++++++++++++++ > mysql-test/t/analyze_format_json.test | 11 ++++++ > sql/sql_analyze_stmt.cc | 11 +++--- > 3 files changed, 78 insertions(+), 4 deletions(-) > > diff --git a/mysql-test/r/analyze_format_json.result b/mysql-test/r/analyze_format_json.result > index 686da92..436b13b 100644 > --- a/mysql-test/r/analyze_format_json.result > +++ b/mysql-test/r/analyze_format_json.result > @@ -757,3 +757,63 @@ ANALYZE > } > } > drop table t1,t2,t3; > +# > +# MDEV-13286: Floating point exception in Filesort_tracker::print_json_members(Json_writer*) > +# > +create table t0(a int); > +insert into t0 values (0),(1),(2),(3),(4),(5),(6),(7),(8),(9); > +create table t1 (a int, b int, c int); > +insert into t1 select a,a,a from t0; > +create table t2 as select * from t1; > +analyze format=json select a, (select t2.b from t2 where t2.a<t1.a order by t2.c limit 1) from t1 where t1.a<0; > +ANALYZE > +{ > + "query_block": { > + "select_id": 1, > + "r_loops": 1, > + "r_total_time_ms": 0.0324, > + "table": { > + "table_name": "t1", > + "access_type": "ALL", > + "r_loops": 1, > + "rows": 10, > + "r_rows": 10, > + "r_total_time_ms": 0.0098, > + "filtered": 100, > + "r_filtered": 0, > + "attached_condition": "t1.a < 0" > + }, > + "subqueries": [ > + { > + "expression_cache": { > + "state": "uninitialized", > + "r_loops": 0, > + "query_block": { > + "select_id": 2, > + "read_sorted_file": { > + "r_rows": null, > + "filesort": { > + "sort_key": "t2.c", > + "r_loops": 0, > + "r_limit": "(varied across executions)", > + "r_used_priority_queue": false, Look at the "table" field below: the parameters whose values are not defined because of r_loop=0 had value of null (e.g. r_filtered, r_rows). Let's follow that pattern here: r_loops:null, r_limit:null, r_used_priority_queue: null. Please change this. Ok to push after this is addressed. > + "r_output_rows": 0, > + "table": { > + "table_name": "t2", > + "access_type": "ALL", > + "r_loops": 0, > + "rows": 10, > + "r_rows": null, > + "filtered": 100, > + "r_filtered": null, > + "attached_condition": "t2.a < t1.a" > + } > + } > + } > + } > + } > + } > + ] > + } > +} > +drop table t0,t1,t2; > diff --git a/mysql-test/t/analyze_format_json.test b/mysql-test/t/analyze_format_json.test > index 88a9077..02603bc 100644 > --- a/mysql-test/t/analyze_format_json.test > +++ b/mysql-test/t/analyze_format_json.test > @@ -212,3 +212,14 @@ GROUP BY sq ORDER BY gc; > > drop table t1,t2,t3; > > + > +--echo # > +--echo # MDEV-13286: Floating point exception in Filesort_tracker::print_json_members(Json_writer*) > +--echo # > +create table t0(a int); > +insert into t0 values (0),(1),(2),(3),(4),(5),(6),(7),(8),(9); > +create table t1 (a int, b int, c int); > +insert into t1 select a,a,a from t0; > +create table t2 as select * from t1; > +analyze format=json select a, (select t2.b from t2 where t2.a<t1.a order by t2.c limit 1) from t1 where t1.a<0; > +drop table t0,t1,t2; > diff --git a/sql/sql_analyze_stmt.cc b/sql/sql_analyze_stmt.cc > index cc6e0f8..3e49af1 100644 > --- a/sql/sql_analyze_stmt.cc > +++ b/sql/sql_analyze_stmt.cc > @@ -43,15 +43,18 @@ void Filesort_tracker::print_json_members(Json_writer *writer) > } > > writer->add_member("r_used_priority_queue"); > - if (r_used_pq == get_r_loops()) > - writer->add_bool(true); > - else if (r_used_pq == 0) > + if (r_used_pq == 0) > writer->add_bool(false); > + else if (r_used_pq == get_r_loops()) > + writer->add_bool(true); > else > writer->add_str(varied_str); > > - writer->add_member("r_output_rows").add_ll((longlong) rint(r_output_rows / > + if (r_output_rows) > + writer->add_member("r_output_rows").add_ll((longlong) rint(r_output_rows / > get_r_loops())); > + else > + writer->add_member("r_output_rows").add_ll((longlong)0); > > if (sort_passes) > { > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] RFC: new replication feature "per-engine mysql.gtid_slave pos"
by Kristian Nielsen 11 Jul '17

11 Jul '17

I plan to implement MDEV-12179, per-engine mysql.gtid_slave_pos. Here is a description of the high-level design, as a request for comments and/or suggestion for changes. The purpose of this is to fix a serious performance issue in replication when multiple storage engines are used. Every replicated transaction updates the replication GTID position in table mysql.gtid_slave_pos. There is a large overhead if that table is using another storage engine than the rest of the transaction, as then a cross-engine XA transaction needs to be done. But if more than one storage engine is used on a server, some replicated transactions will necessarily end up being cross-engine. This problem becomes more interesting to fix with the new interest in MyRocks. Though it already exists also for eg. people using MyISAM or TokuDB simultaneously with InnoDB, for example. The basic idea is to create multiple copies of the table, eg. mysql.gtid_slave_pos_innodb and mysql.gtid_slave_pos_myrocks. This way, an InnoDB transaction can update the InnoDB version of the table, and similar for other engines. To enable the feature, the DBA would create extra copies of the table for the engines she wants: CREATE TABLE mysql.gtid_slave_pos_myrocks LIKE mysql.gtid_slave_pos; ALTER TABLE mysql.gtid_slave_pos_myrocks ENGINE=myrocks; The server will look for all tables called mysql.gtid_slave_pos* at startup, as well as on each START SLAVE. This method makes migration from earlier versions simple. MariaDB already has the ability to work with multiple redundant rows in the mysql.gtid_slave_pos table. The column "sub_id" provides a version to identify the most recent entry. So the server just needs to read multiple tables (if present), and pick the newest version amongst all rows. Thus, the new feature is compatible with older versions without any schema changes. I think it is better than automatically creating copies of the table for all engines; it is often the case that an engine (eg. TokuDB) is available in the server binary, but the user does not want any tables created in that engine. This means that no speedup is obtained if the user does not create additional mysql.gtid_slave_pos* tables. (However, things should still work fine, thanks to XA synchronising cross-engine transactions). - Kristian.

6 12

[Maria-developers] mariadb_config broken in 10.2.6
by Jeroen Ooms 07 Jul '17

07 Jul '17

Running 'mariadb_config --libs' returns invalid output. See https://github.com/Homebrew/homebrew-core/issues/15333

3 3

[Maria-developers] Confusion between variable / alias and row type.
by jerome brauge 07 Jul '17

07 Jul '17

Hello Alexander, I think there is a regression introduced with row type. Attached stored procedure creation fail with error : ERROR 1105 (HY000) at line 9: 'a' is not a row variable. Regards, Jérôme.

2 2

[Maria-developers] Review of PERCENTILE_CONT && PERCENTILE_DISC
by Vicențiu Ciorbaru 04 Jul '17

04 Jul '17

Hi Varun, Here's the review for your patch. > diff --git a/mysql-test/t/percentile.test b/mysql-test/t/percentile.test > new file mode 100644 > index 00000000000..0958fc05e7d > --- /dev/null > +++ b/mysql-test/t/percentile.test > @@ -0,0 +1,41 @@ Whenever you are adding a test case, please add the result file too. It makes things easier to review. > +CREATE TABLE student (name CHAR(10), test double, score TINYINT); > + > +INSERT INTO student VALUES > +('Chun', 0, null), ('Chun', 0, 4), > +('Esben', 1, null), ('Esben', 1, null), > +('Kaolin', 0.5, 56), ('Kaolin', 0.5, 88), > +('Tatiana', 0.8, 2), ('Tatiana', 0.8, 1); > + > + > + > +select name, percentile_disc(0.6) within group(order by score) over (partition by name) from student; > +select name, percentile_disc(test) within group(order by score) over (partition by name) from student; > +select name, percentile_disc(0.4) within group(order by score) over (partition by name) from student; > + > + > +#select name, percentile_cont(null) within group(order by score) over (partition by name) from student; > +#select name, cume_dist() over (partition by name order by score) from student; > + > + > +#normal parsing > +#select percentile_cont(0.5) within group(order by score) over w1 from student > +#window w1 AS (partition by name); > + > +# no partition clause > +#select percentile_cont(0.5) within group(order by score) over () from student; > + > + > +# only one sort allowed > +#select percentile_cont(0.5) within group(order by score) over (partition by name); > + > +#parameter value should be in the range of 0 to 1 > +#select percentile_cont(1.5) within group(order by score) over (partition by name); > + > + > +# > +#select rank() over (partition by name order by score ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) from student; > + > + > + > +drop table student; > + > diff --git a/sql/item.h b/sql/item.h > index 76ce4aa935f..3df4ec4f358 100644 > --- a/sql/item.h > +++ b/sql/item.h > @@ -5215,6 +5217,14 @@ class Cached_item_item : public Cached_item > cmp(); > item= save; > } This is not used in the patch. Is there a need for it any more? > + Item* get_item() > + { > + return item; > + } Add an empty line here here. > + void clear() > + { > + null_value= false; > + } > }; > > class Cached_item_str :public Cached_item_item > @@ -5236,6 +5250,12 @@ class Cached_item_real :public Cached_item_item > Cached_item_real(Item *item_par) :Cached_item_item(item_par),value(0.0) {} > bool cmp(void); > int cmp_read_only(); Space after ). Mark this function as const > + double get_value(){ return value;} I don't quite get what this clear is supposed to do. I'd expect clear to set the null_value to true, not to false. Same for other functions in the diff. Add a comment explaining this. > + void clear() > + { Space after = > + value=0.0; > + null_value= false; > + } > }; > > class Cached_item_int :public Cached_item_item > @@ -5245,6 +5265,12 @@ class Cached_item_int :public Cached_item_item > Cached_item_int(Item *item_par) :Cached_item_item(item_par),value(0) {} > bool cmp(void); > int cmp_read_only(); Space after ). Mark this function as const. > + longlong get_value(){ return value;} > + void clear() > + { Space after = > + value=0.0; > + null_value= false; > + } > }; > > > @@ -5255,6 +5281,12 @@ class Cached_item_decimal :public Cached_item_item > Cached_item_decimal(Item *item_par); > bool cmp(void); > int cmp_read_only(); Space after ) Mark this function as const. > + my_decimal *get_value(){ return &value;}; > + void clear() > + { > + null_value= false; > + my_decimal_set_zero(&value); > + } > }; > > class Cached_item_field :public Cached_item > diff --git a/sql/item_windowfunc.cc b/sql/item_windowfunc.cc > index 6ab903a81bb..b711642c52c 100644 > --- a/sql/item_windowfunc.cc > +++ b/sql/item_windowfunc.cc > @@ -93,6 +93,17 @@ Item_window_func::fix_fields(THD *thd, Item **ref) > my_error(ER_NO_ORDER_LIST_IN_WINDOW_SPEC, MYF(0), window_func()->func_name()); > return true; > } > + > + if (only_single_element_order_list()) > + { Mark such comments as TODO, like so: TODO(varunt): <message> > + // need to change the error, the error should say that we have more than one element in the order list > + if (window_spec->order_list->elements != 1) > + { > + my_error(ER_NO_ORDER_LIST_IN_WINDOW_SPEC, MYF(0), window_func()->func_name()); > + return true; > + } > + } > + > /* > TODO: why the last parameter is 'ref' in this call? What if window_func > decides to substitute itself for something else and does *ref=.... ? > @@ -179,6 +190,16 @@ void Item_sum_dense_rank::setup_window_func(THD *thd, Window_spec *window_spec) > clear(); > } > > I don't see any added value for this extra setup_percentile_func, when it's only I guess you can just put the code in directly. > +void Item_sum_percentile_disc::setup_window_func(THD *thd, Window_spec *window_spec) > +{ > + setup_percentile_func(thd, window_spec->order_list); > +} > + This function does nothing. get_handler_by_cmp_type returns an item and then you don't do anything with it. > +void Item_sum_percentile_disc::set_type_handler(Window_spec *window_spec) > +{ > + type_handler()->get_handler_by_cmp_type(window_spec->order_list->first->item[0]->result_type()); > +} > + > bool Item_sum_dense_rank::add() > { > if (peer_tracker->check_if_next_group() || first_add) > diff --git a/sql/item_windowfunc.h b/sql/item_windowfunc.h > index 77cbd556e60..5179d1832e8 100644 > --- a/sql/item_windowfunc.h > +++ b/sql/item_windowfunc.h > @@ -16,10 +17,13 @@ class Group_bound_tracker > > Group_bound_tracker(THD *thd, SQL_I_List<ORDER> *list) > { As discussed on slack, lets keep this Group_bound_tracker as is. Better yet, add a DBUG_ASSERT(list). > - for (ORDER *curr = list->first; curr; curr=curr->next) > + if (list) > { > - Cached_item *tmp= new_Cached_item(thd, curr->item[0], TRUE); > - group_fields.push_back(tmp); > + for (ORDER *curr = list->first; curr; curr=curr->next) > + { > + Cached_item *tmp= new_Cached_item(thd, curr->item[0], TRUE); > + group_fields.push_back(tmp); > + } > } > } > > @@ -71,6 +75,19 @@ class Group_bound_tracker > return 0; > } > Add a comment as to what this function does. I don't quite like how this function works. cmp has a side effect of overwriting the cached item with the compared value. In addition this function stops if we encounter a null value and the group fields are now partially updated. I assume this is only for a single element order list. Why not make something specialised for that and not use this cached_item construct. > + bool compare_with_cache_for_null_values() > + { > + List_iterator<Cached_item> li(group_fields); > + Cached_item *ptr; > + while ((ptr= li++)) > + { > + ptr->cmp(); > + if (ptr->null_value) > + return true; > + } > + return false; > + } > + > private: > List<Cached_item> group_fields; > /* > @@ -678,6 +704,138 @@ class Item_sum_ntile : public Item_sum_window_with_row_count > ulong current_row_count_; > }; > > +class Item_sum_percentile_disc : public Item_sum_cume_dist, > + public Type_handler_hybrid_field_type > +{ > +public: > + Item_sum_percentile_disc(THD *thd, Item* arg) : Item_sum_cume_dist(thd, arg), > + Type_handler_hybrid_field_type(&type_handler_longlong), > + value(NULL), result_value(NULL), val_calculated(FALSE) {} > + > + double val_real() > + { > + if (get_row_count() == 0 || get_arg(0)->is_null()) > + { > + null_value= true; > + return 0; > + } > + null_value= false; Look at Item_sum_hybrid_simple. See how Item_cache * works there. Make use of that instead of Cached_item_xxx. > + return ((Cached_item_real*) result_value)->get_value(); > + } > + > + longlong val_int() > + { > + if (get_row_count() == 0 || get_arg(0)->is_null()) > + { > + null_value= true; > + return 0; > + } > + null_value= false; Same comment as for val_real() > + return ((Cached_item_int*) result_value)->get_value(); > + } > + > + my_decimal* val_decimal(my_decimal* dec) > + { > + if (get_row_count() == 0 || get_arg(0)->is_null()) > + { > + null_value= true; > + return 0; > + } > + null_value= false; Same comment as for val_decimal() > + return ((Cached_item_decimal*) result_value)->get_value(); > + } > + Lets try implementing this with Item_cache, instead of cached_item. Cached_item is designed for comparison, while Item_cache is used for storing something just to cache it. > + bool add() > + { > + Item *arg = get_arg(0); > + if (arg->is_null()) > + return true; > + /* > + need to ensure that the Item arg is constant across the entire partition > + and its value ranges between [0,1] > + */ > + value->cmp(); > + > + /* for the null values of the row, we dont count take those rows in account for calculating > + the CUME_DIST */ > + > + if(value->null_value) > + return false; > + > + Item_sum_cume_dist::add(); > + double val1= Item_sum_cume_dist::val_real(); > + /* need to check type and return value accordingly*/ > + double val2 =arg->val_real_from_decimal(); > + > + /* use Cached_item to do the comparision using cmp_read_only() */ > + > + if( val1 >= val2 && !val_calculated) > + { > + val_calculated= true; > + result_value->cmp(); > + return false; > + } > + return false; > + } > + > + enum Sumfunctype sum_func() const > + { > + return PERCENTILE_DISC_FUNC; > + } > + > + void clear() > + { > + val_calculated= false; > + value->clear(); > + result_value->clear(); > + Item_sum_cume_dist::clear(); > + } > + > + const char*func_name() const > + { > + return "percentile_disc"; > + } > + > + void update_field() {} > + void set_type_handler(Window_spec *window_spec); > + const Type_handler *type_handler() const > + {return Type_handler_hybrid_field_type::type_handler();} > + > + void fix_length_and_dec() > + { > + decimals = 10; // TODO-cvicentiu find out how many decimals the standard > + // requires. > + } > + > + Item *get_copy(THD *thd, MEM_ROOT *mem_root) > + { return get_item_copy<Item_sum_percentile_disc>(thd, mem_root, this); } > + void setup_window_func(THD *thd, Window_spec *window_spec); > + void setup_percentile_func(THD *thd, SQL_I_List<ORDER> *list) > + { > + value= new_Cached_item(thd, list->first->item[0], FALSE); > + result_value= new_Cached_item(thd, list->first->item[0], FALSE); > + } > + void cleanup() > + { > + if (value) > + { > + delete value; > + value= NULL; > + } > + if(result_value) > + { > + delete result_value; > + result_value= NULL; > + } > + Item_sum_num::cleanup(); > + } > + > +private: > + Cached_item *value; > + Cached_item *result_value; > + bool val_calculated; > +}; > + > > class Item_window_func : public Item_func_or_sum > { > diff --git a/sql/sql_window.cc b/sql/sql_window.cc > index 74e1ad25183..9fda4520a05 100644 > --- a/sql/sql_window.cc > +++ b/sql/sql_window.cc > @@ -868,9 +873,39 @@ class Partition_read_cursor : public Table_read_cursor > } > return 0; > } This is not something we want in Partition_read_cursor. Move the contents of it to the most derived class that needs it. > + bool next_func(ha_rows *counter) > + { > + if (next()) > + return true; > + if (!check_for_null_row()) > + { > + (*counter)++; > + } > + return false; > + } This is also not something we want in Partition_read_cursor. > + bool fetch_func(ha_rows *counter) > + { > + if (fetch()) > + return true; > + if (!check_for_null_row()) > + { > + (*counter)++; > + } > + return false; > + } Null row here has no meaning. Partition_read_cursor looks at the bound_tracker for partitions. What "row" are we talking about here. Then again, this logic can probably go into our "special" frame cursor below. > + bool check_for_null_row() > + { > + if (!end_of_partition) > + { Check compare_with_cache_for_null_values comment. > + if (order_tracker.compare_with_cache_for_null_values()) > + return true; > + } > + return false; > + } > > private: > Group_bound_tracker bound_tracker; Not something we want in Partition_read_cursor. If you need an order_tracker define a different class. > + Group_bound_tracker order_tracker; > bool end_of_partition; > }; > > @@ -1102,7 +1137,7 @@ class Frame_range_n_top : public Frame_cursor > SQL_I_List<ORDER> *partition_list, > SQL_I_List<ORDER> *order_list, > bool is_preceding_arg, Item *n_val_arg) : So this gets reverted as discussed on slack. > - cursor(thd, partition_list), n_val(n_val_arg), item_add(NULL), > + cursor(thd, partition_list, NULL), n_val(n_val_arg), item_add(NULL), > is_preceding(is_preceding_arg) > { > DBUG_ASSERT(order_list->elements == 1); > @@ -1241,7 +1276,7 @@ class Frame_range_n_bottom: public Frame_cursor > SQL_I_List<ORDER> *partition_list, > SQL_I_List<ORDER> *order_list, > bool is_preceding_arg, Item *n_val_arg) : So this gets reverted as discussed on slack. > - cursor(thd, partition_list), n_val(n_val_arg), item_add(NULL), > + cursor(thd, partition_list, NULL), n_val(n_val_arg), item_add(NULL), > is_preceding(is_preceding_arg), added_values(false) > { > DBUG_ASSERT(order_list->elements == 1); > @@ -1371,7 +1406,7 @@ class Frame_range_current_row_bottom: public Frame_cursor > Frame_range_current_row_bottom(THD *thd, > SQL_I_List<ORDER> *partition_list, > SQL_I_List<ORDER> *order_list) : So this gets reverted as discussed on slack. > - cursor(thd, partition_list), peer_tracker(thd, order_list) > + cursor(thd, partition_list, NULL), peer_tracker(thd, order_list) > { > } > > @@ -1658,6 +1693,35 @@ class Frame_unbounded_following_set_count : public Frame_unbounded_following > } > }; > As discussed on slack, this needs to be renamed to something that makes sense. > +class Frame_unbounded_following_set_count_special : public Frame_unbounded_following_set_count > +{ > +public: > + Frame_unbounded_following_set_count_special( > + THD *thd, > + SQL_I_List<ORDER> *partition_list, SQL_I_List<ORDER> *order_list) : > + Frame_unbounded_following_set_count(thd, partition_list, order_list) > + {} > + next_partition can have the logic of fetch_func directly in it. > + void next_partition(ha_rows rownum) > + { > + ha_rows num_rows_in_partition= 0; > + if (cursor.fetch_func(&num_rows_in_partition)) > + return; > + Dangerous to have a while loop like this. Lets put next_func code here directly. > + /* Walk to the end of the partition, find how many rows there are. */ > + while (!cursor.next_func(&num_rows_in_partition)); > + > + List_iterator_fast<Item_sum> it(sum_functions); > + Item_sum* item; > + while ((item= it++)) > + { > + Item_sum_window_with_row_count* item_with_row_count = > + static_cast<Item_sum_window_with_row_count *>(item); > + item_with_row_count->set_row_count(num_rows_in_partition); > + } > + } > +}; > + > ///////////////////////////////////////////////////////////////////////////// > // ROWS-type frame bounds > ///////////////////////////////////////////////////////////////////////////// > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index a0bbf39b138..6c64477f08d 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -10671,6 +10676,45 @@ simple_window_func: > } > ; > > +inverse_distribution_function: > + inverse_distribution_function_def WITHIN GROUP_SYM > + '(' > + { Select->prepare_add_window_spec(thd); } > + order_by_single_element_list ')' OVER_SYM > + '(' opt_window_ref opt_window_partition_clause ')' > + { > + LEX *lex= Lex; > + if (Select->add_window_spec(thd, lex->win_ref, > + Select->group_list, > + Select->order_list, > + NULL)) > + MYSQL_YYABORT; > + $$= new (thd->mem_root) Item_window_func(thd, (Item_sum *) $1, > + thd->lex->win_spec); > + if ($$ == NULL) > + MYSQL_YYABORT; > + if (Select->add_window_func((Item_window_func *) $$)) > + MYSQL_YYABORT; > + } > + ; > + > +inverse_distribution_function_def: > + PERCENTILE_CONT_SYM '(' expr ')' Indentation for { and } is not correct here. Please check this file and do the same. > + { mark it as TODO(varun): > + //Not yet started implementing > + } > + | PERCENTILE_DISC_SYM '(' expr ')' Same, check indentation for { and } > + { > + $$= new (thd->mem_root) Item_sum_percentile_disc(thd, $3); > + if ($$ == NULL) > + MYSQL_YYABORT; > + } > + ; > + Indentation check! > +order_by_single_element_list: > + ORDER_SYM BY order_list > + ; > + > window_name: > ident > {

1 0

[Maria-developers] MDEV-13240 Wrong warning with MAX(datetime_field) OVER (...)
by Alexander Barkov 04 Jul '17

04 Jul '17

Hello Vicentiu, can you please review a patch for MDEV-13240? The patch is for bb-10.2-ext / 10.3. Thanks.

1 0

Re: [Maria-developers] [Commits] 2d6649aa24c: MDEV-13186: main.win failure post MDEV-12336
by Oleksandr Byelkin 04 Jul '17

04 Jul '17

Hi, Vicentiu! Am 27.06.2017 um 11:31 schrieb vicentiu: > revision-id: 2d6649aa24c77321df349de5cee576dab1e344f4 (mariadb-10.2.6-55-g2d6649aa24c) > parent(s): c036d5ada7a5816d25915a2c6e0b6836b0cd6743 > author: Vicențiu Ciorbaru > committer: Vicențiu Ciorbaru > timestamp: 2017-06-27 12:26:38 +0300 > message: > > MDEV-13186: main.win failure post MDEV-12336 > > During statement preparation st_order::item gets set to a value in > ref_ptr_array. During statement execution we were overriding that value, > causing subsequent checks for window functions to return true. > > Whenever we do any setting from ref_ptr_array, make sure to always > store the value in all_fields as well. > > For function items containing window functions, as MDEV-12336 has > discovered, we don't need to create a separate Item_direct_ref or > Item_aggregate_ref as they will be computed directly from the top-level > item once the window function argument columns are computed. OK to push but add comment before IF with return explaining it. > > --- > mysql-test/r/win.result | 12 ++++++++++++ > mysql-test/t/win.test | 10 ++++++++++ > sql/item.cc | 6 +++--- > 3 files changed, 25 insertions(+), 3 deletions(-) > > diff --git a/mysql-test/r/win.result b/mysql-test/r/win.result > index ec83998aa98..a67c336e5b4 100644 > --- a/mysql-test/r/win.result > +++ b/mysql-test/r/win.result > @@ -3085,3 +3085,15 @@ select max(id), rank() over (order by max(id)) from t1 where id < 3; > max(id) rank() over (order by max(id)) > 2 1 > drop table t1; > +# > +# main.win failure post MDEV-12336 > +# > +create table t(a decimal(35,10), b int); > +insert into t values (1, 10), (2, 20), (3, 30); > +prepare stmt from "SELECT (CASE WHEN sum(t.a) over (partition by t.b)=1 THEN 1000 ELSE 300 END) AS a FROM t"; > +execute stmt; > +a > +1000 > +300 > +300 > +drop table t; > diff --git a/mysql-test/t/win.test b/mysql-test/t/win.test > index 95d32c5bd14..0ecb2b35074 100644 > --- a/mysql-test/t/win.test > +++ b/mysql-test/t/win.test > @@ -1877,3 +1877,13 @@ select count(max(id)) over (order by max(id)) from t1 where id < 3; > select max(id), rank() over (order by max(id)) from t1 where id < 3; > > drop table t1; > + > +--echo # > +--echo # main.win failure post MDEV-12336 > +--echo # > +create table t(a decimal(35,10), b int); > +insert into t values (1, 10), (2, 20), (3, 30); > + > +prepare stmt from "SELECT (CASE WHEN sum(t.a) over (partition by t.b)=1 THEN 1000 ELSE 300 END) AS a FROM t"; > +execute stmt; > +drop table t; > diff --git a/sql/item.cc b/sql/item.cc > index df615b5ace9..05e5fd1ce45 100644 > --- a/sql/item.cc > +++ b/sql/item.cc > @@ -1920,6 +1920,9 @@ void Item::split_sum_func2(THD *thd, Ref_ptr_array ref_pointer_array, > point to the temporary table. > */ > split_sum_func(thd, ref_pointer_array, fields, split_flags); > + if (type() == FUNC_ITEM) { > + return; > + } > } > else > { > @@ -1979,9 +1982,6 @@ void Item::split_sum_func2(THD *thd, Ref_ptr_array ref_pointer_array, > &ref_pointer_array[el], 0, name)))) > return; // fatal_error is set > } > - else if (type() == FUNC_ITEM && > - ((Item_func *) this)->with_window_func) > - return; > else > { > if (!(item_ref= (new (thd->mem_root) > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

[Maria-developers] Project Progress Report[PHASE-1]
by ANKIT KUMAR 30 Jun '17

30 Jun '17

Hi, I have newly started contributing to opensource projects, so these 3.5 weeks have been really awesome from learning point of view. Everyday I learnt something new, kept experimenting various approaches to a particular problem. I have come a long way as I am about to finish 2 subtasks among 4(4th being optional as it already does what it's expected to). Whole MariaDB community has been really helpful, and special thanks to my Mentor without whom support it wouldn't have been possible. Just recollecting everything I did in this phase- 1. Learnt how bison works, how grammar is written, how actions can be written(w.r.t to extended syntax), how parsing is done, 2. Implemented Extended CREATE USER and GRANT USER syntax. 3. Created tests for the acceptance of newly implemented grammar. 4. Designed in-memory structures that will store value of plugins and perform AND/OR operations as per precedence and authentication will be provided accordingly. 5. Wrote actions for this complex syntax(fixed some of the errors, some are yet to be fixed). In the remaining 3-4 days of Phase-1, I will try to complete this bridge between grammar and in-memory structure after which storing data in memory will be done. Now coming on to phase-2 deliverables, I am planning to finish off with Storage on Disk. We have found a simplest solution to get this done(probably in a week or 2), and in the remaining days, we will check the working of this new feature. I will then migrate this code to 10.3 branch. (Phase-3 for unexpected delay) This project involve lots of thinking than actual coding, so I think I have maintained good pace throughout. I believe I will maintain this enthusiasm and confidence level in upcoming phases too. Hopefully, I will get a chance to continue with this project. Kind regards, Ankit Kumar

2 2

[Maria-developers] Fwd: ecbad0a07a8: MDEV-13189: Window functions crash when using INTERVAL
by Vicențiu Ciorbaru 28 Jun '17

28 Jun '17

Hi Sergey, Can you please review this change that makes window functions work with INTERVAL function? Thanks, Vicentiu ---------- Forwarded message --------- From: vicentiu <vicentiu(a)mariadb.org> Date: Wed, 28 Jun 2017 at 21:08 Subject: ecbad0a07a8: MDEV-13189: Window functions crash when using INTERVAL To: <commits(a)mariadb.org> revision-id: ecbad0a07a848c1f9f474c4c749c87b6bfe7986c (mariadb-10.2.6-74-gecbad0a07a8) parent(s): 31ba0fa48d27715e82258b1e74401093e0ee17a2 author: Vicențiu Ciorbaru committer: Vicențiu Ciorbaru timestamp: 2017-06-28 21:07:55 +0300 message: MDEV-13189: Window functions crash when using INTERVAL Interval function makes use of Item_row. Item_row did not correctly mark with_window_func flag according to its arguments. Fix it by making Item_row aware of this flag. --- mysql-test/r/win.result | 14 ++++++++++++++ mysql-test/t/win.test | 9 +++++++++ sql/item_row.cc | 1 + 3 files changed, 24 insertions(+) diff --git a/mysql-test/r/win.result b/mysql-test/r/win.result index 952bd82ac7f..7f8777ed491 100644 --- a/mysql-test/r/win.result +++ b/mysql-test/r/win.result @@ -3127,3 +3127,17 @@ NULL 1 0 1 0 1 2 0 1 drop table t1; +# +# MDEV-13189: Window functions crash when using INTERVAL function +# +create table t1(i int); +insert into t1 values (1),(2),(10),(20),(30); +select sum(i) over (order by i), interval(sum(i) over (order by i), 10, 20) +from t1; +sum(i) over (order by i) interval(sum(i) over (order by i), 10, 20) +1 0 +3 0 +13 1 +33 2 +63 2 +drop table t1; diff --git a/mysql-test/t/win.test b/mysql-test/t/win.test index 45653b18682..aff717e3151 100644 --- a/mysql-test/t/win.test +++ b/mysql-test/t/win.test @@ -1914,3 +1914,12 @@ select max(i) over (order by i), max(i) over (order by i) is not null from t1; drop table t1; + +--echo # +--echo # MDEV-13189: Window functions crash when using INTERVAL function +--echo # +create table t1(i int); +insert into t1 values (1),(2),(10),(20),(30); +select sum(i) over (order by i), interval(sum(i) over (order by i), 10, 20) +from t1; +drop table t1; diff --git a/sql/item_row.cc b/sql/item_row.cc index ddbb0736d54..fc484f560ee 100644 --- a/sql/item_row.cc +++ b/sql/item_row.cc @@ -63,6 +63,7 @@ bool Item_row::fix_fields(THD *thd, Item **ref) } maybe_null|= item->maybe_null; with_sum_func= with_sum_func || item->with_sum_func; + with_window_func = with_window_func || item->with_window_func; with_field= with_field || item->with_field; with_subselect|= item->with_subselect; }

1 0

[Maria-developers] MDEV-13197 Parser refactoring for CREATE VIEW, TRIGGER, SP, UDF, EVENT
by Alexander Barkov 28 Jun '17

28 Jun '17

Hello Sanja, Can you please review a patch for MDEV-13197? It's a dependency change for MDEV-10591 Oracle-style packages, which I'm currently working on. The patch is for bb-10.2-ext. Thanks.

1 0

[Maria-developers] MDEV-12179: Per-engine mysql.gtid_slave_pos: auto-configuring/packaging
by Sergey Petrunia 26 Jun '17

26 Jun '17

Hello, Suppose there is a transactional storage engine that is shipped as a loadable module. The examples are MyRocks and TokuDB. I think it the default behavior for such engine after MDEV-12179 should be that the engine is listed in @@gtid_pos_auto_engines. That is, as soon as a transaction that uses the storage engine is replicated, mysql.slave_gtid_pos{_$engine} is created for it. gtid_pos_auto_engines makes it easy to do this manually, but it's difficult to achieve as default behavior. Here's an explanation why. Loadable storage engines are put into their separate packages, for example mariadb-plugin-rocksdb_10.2.7+maria-zesty_amd64.deb. The two important files in the package are : ./etc/mysql/mariadb.conf.d/rocksdb.cnf ./usr/lib/mysql/plugin/ha_rocksdb.so The first one is a .cnf that gets included into the server's cnf file. It would be nice to put there something like SET global gtid_pos_auto_engines= CONCAT(@@gtid_pos_auto_engines, 'rocksdb'); but that is not possible (it's a .cnf not .sql) Possible solutions: == [Ab]use the Storage Engine API == Make the storage engine put itself in @@gtid_pos_auto_engines. I think this is technically possible, but - It's a storage engine API abuse - how does one explain @@gtid_pos_auto_engines to the users? == Let gtid_pos_auto_engines include unknown engine == - Allow @@gtid_pos_auto_engines to list unknown engines - set the default to include all known transactional storage engines, that is it should be : gtid_pos_auto_engines='InnoDB,TokuDB,RocksDB' // (*) and the server should allow this even if it is built without TokuDB or RocksDB (*) -- Aria is not needed as it is not a "true" transactional storage engine as far as interplay with the binlog is concerned. == Let packaging use some other solution == I suppose it's possible for packaging system to run some script after a package has been installed. that script might find/adjust my.cnf file to have the desired @@gtid_pos_auto_engines setting. This solution might be complex and fragile, though. Any thoughts ? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

2 1

[Maria-developers] exact commands to produce similar mariadb release tarball
by Phil Sweeney 26 Jun '17

26 Jun '17

Hi there, I'm trying to do some custom builds of MariaDB for myself, but want to emulate the release builds as much as possible. I'm doing this on Ubuntu 16.04 LTS and following steps here: https://mariadb.com/kb/en/mariadb/creating-the-mariadb-binary-tarball/ However, my tarball ends up being 227MB compared to the downloads that are ~500MB, plus it won't start because it's missing jemalloc. Where can I find info about the environment and the commandline used to build the official releases e.g. mariadb-10.1.24-linux-glibc_214-x86_64.tar.gz (535.6MB) Thanks! Phil

2 2

Re: [Maria-developers] [Commits] ee73b33: MDEV-8840: ANALYZE FORMAT=JSON produces wrong data with BKA
by Sergey Petrunia 25 Jun '17

25 Jun '17

Hi Varun, On Sat, Jun 17, 2017 at 05:35:41PM +0530, Varun wrote: > revision-id: ee73b331d97d33ae7aa1403a4ac7312f2d261754 (mariadb-10.1.20-323-gee73b33) > parent(s): 056bab0880544d91ea67d18fe8db65b4f6625482 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2017-06-17 17:34:07 +0530 > message: > > MDEV-8840: ANALYZE FORMAT=JSON produces wrong data with BKA > > The issue was that r_loops, r_rows and r_filtered in ANALYZE FORMAT= JSON were not > calculated for the table on which we were performing the MRR scan in the BKA join > Fixed this by adding respective counter in the JOIN_CACHE_MRR::open and JOIN_CACHE::next > So I apply the patch and run this example (the dataset is from explain_json.test, also pasted below): MariaDB [j5]> analyze format=json select * from t3,t4 where t3.a=t4.a and (t4.c+1 < t3.b+1)\G *************************** 1. row *************************** ANALYZE: { "query_block": { "select_id": 1, "r_loops": 1, "r_total_time_ms": 0.8859, "table": { "table_name": "t3", "access_type": "ALL", "r_loops": 1, "rows": 10, "r_rows": 10, "r_total_time_ms": 0.0202, "filtered": 100, "r_filtered": 100, "attached_condition": "t3.a is not null" }, "block-nl-join": { "table": { "table_name": "t4", "access_type": "ref", "possible_keys": ["a"], "key": "a", "key_length": "5", "used_key_parts": ["a"], "ref": ["j5.t3.a"], "r_loops": 1, "rows": 1, "r_rows": 11, "r_total_time_ms": 0.0379, "filtered": 100, "r_filtered": 90.909 }, "buffer_type": "flat", "buffer_size": "256Kb", "join_type": "BKA", "mrr_type": "Rowid-ordered scan", "attached_condition": "t4.c + 1 < t3.b + 1", "r_filtered": 0 } } } I think, these value are wrong: > "r_rows": 11, > "r_filtered": 90.909 r_rows is actually 10, r_filtered=100 (no attached condition) I guess wrong r_filtered is caused by r_rows being wrong. ## Dataset: set storage_engine=myisam; set default_storage_engine=myisam; create table t1(a int); insert into t1 values (0),(1),(2),(3),(4),(5),(6),(7),(8),(9); create table t2(a int); insert into t2 select A.a + B.a* 10 + C.a * 100 from t1 A, t1 B, t1 C; create table t3(a int, b int); insert into t3 select a,a from t1; create table t4(a int, b int, c int, filler char(100), key (a,b)); insert into t4 select a,a,a, 'filler-data' from t2; set optimizer_switch='mrr=on'; set join_cache_level=6; BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] [Commits] 62021f3: MDEV-12070 - Introduce thd_query_safe() from MySQL 5.7
by Sergei Golubchik 23 Jun '17

23 Jun '17

Hi, Sergey! On Jun 22, Sergey Vojtovich wrote: > revision-id: 62021f391a42c5577190aa43cb8ad91e56235b46 (mariadb-10.2.6-57-g62021f3) > parent(s): 557e1bd472612848a42e772c1fb6f8ed32ab33b4 > committer: Sergey Vojtovich > timestamp: 2017-06-22 17:15:10 +0400 > message: > > MDEV-12070 - Introduce thd_query_safe() from MySQL 5.7 > > Merged relevant part of MySQL revision: > https://github.com/mysql/mysql-server/commit/565d20b44f24fcc855dc616164d87b… > > diff --git a/sql/sql_class.cc b/sql/sql_class.cc > index f8cf829..db65eb3 100644 > --- a/sql/sql_class.cc > +++ b/sql/sql_class.cc > @@ -4529,6 +4529,32 @@ extern "C" LEX_STRING * thd_query_string (MYSQL_THD thd) > return(&thd->query_string.string); > } > > + > +/** > + Get the current query string for the thread. > + > + @param thd The MySQL internal thread pointer > + @param buf Buffer where the query string will be copied > + @param buflen Length of the buffer > + > + @return Length of the query > + > + @note This function is thread safe as the query string is > + accessed under mutex protection and the string is copied > + into the provided buffer. @see thd_query_string(). > +*/ > + > +extern "C" size_t thd_query_safe(MYSQL_THD thd, char *buf, size_t buflen) > +{ > + mysql_mutex_lock(&thd->LOCK_thd_data); > + size_t len= MY_MIN(buflen - 1, thd->query_length()); > + memcpy(buf, thd->query(), len); > + mysql_mutex_unlock(&thd->LOCK_thd_data); > + buf[len]= '\0'; > + return len; > +} Okay. But as neither thd_query_string() nor thd_query() are part of the plugin API, I'd suggest to remove them, they're inherently unsafe and should not be used. Few other engines use thd_query_string() need to be fixed too (but they bypass the plugin API and we don't promise stability for internal functions). Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 2

Re: [Maria-developers] [Commits] 922f970: MDEV-12882 - Assertion failed in MDL_context::upgrade_shared_lock
by Sergei Golubchik 23 Jun '17

23 Jun '17

Hi, Sergey! On Jun 22, Sergey Vojtovich wrote: > revision-id: 922f9700a7fb084e3f7d6cff56e27e24fc603ed7 (mariadb-10.2.6-58-g922f970) > parent(s): 62021f391a42c5577190aa43cb8ad91e56235b46 > committer: Sergey Vojtovich > timestamp: 2017-06-22 17:35:36 +0400 > message: > > MDEV-12882 - Assertion failed in MDL_context::upgrade_shared_lock > > Relaxed assertion (in MySQL it was removed). > For "LOCK TABLES t1 WRITE CONCURRENT, t1 READ" upgrade lock to weakest > existing suitable lock, which is MDL_SHARED_NO_READ_WRITE. > > diff --git a/mysql-test/t/mdl.test b/mysql-test/t/mdl.test > new file mode 100644 > index 0000000..5a74ae5 > --- /dev/null > +++ b/mysql-test/t/mdl.test > @@ -0,0 +1,15 @@ Hm. I'd totally expect main.mdl test to exist for years :) > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > index 412cd1d..7c910e8 100644 > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -2787,6 +2787,7 @@ static bool lock_tables_open_and_lock_tables(THD *thd, TABLE_LIST *tables) > ! table->prelocking_placeholder && > table->table->file->lock_count() == 0) > { > + enum enum_mdl_type lock_type; > /* > In case when LOCK TABLE ... READ LOCAL was issued for table with > storage engine which doesn't support READ LOCAL option and doesn't > @@ -2799,9 +2800,12 @@ static bool lock_tables_open_and_lock_tables(THD *thd, TABLE_LIST *tables) > deadlock_handler.init(); > thd->push_internal_handler(&deadlock_handler); > > + lock_type= table->table->mdl_ticket->get_type() == MDL_SHARED_WRITE ? > + MDL_SHARED_NO_READ_WRITE : MDL_SHARED_READ_ONLY; > + > bool result= thd->mdl_context.upgrade_shared_lock( > table->table->mdl_ticket, > - MDL_SHARED_READ_ONLY, > + lock_type, > thd->variables.lock_wait_timeout); This only works if you lock WRITE, then READ. If you do it the other way around, you'll get an error. Isn't it inconsistent? Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 1

[Maria-developers] Condition pushdown into derived tables and outer joins.
by Sergey Petrunia 23 Jun '17

23 Jun '17

Hello Igor, While reviewing MDEV-12845, I've found this piece in sql_select.cc: while ((tbl= li++)) { /* Do not push conditions from where into materialized inner tables of outer joins: this is not valid. */ if (tbl->is_materialized_derived()) { /* Do not push conditions from where into materialized inner tables of outer joins: this is not valid. */ if (!tbl->is_inner_table_of_outer_join()) Q1: Is int meaningful to have two identical comments? Q2: It is not valid push parts of WHERE, but it is valid to push the parts of ON expression. An example: explain format=json select * from t21 LEFT JOIN (select a,b, COUNT(*) as CNT from t22 group by a,b) TBL ON (t21.a=TBL.a AND TBL.a<5); here, "TBL.a<5" can be pushed down into the subquery, but it is not. EXPLAIN EXTENDED confirms it: "query_block": { "select_id": 1, "const_condition": "1", "table": { "table_name": "t21", "access_type": "ALL", "rows": 10, "filtered": 100 }, "table": { "table_name": "<derived2>", "access_type": "ref", "possible_keys": ["key0"], "key": "key0", "key_length": "5", "used_key_parts": ["a"], "ref": ["test.t21.a"], "rows": 2, "filtered": 100, "attached_condition": "trigcond(trigcond(t21.a < 5 and t21.a is not null))", "materialized": { "query_block": { "select_id": 2, "filesort": { "sort_key": "t22.a, t22.b", "temporary_table": { "table": { "table_name": "t22", "access_type": "ALL", "rows": 10, "filtered": 100 } } } } } } Just in case, the commands to create the dataset were: create table ten(a int); insert into ten values (0),(1),(2),(3),(4),(5),(6),(7),(8),(9); create table t21 (a int, b int, c int); insert into t21 select a,a,a from ten; create table t22 (a int, b int, c int); insert into t22 select a,a,a from ten; BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Project Progress Report [Week 3]
by ANKIT KUMAR 20 Jun '17

20 Jun '17

Hi, In this week, we worked on in-memory structures:- Which representation to choose(prefix or expression tree), How to store values and how authentication will be done. I have prepared pseudocode to implement this subtask and have made implemented some part of code. Hopefully by 25th May, I will finish off with >50% of the project. I will declare Phase-2 deliverables next Monday(26 May). Kind regards, Ankit Kumar

2 1

Re: [Maria-developers] MDEV-12620 - set lock_wait_timeout = 1; flush tables with read lock;
by Sergei Golubchik 16 Jun '17

16 Jun '17

Hi, Sergey! On Jun 16, Sergey Vojtovich wrote: > revision-id: 056bab0880544d91ea67d18fe8db65b4f6625482 (mariadb-10.1.24-22-g056bab0) > parent(s): 58f87a41bd8de7370cc05c41977fadc685826c9e > committer: Sergey Vojtovich > timestamp: 2017-06-16 15:47:46 +0400 > message: > > MDEV-12620 - set lock_wait_timeout = 1;flush tables with read lock; > lock not released after timeout > > Release GRL if FLUSH TABLES phase failed. > > diff --git a/mysql-test/t/mdl_sync.test b/mysql-test/t/mdl_sync.test > index 8e80978..c479c2a 100644 > --- a/mysql-test/t/mdl_sync.test > +++ b/mysql-test/t/mdl_sync.test > @@ -4806,6 +4806,30 @@ disconnect con2; > disconnect con3; > > +--echo # > +--echo # MDEV-12620 - set lock_wait_timeout = 1;flush tables with read lock; > +--echo # lock not released after timeout > +--echo # > +CREATE TABLE t1(a INT) ENGINE=InnoDB; > +SET debug_sync='open_tables_after_open_and_process_table SIGNAL ready WAIT_FOR go'; > +send SELECT * FROM t1; > + > +connect (con1,localhost,root,,); > +SET debug_sync='now WAIT_FOR ready'; > +# lock_wait_timeout should be 0 in 10.3, so that we don't have to wait at all what's special in 10.3? > +SET lock_wait_timeout=1; > +--error ER_LOCK_WAIT_TIMEOUT > +FLUSH TABLES WITH READ LOCK; > +SET debug_sync='now SIGNAL go'; > + > +connection default; > +reap; > +SET debug_sync='RESET'; > +DROP TABLE t1; What should happen without a bug fix? DROP TABLE hangs? > + > +disconnect con1; > + > # Check that all connections opened by test cases in this file are really > # gone so execution of other tests won't be affected by their presence. > --source include/wait_until_count_sessions.inc > diff --git a/sql/sql_reload.cc b/sql/sql_reload.cc > index 376dfeb..d68ce96 100644 > --- a/sql/sql_reload.cc > +++ b/sql/sql_reload.cc > @@ -253,7 +253,8 @@ bool reload_acl_and_cache(THD *thd, unsigned long long options, > NOTE: my_error() has been already called by reopen_tables() within > close_cached_tables(). > */ > - result= 1; > + thd->global_read_lock.unlock_global_read_lock(thd); > + return 1; > } Sure. Looks correct. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 1

[Maria-developers] MDEV-10306 Wrong results with combination of CONCAT, SUBSTR and CONVERT in subquery
by Alexander Barkov 16 Jun '17

16 Jun '17

Hello Sergei, Please review a fix for MDEV-10306. Thanks!

4 13

Re: [Maria-developers] [Commits] 01b2bb79bb78e42108e32af02f1418a660ca861d Fixed the bug mdev-13064.
by Oleksandr Byelkin 16 Jun '17

16 Jun '17

Hi, Igor! Am 15.06.2017 um 23:43 schrieb Igor Babaev: > commit 01b2bb79bb78e42108e32af02f1418a660ca861d > Author: Igor Babaev <igor(a)askmonty.org> > Commit: Igor Babaev <igor(a)askmonty.org> > > Fixed the bug mdev-13064. > > This is another attempt to fix the bug mdev-12992. > This patch introduces st_select_lex::context_analysis_place for > the place in SELECT where context analysis is currently performed. > It's similar to st_select_lex::parsing_place, but it is used at It is OK to push. > the preparation stage. > --- > mysql-test/r/func_group.result | 9 +++++++++ > mysql-test/t/func_group.test | 13 +++++++++++++ > sql/item.cc | 5 ++--- > sql/item_windowfunc.cc | 2 +- > sql/sql_lex.h | 1 + > sql/sql_select.cc | 33 ++++++++++++++++++--------------- > 6 files changed, 44 insertions(+), 19 deletions(-) > > diff --git a/mysql-test/r/func_group.result b/mysql-test/r/func_group.result > index cb97ea2..c4e991e 100644 > --- a/mysql-test/r/func_group.result > +++ b/mysql-test/r/func_group.result > @@ -2447,3 +2447,12 @@ DROP TABLE t1; > # > # End of 10.1 tests > # > +# > +# MDEV-13064: assertion `n < m_size' fails in Item::split_sum_func2() > +# > +create table t1 (i int) engine=MyISAM; > +insert into t1 value (1),(2); > +select count(*)+sleep(0) from t1; > +count(*)+sleep(0) > +2 > +drop table t1; > diff --git a/mysql-test/t/func_group.test b/mysql-test/t/func_group.test > index 1e75099..8bbc9e6 100644 > --- a/mysql-test/t/func_group.test > +++ b/mysql-test/t/func_group.test > @@ -1691,3 +1691,16 @@ DROP TABLE t1; > --echo # > --echo # End of 10.1 tests > --echo # > + > +--echo # > +--echo # MDEV-13064: assertion `n < m_size' fails in Item::split_sum_func2() > +--echo # > + > +create table t1 (i int) engine=MyISAM; > +insert into t1 value (1),(2); > + > +select count(*)+sleep(0) from t1; > + > +drop table t1; > + > + > diff --git a/sql/item.cc b/sql/item.cc > index 8db5cfc..df615b5 100644 > --- a/sql/item.cc > +++ b/sql/item.cc > @@ -490,8 +490,7 @@ int Item::save_str_value_in_field(Field *field, String *result) > command => we should check thd->lex->current_select on zero (thd->lex > can be uninitialised) > */ > - if (thd->lex->current_select && > - thd->stmt_arena->is_stmt_prepare_or_first_sp_execute()) > + if (thd->lex->current_select) > { > enum_parsing_place place= > thd->lex->current_select->parsing_place; > @@ -5513,7 +5512,7 @@ bool Item_field::fix_fields(THD *thd, Item **reference) > > SELECT_LEX *select= thd->lex->current_select; > thd->change_item_tree(reference, > - select->parsing_place == IN_GROUP_BY && > + select->context_analysis_place == IN_GROUP_BY && > alias_name_used ? *rf->ref : rf); > > /* > diff --git a/sql/item_windowfunc.cc b/sql/item_windowfunc.cc > index 59a22c6..27b0072 100644 > --- a/sql/item_windowfunc.cc > +++ b/sql/item_windowfunc.cc > @@ -71,7 +71,7 @@ > { > DBUG_ASSERT(fixed == 0); > > - enum_parsing_place place= thd->lex->current_select->parsing_place; > + enum_parsing_place place= thd->lex->current_select->context_analysis_place; > > if (!(place == SELECT_LIST || place == IN_ORDER_BY)) > { > diff --git a/sql/sql_lex.h b/sql/sql_lex.h > index face3bc..0c55ffc 100644 > --- a/sql/sql_lex.h > +++ b/sql/sql_lex.h > @@ -855,6 +855,7 @@ class st_select_lex: public st_select_lex_node > /* reserved for exists 2 in */ > uint select_n_reserved; > enum_parsing_place parsing_place; /* where we are parsing expression */ > + enum_parsing_place context_analysis_place; /* where we are in prepare */ > bool with_sum_func; /* sum function indicator */ > > ulong table_join_options; > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index 4c64cf2..6c56f01 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -648,17 +648,15 @@ void remove_redundant_subquery_clauses(st_select_lex *subq_select_lex) > > thd->lex->allow_sum_func|= (nesting_map)1 << select->nest_level; > > - save_place= thd->lex->current_select->parsing_place; > - thd->lex->current_select->parsing_place= IN_ORDER_BY; > + save_place= thd->lex->current_select->context_analysis_place; > + thd->lex->current_select->context_analysis_place= IN_ORDER_BY; > res= res || setup_order(thd, ref_pointer_array, tables, fields, all_fields, > order); > - thd->lex->current_select->parsing_place= save_place; > - thd->lex->allow_sum_func&= ~((nesting_map)1 << select->nest_level); > - save_place= thd->lex->current_select->parsing_place; > - thd->lex->current_select->parsing_place= IN_GROUP_BY; > + thd->lex->allow_sum_func&= ~((nesting_map)1 << select->nest_level); > + thd->lex->current_select->context_analysis_place= IN_GROUP_BY; > res= res || setup_group(thd, ref_pointer_array, tables, fields, all_fields, > group, hidden_group_fields); > - thd->lex->current_select->parsing_place= save_place; > + thd->lex->current_select->context_analysis_place= save_place; > thd->lex->allow_sum_func|= (nesting_map)1 << select->nest_level; > res= res || setup_windows(thd, ref_pointer_array, tables, fields, all_fields, > win_specs, win_funcs); > @@ -712,6 +710,7 @@ void remove_redundant_subquery_clauses(st_select_lex *subq_select_lex) > if (select_lex->handle_derived(thd->lex, DT_PREPARE)) > DBUG_RETURN(1); > > + thd->lex->current_select->context_analysis_place= NO_MATTER; > thd->lex->current_select->is_item_list_lookup= 1; > /* > If we have already executed SELECT, then it have not sense to prevent > @@ -801,12 +800,13 @@ void remove_redundant_subquery_clauses(st_select_lex *subq_select_lex) > > ref_ptrs= ref_ptr_array_slice(0); > > - enum_parsing_place save_place= thd->lex->current_select->parsing_place; > - thd->lex->current_select->parsing_place= SELECT_LIST; > + enum_parsing_place save_place= > + thd->lex->current_select->context_analysis_place; > + thd->lex->current_select->context_analysis_place= SELECT_LIST; > if (setup_fields(thd, ref_ptrs, fields_list, MARK_COLUMNS_READ, > &all_fields, 1)) > DBUG_RETURN(-1); > - thd->lex->current_select->parsing_place= save_place; > + thd->lex->current_select->context_analysis_place= save_place; > > if (setup_without_group(thd, ref_ptrs, tables_list, > select_lex->leaf_tables, fields_list, > @@ -22235,14 +22235,16 @@ int setup_order(THD *thd, Ref_ptr_array ref_pointer_array, TABLE_LIST *tables, > List<Item> &fields, List<Item> &all_fields, ORDER *order, > bool from_window_spec) > { > - enum_parsing_place parsing_place= thd->lex->current_select->parsing_place; > + enum_parsing_place context_analysis_place= > + thd->lex->current_select->context_analysis_place; > thd->where="order clause"; > for (; order; order=order->next) > { > if (find_order_in_list(thd, ref_pointer_array, tables, order, fields, > all_fields, FALSE, from_window_spec)) > return 1; > - if ((*order->item)->with_window_func && parsing_place != IN_ORDER_BY) > + if ((*order->item)->with_window_func && > + context_analysis_place != IN_ORDER_BY) > { > my_error(ER_WINDOW_FUNCTION_IN_WINDOW_SPEC, MYF(0)); > return 1; > @@ -22284,7 +22286,8 @@ int setup_order(THD *thd, Ref_ptr_array ref_pointer_array, TABLE_LIST *tables, > List<Item> &fields, List<Item> &all_fields, ORDER *order, > bool *hidden_group_fields, bool from_window_spec) > { > - enum_parsing_place parsing_place= thd->lex->current_select->parsing_place; > + enum_parsing_place context_analysis_place= > + thd->lex->current_select->context_analysis_place; > *hidden_group_fields=0; > ORDER *ord; > > @@ -22300,14 +22303,14 @@ int setup_order(THD *thd, Ref_ptr_array ref_pointer_array, TABLE_LIST *tables, > all_fields, TRUE, from_window_spec)) > return 1; > (*ord->item)->marker= UNDEF_POS; /* Mark found */ > - if ((*ord->item)->with_sum_func && parsing_place == IN_GROUP_BY) > + if ((*ord->item)->with_sum_func && context_analysis_place == IN_GROUP_BY) > { > my_error(ER_WRONG_GROUP_FIELD, MYF(0), (*ord->item)->full_name()); > return 1; > } > if ((*ord->item)->with_window_func) > { > - if (parsing_place == IN_GROUP_BY) > + if (context_analysis_place == IN_GROUP_BY) > my_error(ER_WRONG_PLACEMENT_OF_WINDOW_FUNCTION, MYF(0)); > else > my_error(ER_WINDOW_FUNCTION_IN_WINDOW_SPEC, MYF(0)); > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits

1 0

[Maria-developers] Review for MDEV-12666: CURRENT_ROLE() and DATABASE() does not work in a view
by Alexander Barkov 13 Jun '17

13 Jun '17

Hi Vicentiu, The patch looks fine. Thanks! I have only small comments below: > > [Commits] 974064cde3c: MDEV-12666: CURRENT_ROLE() and DATABASE() does not work in a view > vicentiu vicentiu at mariadb.org > Thu Jun 8 10:11:38 EEST 2017 > > Previous message (by thread): [Commits] 7f2d3c3: MDEV-11084 Select statement with partition selection against MyISAM table opens all partitions. > Next message (by thread): [Commits] d03abc71a49: MDEV-12609: Allow suppression of InnoDB log messages about reserving extents > Messages sorted by: [ date ] [ thread ] [ subject ] [ author ] > > revision-id: 974064cde3c1cd1be9157bc11ce410604807f68b (mariadb-10.0.31-2-g974064cde3c) > parent(s): 54cd5bc703da08be45894495d7f3e2c8117617cd > author: Vicențiu Ciorbaru > committer: Vicențiu Ciorbaru > timestamp: 2017-06-08 10:10:07 +0300 > message: > > MDEV-12666: CURRENT_ROLE() and DATABASE() does not work in a view > > The problem lies in how CURRENT_ROLE is defined. The > Item_func_current_role inherits from Item_func_sysconst, which defines > a safe_charset_converter to be a const_charset_converter. > > During view creation, if there is no role previously set, the current_role() > function returns NULL. > > This is captured on item instantiation and the > const_charset_converter call subsequently returns an Item_null. > In turn, the function is replaced with Item_null and the view is > then created with an Item_null instead of Item_func_current_role. > > Without this patch, the first SHOW CREATE VIEW from the testcase would > have a where clause of WHERE role_name = NULL, while the second SHOW > CREATE VIEW would show a correctly created view. > > The same applies for the DATABASE function, as it can change as well. > > There is an additional problem with CURRENT_ROLE() when used in a > prepared statement. During prepared statement creation we used to set > the string_value of the function to the current role as well as the > null_value flag. During execution, if CURRENT_ROLE was not null, the > null_value flag was never set to not-null during fix_fields. > > Item_func_current_user however can never be NULL so it did not show this > problem in a view before. At the same time, the CURRENT_USER() can not > be changed between prepared statement execution and creation so the > implementation where the value is stored during fix_fields is > sufficient. > > Note also that DATABASE() function behaves differently during prepared > statements. See bug 25843 for details or commit > 7e0ad09edff587dadc3e9855fc81e1b7de8f2199 > <cut> > --- a/sql/item_strfunc.cc > +++ b/sql/item_strfunc.cc > @@ -2344,6 +2344,7 @@ String *Item_func_database::val_str(String *str) > } > else > str->copy(thd->db, thd->db_length, system_charset_info); > + null_value= 0; > return str; > } > > @@ -2378,6 +2379,30 @@ bool Item_func_user::init(const char *user, const char *host) > } > > > +Item *Item_func_sysconst::safe_charset_converter(CHARSET_INFO *tocs) > +{ > + /* > + During view or prepared statement creation, the item should not > + make use of const_charset_converter as it would imply substitution > + with constant items which is not correct. Functions can have different > + values during view creation and view execution based on context. > + > + Return the identical item during view creation and prepare. > + */ > + if (current_thd->lex->context_analysis_only & > + (CONTEXT_ANALYSIS_ONLY_PREPARE | CONTEXT_ANALYSIS_ONLY_VIEW)) > + return this; > + return const_charset_converter(tocs, true, fully_qualified_func_name()); I propose to avoid code duplication and do like this: if (!Item_func_sysconst::const_item()) return this; return return const_charset_converter(tocs, true, fully_qualified_func_name()); > +} > + > +bool Item_func_sysconst::const_item() const > +{ > + if (current_thd->lex->context_analysis_only & > + (CONTEXT_ANALYSIS_ONLY_PREPARE | CONTEXT_ANALYSIS_ONLY_VIEW)) > + return false; > + return true; > +} > + > bool Item_func_user::fix_fields(THD *thd, Item **ref) > { > return (Item_func_sysconst::fix_fields(thd, ref) || > @@ -2403,20 +2428,25 @@ bool Item_func_current_role::fix_fields(THD *thd, Item **ref) > > Security_context *ctx= context->security_ctx > ? context->security_ctx : thd->security_ctx; > - > if (ctx->priv_role[0]) > { > if (str_value.copy(ctx->priv_role, strlen(ctx->priv_role), > system_charset_info)) > return 1; > - > str_value.mark_as_const(); > + null_value= maybe_null= 0; > return 0; > } > null_value= maybe_null= 1; > return 0; > } > > +String* Item_func_current_role::val_str(String * str) > +{ > + DBUG_ASSERT(fixed == 1); > + return null_value ? NULL : &str_value; > +} > + Why did you move val_str() from *.h to *.cc ? The method is just two lines, I'd leave it in *.h <cut>

1 0

[Maria-developers] MDEV-12962 Testing MDEV-10142
by Alexander Barkov 13 Jun '17

13 Jun '17

Hi Andrii, Thanks for looking into this. Please find my comments inline: > > > Below are mostly minor notes for which clarification is needed. > > Conceptual: are we sure that ORACLE mode should invalidate native MariaDB constructs? We need some consistency and in my understanding following should still work in Oracle mode: > ELSE / ELSEIF > % operation (if possible) I'm not sure. Monty, what do you think? > > > 2. MDEV-10343 sql_mode=ORACLE: Providing compatibility for basic SQL data types > > following are valid build-in datatypes declaration in oracle which cause syntax error at the moment (needs to be documented or created another task): > > these probably had to be implemented inside MDEV-10343 > > create or replace table t1(a binary_float); > create or replace table t1(a binary_double); > create or replace table t1(a long raw); > create or replace table t1(a NCLOB); > these need more effort or maybe just must be documented or separate task is needed > create or replace table t1(a varchar(30 byte)); > create or replace table t1(a varchar(30 char)); > create or replace table t1(a timestamp with local time zone); > create or replace table t1(a timestamp with time zone); > create or replace table t1(a interval day); > create or replace table t1(a interval year); > create or replace table t1(a BFILE); > create or replace table t1(a rowid); > create or replace table t1(a urowid(5)); > this should give an error (oracle will return ORA-02017: integer value required ) > MariaDB [test]> create or replace table t1(a number(2.2)); > Query OK, 0 rows affected (0.24 sec) There is a task for TIMESTAMP WITH TIME ZONE: MDEV-10018 Timestamp with time zone For other data types, should we create one task for every data type? > > > 3. MDEV-10411 Providing compatibility for basic PL/SQL constructs > 3.6 Assignment operator > It doesn't sound intuitive that one may write "wait_timeout:=a", but "a:=wait_timeout" shows syntax error. > > begin declare a int; begin a:=wait_timeout; end; end;; One needs to use @ when referring a system variable in the right side: a:=@wait_timeout; It's the same in the default mode: SET wait_timeount=100; but: SET xxx=@wait_timeout; Not sure if we should do anything with that. > > > 3.7 Variable declarations > Oracle allows NOT NULL in variable declaration. Maybe we should allow that syntax as well? > > begin declare a NOT NULL int; begin a:=1; a:=NULL; end; end;; I have created a task for this: MDEV-13078 NOT NULL routine variables > > > 10. MDEV-10585 EXECUTE IMMEDIATE > In ORACLE the command supports 'RETURNING INTO' clause, we should probably document the difference? > > sql_stmt := 'UPDATE emp SET sal = 2000 WHERE empno = :1 > > RETURNING sal INTO :2'; > > EXECUTE IMMEDIATE sql_stmt USING emp_id RETURNING INTO salary; In Oracle, EXECUTE IMMEDIATE also has the "INTO" clause: sql_stmt := 'SELECT * FROM emp WHERE empno = :id'; EXECUTE IMMEDIATE sql_stmt INTO emp_rec USING emp_id; Should we create two new tasks, for RETURNING INTO, and for INTO? > > 12. MDEV-10588 TRUNCATE TABLE t1 DROP|REUSE STORAGE > Following clauses should be allowed in TRUNCATE command as well: > [ PRESERVE MATERIALIZED VIEW LOG | PURGE MATERIALIZED VIEW LOG ] Should we create a task to ignore this syntax? > > 18. MDEV-10577 sql_mode=ORACLE: %TYPE in variable declarations > It is not clear what is expected to happen if table's column definition is altered after stored program creation. Nothing. Data type resolution is done during execution.

1 0

[Maria-developers] Progress report of the Project[Week 2]
by ANKIT KUMAR 12 Jun '17

12 Jun '17

Hi, In this week, I managed to do few contributions and finally completed subtask-1. These are few things that I did this week- 1. Extended CREATE USER syntax- Now there will be multiple authentication mechanisms per user. 2. Created tests for the newly implemented syntax and some tests for previously existing functonalities. Need to give final touch up to the code and will commit tomorrow. Now I will proceed to subtask-2 Week-3[TODO's]- 1. Implement in-memory structures for this new syntax-[Providing it to ACL_USER] 2. Will be able to write data in memory(By the end of Phase-1 evaluation). Kind regards, Ankit Kumar.

1 0

[Maria-developers] libmariadb.so.3 compatibility
by Michal Schorm 08 Jun '17

08 Jun '17

Hello, I'm currently quite confused about the *libmariadb* library. In MariaDB 10.1.24 there are: " libmysqlclient_r.so -> libmysqlclient.so " " libmysqlclient.so -> libmysqlclient.so.18 " " libmysqlclient.so.18 -> libmysqlclient.so.18.0.0 " " libmysqlclient.so.18.0.0 " In MariaDB 10.2.6 there are: " libmariadb.so -> libmariadb.so.3 " In MariaDB-connector-c v2.3.3 there are: " libmysqlclient.so -> libmariadb.so " " libmysqlclient_r.so -> libmariadb.so " " libmariadb.so -> libmariadb.so.2 " " libmariadb.so.2 " -- 1) is "libmariadb.so.2" compatibile with "libmysqlclient.so.18.0.0"? Both API and ABI? 2) is "libmariadb.so.3" compatibile with "libmysqlclient.so.18.0.0"? Both API and ABI? 3) is "libmariadb.so.2" compatibile with "libmariadb.so.3"? Both API and ABI? Many packages in Fedora depends on "libmysqlclient.so". I would like to pack 10.2 to Fedora, but I can't before all those packages move to the libmariadb library. It would be considerably easier transfer for me, if all those libraries were compatibile, so I could provide symlinks (simmilar as in the connector C), and provide time for all those packages in Fedora to move safely. -- Michal Schorm Associate Software Engineer Core Services - Databases Team Red Hat

2 2

Re: [Maria-developers] [Commits] 4d6b16f: Fixed the bug mdev-12838.
by Sergey Petrunia 07 Jun '17

07 Jun '17

Hello Igor, On Wed, May 31, 2017 at 10:45:07AM -0700, IgorBabaev wrote: > revision-id: 4d6b16f10f9991821c128c382d32eb85248b3926 (mariadb-5.5.56-14-g4d6b16f) > parent(s): b8405c853fa30002d164d5fe2b4f8ea8979c09b8 > author: Igor Babaev > committer: Igor Babaev > timestamp: 2017-05-31 10:45:07 -0700 > message: > > Fixed the bug mdev-12838. > ... > +bool JOIN_TAB::keyuse_is_valid_for_access_in_chosen_plan(JOIN *join, > + KEYUSE *keyuse) > +{ > + if (!access_from_tables_is_allowed(keyuse->used_tables, > + join->sjm_lookup_tables)) > + return false; > + if (join->sjm_scan_tables & table->map) > + return true; > + table_map keyuse_sjm_scan_tables= keyuse->used_tables & > + join->sjm_scan_tables; > + if (!keyuse_sjm_scan_tables) > + return true; > + uint sjm_tab_nr= 0; > + while (!(keyuse_sjm_scan_tables & 1 << sjm_tab_nr)) The above "1" is a 32-bit number, while it should have the same bit length as table_map (that is, 64). Please use "table_map(1)". BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 1

[Maria-developers] the Lock on supremum is neccessary ?
by jhx1008 07 Jun '17

07 Jun '17

hi everyone: I use sysbench to do some tests on MySQL 5.7.x, and then I got some problems with the lock added on supremum, I hope someone can give me some advice. my table like this: CREATE TABLE `sbtest1` ( `id` int(11) NOT NULL AUTO_INCREMENT, `k` int(11) NOT NULL DEFAULT '0', `c` char(120) NOT NULL DEFAULT '', `pad` char(60) NOT NULL DEFAULT '', PRIMARY KEY (`id`), ) ENGINE=InnoDB; secondary index is not neccessary, I do not create it!! I have two sessions: session1: begin; select * from sbtest1 where id=123456 for update session2: insert into sbtest1 values(123450, 0,'aaa','bbb'); the records in the table like below: (...),(123440, 264454, xxxxxx, xxxxx), (123456, 251028, xxxxx, xxxxxx),(....) in my opinion, the record with the id=123456 existed, so session1 add the X lock on this record, session2 add the insert intention lock between(123440, 123456), these two locks are not conflicted. session2 will not be blocked by session1. unfortunately, in my case, session2 is blocked by session1, what happens? when session2 is blocked, I do "select * from information_schema.INNODB_LOCKS" use another session, It shows : mysql> select * from information_schema.INNODB_LOCKS; +----------------+-------------+-----------+-----------+------------------+------------+------------+-----------+----------+------------------------+ | lock_id | lock_trx_id | lock_mode | lock_type | lock_table | lock_index | lock_space | lock_page | lock_rec | lock_data | +----------------+-------------+-----------+-----------+------------------+------------+------------+-----------+----------+------------------------+ | 2388:29:2772:1 | 2388 | X | RECORD | `mydb`.`sbtest1` | PRIMARY | 29 | 2772 | 1 | supremum pseudo-record | | 2387:29:2772:1 | 2387 | X | RECORD | `mydb`.`sbtest1` | PRIMARY | 29 | 2772 | 1 | supremum pseudo-record | +----------------+-------------+-----------+-----------+------------------+------------+------------+-----------+----------+------------------------+ both the two sessions want to add X lock on "supremum pseudo-record" in the same time. then I have debugged this instance(row0sel.cc::row_search_mvcc), I got that: when a record existed on the page for the first one, if you want to add the X Lock on it, the supremum of pre-page will be add X Lock(maybe next key lock, supremum can be add gap lock only) also. In my case, the records' primary key stored like this: (..., 123440, supremum), (infimum, 123456, ...)， the id=123456 is the first record of that page, when add X Lock, first, find the pre-record id=123440, and then add the X lock after this record(id=123440) until match the record witch id=123456, so the supremum is locked also...， because of this lock(supremum)， session2 is blocked. if id=123456 is not the first record of page, session2 will not be blocked. so my doubt is: in my case, the session1 lock the supremum is necessary? if it's necessary, the lock on supremum maybe resulting the difference even if two mysql instances have the same records, just as in a replication. the record located on the page not all the same even if two instances have the same table sturcture & data(if slave define the table compressed, or innodb page_size not set as the same as master, etc..). if lock on supremum is neccessary, two instances execute the same SQLs, master is OK, but slave conflicted? on master， the two transactions may be committed in one group, but on slave maybe conflicted, slave's parallell replication is OK? also in MySQL 5.7.x, the xa prepare recorded in binlog, if on the master, execute two xa transactions: xa start 'aaa'; delete from table_name where id=xxxx; xa end 'aaa';xa prepare 'aaa'; xa start 'bbb';insert into table_name values(xxxx); end 'bbb'; xa prepare 'bbb'; xa commit 'aaa'; xa commit 'bbb' on the master, these xa not conflicted, I can committed these transaction, but on the slave, maybe conflicted sometimes, the replication is broken. how to repeat: create a table with primary key(id auto_increment is better) insert some records in this table use 3 sessions to find the first record on one page(perhaps, there are many pages, any one is OK).how to? session1: begin; select * from table_name where id=xxx for update // id=xxx must be exist session2: select * from table_name where id=xxx for update session3: when session2 is blocked, execute: show engine innodb status\G; in the transaction section, you can find some details like this: ------- TRX HAS BEEN WAITING 2 SEC FOR THIS LOCK TO BE GRANTED: RECORD LOCKS space id 29 page no 2772 n bits 144 index PRIMARY of table `mydb`.`sbtest1` trx id 2388 lock_mode X insert intention waiting Record lock, heap no 1 PHYSICAL RECORD: n_fields 1; compact format; info bits 0 0: len 8; hex 73757072656d756d; asc supremum;; from the information "space id 29 page no 2772", we can find the page of this record. then test for next id until the page no become to 2773, remember this id, this record is the first one of page 2773 delete some records before this id witch get in step 3 use 2 sessions to do what i described I repeat it by sysbench(version 1.0). 1. use sysbench to create a table with 500,000 records: sysbench --table-size=500000 --tables=1 --threads=64 --create_secondary=false --db-driver=mysql --mysql-host=127.0.0.1 --mysql-port=3306 --mysql-user=root --mysql-db=mydb oltp_read_write.lua prepare 2. delete the records between(200000, 200057), delete from sbtest1 where id>200000 and id < 200057 3. two sessions: session1: begin;select * from sbtest1 where id=200057 for update; session2: insert into sbtest1 values(200050, 0,'a','a') session2 will be blocked by session1. Because this problem is related to the localtion of the record in the page, repeat it like my operations, do not use the compressed table and make sure the innodb_page_size = 16384. in my case, the Lock on supremum is neccessary? 2017-06-05 boyce@netease

2 4

[Maria-developers] Progress report of the Project
by ANKIT KUMAR 05 Jun '17

05 Jun '17

Hi, I am Working on MDEV-11340 and by the end of the first week of Coding Period I have managed to learn How to use Bison. Currently thinking about How to implement subtask-1 of my project. Kind Regards,

1 0

[Maria-developers] header change in MariaDB 10.2.6 and mysqlclient python binding
by Kazuhiko Shiozaki 30 May '17

30 May '17

Hi ! First of all, congratulations to MariaDB 10.2 GA. mysqlclient (python binding : https://pypi.python.org/pypi/mysqlclient/1.3.10 ) cannot be built with MariaDB 10.2.6 like the following : _mysql.c: In function '_mysql_ConnectionObject_ping': _mysql.c:1911:41: error: 'MYSQL' has no member named 'reconnect' if ( reconnect != -1 ) self->connection.reconnect = reconnect; ^ error: Setup script exited with error: command 'gcc' failed with exit status 1 This is because the change in include/mysql/mysql.h. In MariaDB 10.2.5, it is same as mariadb-10.2.5/include/mysql.h in the source tree. In MariaDB 10.2.6, it is same as mariadb-10.2.6/libmariadb/include/mysql.h in the source tree. Is this change intentional ? Should I contact mysqlclient developer for further fix of this issue ? Thanks in advance ! Kazuhiko

2 1

[Maria-developers] review for SP stack trace
by Alexander Barkov 30 May '17

30 May '17

Hello Jerome, here's Monty's review on your stack trace patch. Thank you very much. We're sorry for delay. > diff --git a/mysql-test/r/commit_1innodb.result b/mysql-test/r/commit_1innodb.result > index 1adba7b..8e40ec0 100644 > <cut> > > --- a/mysql-test/r/get_diagnostics.result > +++ b/mysql-test/r/get_diagnostics.result > @@ -595,7 +595,7 @@ SELECT var, @var; > END| > CALL p1(); > var @var > -1 1 > +2 2 > DROP PROCEDURE p1; > > What is the reason for this going from 1 to 2 ? > I would have understood if there would have been some warnings from before > in the code, but there wasn't. > > Does this come from the error from 'call p1()' earlier that does now have > a new 'note message'. > > If yes, it would be good to make this clear by adding a 'show > warnings' just before the above create procedure p1() line. > > # Setting TABLE_NAME is currently not implemented. > diff --git a/mysql-test/r/signal.result b/mysql-test/r/signal.result > index f05e357..a796cc6 100644 > --- a/mysql-test/r/signal_demo3.result > +++ b/mysql-test/r/signal_demo3.result > @@ -79,14 +79,23 @@ show warnings; > @@ -95,6 +104,4 @@ call proc_1(); > ERROR 45000: Oops in proc_1 > show warnings; > Level Code Message > -Error 1644 Oops in proc_5 > -Error 1644 Oops in proc_4 > -Error 1644 Oops in proc_3 > +Note 4068 At line 4 in demo.proc_3 > > Why has 3 error disappered and why is there a line number note for > demo.proc_3 but no error line? > > I assume this comes from the following line: > SET @@session.max_error_count = 5 > > The question is if max_error_count's should really be counted for notes too, > especially for "At line" notes. > The documentation for the variable says > "Specifies the maximum number of messages stored for display by" > > Still, the question is if we should include the 'At line' or not in counting. > (Not important for this patch, but good to think about) > > > --- a/sql/share/errmsg-utf8.txt > +++ b/sql/share/errmsg-utf8.txt > @@ -7484,3 +7484,5 @@ ER_UNKNOWN_SEQUENCES 42S02 > eng "Unknown SEQUENCE: '%-.300s'" > ER_UNKNOWN_VIEW 42S02 > eng "Unknown VIEW: '%-.300s'" > +ER_SP_STACK_TRACE > + eng "At line %d in %s" > > Should be %u as lineno is uint > > > Please add the "SHOW WARNINGS" and fix the error message and it's ok to push. > Thanks for a great addition to MariaDB!

2 2

[Maria-developers] Weekly Progress Report of the Project
by ANKIT KUMAR 29 May '17

29 May '17

Hi all, I am Ankit Kumar and I have been selected to work on allowing multiple authentication methods to same user. These are the few things I did this week- 1. Setted up Development environment on Windows. Resolved few issues which I was facing, some MariaDB members really helped me whenever I needed their help. 2. Read the Testing Framework Documentation of MariaDB in order to see how testing is performed. 3. I have been regularly communicating with the Mentor and other MariaDB Contributors in order in get deep insight of the problems that I would be working on during Coding Period. These are some basic things that I would need during official coding period, so I wont be wasting time on such things. Now I can get Started with my project. IRC Nick- AspiringMind

2 4

Re: [Maria-developers] GSoC NUMA support Project
by Daniel Black 29 May '17

29 May '17

On 29/05/17 06:03, Sumit Lakra wrote: > I think I am near done with dividing the buffer pool into the number of > nodes available in a NUMA machine. There is one thing left to do - to > check for the presence of 'libnuma' library in the server. Based on https://jira.mariadb.org/browse/MDEV-12924 I probably broke it in ab0e5031e41d0aa72dd77a6dced9eacc8be5c548 > As of now I > have explicitly defined HAVE_LIBNUMA in the > CMakeList.txt to check my code. CMakeList.txt in which directory? I'd suggest refering to publicly pushed code/branches for review. A branch on https://github.com/theGodlessLakra/server would be good. > Also, to test whether the control of execution reaches within a function > A, I tried to print out a message on the scree like 'fprintf(stderr, In > function A);' This seems to work in main(), my_init(), sys_var_init() > etc, but not in buf0buf.cc. The function which creates the buffer pool > is called by 'buf_pool_init()', which is called by > 'innobase_start_on_create_for_mysql()', which in turn is called by > 'innobase_init()', but I couldn't find where an d how is this function > called. Its called by function pointer -> see maria_declare_plugin in storage/innobase/handler/ha_innodb.cc which is called by the plugin initialization code. > Seems like the output to bot stdout and stderr is being > redirected somewhere and I can't find where (in code). reopen_fstreams in sql/{mysqld,log}.cc > Having this printing stuff is a quick way of testing the flow of control > and although my code is working, You can also run mysqld in gdb and the use "break" to set a breakpoint at a function/file:line and step through form there. > I would like to know about this > redirections if someone can tell me. add --verbose to the mysqld argument. > Also, which part of code calls the 'innobase_init()' function ? sql/plugin.cc:plugin_initialize in the call to plugin->init() > I have attached a screenshot of the server running on my NUMA emulated > pc with and without --numa-interleave=all. [ PID - 21016 without NUMA, > PID - 21058 with NUMA] > > At present it just ALLOCATES a buffer pool instance on each numa node. > Need to work on specific node binding and stuff.

1 0

[Maria-developers] GSoC NUMA support Project
by Sumit Lakra 28 May '17

28 May '17

I think I am near done with dividing the buffer pool into the number of nodes available in a NUMA machine. There is one thing left to do - to check for the presence of 'libnuma' library in the server. As of now I have explicitly defined HAVE_LIBNUMA in the CMakeList.txt to check my code. Also, to test whether the control of execution reaches within a function A, I tried to print out a message on the scree like 'fprintf(stderr, In function A);' This seems to work in main(), my_init(), sys_var_init() etc, but not in buf0buf.cc. The function which creates the buffer pool is called by 'buf_pool_init()', which is called by 'innobase_start_on_create_for_mysql()', which in turn is called by 'innobase_init()', but I couldn't find where an d how is this function called. Seems like the output to bot stdout and stderr is being redirected somewhere and I can't find where (in code). Having this printing stuff is a quick way of testing the flow of control and although my code is working, I would like to know about this redirections if someone can tell me. Also, which part of code calls the 'innobase_init()' function ? I have attached a screenshot of the server running on my NUMA emulated pc with and without --numa-interleave=all. [ PID - 21016 without NUMA, PID - 21058 with NUMA] At present it just ALLOCATES a buffer pool instance on each numa node. Need to work on specific node binding and stuff.

1 0

[Maria-developers] Building LibraOffice w/ MariaDB installed fails apt-get build-dep
by Howard Johnson 27 May '17

27 May '17

Hi, I'm having trouble with MariaDB as is described here: https://ask.libreoffice.org/en/question/96902/building-lo-w-mariadb-install… I suspect that there might be an issue with the virtual package or something like that. Any idea of what is going wrong or how to fix it? Thanks -- howard

2 1

Re: [Maria-developers] [Commits] df94429: MDEV-11196: Error:Run-Time Check Failure #2 - Stack around the variable 'key_buff'
by Sergey Petrunia 26 May '17

26 May '17

Hi Varun, Ok to push. On Thu, May 25, 2017 at 07:41:50PM +0530, Varun wrote: > revision-id: df944293721a7ad93d115a58f2b574067d1dc79f (mariadb-10.1.20-282-gdf94429) > parent(s): a1b6128dedb4419db9fadaf94c356d3477d4e06f > author: Varun Gupta > committer: Varun Gupta > timestamp: 2017-05-25 19:40:08 +0530 > message: > > MDEV-11196: Error:Run-Time Check Failure #2 - Stack around the variable 'key_buff' > was corrupted, server crashes in opt_sum_query > > extended keys feature disabled if the length of extended key is longer than MAX_KEY_LEN > > --- > mysql-test/r/innodb_ext_key.result | 81 ++++++++++++++++++++++++++++++++++++++ > mysql-test/t/innodb_ext_key.test | 51 ++++++++++++++++++++++++ > sql/table.cc | 39 +++++++++++++++++- > 3 files changed, 170 insertions(+), 1 deletion(-) > > diff --git a/mysql-test/r/innodb_ext_key.result b/mysql-test/r/innodb_ext_key.result > index 1305be8..de1323e 100644 > --- a/mysql-test/r/innodb_ext_key.result > +++ b/mysql-test/r/innodb_ext_key.result > @@ -1133,5 +1133,86 @@ where index_date_updated= 10 and index_id < 800; > id select_type table type possible_keys key key_len ref rows Extra > 1 SIMPLE t2 range index_date_updated index_date_updated 13 NULL # Using index condition > drop table t0,t1,t2; > +# > +# MDEV-11196: Error:Run-Time Check Failure #2 - Stack around the variable 'key_buff' > +# was corrupted, server crashes in opt_sum_query > +set @save_innodb_file_format= @@innodb_file_format; > +set @save_innodb_large_prefix= @@innodb_large_prefix; > +set global innodb_file_format = BARRACUDA; > +set global innodb_large_prefix = ON; > +CREATE TABLE t1 ( > +pk INT, > +f1 VARCHAR(3), > +f2 VARCHAR(1024), > +PRIMARY KEY (pk), > +KEY(f2) > +) ENGINE=InnoDB CHARSET utf8 ROW_FORMAT= DYNAMIC; > +INSERT INTO t1 VALUES (1,'foo','abc'),(2,'bar','def'); > +SELECT MAX(t2.pk) FROM t1 t2 INNER JOIN t1 t3 ON t2.f1 = t3.f1 WHERE t2.pk <= 4; > +MAX(t2.pk) > +2 > +drop table t1; > +CREATE TABLE t1 ( > +pk1 INT, > +pk2 INT, > +f1 VARCHAR(3), > +f2 VARCHAR(1021), > +PRIMARY KEY (pk1,pk2), > +KEY(f2) > +) ENGINE=InnoDB CHARSET utf8 ROW_FORMAT= DYNAMIC; > +INSERT INTO t1 VALUES (1,2,'2','abc'),(2,3,'3','def'); > +explain format= json > +select * from t1 force index(f2) where pk1 <= 5 and pk2 <=5 and f2 = 'abc' and f1 <= '3'; > +EXPLAIN > +{ > + "query_block": { > + "select_id": 1, > + "table": { > + "table_name": "t1", > + "access_type": "range", > + "possible_keys": ["f2"], > + "key": "f2", > + "key_length": "3070", > + "used_key_parts": ["f2", "pk1"], > + "rows": 1, > + "filtered": 100, > + "index_condition": "((t1.pk1 <= 5) and (t1.pk2 <= 5) and (t1.f2 = 'abc'))", > + "attached_condition": "(t1.f1 <= '3')" > + } > + } > +} > +drop table t1; > +CREATE TABLE t1 ( > +f2 INT, > +pk2 INT, > +f1 VARCHAR(3), > +pk1 VARCHAR(1000), > +PRIMARY KEY (pk1,pk2), > +KEY k1(pk1,f2) > +) ENGINE=InnoDB CHARSET utf8 ROW_FORMAT= DYNAMIC; > +INSERT INTO t1 VALUES (1,2,'2','abc'),(2,3,'3','def'); > +explain format= json > +select * from t1 force index(k1) where f2 <= 5 and pk2 <=5 and pk1 = 'abc' and f1 <= '3'; > +EXPLAIN > +{ > + "query_block": { > + "select_id": 1, > + "table": { > + "table_name": "t1", > + "access_type": "range", > + "possible_keys": ["k1"], > + "key": "k1", > + "key_length": "3011", > + "used_key_parts": ["pk1", "f2", "pk2"], > + "rows": 1, > + "filtered": 100, > + "index_condition": "((t1.f2 <= 5) and (t1.pk2 <= 5) and (t1.pk1 = 'abc'))", > + "attached_condition": "(t1.f1 <= '3')" > + } > + } > +} > +drop table t1; > set optimizer_switch=@save_ext_key_optimizer_switch; > +set global innodb_file_format = @save_innodb_file_format; > +set global innodb_large_prefix = @save_innodb_large_prefix; > SET SESSION STORAGE_ENGINE=DEFAULT; > diff --git a/mysql-test/t/innodb_ext_key.test b/mysql-test/t/innodb_ext_key.test > index bf94b7d..c8acfc5 100644 > --- a/mysql-test/t/innodb_ext_key.test > +++ b/mysql-test/t/innodb_ext_key.test > @@ -778,5 +778,56 @@ where index_date_updated= 10 and index_id < 800; > > drop table t0,t1,t2; > > + > +--echo # > +--echo # MDEV-11196: Error:Run-Time Check Failure #2 - Stack around the variable 'key_buff' > +--echo # was corrupted, server crashes in opt_sum_query > + > +set @save_innodb_file_format= @@innodb_file_format; > +set @save_innodb_large_prefix= @@innodb_large_prefix; > +set global innodb_file_format = BARRACUDA; > +set global innodb_large_prefix = ON; > + > +CREATE TABLE t1 ( > + pk INT, > + f1 VARCHAR(3), > + f2 VARCHAR(1024), > + PRIMARY KEY (pk), > + KEY(f2) > +) ENGINE=InnoDB CHARSET utf8 ROW_FORMAT= DYNAMIC; > + > +INSERT INTO t1 VALUES (1,'foo','abc'),(2,'bar','def'); > +SELECT MAX(t2.pk) FROM t1 t2 INNER JOIN t1 t3 ON t2.f1 = t3.f1 WHERE t2.pk <= 4; > +drop table t1; > + > +CREATE TABLE t1 ( > + pk1 INT, > + pk2 INT, > + f1 VARCHAR(3), > + f2 VARCHAR(1021), > + PRIMARY KEY (pk1,pk2), > + KEY(f2) > +) ENGINE=InnoDB CHARSET utf8 ROW_FORMAT= DYNAMIC; > + > +INSERT INTO t1 VALUES (1,2,'2','abc'),(2,3,'3','def'); > +explain format= json > +select * from t1 force index(f2) where pk1 <= 5 and pk2 <=5 and f2 = 'abc' and f1 <= '3'; > +drop table t1; > + > +CREATE TABLE t1 ( > +f2 INT, > +pk2 INT, > +f1 VARCHAR(3), > +pk1 VARCHAR(1000), > +PRIMARY KEY (pk1,pk2), > +KEY k1(pk1,f2) > +) ENGINE=InnoDB CHARSET utf8 ROW_FORMAT= DYNAMIC; > +INSERT INTO t1 VALUES (1,2,'2','abc'),(2,3,'3','def'); > +explain format= json > +select * from t1 force index(k1) where f2 <= 5 and pk2 <=5 and pk1 = 'abc' and f1 <= '3'; > +drop table t1; > + > set optimizer_switch=@save_ext_key_optimizer_switch; > +set global innodb_file_format = @save_innodb_file_format; > +set global innodb_large_prefix = @save_innodb_large_prefix; > SET SESSION STORAGE_ENGINE=DEFAULT; > diff --git a/sql/table.cc b/sql/table.cc > index 37c0b63..a1e9ebf 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -1721,6 +1721,7 @@ int TABLE_SHARE::init_from_binary_frm_image(THD *thd, bool write, > keyinfo= share->key_info; > uint primary_key= my_strcasecmp(system_charset_info, share->keynames.type_names[0], > primary_key_name) ? MAX_KEY : 0; > + KEY* key_first_info; > > if (primary_key >= MAX_KEY && keyinfo->flags & HA_NOSAME) > { > @@ -1800,19 +1801,38 @@ int TABLE_SHARE::init_from_binary_frm_image(THD *thd, bool write, > keyinfo->name_length+1); > } > > + if (!key) > + key_first_info= keyinfo; > + > if (ext_key_parts > share->key_parts && key) > { > KEY_PART_INFO *new_key_part= (keyinfo-1)->key_part + > (keyinfo-1)->ext_key_parts; > uint add_keyparts_for_this_key= add_first_key_parts; > + uint length_bytes= 0, len_null_byte= 0, ext_key_length= 0; > + Field *field; > > /* > Do not extend the key that contains a component > defined over the beginning of a field. > */ > for (i= 0; i < keyinfo->user_defined_key_parts; i++) > - { > + { > uint fieldnr= keyinfo->key_part[i].fieldnr; > + field= share->field[keyinfo->key_part[i].fieldnr-1]; > + > + if (field->null_ptr) > + len_null_byte= HA_KEY_NULL_LENGTH; > + > + if (field->type() == MYSQL_TYPE_BLOB || > + field->real_type() == MYSQL_TYPE_VARCHAR || > + field->type() == MYSQL_TYPE_GEOMETRY) > + { > + length_bytes= HA_KEY_BLOB_LENGTH; > + } > + > + ext_key_length+= keyinfo->key_part[i].length + len_null_byte > + + length_bytes; > if (share->field[fieldnr-1]->key_length() != > keyinfo->key_part[i].length) > { > @@ -1821,6 +1841,23 @@ int TABLE_SHARE::init_from_binary_frm_image(THD *thd, bool write, > } > } > > + if (add_keyparts_for_this_key) > + { > + for (i= 0; i < add_keyparts_for_this_key; i++) > + { > + uint pk_part_length= key_first_info->key_part[i].store_length; > + if (keyinfo->ext_key_part_map & 1<<i) > + { > + if (ext_key_length + pk_part_length > MAX_KEY_LENGTH) > + { > + add_keyparts_for_this_key= i; > + break; > + } > + ext_key_length+= pk_part_length; > + } > + } > + } > + > if (add_keyparts_for_this_key < (keyinfo->ext_key_parts - > keyinfo->user_defined_key_parts)) > { > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Cannot push branch to github
by Valentin Rakush 26 May '17

26 May '17

Hi all, I am trying to do first pull request to https://github.com/MariaDB/server but I cannot push my new branch to github. Git says "ERROR: Permission to MariaDB/server.git denied to alpes214. fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists." Should I talk to somebody to get access rights first? -- Kind regards, Valentin Rakush

2 3

Re: [Maria-developers] MDEV-12803 Improve function parameter data type control
by Alexander Barkov 26 May '17

26 May '17

Hello Alexey, Thanks for your feedback. Sending a new version, as discussed in this email and on IRC. Please find attached. - Removed Item_geometry_func_importer - Renamed Item_xxx_geometry_property to Item_xxx_args_geometry - Renamed Item_xxx_dyadic_operation to Item_xxx_args_geometry_geometry. This is the new hierarchy: Item_str_func Item_geometry_func Item_geometry_func_args_geometry Item_func_centroid Item_func_envelope Item_func_boundary Item_func_spatial_decomp Item_func_spatial_decomp_n Item_func_buffer Item_func_pointonsurface Item_func_geometry_from_text Item_func_geometry_from_wkb Item_func_geometry_from_json Item_func_point Item_func_spatial_collection Item_func_spatial_operation Item_binary_func_args_geometry Item_func_as_wkb Item_real_func Item_real_func_args_geometry Item_func_x Item_func_y Item_func_area Item_func_glength Item_real_func_args_geometry_geometry Item_func_distance Item_long_func Item_long_func_args_geometry Item_func_issimple Item_func_isring Item_func_isclosed Item_func_dimension Item_func_numgeometries Item_func_numinteriorring Item_func_numpoints Item_func_srid Item_bool_func Item_bool_func_args_geometry Item_func_isempty Item_bool_func_args_geometry_geometry Item_func_spatial_relate Item_str_ascii_func Item_str_ascii_func_args_geometry Item_func_as_wkt Item_func_as_geojson Item_func_geometry_type Thanks! On 05/26/2017 11:50 AM, Alexey Botchkov wrote: >>1. It will look like a C fragment in a C++ code. >> Why is it more natural than a virtual method? > > The Item hierarchy and 'check_type' hierarchy are parallel. > So it seems natural to me if we can assign an already-prepared > parameter-validation function to the type. > >> 2. It will make the Item size bigger. > But it will make Item** virtual tables smaller. > >> 4. It will need more CPU for m_check_arguments_function initialization. > Must be really small overhead. > >> Are you afraid that I'll have to modify class hierarchy? > Yes. My only concern is that the class hierarchy becomes more complicated > and with repeating code. > > But after a while i sort of used to it so now i'm fine with the virtual > method. > > >> We have a convention that functions return true on error and false on >> success. > For the check_something function returning TRUE or FALSE doesn't mean an > 'error'. > It's just the condition fits or doesn't. > But fine, our code has many examples of 'check' functions returning > results this way. > > Typenames discussed. > So otherwise i'm ok with the patch. > > Best regards. > HF > > On Wed, May 24, 2017 at 4:22 PM, Alexander Barkov <bar(a)mariadb.org > <mailto:bar@mariadb.org>> wrote: > > Hi Alexey, > > Thank you very much for your review! > > Please see my comments below: > > > On 05/23/2017 12:16 AM, Alexey Botchkov wrote: > > Firstly I'd say i don't like the idea of the 'virtual check_arguments()' > > function. > > I belive it's more natural to have a member 'm_check_arguments_function' > > that would be set in constructor (or be an argument to the constructor) > > and the non-virtual 'check_arguments()' that would just call that member. > > This would make Item class structure much nicer and would even save us > > memory. > > > > If you have good arguments for your approach, I have some > > comments/questions to your patch :) > > I'm not sure that a function member is better. > > 1. It will look like a C fragment in a C++ code. > Why is it more natural than a virtual method? > > 2. It will make the Item size bigger. > > 3. It will need more coding. > Now we have to declare only virtual functions. > With a function member, we'll have to do: > - declare function anyway > - add a code into constructors > > 4. It will need more CPU for m_check_arguments_function initialization. > On the contrary, having a virtual methods costs nothing, > except a small initialization in VMTs during mysqld startup. > > > What is your main concern? > Are you afraid that I'll have to modify class hierarchy > for other Item_xxx_func? It should not be necessary. > We'll just have a set of protected methods on the Item_func level, > for most typical cases. > > > > > > Why this change? > > +++ b/mysql-test/t/gis-rtree.test > > @@ -61,7 +61,7 @@ while ($1) > > let $2=10; > > while ($2) > > { > > - eval DELETE FROM t2 WHERE Within(g, > > Envelope(GeometryFromWKB(Point($1 * 10 - 9, $2 * 10 - 9), Point($1 * 10, > > $2 * 10)))); > > + eval DELETE FROM t2 WHERE Within(g, > > Envelope(GeometryFromWKB(Point($1 * 10 - 9, $2 * 10 - 9), 0))); > > SELECT count(*) FROM t2; > > dec $2; > > > It passed POINT() as the second argiment to GeometryFromWKB. > The second argument is for SRID. After my changes it's not allowed. > > > > } > > > > > > It looks weird when the 'check_something' functions returns FALSE if > > the argument fits the check. I'd make them returning TRUE in this case > > as people normally expect. > > We have a convention that functions return true on error and false on > success. > > Please have a look into: > > check_db_name() > check_for_broken_triggers() > check_unique_table() > > They all return true on error. > I did the same. > > > > > > +bool Item_func::check_argument_types_scalar(uint start, uint end) const > > +{ > > + for (uint i= start; i < end; i++) > > + { > > + DBUG_ASSERT(i < arg_count); > > > > In code like this i'd rather do > > bool Item_func::check_argument_types_scalar(uint start, uint end) const > > { > > DBUG_ASSERT(end < arg_count); > > > > for (uint i= start; i < end; i++) > > { > > // no DBUG_ASSERT here > > ... > > Agree. > > > > > > > > > Some typenames feel misleading. I'd suggest changes > > Item_real_func_geometry_property -> Item_real_func_arg_geometry > > Item_int_func_geometry_property -> Item_int_func_arg_geometry > > Item_bool_func_geometry_property -> Item_bool_func_arg_geometry > > Item_str_ascii_func_geometry_property -> Item_ascii_func_arg_geometry > > Item_geometry_func_geometry_property -> Item_geometry_func_arg_geometry > > This looks like a property. > The first argument is an object of some complex type (GEOMETRY in > our case). > The other arguments are simple type additional optional parameters. > > In C++ this would look like: > > geom->IsSimple(); > geom->Buffer(10); > > > OpenGIS also calls these methods "properties", > so do we in the documentation: > > https://dev.mysql.com/doc/refman/5.7/en/gis-class-point.html > <https://dev.mysql.com/doc/refman/5.7/en/gis-class-point.html> > > > > > > > Item_binary_func_geometry_property - do we really need this type? > > Right, there is only one binary string property: AsWKB(). > > I added Item_xxx_geometry_property for regularity > (to have a symmetric class hierarchy with other "properties" ). > > Here's the class structure: > > > Item_str_func > Item_geometry_func > Item_geometry_func_geometry_property > Item_func_centroid > Item_func_envelope > Item_func_boundary > Item_func_spatial_decomp > Item_func_spatial_decomp_n > Item_func_buffer > Item_func_pointonsurface > Item_geometry_func_importer > Item_func_geometry_from_text > Item_func_geometry_from_wkb > Item_func_geometry_from_json > Item_func_point > Item_func_spatial_collection > Item_func_spatial_operation > Item_binary_func_geometry_property > Item_func_as_wkb > Item_real_func > Item_real_func_geometry_property > Item_func_x > Item_func_y > Item_func_area > Item_func_glength > Item_real_func_geometry_dyadic_operation > Item_func_distance > Item_int_func > Item_int_func_geometry_property > Item_func_issimple > Item_func_isring > Item_func_isclosed > Item_func_dimension > Item_func_numgeometries > Item_func_numinteriorring > Item_func_numpoints > Item_func_srid > Item_func_gis_debug > Item_bool_func > Item_bool_func_geometry_property > Item_func_isempty > Item_bool_func_geometry_dyadic_operation > Item_func_spatial_relate > Item_str_ascii_func > Item_str_ascii_func_geometry_property > Item_func_as_wkt > Item_func_as_geojson > Item_func_geometry_type > > Btw, do you need a program to print this hierarchy? :) > I wrote it a few years ago, and use it almost every day. > > > > > Item_geometry_func_importer - do we need this type? > > I can remove this. It does not have shared methods indeed. > > > > > Item_real_func_geometry_dyadic_operation -> Item_real_func_arg_2_geometries > > Item_bool_func_geometry_dyadic_operation -> Item_bool_func_arg_2_geometries > > It's an operation on two geometries. > My names is quite self-descriptive. > Also, it tells that: > - there is no any other GEOMETRY arguments > - there can be optional non-GEOMETRY additional arguments > > > > > > > + bool check_arguments() const > > + { > > + DBUG_ASSERT(arg_count >= 2); > > + return check_argument_types_or_binary(&type_handler_geometry, 0, 2); > > > > the check_argument_types_or_binary() does the check of the arg_count so > > no need for the DBUG_ASSERT(). > > But i don't insist on removing this line :) > > I can remove it :) > > > > > > > On Mon, May 15, 2017 at 5:56 PM, Alexander Barkov <bar(a)mariadb.org <mailto:bar@mariadb.org> > > <mailto:bar@mariadb.org <mailto:bar@mariadb.org>>> wrote: > > > > Hello Alexey, > > > > can you please review a patch for MDEV-12803? > > > > Thanks! > > > > > >

1 0

Re: [Maria-developers] [Commits] 5ac7ee3: MDEV-11196: Error:Run-Time Check Failure #2 - Stack around the variable 'key_buff'
by Sergey Petrunia 25 May '17

25 May '17

Hi Varun, Good to see that the feedback from the last time was addressed. There is another issue, though. Consider a case where some (but not necessarily all) PK columns are present in the secondary key: create table t1 ( pk1 VARCHAR(1000) not null, pk2 INT not null, col2 INT not null, KEY key1(pk1, col2), PRIMARY KEY(pk1, pk2) ) CHARSET utf8 engine=innodb; insert into t1 select a,a,a from ten; In this case, key1 still has a suffix, but the suffix only includes columns that are not already present in the key. That is, the "real" definition of key1 is (pk1, col2, pk2) and not (pk1, col2, pk1, pk2) . Let's try that. explain select * from t1 force index(key1) where pk1 < 'zz' and col2 <=2 and pk2 <=4; Without the patch, I have all 3 key parts used: | { "query_block": { "select_id": 1, "table": { "table_name": "t1", "access_type": "range", "possible_keys": ["key1"], "key": "key1", "key_length": "3010", "used_key_parts": ["pk1", "col2", "pk2"], "rows": 1, "filtered": 100, "attached_condition": "t1.pk1 <= '0' and t1.col2 <= 2 and t1.pk2 <= 4", "using_index": true } } } | With the patch, I only see the explicitly defined key parts used: | { "query_block": { "select_id": 1, "table": { "table_name": "t1", "access_type": "range", "possible_keys": ["key1"], "key": "key1", "key_length": "3006", "used_key_parts": ["pk1", "col2"], "rows": 1, "filtered": 100, "attached_condition": "t1.pk1 <= '0' and t1.col2 <= 2 and t1.pk2 <= 4", "using_index": true } } } | Which I think is wrong, because the key length never exceeds MAX_KEY_LEN. This can be easily fixed by appyling this patch over your patch: --- table.cc.varun 2017-05-24 18:30:39.076341082 +0300 +++ table.cc 2017-05-24 19:04:25.637740027 +0300 @@ -2164,10 +2164,13 @@ for (i= 0; i < add_keyparts_for_this_key; i++) { uint pk_part_length= key_first_info->key_part[i].store_length; - if (ext_key_length + pk_part_length > MAX_KEY_LENGTH) + if (keyinfo->ext_key_part_map & 1<<i) { - add_keyparts_for_this_key= i; - break; + if (ext_key_length + pk_part_length > MAX_KEY_LENGTH) + { + add_keyparts_for_this_key= i; + break; + } } ext_key_length+= pk_part_length; } Do you agree with this change? Any amendments? If it's ok, please add this (together with the testcase) into your patch. I've checked MySQL 5.7 also, and they do something weird: ... insert into t1 select A.a + 1000*B.a, A.a + 1000*B.a, A.a + 1000*B.a from one_k A, ten B; explain format=json select * from t1 force index(key1) where pk1 <= '0' and col2 <=2 and pk2 <=4; EXPLAIN { "query_block": { "select_id": 1, "cost_info": { "query_cost": "1.41" }, "table": { "table_name": "t1", "access_type": "range", "possible_keys": [ "key1" ], "key": "key1", "used_key_parts": [ "pk1" ], "key_length": "3010", "rows_examined_per_scan": 1, "rows_produced_per_join": 0, "filtered": "11.11", "using_index": true, "cost_info": { "read_cost": "1.39", "eval_cost": "0.02", "prefix_cost": "1.41", "data_read_per_join": "335" }, "used_columns": [ "pk1", "pk2", "col2" ], "attached_condition": "((`test`.`t1`.`pk1` <= '0') and (`test`.`t1`.`col2` <= 2) and (`test`.`t1`.`pk2` <= 4))" } } } key_length is 3010, but used_key_parts only includes "pk1"... The numbers do not match. On Tue, May 23, 2017 at 07:08:48PM +0530, Varun wrote: > revision-id: 5ac7ee32a8e373d989bf9d665b5e01f770ac583f (mariadb-10.1.20-282-g5ac7ee3) > parent(s): a1b6128dedb4419db9fadaf94c356d3477d4e06f > author: Varun Gupta > committer: Varun Gupta > timestamp: 2017-05-23 19:00:15 +0530 > message: > > MDEV-11196: Error:Run-Time Check Failure #2 - Stack around the variable 'key_buff' > was corrupted, server crashes in opt_sum_query > > extended keys feature disabled if the length of extended key is longer than MAX_KEY_LEN > > --- > mysql-test/r/innodb_ext_key.result | 51 ++++++++++++++++++++++++++++++++++++++ > mysql-test/t/innodb_ext_key.test | 38 ++++++++++++++++++++++++++++ > sql/table.cc | 36 ++++++++++++++++++++++++++- > 3 files changed, 124 insertions(+), 1 deletion(-) > > diff --git a/mysql-test/r/innodb_ext_key.result b/mysql-test/r/innodb_ext_key.result > index 1305be8..c76660c 100644 > --- a/mysql-test/r/innodb_ext_key.result > +++ b/mysql-test/r/innodb_ext_key.result > @@ -1133,5 +1133,56 @@ where index_date_updated= 10 and index_id < 800; > id select_type table type possible_keys key key_len ref rows Extra > 1 SIMPLE t2 range index_date_updated index_date_updated 13 NULL # Using index condition > drop table t0,t1,t2; > +# > +# MDEV-11196: Error:Run-Time Check Failure #2 - Stack around the variable 'key_buff' > +# was corrupted, server crashes in opt_sum_query > +set @save_innodb_file_format= @@innodb_file_format; > +set @save_innodb_large_prefix= @@innodb_large_prefix; > +set global innodb_file_format = BARRACUDA; > +set global innodb_large_prefix = ON; > +CREATE TABLE t1 ( > +pk INT, > +f1 VARCHAR(3), > +f2 VARCHAR(1024), > +PRIMARY KEY (pk), > +KEY(f2) > +) ENGINE=InnoDB CHARSET utf8 ROW_FORMAT= DYNAMIC; > +INSERT INTO t1 VALUES (1,'foo','abc'),(2,'bar','def'); > +SELECT MAX(t2.pk) FROM t1 t2 INNER JOIN t1 t3 ON t2.f1 = t3.f1 WHERE t2.pk <= 4; > +MAX(t2.pk) > +2 > +drop table t1; > +CREATE TABLE t1 ( > +pk1 INT, > +pk2 INT, > +f1 VARCHAR(3), > +f2 VARCHAR(1021), > +PRIMARY KEY (pk1,pk2), > +KEY(f2) > +) ENGINE=InnoDB CHARSET utf8 ROW_FORMAT= DYNAMIC; > +INSERT INTO t1 VALUES (1,2,'2','abc'),(2,3,'3','def'); > +explain format= json > +select * from t1 force index(f2) where pk1 <= 5 and pk2 <=5 and f2 = 'abc' and f1 <= '3'; > +EXPLAIN > +{ > + "query_block": { > + "select_id": 1, > + "table": { > + "table_name": "t1", > + "access_type": "range", > + "possible_keys": ["f2"], > + "key": "f2", > + "key_length": "3070", > + "used_key_parts": ["f2", "pk1"], > + "rows": 1, > + "filtered": 100, > + "index_condition": "((t1.pk1 <= 5) and (t1.pk2 <= 5) and (t1.f2 = 'abc'))", > + "attached_condition": "(t1.f1 <= '3')" > + } > + } > +} > +drop table t1; > set optimizer_switch=@save_ext_key_optimizer_switch; > +set global innodb_file_format = @save_innodb_file_format; > +set global innodb_large_prefix = @save_innodb_large_prefix; > SET SESSION STORAGE_ENGINE=DEFAULT; > diff --git a/mysql-test/t/innodb_ext_key.test b/mysql-test/t/innodb_ext_key.test > index bf94b7d..adb032f 100644 > --- a/mysql-test/t/innodb_ext_key.test > +++ b/mysql-test/t/innodb_ext_key.test > @@ -778,5 +778,43 @@ where index_date_updated= 10 and index_id < 800; > > drop table t0,t1,t2; > > + > +--echo # > +--echo # MDEV-11196: Error:Run-Time Check Failure #2 - Stack around the variable 'key_buff' > +--echo # was corrupted, server crashes in opt_sum_query > + > +set @save_innodb_file_format= @@innodb_file_format; > +set @save_innodb_large_prefix= @@innodb_large_prefix; > +set global innodb_file_format = BARRACUDA; > +set global innodb_large_prefix = ON; > + > +CREATE TABLE t1 ( > + pk INT, > + f1 VARCHAR(3), > + f2 VARCHAR(1024), > + PRIMARY KEY (pk), > + KEY(f2) > +) ENGINE=InnoDB CHARSET utf8 ROW_FORMAT= DYNAMIC; > + > +INSERT INTO t1 VALUES (1,'foo','abc'),(2,'bar','def'); > +SELECT MAX(t2.pk) FROM t1 t2 INNER JOIN t1 t3 ON t2.f1 = t3.f1 WHERE t2.pk <= 4; > +drop table t1; > + > +CREATE TABLE t1 ( > + pk1 INT, > + pk2 INT, > + f1 VARCHAR(3), > + f2 VARCHAR(1021), > + PRIMARY KEY (pk1,pk2), > + KEY(f2) > +) ENGINE=InnoDB CHARSET utf8 ROW_FORMAT= DYNAMIC; > + > +INSERT INTO t1 VALUES (1,2,'2','abc'),(2,3,'3','def'); > +explain format= json > +select * from t1 force index(f2) where pk1 <= 5 and pk2 <=5 and f2 = 'abc' and f1 <= '3'; > +drop table t1; > + > set optimizer_switch=@save_ext_key_optimizer_switch; > +set global innodb_file_format = @save_innodb_file_format; > +set global innodb_large_prefix = @save_innodb_large_prefix; > SET SESSION STORAGE_ENGINE=DEFAULT; > diff --git a/sql/table.cc b/sql/table.cc > index 37c0b63..c8c9696 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -1721,6 +1721,7 @@ int TABLE_SHARE::init_from_binary_frm_image(THD *thd, bool write, > keyinfo= share->key_info; > uint primary_key= my_strcasecmp(system_charset_info, share->keynames.type_names[0], > primary_key_name) ? MAX_KEY : 0; > + KEY* key_first_info; > > if (primary_key >= MAX_KEY && keyinfo->flags & HA_NOSAME) > { > @@ -1800,19 +1801,38 @@ int TABLE_SHARE::init_from_binary_frm_image(THD *thd, bool write, > keyinfo->name_length+1); > } > > + if (!key) > + key_first_info= keyinfo; > + > if (ext_key_parts > share->key_parts && key) > { > KEY_PART_INFO *new_key_part= (keyinfo-1)->key_part + > (keyinfo-1)->ext_key_parts; > uint add_keyparts_for_this_key= add_first_key_parts; > + uint length_bytes= 0, len_null_byte= 0, ext_key_length= 0; > + Field *field; > > /* > Do not extend the key that contains a component > defined over the beginning of a field. > */ > for (i= 0; i < keyinfo->user_defined_key_parts; i++) > - { > + { > uint fieldnr= keyinfo->key_part[i].fieldnr; > + field= share->field[keyinfo->key_part[i].fieldnr-1]; > + > + if (field->null_ptr) > + len_null_byte= HA_KEY_NULL_LENGTH; > + > + if (field->type() == MYSQL_TYPE_BLOB || > + field->real_type() == MYSQL_TYPE_VARCHAR || > + field->type() == MYSQL_TYPE_GEOMETRY) > + { > + length_bytes= HA_KEY_BLOB_LENGTH; > + } > + > + ext_key_length+= keyinfo->key_part[i].length + len_null_byte > + + length_bytes; > if (share->field[fieldnr-1]->key_length() != > keyinfo->key_part[i].length) > { > @@ -1821,6 +1841,20 @@ int TABLE_SHARE::init_from_binary_frm_image(THD *thd, bool write, > } > } > > + if (add_keyparts_for_this_key) > + { > + for (i= 0; i < add_keyparts_for_this_key; i++) > + { > + uint pk_part_length= key_first_info->key_part[i].store_length; > + if (ext_key_length + pk_part_length > MAX_KEY_LENGTH) > + { > + add_keyparts_for_this_key= i; > + break; > + } > + ext_key_length+= pk_part_length; > + } > + } > + > if (add_keyparts_for_this_key < (keyinfo->ext_key_parts - > keyinfo->user_defined_key_parts)) > { > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

2 1

Re: [Maria-developers] MDEV-12803 Improve function parameter data type control
by Alexander Barkov 24 May '17

24 May '17

Hi Alexey, Thank you very much for your review! Please see my comments below: On 05/23/2017 12:16 AM, Alexey Botchkov wrote: > Firstly I'd say i don't like the idea of the 'virtual check_arguments()' > function. > I belive it's more natural to have a member 'm_check_arguments_function' > that would be set in constructor (or be an argument to the constructor) > and the non-virtual 'check_arguments()' that would just call that member. > This would make Item class structure much nicer and would even save us > memory. > > If you have good arguments for your approach, I have some > comments/questions to your patch :) I'm not sure that a function member is better. 1. It will look like a C fragment in a C++ code. Why is it more natural than a virtual method? 2. It will make the Item size bigger. 3. It will need more coding. Now we have to declare only virtual functions. With a function member, we'll have to do: - declare function anyway - add a code into constructors 4. It will need more CPU for m_check_arguments_function initialization. On the contrary, having a virtual methods costs nothing, except a small initialization in VMTs during mysqld startup. What is your main concern? Are you afraid that I'll have to modify class hierarchy for other Item_xxx_func? It should not be necessary. We'll just have a set of protected methods on the Item_func level, for most typical cases. > > Why this change? > +++ b/mysql-test/t/gis-rtree.test > @@ -61,7 +61,7 @@ while ($1) > let $2=10; > while ($2) > { > - eval DELETE FROM t2 WHERE Within(g, > Envelope(GeometryFromWKB(Point($1 * 10 - 9, $2 * 10 - 9), Point($1 * 10, > $2 * 10)))); > + eval DELETE FROM t2 WHERE Within(g, > Envelope(GeometryFromWKB(Point($1 * 10 - 9, $2 * 10 - 9), 0))); > SELECT count(*) FROM t2; > dec $2; It passed POINT() as the second argiment to GeometryFromWKB. The second argument is for SRID. After my changes it's not allowed. > } > > > It looks weird when the 'check_something' functions returns FALSE if > the argument fits the check. I'd make them returning TRUE in this case > as people normally expect. We have a convention that functions return true on error and false on success. Please have a look into: check_db_name() check_for_broken_triggers() check_unique_table() They all return true on error. I did the same. > > +bool Item_func::check_argument_types_scalar(uint start, uint end) const > +{ > + for (uint i= start; i < end; i++) > + { > + DBUG_ASSERT(i < arg_count); > > In code like this i'd rather do > bool Item_func::check_argument_types_scalar(uint start, uint end) const > { > DBUG_ASSERT(end < arg_count); > > for (uint i= start; i < end; i++) > { > // no DBUG_ASSERT here > ... Agree. > > > > Some typenames feel misleading. I'd suggest changes > Item_real_func_geometry_property -> Item_real_func_arg_geometry > Item_int_func_geometry_property -> Item_int_func_arg_geometry > Item_bool_func_geometry_property -> Item_bool_func_arg_geometry > Item_str_ascii_func_geometry_property -> Item_ascii_func_arg_geometry > Item_geometry_func_geometry_property -> Item_geometry_func_arg_geometry This looks like a property. The first argument is an object of some complex type (GEOMETRY in our case). The other arguments are simple type additional optional parameters. In C++ this would look like: geom->IsSimple(); geom->Buffer(10); OpenGIS also calls these methods "properties", so do we in the documentation: https://dev.mysql.com/doc/refman/5.7/en/gis-class-point.html > > Item_binary_func_geometry_property - do we really need this type? Right, there is only one binary string property: AsWKB(). I added Item_xxx_geometry_property for regularity (to have a symmetric class hierarchy with other "properties" ). Here's the class structure: Item_str_func Item_geometry_func Item_geometry_func_geometry_property Item_func_centroid Item_func_envelope Item_func_boundary Item_func_spatial_decomp Item_func_spatial_decomp_n Item_func_buffer Item_func_pointonsurface Item_geometry_func_importer Item_func_geometry_from_text Item_func_geometry_from_wkb Item_func_geometry_from_json Item_func_point Item_func_spatial_collection Item_func_spatial_operation Item_binary_func_geometry_property Item_func_as_wkb Item_real_func Item_real_func_geometry_property Item_func_x Item_func_y Item_func_area Item_func_glength Item_real_func_geometry_dyadic_operation Item_func_distance Item_int_func Item_int_func_geometry_property Item_func_issimple Item_func_isring Item_func_isclosed Item_func_dimension Item_func_numgeometries Item_func_numinteriorring Item_func_numpoints Item_func_srid Item_func_gis_debug Item_bool_func Item_bool_func_geometry_property Item_func_isempty Item_bool_func_geometry_dyadic_operation Item_func_spatial_relate Item_str_ascii_func Item_str_ascii_func_geometry_property Item_func_as_wkt Item_func_as_geojson Item_func_geometry_type Btw, do you need a program to print this hierarchy? :) I wrote it a few years ago, and use it almost every day. > > Item_geometry_func_importer - do we need this type? I can remove this. It does not have shared methods indeed. > > Item_real_func_geometry_dyadic_operation -> Item_real_func_arg_2_geometries > Item_bool_func_geometry_dyadic_operation -> Item_bool_func_arg_2_geometries It's an operation on two geometries. My names is quite self-descriptive. Also, it tells that: - there is no any other GEOMETRY arguments - there can be optional non-GEOMETRY additional arguments > > > + bool check_arguments() const > + { > + DBUG_ASSERT(arg_count >= 2); > + return check_argument_types_or_binary(&type_handler_geometry, 0, 2); > > the check_argument_types_or_binary() does the check of the arg_count so > no need for the DBUG_ASSERT(). > But i don't insist on removing this line :) I can remove it :) > > > On Mon, May 15, 2017 at 5:56 PM, Alexander Barkov <bar(a)mariadb.org > <mailto:bar@mariadb.org>> wrote: > > Hello Alexey, > > can you please review a patch for MDEV-12803? > > Thanks! > >

1 0

[Maria-developers] Oracle compatibility : Update with same source and target
by jerome brauge 23 May '17

23 May '17

Hello Alexander, In the same spirit of MDEV-12137, can you create a task for update with same source and target ? Regard, Jérôme.

2 3

[Maria-developers] MDEV-10142 - Replace function
by jerome brauge 23 May '17

23 May '17

Hello Alexander, Here is patch for replace function in oracle mode. Do I have to add a test for replication ? Regards, Jérôme.

2 4

Re: [Maria-developers] [Commits] 55a4858fe6f: MDEV-12666: CURRENT_ROLE() does not work in a view
by Alexander Barkov 23 May '17

23 May '17

Hi Vicentiu, It seems that Item_func_sysconst abuses const_charset_converter(). I have the following proposals: - Instead of adding a new code into Item::const_charset_converter(), can you please override Item_func_sysconst::safe_charset_converter()? The direction proposed by Sergei's looks good: Item *safe_charset_converter(THD *thd, CHARSET_INFO *tocs) { + if (thd->lex->context_analysis_only & CONTEXT_ANALYSIS_ONLY_VIEW) + return Item_str_func::safe_charset_converter(thd, tocs); return const_charset_converter(thd, tocs, true, fully_qualified_func_name() } But it probably needs to be extended to handle prepared statement. At PREPARE-time it should not call const_charset_converter(). At EXECUTE-time it should be Ok to call const_charset_converter(). So can you please change Item_func_sysconst::const_item() to report false when in views and during PREPARE-time? And then change safe_charset_converter() as follows: Item *safe_charset_converter(THD *thd, CHARSET_INFO *tocs) { if (!const_item()) return Item_str_func::safe_charset_converter(thd, tocs); return const_charset_converter(thd, tocs, true, fully_qualified_func_name() } - Please add tests for DATABASE() and CURRENT_USER - Please add tests for PS with changing context between PREPARE and EXECUTE. Thanks. On 05/22/2017 06:15 PM, vicentiu wrote: > revision-id: 55a4858fe6f3fdc63a072b93b9bc3d8e51dc5e94 (mariadb-10.0.30-76-g55a4858fe6f) > parent(s): 8d1827f5540a0def82b1e2cbc557245c3e5e14ca > author: Vicențiu Ciorbaru > committer: Vicențiu Ciorbaru > timestamp: 2017-05-22 17:06:01 +0300 > message: > > MDEV-12666: CURRENT_ROLE() does not work in a view > > The problem lies in how CURRENT_ROLE is defined. The > Item_func_current_role inherits from Item_func_sysconst, which defines > a safe_charset_converter to be a const_charset_converter. > > During view creation, if there is no role previously set, the current_role() > function returns NULL. > > This is captured on item instantiation and the > const_charset_converter call subsequently returns an Item_null. > In turn, the function is replaced with Item_null and the view is > then created with an Item_null instead of Item_func_current_role. > > Without this patch, the first SHOW CREATE VIEW from the testcase would > have a where clause of WHERE role_name = NULL, while the second SHOW > CREATE VIEW would show a correctly created view. > > The solution proposed is to not replace any function with an Item_null, > but instead use an Item_static_string_func. > > --- > .../suite/roles/current_role_view-12666.result | 48 ++++++++++++++++++++ > .../suite/roles/current_role_view-12666.test | 52 ++++++++++++++++++++++ > sql/item.cc | 13 +++++- > 3 files changed, 112 insertions(+), 1 deletion(-) > > diff --git a/mysql-test/suite/roles/current_role_view-12666.result b/mysql-test/suite/roles/current_role_view-12666.result > new file mode 100644 > index 00000000000..525748b4875 > --- /dev/null > +++ b/mysql-test/suite/roles/current_role_view-12666.result > @@ -0,0 +1,48 @@ > +CREATE USER has_role@'localhost'; > +GRANT ALL PRIVILEGES ON *.* TO has_role@'localhost'; > +CREATE ROLE test_role; > +GRANT test_role TO has_role@'localhost'; > +CREATE USER no_role@'localhost'; > +GRANT ALL PRIVILEGES ON *.* TO no_role@'localhost'; > +CREATE TABLE view_role_test ( > +id int primary key, > +role_name varchar(50) > +); > +INSERT INTO view_role_test VALUES (1, 'test_role'); > +CREATE OR REPLACE > +DEFINER = no_role@localhost > +SQL SECURITY INVOKER > +VIEW v_view_role_test > +AS > +SELECT * FROM view_role_test WHERE role_name = CURRENT_ROLE(); > +show create view v_view_role_test; > +View Create View character_set_client collation_connection > +v_view_role_test CREATE ALGORITHM=UNDEFINED DEFINER=`no_role`@`localhost` SQL SECURITY INVOKER VIEW `v_view_role_test` AS select `view_role_test`.`id` AS `id`,`view_role_test`.`role_name` AS `role_name` from `view_role_test` where (`view_role_test`.`role_name` = current_role()) latin1 latin1_swedish_ci > +SET ROLE test_role; > +CREATE OR REPLACE > +DEFINER = no_role@localhost > +SQL SECURITY INVOKER > +VIEW v_view_role_test_2 > +AS > +SELECT * FROM view_role_test WHERE role_name = CURRENT_ROLE(); > +show create view v_view_role_test_2; > +View Create View character_set_client collation_connection > +v_view_role_test_2 CREATE ALGORITHM=UNDEFINED DEFINER=`no_role`@`localhost` SQL SECURITY INVOKER VIEW `v_view_role_test_2` AS select `view_role_test`.`id` AS `id`,`view_role_test`.`role_name` AS `role_name` from `view_role_test` where (`view_role_test`.`role_name` = current_role()) latin1 latin1_swedish_ci > +SELECT CURRENT_USER(); > +CURRENT_USER() > +root@localhost > +SELECT CURRENT_ROLE(); > +CURRENT_ROLE() > +test_role > +SELECT * FROM view_role_test WHERE role_name = CURRENT_ROLE(); > +id role_name > +1 test_role > +SELECT * FROM v_view_role_test; > +id role_name > +1 test_role > +drop user has_role@'localhost'; > +drop user no_role@'localhost'; > +drop role test_role; > +drop table view_role_test; > +drop view v_view_role_test; > +drop view v_view_role_test_2; > diff --git a/mysql-test/suite/roles/current_role_view-12666.test b/mysql-test/suite/roles/current_role_view-12666.test > new file mode 100644 > index 00000000000..86c74260b6e > --- /dev/null > +++ b/mysql-test/suite/roles/current_role_view-12666.test > @@ -0,0 +1,52 @@ > +CREATE USER has_role@'localhost'; > +GRANT ALL PRIVILEGES ON *.* TO has_role@'localhost'; > + > +CREATE ROLE test_role; > +GRANT test_role TO has_role@'localhost'; > + > +CREATE USER no_role@'localhost'; > +GRANT ALL PRIVILEGES ON *.* TO no_role@'localhost'; > + > +CREATE TABLE view_role_test ( > + id int primary key, > + role_name varchar(50) > + ); > + > +INSERT INTO view_role_test VALUES (1, 'test_role'); > + > + > +CREATE OR REPLACE > +DEFINER = no_role@localhost > +SQL SECURITY INVOKER > +VIEW v_view_role_test > +AS > +SELECT * FROM view_role_test WHERE role_name = CURRENT_ROLE(); > + > +show create view v_view_role_test; > + > +SET ROLE test_role; > + > +CREATE OR REPLACE > +DEFINER = no_role@localhost > +SQL SECURITY INVOKER > +VIEW v_view_role_test_2 > +AS > +SELECT * FROM view_role_test WHERE role_name = CURRENT_ROLE(); > + > +show create view v_view_role_test_2; > + > + > +SELECT CURRENT_USER(); > +SELECT CURRENT_ROLE(); > + > +SELECT * FROM view_role_test WHERE role_name = CURRENT_ROLE(); > +SELECT * FROM v_view_role_test; > + > + > +drop user has_role@'localhost'; > +drop user no_role@'localhost'; > +drop role test_role; > + > +drop table view_role_test; > +drop view v_view_role_test; > +drop view v_view_role_test_2; > diff --git a/sql/item.cc b/sql/item.cc > index 4ce8396f71e..2ee3a6d61dc 100644 > --- a/sql/item.cc > +++ b/sql/item.cc > @@ -1263,9 +1263,20 @@ Item *Item::const_charset_converter(CHARSET_INFO *tocs, > DBUG_ASSERT(fixed); > StringBuffer<64>tmp; > String *s= val_str(&tmp); > - if (!s) > + > + /* Functions that return NULL during prepare must not be replaced with > + Item_null as they may return non-null values during execution. > + Ex: CURRENT_ROLE() during view creation time can be NULL if no role > + is set. */ > + if (!s && !func_name) > return new Item_null((char *) func_name, tocs); > > + if (!s) > + { > + tmp.set_ascii("", 0); > + s = &tmp; > + } > + > if (!needs_charset_converter(s->length(), tocs)) > { > if (collation.collation == &my_charset_bin && tocs != &my_charset_bin && > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits >

1 0

Re: [Maria-developers] [Commits] b400696: MDEV-11196: Error:Run-Time Check Failure #2 - Stack around the variable 'key_buff'
by Sergey Petrunia 22 May '17

22 May '17

Hi Varun, Please find my input below. I would like again to voice my dissatisfaction that I have to point out coding style violations in pretty much every patch I get. The other error in the patch is also pretty basic, ok to have it once but I think it is reasonable to expect more polished patches to be submitted for code review. On Sat, May 20, 2017 at 01:50:42AM +0530, Varun wrote: > revision-id: b400696386529fa63e79f723bbb06c05f7bae820 (mariadb-10.2.2-416-gb400696) > parent(s): fff61e31eca83a0a9af7c30e2bcda8309e9c695a > author: Varun Gupta > committer: Varun Gupta > timestamp: 2017-05-20 01:49:34 +0530 > message: > > MDEV-11196: Error:Run-Time Check Failure #2 - Stack around the variable 'key_buff' > was corrupted, server crashes in opt_sum_query > > extended keys feature disabled if the length of extended key is longer than MAX_KEY_LEN > > --- > mysql-test/r/innodb_ext_key.result | 45 ++++++++++++++++++++++++++++++++++++++ > mysql-test/t/innodb_ext_key.test | 31 ++++++++++++++++++++++++++ > sql/table.cc | 36 +++++++++++++++++++++++++++++- > 3 files changed, 111 insertions(+), 1 deletion(-) > > diff --git a/mysql-test/r/innodb_ext_key.result b/mysql-test/r/innodb_ext_key.result > index 1305be8..66391d6 100644 > --- a/mysql-test/r/innodb_ext_key.result > +++ b/mysql-test/r/innodb_ext_key.result > @@ -1133,5 +1133,50 @@ where index_date_updated= 10 and index_id < 800; > id select_type table type possible_keys key key_len ref rows Extra > 1 SIMPLE t2 range index_date_updated index_date_updated 13 NULL # Using index condition > drop table t0,t1,t2; > +# > +# MDEV-11196: Error:Run-Time Check Failure #2 - Stack around the variable 'key_buff' > +# was corrupted, server crashes in opt_sum_query > +CREATE TABLE t1 ( > +pk INT, > +f1 VARCHAR(3), > +f2 VARCHAR(1024), > +PRIMARY KEY (pk), > +KEY(f2) > +) ENGINE=InnoDB CHARSET utf8; > +INSERT INTO t1 VALUES (1,'foo','abc'),(2,'bar','def'); > +SELECT MAX(t2.pk) FROM t1 t2 INNER JOIN t1 t3 ON t2.f1 = t3.f1 WHERE t2.pk <= 4; > +MAX(t2.pk) > +2 > +drop table t1; > +CREATE TABLE t1 ( > +pk1 INT, > +pk2 INT, > +f1 VARCHAR(3), > +f2 VARCHAR(1021), > +PRIMARY KEY (pk1,pk2), > +KEY(f2) > +) ENGINE=InnoDB CHARSET utf8; > +INSERT INTO t1 VALUES (1,2,'2','abc'),(2,3,'3','def'); > +explain format= json > +select * from t1 force index(f2) where pk1 <= 5 and pk2 <=5 and f2 = 'abc' and f1 <= '3'; > +EXPLAIN > +{ > + "query_block": { > + "select_id": 1, > + "table": { > + "table_name": "t1", > + "access_type": "range", > + "possible_keys": ["f2"], > + "key": "f2", > + "key_length": "3070", > + "used_key_parts": ["f2", "pk1"], > + "rows": 1, > + "filtered": 100, > + "index_condition": "t1.pk1 <= 5 and t1.pk2 <= 5 and t1.f2 = 'abc'", > + "attached_condition": "t1.f1 <= '3'" > + } > + } > +} > +drop table t1; > set optimizer_switch=@save_ext_key_optimizer_switch; > SET SESSION STORAGE_ENGINE=DEFAULT; ... > diff --git a/sql/table.cc b/sql/table.cc > index 398383e..7d88eba 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -2104,6 +2104,7 @@ int TABLE_SHARE::init_from_binary_frm_image(THD *thd, bool write, > for (uint key=0 ; key < keys ; key++,keyinfo++) > { > uint usable_parts= 0; > + KEY* key_first_info; This variable may be used un-initialized. If I change the above line to be KEY* key_first_info=(KEY*)0x1; and run this statement: CREATE TABLE t1a ( pk INT, f1 VARCHAR(3), f2 VARCHAR(1024), f2a VARCHAR(1024), PRIMARY KEY (pk), KEY(f2), KEY(f2a) ) ENGINE=InnoDB CHARSET utf8; Then I get a crash. > keyinfo->name=(char*) share->keynames.type_names[key]; > keyinfo->name_length= strlen(keyinfo->name); > keyinfo->cache_name= > @@ -2118,19 +2119,38 @@ int TABLE_SHARE::init_from_binary_frm_image(THD *thd, bool write, > keyinfo->name_length+1); > } > > + if(!key) Coding style: 'if (' ... > + key_first_info= keyinfo; > + > if (ext_key_parts > share->key_parts && key) > { > KEY_PART_INFO *new_key_part= (keyinfo-1)->key_part + > (keyinfo-1)->ext_key_parts; > uint add_keyparts_for_this_key= add_first_key_parts; > + uint length_bytes= 0, len_null_byte= 0, ext_key_length= 0; > + Field *field; > > /* > Do not extend the key that contains a component > defined over the beginning of a field. > */ > for (i= 0; i < keyinfo->user_defined_key_parts; i++) > - { > + { Coding style: wrong identation. > uint fieldnr= keyinfo->key_part[i].fieldnr; > + field= share->field[keyinfo->key_part[i].fieldnr-1]; > + > + if (field->null_ptr) > + len_null_byte= HA_KEY_NULL_LENGTH; > + > + if (field->type() == MYSQL_TYPE_BLOB || > + field->real_type() == MYSQL_TYPE_VARCHAR || > + field->type() == MYSQL_TYPE_GEOMETRY) > + { > + length_bytes= HA_KEY_BLOB_LENGTH; > + } > + > + ext_key_length+= keyinfo->key_part[i].length + len_null_byte > + + length_bytes; > if (share->field[fieldnr-1]->key_length() != > keyinfo->key_part[i].length) > { > @@ -2139,6 +2159,20 @@ int TABLE_SHARE::init_from_binary_frm_image(THD *thd, bool write, > } > } > > + if (add_keyparts_for_this_key) > + { > + for (i= 0; i < add_keyparts_for_this_key; i++) > + { > + uint pk_part_length= key_first_info->key_part[i].store_length; > + if (ext_key_length + pk_part_length > MAX_KEY_LENGTH) > + { > + add_keyparts_for_this_key= i; > + break; > + } > + ext_key_length+= pk_part_length; > + } > + } > + > if (add_keyparts_for_this_key < (keyinfo->ext_key_parts - > keyinfo->user_defined_key_parts)) > { BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] 55a4858fe6f: MDEV-12666: CURRENT_ROLE() does not work in a view
by Sergei Golubchik 22 May '17

22 May '17

Hi, Vicentiu! Another possible approach (that doesn't necessarily penalizes other const functions that happen to return NULL) would be for Item_func_sysconst not to behave as const during view creation. Because they really aren't, sysconst functions can change between view creation and a select from the view. The fix would be, then Item *safe_charset_converter(THD *thd, CHARSET_INFO *tocs) { + if (thd->lex->context_analysis_only & CONTEXT_ANALYSIS_ONLY_VIEW) + return Item_str_func::safe_charset_converter(thd, tocs); return const_charset_converter(thd, tocs, true, fully_qualified_func_name() } And possibly (not sure) other values of context_analysis_only would need a similar treatment here. On May 22, vicentiu wrote: > revision-id: 55a4858fe6f3fdc63a072b93b9bc3d8e51dc5e94 (mariadb-10.0.30-76-g55a4858fe6f) > parent(s): 8d1827f5540a0def82b1e2cbc557245c3e5e14ca > author: Vicențiu Ciorbaru > committer: Vicențiu Ciorbaru > timestamp: 2017-05-22 17:06:01 +0300 > message: > > MDEV-12666: CURRENT_ROLE() does not work in a view > > The problem lies in how CURRENT_ROLE is defined. The > Item_func_current_role inherits from Item_func_sysconst, which defines > a safe_charset_converter to be a const_charset_converter. > > During view creation, if there is no role previously set, the current_role() > function returns NULL. > > This is captured on item instantiation and the > const_charset_converter call subsequently returns an Item_null. > In turn, the function is replaced with Item_null and the view is > then created with an Item_null instead of Item_func_current_role. > > Without this patch, the first SHOW CREATE VIEW from the testcase would > have a where clause of WHERE role_name = NULL, while the second SHOW > CREATE VIEW would show a correctly created view. > > The solution proposed is to not replace any function with an Item_null, > but instead use an Item_static_string_func. > Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] MariaDB License Clarification
by Ryan Coe 19 May '17

19 May '17

Hello, My name is Ryan Coe and I've been maintaining the MariaDB package for Buildroot (https://buildroot.org) In commit 577915def8d39223aa23a06f898e11e82e72365a, COPYING.LESSER was removed. Our packaging currently states the MariaDB license as GPL-2.0 (server), GPL-2.0 with FLOSS exception (GPL client library), LGPL-2.0 (LGPL client library). Could I please get clarification on how the different components are licensed? Also, the GitHub page is still referencing COPYING.LESSER in the README. Thank you.

2 1

Re: [Maria-developers] strange IF in translog_flush_buffers()
by Sergei Golubchik 18 May '17

18 May '17

Hi, Sanja! On May 17, Oleksandr Byelkin wrote: > Hi! > > Actually code should work as it is, but it check unnecessary condition and > also very dificult to understand. I propose following fix (comment in it > explains why it work now and why condition is always TRUE) I cannot review it, this is between you and Monty. If you think it's ok, please push it (into 10.0). Or I can push it together with all other similar fixes. > diff --git a/storage/maria/ma_loghandler.c b/storage/maria/ma_loghandler.c > index 86e51cc3526..e680e830703 100644 > --- a/storage/maria/ma_loghandler.c > +++ b/storage/maria/ma_loghandler.c > @@ -7789,8 +7789,23 @@ void translog_flush_buffers(TRANSLOG_ADDRESS *lsn, > translog_force_current_buffer_to_finish(); > translog_buffer_unlock(buffer); > } > - else if (log_descriptor.bc.buffer->prev_last_lsn != LSN_IMPOSSIBLE) > + else > { > + if (log_descriptor.bc.buffer->last_lsn == LSN_IMPOSSIBLE) > + { > + /* > + In this case both last_lsn & prev_last_lsn are LSN_IMPOSSIBLE > + otherwise it will go in the first IF because LSN_IMPOSSIBLE less > + then any real LSN and cmp_translog_addr(*lsn, > + log_descriptor.bc.buffer->prev_last_lsn) will be TRUE > + */ > + DBUG_ASSERT(log_descriptor.bc.buffer->prev_last_lsn == > + LSN_IMPOSSIBLE); > + DBUG_PRINT("info", ("There is no LSNs yet generated => do nothing")); > + translog_unlock(); > + DBUG_VOID_RETURN; > + } > + > /* fix lsn if it was horizon */ > *lsn= log_descriptor.bc.buffer->prev_last_lsn; > DBUG_PRINT("info", ("LSN to flush fixed to prev last lsn: (%lu,0x%lx)", > @@ -7799,13 +7814,6 @@ void translog_flush_buffers(TRANSLOG_ADDRESS *lsn, > TRANSLOG_BUFFERS_NO); > translog_unlock(); > } > - else if (log_descriptor.bc.buffer->last_lsn == LSN_IMPOSSIBLE) > - { > - DBUG_PRINT("info", ("There is no LSNs yet generated => do nothing")); > - translog_unlock(); > - DBUG_VOID_RETURN; > - } > - > /* flush buffers */ > *sent_to_disk= translog_get_sent_to_disk(); > if (cmp_translog_addr(*lsn, *sent_to_disk) > 0) Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

[Maria-developers] MDEV-12803 Improve function parameter data type control
by Alexander Barkov 15 May '17

15 May '17

Hello Alexey, can you please review a patch for MDEV-12803? Thanks!

1 0

[Maria-developers] String functions have their item->maybe_null=true unnecessarily, blocking optimizations
by Sergey Petrunia 15 May '17

15 May '17

Hi Alexander, I was looking at https://jira.mariadb.org/browse/MDEV-11893 and found this piece of code: bool Item_str_func::fix_fields(THD *thd, Item **ref) { bool res= Item_func::fix_fields(thd, ref); /* In Item_str_func::check_well_formed_result() we may set null_value flag on the same condition as in test() below. */ maybe_null= maybe_null || thd->is_strict_mode(); return res; } This has been introduced by this cset: https://github.com/MariaDB/server/commit/af22eb35e577ef17226faf662f2cffc470… Which says: Add Item_str_func::fix_fields() implementation, and set maybe_null to TRUE if we are in the SQL mode that requires some functions to return null even if they normally do not. The patch has only one example of CHAR() function doing that. At the same time, not having NOT-NULL attribute disallows query opimizations (see MDEV-11893 for an example). I think we should have this maybe_null= maybe_null || thd->is_strict_mode(); logic on a case-by-case basis only. I wanted to check with you - are there any other known string functions that, as the patch puts it "return null even if they normally do not"? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

2 1

[Maria-developers] MDEV-11340 Allow multiple alternative authentication methods for the same user
by serg＠mariadb.org 15 May '17

15 May '17

Hi, Ankit! I've started writing down my thoughts about MDEV-11340 "Allow multiple alternative authentication methods for the same user" See MDEV-11340 and feel free to comment on it. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] [Commits] 8edbb1117a9: MDEV-12741: innodb.ibuf_not_empty failed in buildbot with "InnoDB: Trying to do I/O to a tablespace which does not exist"
by Marko Mäkelä 15 May '17

15 May '17

On Thu, May 11, 2017 at 11:25 AM, jan <jan.lindstrom(a)mariadb.com> wrote: > revision-id: 8edbb1117a9e1fd81fbd08b8f1d06c72efe38f44 (mariadb-10.2.5-168- > g8edbb1117a9) > parent(s): f4df8c00c957146f04db031f1f9f80aab229d1d2 > author: Jan Lindström > committer: Jan Lindström > timestamp: 2017-05-11 11:15:37 +0300 > message: > > MDEV-12741: innodb.ibuf_not_empty failed in buildbot with "InnoDB: Trying > to do I/O to a tablespace which does not exist" > > Background thread is doing ibuf merge, in buf0rea.cc > buf_read_ibuf_merge_pages(). > It first tries to get page_size and if space is not found it deletes them, > but > as we do not hold any mutexes, space can be marked as stopped between that > and buf_read_page_low() for same space. This naturally leads seen error > message on log. > > buf_read_page_low(): Add parameter ignore_missing_space = false that > is passed to fil_io() > [snip] > @@ -178,7 +180,7 @@ buf_read_page_low( > > *err = fil_io( > request, sync, page_id, page_size, 0, page_size.physical(), > - dst, bpage); > + dst, bpage, ignore_missing_space); > > if (sync) { > thd_wait_end(NULL); > Can you please add the parameter to the IORequest? It would seem to be a more appropriate place for this. (Ideally, fil_io() would only take the request as a parameter, but we do not want to do such a big refactoring right now.) Marko

1 0

Re: [Maria-developers] [Commits] e6dabcd: MDEV-11196: Error:Run-Time Check Failure #2 - Stack around the variable 'key_buff'
by Sergey Petrunia 11 May '17

11 May '17

Hi Varun, So I am debugging the testcase from the bug report. I reach these lines: > if(i) > ext_key_length+= key_part->length; (Why there is no indentation btw?) and I see: (gdb) p key_part->length $40 = 3072 3072=1024*3, that is, the NULL-byte and length bytes are NOT counted. The code then proceeds to check if pk_length + ext_key_length <= MAX_KEY_LENGTH but that check could produce a wrong result if NULL- and length- bytes are not counted. The second issue is that I don't see where ext_key_length is zero'ed. I try debugging this query: CREATE TABLE t2 ( pk INT, f1 VARCHAR(3), f2 VARCHAR(1024), f2a VARCHAR(1024), PRIMARY KEY (pk), KEY(f2), KEY(f2a) ) ENGINE=InnoDB CHARSET utf8; I put a breakpoint at this line: > if (share->use_ext_keys && i && !(keyinfo->flags & HA_NOSAME) > && (pk_length + ext_key_length <= MAX_KEY_LENGTH)) (84 chars! where's the coding style again?) and I observe: (gdb) print ext_key_length $45 = 6144 This is clearly more than any [possible] key length for the table. So, this particular patch is not a solution (even if the testcase happens to pass with it). On Mon, May 08, 2017 at 04:23:11PM +0530, Varun wrote: > revision-id: e6dabcd3a7e994feaa1a1dc5d0c1a0d621272f72 (mariadb-10.2.2-387-ge6dabcd) > parent(s): 39bce67b7aa27fc7e5458f36838148f6e0f4c3af > author: Varun Gupta > committer: Varun Gupta > timestamp: 2017-05-08 16:21:32 +0530 > message: > > MDEV-11196: Error:Run-Time Check Failure #2 - Stack around the variable 'key_buff' > was corrupted, server crashes in opt_sum_query > > extended keys feature disabled if the length of extended key is longer than MAX_KEY_LEN > > --- > sql/table.cc | 8 ++++++-- > 1 file changed, 6 insertions(+), 2 deletions(-) > > diff --git a/sql/table.cc b/sql/table.cc > index 398383e..554fc78 100644 > --- a/sql/table.cc > +++ b/sql/table.cc > @@ -695,7 +695,7 @@ static bool create_key_infos(const uchar *strpos, const uchar *frm_image_end, > KEY_PART_INFO *key_part= NULL; > ulong *rec_per_key= NULL; > KEY_PART_INFO *first_key_part= NULL; > - uint first_key_parts= 0; > + uint first_key_parts= 0, pk_length=0, ext_key_length=0; > > if (!keys) > { > @@ -765,6 +765,7 @@ static bool create_key_infos(const uchar *strpos, const uchar *frm_image_end, > keyinfo->algorithm= first_keyinfo->algorithm; > if (new_frm_ver >= 3) > keyinfo->block_size= first_keyinfo->block_size; > + pk_length= keyinfo->key_length; > } > > keyinfo->key_part= key_part; > @@ -796,6 +797,8 @@ static bool create_key_infos(const uchar *strpos, const uchar *frm_image_end, > strpos+=7; > } > key_part->store_length=key_part->length; > + if(i) > + ext_key_length+= key_part->length; > } > > /* > @@ -805,7 +808,8 @@ static bool create_key_infos(const uchar *strpos, const uchar *frm_image_end, > keyinfo->ext_key_parts= keyinfo->user_defined_key_parts; > keyinfo->ext_key_flags= keyinfo->flags; > keyinfo->ext_key_part_map= 0; > - if (share->use_ext_keys && i && !(keyinfo->flags & HA_NOSAME)) > + if (share->use_ext_keys && i && !(keyinfo->flags & HA_NOSAME) > + && (pk_length + ext_key_length <= MAX_KEY_LENGTH)) > { > for (j= 0; > j < first_key_parts && keyinfo->ext_key_parts < MAX_REF_PARTS; > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] MDEV-10764 PL/SQL parser - Phase 2 / MDEV-12137
by jerome brauge 11 May '17

11 May '17

2 1

[Maria-developers] mariadb-java-client 1.6.0 does not work from yesterday update ?
by Krishnan, Gnanavel (Nokia - IN/Chennai) 11 May '17

11 May '17

Hi All, I was using mariadb java client jar version 1.6.0. but looks it is not present in release document Also from yesterday 1.6.0 jar gets updated not sure why Should use only 1.5.9 or 2.0.0 rc ? https://repo1.maven.org/maven2/org/mariadb/jdbc/mariadb-java-client/1.6.0/ Thank you, Gnanavel.

1 0

Re: [Maria-developers] [Commits] 39bce67: MDEV-11196: Error:Run-Time Check Failure #2 - Stack around the variable 'key_buff'
by Sergey Petrunia 10 May '17

10 May '17

Hi Varun, First reaction: > diff --git a/sql/sql_table.cc b/sql/sql_table.cc > index 84e4018..63636f9 100644 > --- a/sql/sql_table.cc > +++ b/sql/sql_table.cc > @@ -3948,7 +3948,11 @@ mysql_prepare_create_table(THD *thd, HA_CREATE_INFO *create_info, > key_part_info->fieldnr= field; > key_part_info->offset= (uint16) sql_field->offset; > key_part_info->key_type=sql_field->pack_flag; > - uint key_part_length= sql_field->key_length; > + uint key_part_length= MY_MAX(sql_field->key_length, > + sql_field->pack_length); What does the above do? It is not clear why MAX should be used. (Let's denote the compared variables X and Y. Can X>Y and X<Y both happen in practice? This requires a comment!) > + > + if (!(sql_field->flags & NOT_NULL_FLAG)) > + key_part_length++; > > if (column->length) > { I am concerned that the above may have an undesired effect: key_part_kength counts NULL-byte while previously it did not. Let's try this example: create table t1 (pk int not null primary key, col1 int, key(col1) ) engine=innodb; Run it, and put a breakpoint below on this code: /* Check if the key segment is partial, set the key flag accordingly */ if (key_part_length != sql_field->key_length) key_info->flags|= HA_KEY_HAS_PART_KEY_SEG; You will see that for filed 'col1' HA_KEY_HAS_PART_KEY_SEG flag is now set. This didn't happen before. Is this really intended? I wasn't able to find what exactly may get wrong, but if I run this query: explain select * from t1 where col1=1; then it hangs. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] MDEV-12775 Reuse data type aggregation code for hybrid functions and UNION
by Alexander Barkov 10 May '17

10 May '17

Hello Sanja, Please review a patch for MDEV-12775, also fixing a number of bugs: - MDEV-9495 Wrong field type for a UNION of a signed and an unsigned INT expression - MDEV-9497 UNION and COALESCE produce different field types for DECIMAL+INT - MDEV-12594 UNION between fixed length double columns does not always preserve scale - MDEV-12595 UNION converts INT to BIGINT - MDEV-12599 UNION is not symmetric when mixing INT and CHAR Thanks!

1 0

[Maria-developers] galera source and direction
by Daniel Black 09 May '17

09 May '17

With Codership's galera repository showing spelling fixes and test case changes as the only changes this year, and Percona's galera repository streaming ahead with improvements, is Percona's galera becoming the new base? I did a quick test though and the mtr tests all pass with Percona's galera on the 10.2 branch head.

1 0

[Maria-developers] Google Summer of Code 2017 students
by Daniel Black 05 May '17

05 May '17

Folks I'd like to welcome our Google Summer of Code students. https://summerofcode.withgoogle.com/organizations/5889423829893120/#projects Ankit Kumar - Allow multiple alternative authentication methods for the same user mentor: Sergei Golubchik Daniil Medvedev - Full DECIMAL support in ColumnStore mentor: Andrew Hutchings Sumit Lakra - NUMA GSoC Project Proposal mentors: Daniel Black and Jan Lindström Galina Shalygina - Transformation of IN predicate into IN subquery based on implementation of table value constructor mentor: Igor Babaev Please make them welcome and we are looking forward to mentoring them to develop their contributions.

1 0

[Maria-developers] MDEV-7049 MySQL#74585 - InnoDB: Failing assertion: *mbmaxlen < 5 in file ha_innodb.cc line 1904
by Alexander Barkov 04 May '17

04 May '17

Hello Marko, InnoDB uses 5 bits to store the combination of mbminlen and mbmaxlen, and its calculated as: mbmaxlen * 5 + mbminlen Note, combinations with mbminlen=0 and mbmaxlen=0 are not possible. Also, combinations with mbminlen>mbmaxlen are not possible. This is a summary: mbminlen mbmaxlen 0 1 2 3 4 0 (0) (1) (2) (3) (4) 1 (5) 6 (7) (8) (9) 2 (10) 11 12 (13) (14) 3 (15) 16 17 18 (19) 4 (20) 21 22 23 24 I marked impossible numbers with parentheses. In the range 0..24: - 10 pairs are valid - 15 pairs are not valid It should be safe to use any impossible number to mean mbmaxlen=5 & mbminlen=1 It seems there is no really a need to extend the header to use separate 3 bits for both mbminlen and mbmaxlen.

2 2

[Maria-developers] MDEV-10142 - bb-10.2-compatibility
by jerome brauge 04 May '17

04 May '17

Hello Alexander, This is the patch for CHR(). Function CHR(n) returns the character having the binary equivalent to n in the database character set. Returns null on null input. Regard, Jérôme. PS : I share this code under terms of MCA. > -----Message d'origine----- > De : Alexander Barkov [mailto:bar@mariadb.org] > Envoyé : mercredi 26 avril 2017 06:12 > À : jerome brauge > Cc : MariaDB Developers > Objet : Re: MDEV-10142 - bb-10.2-compatibility / MDEV-10574 > > Hello Jerome, > > Thanks for working on this! > > On 04/25/2017 11:42 PM, jerome brauge wrote: > > Hello Alexander, > > Some times ago, we have had the below discussion. > > I made a patch following my first idea (it's enough from our point of view) : > replace empty string by null in literals and in make_empty_string. > > I think this patch can be complementary to your suggestion of adding > > an Item_func_eq_oracle (for data inserted in default sql_mode) > > > > This patch also contains following some functions changes: > > - replace ('ab','a',null) returns 'a' instead null > > - trim / ltrim / rtrim returns null when input only contains spaces > > - lpad /rpad can accept 2 args (default padding char is space) and if > > length equals to 0, returns null -substring : if start index is equal > > to 0, act as if it's equals to 1 -add a function chr() as a synonym of > > char() (but accept only one arg) -add a function lengthb() as a > > synonym of lentgh() -change the semantic of length for oracle (where > > length() is the character length and not the byte length) > > > > What do you think about this patch ? > > > > (Perhaps I have to make 2 patchs, one for empty string, and one for > > functions changes with separate test file for each modified function) > > Right, can you please split the patch into pieces? > I suggest even more pieces than two. One patch for one function. > I'll create MDEVs, one MDEV per function and put examples how Oracle > works. > > Can we do it in the following order: > > 1. LPAD / RPAD - making the third parameter optional. > This extension will be good for all sql_modes. > I checked IBM DB2, Oracle PostgreSQL, Firebird - they all have > the third parameter to LPAD/RPAD optional. > > 2. CHR(). > > 3. LENGTH() and LENGTHB(). > > > Then we can continue with REPLACE, TRIM. SUBSTRING in no particular > order, one patch per function. > > > Some general notes: > > > 1. Unify *.yy files > > Please put sql_mode dependent code into LEX rather than *.yy files directly. > We're going to unify the two files soon. See here for details: > MDEV-12518) Unify sql_yacc.yy and sql_yacc_ora.yy > > So instead of doing: > > | SUBSTRING '(' expr ',' expr ',' expr ')' > { > $$= new (thd->mem_root) Item_func_substr(thd, $3, $5, $7); > if ($$ == NULL) > MYSQL_YYABORT; > } > > > Please do something like this in both sql_yacc.yy and sql_yacc_ora.yy: > > > | SUBSTRING '(' expr ',' expr ',' expr ')' > { > if (!($$= Lex->make_item_func_substr(thd, $3, $5, $7))) > MYSQL_YYABORT; > } > > where LEX::make_item_func_substr() is a new method, creating either > Item_func_substr or Item_func_substr_oracle depending on sql_mode. > > The same applies to all other functions, and to text_literal. > > > 2. Don't use Item::is_null() when you call val_str() anyway. > > There are two reasons that: > - Performance > is_null() involves the second val_str() call, which can be resource consuming > in case of complex functions and subselect. > > - Side effect > Expressions can have some side effects. > For example, in case if the argument is a stored function which counts how > many times it was called and stores the counter in a user variable, or writes > something to a table on every execution. > If we do is_null() followed by val_str(), the side effect will done two times. > We need one time only. > > So instead of: > > String *Item_func_replace_oracle::val_str(String *str) { > String * res; > if (res = replace_value(str, > args[1]->is_null() ? &tmp_emtpystr : > rgs[1]->val_str(&tmp_value2), > args[2]->is_null() ? &tmp_emtpystr : > rgs[2]->val_str(&tmp_value3)) > > return (res->length() == 0) ? make_empty_result () : res; > else > return 0; > } > > > Please do something like this: > > String *Item_func_replace_oracle::val_str(String *str) { > String * res; > String *str1= args[1]->val_str(&tmp_value1); > String *str2= args[2]->val_str(&tmp_value2); > if (res= replace_value(str, > str1 ? str1 : &tmp_emptystr, > str2 ? str2 : &tmp_emptystr) > return (res->length() == 0) ? make_empty_result () : res; > else > return 0; > } > > > Thanks! > > > > > > > Best regards, > > Jérôme. > > > >> -----Message d'origine----- > >> De : Alexander Barkov [mailto:bar@mariadb.org] Envoyé : lundi 23 > >> janvier 2017 12:49 À : jerome brauge Cc : MariaDB Developers Objet : > >> Re: MDEV-10142 - bb-10.2-compatibility / MDEV-10574 > >> > >> Hi Jerome, > >> > >> On 01/18/2017 01:22 PM, jerome brauge wrote: > >>> Hello Alexander, > >>> Sometime ago, when I ask you about plan for MDEV-10574, you replied : > >>> > >>>> The current plan is to do these transformations: > >>>> > >>>> 1. Transform Insert > >>>> - insert values ("") -> insert values (null) > >>>> > >>>> 2. Transform Select > >>>> > >>>> - where v=x => (v <> "" and V=X) > >>>> - where v is null => (v="" or v is null) > >>>> > >>>> We didn't plan to change functions yet. Thanks for bringing this up. > >>>> We'll discuss this. > >>> > >>> I've done some tests just by changing : > >>> - insert an Item_null instead of an Item_string when $1.length==0 in > >>> rule text_literal of sql_yacc_ora.yy > >>> - return null instead of an empty string in > >>> Item_str_func::make_empty_result > >>> > >>> My first tests seem promising. > >>> > >>> Of course this solution does not allow to "see" the records created > >>> with > >> empty strings as null values. > >>> I don't see the importance of being able to do this in a transparent way. > >>> We can explicitly select these row by adding rtrim on these columns. > >>> > >>> If you are interesting, I can begin to write a test to evaluate the > >>> coverage of > >> this solution. > >> > >> I'm afraid this will fix the behavior only for literals. > >> > >> This script: > >> > >> DROP TABLE t1; > >> CREATE TABLE t1 (a VARCHAR(10), b VARCHAR(10)); INSERT INTO t1 > VALUES > >> ('',''); SELECT COUNT(*) FROM t1 WHERE a=''; SELECT COUNT(*) FROM t1 > >> WHERE a=b; > >> > >> returns 0 for both SELECT queries in Oracle, and returns 1 for both > >> SELECT queries in MariaDB, Your approach will fix the first SELECT only. > >> The second will still return 1. > >> > >> > >> For now, I'm not sure how to do this correctly. Some ways possible: > >> > >> 1. We could add new classes, e.g. Item_func_eq_oracle as a pair for > >> Item_func_eq, which will handle both empty strings and NULLs inside > >> their > >> val_int() implementations. > >> > >> 2. Or there could be some after-processing on the created Item tree > >> which will replace all things like Item_func_eq to Item_cond_and with > >> Item_func_eq and Item_func_null_predicate passed as arguments. > >> > >> Currently I'm inclined to #1, because we don't have a good mechanism > >> to clone items to implement #2 correctly. We'd need such cloning to > >> pass arguments both to Item_func_eq and Item_func_null_predicate > correctly. > >> Some places in the code don't expect that the same arguments can be > >> passed to two different Item_func instances. > >> > >> On the other hand, implementing #1 will need more changes in the > >> optimizer. > >> > >> We were going to think on this later. > >> > >>> > >>> Best regard. > >>> Jérôme. > >>>

2 6

[Maria-developers] cte page
by Galina Shalygina 04 May '17

04 May '17

мои артиклы

4 5

[Maria-developers] Please review a patch for MDEV-12665 Hybrid functions do not preserve geometry type
by Alexander Barkov 02 May '17

02 May '17

Hello Alexey, Please review a joint patch fixing these two problems: MDEV-12665 Hybrid functions do not preserve geometry type MDEV-12560 Wrong data type for SELECT NULL UNION SELECT Point(1,1) Thanks!

1 0

[Maria-developers] Race Condition in Seconds behind master.
by Sachin Setiya 02 May '17

02 May '17

Hello Everyone! I setup a classical master-slave server. Master is constantly running query like this insert into t1 values(1,1,1,1,1); insert into t1 values(1,1,1,1,1); insert into t1 values(1,1,1,1,1); insert into t1 values(1,1,1,1,1); insert into t1 values(1,1,1,1,1); insert into t1 values(1,1,1,1,1); insert into t1 values(1,1,1,1,1); insert into t1 values(1,1,1,1,1); insert into t1 values(1,1,1,1,1); flush logs; insert into t1 values(1,1,1,1,1); {there are arround million querys} Then I executed Show slave status (couple of times in a second), and found Seconds_behind_master is changing arbitrarily MariaDB [(none)]> show slave status\G *************************** 1. row *************************** Slave_IO_State: Queueing master event to the relay log Master_Host: localhost Master_User: root Master_Port: 16000 Connect_Retry: 60 Master_Log_File: master-bin.014595 Read_Master_Log_Pos: 4 Relay_Log_File: slave-relay-bin.029186 Relay_Log_Pos: 3557 Relay_Master_Log_File: master-bin.014593 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 3257 Relay_Log_Space: 184375 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 148 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: Master_Server_Id: 1 Master_SSL_Crl: Master_SSL_Crlpath: Using_Gtid: Current_Pos Gtid_IO_Pos: 0-1-145942 Replicate_Do_Domain_Ids: Replicate_Ignore_Domain_Ids: Parallel_Mode: conservative SQL_Delay: 0 SQL_Remaining_Delay: NULL Slave_SQL_Running_State: Reading event from the relay log 1 row in set (0.01 sec) MariaDB [(none)]> show slave status\G *************************** 1. row *************************** Slave_IO_State: Waiting for master to send event Master_Host: localhost Master_User: root Master_Port: 16000 Connect_Retry: 60 Master_Log_File: master-bin.014632 Read_Master_Log_Pos: 3257 Relay_Log_File: slave-relay-bin.029264 Relay_Log_Pos: 1548 Relay_Master_Log_File: master-bin.014632 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 1248 Relay_Log_Space: 179996 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 0 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: Master_Server_Id: 1 Master_SSL_Crl: Master_SSL_Crlpath: Using_Gtid: Current_Pos Gtid_IO_Pos: 0-1-146322 Replicate_Do_Domain_Ids: Replicate_Ignore_Domain_Ids: Parallel_Mode: conservative SQL_Delay: 0 SQL_Remaining_Delay: NULL Slave_SQL_Running_State: init 1 row in set (0.00 sec) MariaDB [(none)]> show slave status\G *************************** 1. row *************************** Slave_IO_State: Queueing master event to the relay log Master_Host: localhost Master_User: root Master_Port: 16000 Connect_Retry: 60 Master_Log_File: master-bin.014670 Read_Master_Log_Pos: 299 Relay_Log_File: slave-relay-bin.029336 Relay_Log_Pos: 3557 Relay_Master_Log_File: master-bin.014668 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 3257 Relay_Log_Space: 184569 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 150 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: Master_Server_Id: 1 Master_SSL_Crl: Master_SSL_Crlpath: Using_Gtid: Current_Pos Gtid_IO_Pos: 0-1-146692 Replicate_Do_Domain_Ids: Replicate_Ignore_Domain_Ids: Parallel_Mode: conservative SQL_Delay: 0 SQL_Remaining_Delay: NULL Slave_SQL_Running_State: Reading event from the relay log -- Regards Sachin Setiya Software Engineer at MariaDB

3 2

[Maria-developers] bb-10.2-ext / rpad lpad with 2 args
by jerome brauge 02 May '17

02 May '17

Hi Alexander, This is the patch for rpad/lpad with 2 args (call of make_empty_string() in val_str() is to prepare MDEV-10574). Regard, Jérôme. PS : I share this code under terms of MCA. > -----Message d'origine----- > De : Alexander Barkov [mailto:bar@mariadb.org] > Envoyé : mercredi 26 avril 2017 06:12 > À : jerome brauge > Cc : MariaDB Developers > Objet : Re: MDEV-10142 - bb-10.2-compatibility / MDEV-10574 > > Hello Jerome, > > Thanks for working on this! > > On 04/25/2017 11:42 PM, jerome brauge wrote: > > Hello Alexander, > > Some times ago, we have had the below discussion. > > I made a patch following my first idea (it's enough from our point of view) : > replace empty string by null in literals and in make_empty_string. > > I think this patch can be complementary to your suggestion of adding > > an Item_func_eq_oracle (for data inserted in default sql_mode) > > > > This patch also contains following some functions changes: > > - replace ('ab','a',null) returns 'a' instead null > > - trim / ltrim / rtrim returns null when input only contains spaces > > - lpad /rpad can accept 2 args (default padding char is space) and if > > length equals to 0, returns null -substring : if start index is equal > > to 0, act as if it's equals to 1 -add a function chr() as a synonym of > > char() (but accept only one arg) -add a function lengthb() as a > > synonym of lentgh() -change the semantic of length for oracle (where > > length() is the character length and not the byte length) > > > > What do you think about this patch ? > > > > (Perhaps I have to make 2 patchs, one for empty string, and one for > > functions changes with separate test file for each modified function) > > Right, can you please split the patch into pieces? > I suggest even more pieces than two. One patch for one function. > I'll create MDEVs, one MDEV per function and put examples how Oracle > works. > > Can we do it in the following order: > > 1. LPAD / RPAD - making the third parameter optional. > This extension will be good for all sql_modes. > I checked IBM DB2, Oracle PostgreSQL, Firebird - they all have > the third parameter to LPAD/RPAD optional. > > 2. CHR(). > > 3. LENGTH() and LENGTHB(). > > > Then we can continue with REPLACE, TRIM. SUBSTRING in no particular > order, one patch per function. > > > Some general notes: > > > 1. Unify *.yy files > > Please put sql_mode dependent code into LEX rather than *.yy files directly. > We're going to unify the two files soon. See here for details: > MDEV-12518) Unify sql_yacc.yy and sql_yacc_ora.yy > > So instead of doing: > > | SUBSTRING '(' expr ',' expr ',' expr ')' > { > $$= new (thd->mem_root) Item_func_substr(thd, $3, $5, $7); > if ($$ == NULL) > MYSQL_YYABORT; > } > > > Please do something like this in both sql_yacc.yy and sql_yacc_ora.yy: > > > | SUBSTRING '(' expr ',' expr ',' expr ')' > { > if (!($$= Lex->make_item_func_substr(thd, $3, $5, $7))) > MYSQL_YYABORT; > } > > where LEX::make_item_func_substr() is a new method, creating either > Item_func_substr or Item_func_substr_oracle depending on sql_mode. > > The same applies to all other functions, and to text_literal. > > > 2. Don't use Item::is_null() when you call val_str() anyway. > > There are two reasons that: > - Performance > is_null() involves the second val_str() call, which can be resource consuming > in case of complex functions and subselect. > > - Side effect > Expressions can have some side effects. > For example, in case if the argument is a stored function which counts how > many times it was called and stores the counter in a user variable, or writes > something to a table on every execution. > If we do is_null() followed by val_str(), the side effect will done two times. > We need one time only. > > So instead of: > > String *Item_func_replace_oracle::val_str(String *str) { > String * res; > if (res = replace_value(str, > args[1]->is_null() ? &tmp_emtpystr : > rgs[1]->val_str(&tmp_value2), > args[2]->is_null() ? &tmp_emtpystr : > rgs[2]->val_str(&tmp_value3)) > > return (res->length() == 0) ? make_empty_result () : res; > else > return 0; > } > > > Please do something like this: > > String *Item_func_replace_oracle::val_str(String *str) { > String * res; > String *str1= args[1]->val_str(&tmp_value1); > String *str2= args[2]->val_str(&tmp_value2); > if (res= replace_value(str, > str1 ? str1 : &tmp_emptystr, > str2 ? str2 : &tmp_emptystr) > return (res->length() == 0) ? make_empty_result () : res; > else > return 0; > } > > > Thanks! > > > > > > > Best regards, > > Jérôme. > > > >> -----Message d'origine----- > >> De : Alexander Barkov [mailto:bar@mariadb.org] Envoyé : lundi 23 > >> janvier 2017 12:49 À : jerome brauge Cc : MariaDB Developers Objet : > >> Re: MDEV-10142 - bb-10.2-compatibility / MDEV-10574 > >> > >> Hi Jerome, > >> > >> On 01/18/2017 01:22 PM, jerome brauge wrote: > >>> Hello Alexander, > >>> Sometime ago, when I ask you about plan for MDEV-10574, you replied : > >>> > >>>> The current plan is to do these transformations: > >>>> > >>>> 1. Transform Insert > >>>> - insert values ("") -> insert values (null) > >>>> > >>>> 2. Transform Select > >>>> > >>>> - where v=x => (v <> "" and V=X) > >>>> - where v is null => (v="" or v is null) > >>>> > >>>> We didn't plan to change functions yet. Thanks for bringing this up. > >>>> We'll discuss this. > >>> > >>> I've done some tests just by changing : > >>> - insert an Item_null instead of an Item_string when $1.length==0 in > >>> rule text_literal of sql_yacc_ora.yy > >>> - return null instead of an empty string in > >>> Item_str_func::make_empty_result > >>> > >>> My first tests seem promising. > >>> > >>> Of course this solution does not allow to "see" the records created > >>> with > >> empty strings as null values. > >>> I don't see the importance of being able to do this in a transparent way. > >>> We can explicitly select these row by adding rtrim on these columns. > >>> > >>> If you are interesting, I can begin to write a test to evaluate the > >>> coverage of > >> this solution. > >> > >> I'm afraid this will fix the behavior only for literals. > >> > >> This script: > >> > >> DROP TABLE t1; > >> CREATE TABLE t1 (a VARCHAR(10), b VARCHAR(10)); INSERT INTO t1 > VALUES > >> ('',''); SELECT COUNT(*) FROM t1 WHERE a=''; SELECT COUNT(*) FROM t1 > >> WHERE a=b; > >> > >> returns 0 for both SELECT queries in Oracle, and returns 1 for both > >> SELECT queries in MariaDB, Your approach will fix the first SELECT only. > >> The second will still return 1. > >> > >> > >> For now, I'm not sure how to do this correctly. Some ways possible: > >> > >> 1. We could add new classes, e.g. Item_func_eq_oracle as a pair for > >> Item_func_eq, which will handle both empty strings and NULLs inside > >> their > >> val_int() implementations. > >> > >> 2. Or there could be some after-processing on the created Item tree > >> which will replace all things like Item_func_eq to Item_cond_and with > >> Item_func_eq and Item_func_null_predicate passed as arguments. > >> > >> Currently I'm inclined to #1, because we don't have a good mechanism > >> to clone items to implement #2 correctly. We'd need such cloning to > >> pass arguments both to Item_func_eq and Item_func_null_predicate > correctly. > >> Some places in the code don't expect that the same arguments can be > >> passed to two different Item_func instances. > >> > >> On the other hand, implementing #1 will need more changes in the > >> optimizer. > >> > >> We were going to think on this later. > >> > >>> > >>> Best regard. > >>> Jérôme. > >>>

2 4

[Maria-developers] MDEV-12617 CASE and CASE-alike hybrid functions do not preserve exact data types
by Alexander Barkov 28 Apr '17

28 Apr '17

Hi Sanja, can you please review a patch for MDEV-12617? Thanks!

1 0

[Maria-developers] MDEV-10142 - bb-10.2-compatibility / MDEV-10574
by jerome brauge 28 Apr '17

28 Apr '17

Hello Alexander, Sometime ago, when I ask you about plan for MDEV-10574, you replied : >The current plan is to do these transformations: > >1. Transform Insert >- insert values ("") -> insert values (null) > >2. Transform Select > >- where v=x => (v <> "" and V=X) >- where v is null => (v="" or v is null) > >We didn't plan to change functions yet. Thanks for bringing this up. >We'll discuss this. I've done some tests just by changing : - insert an Item_null instead of an Item_string when $1.length==0 in rule text_literal of sql_yacc_ora.yy - return null instead of an empty string in Item_str_func::make_empty_result My first tests seem promising. Of course this solution does not allow to "see" the records created with empty strings as null values. I don't see the importance of being able to do this in a transparent way. We can explicitly select these row by adding rtrim on these columns. If you are interesting, I can begin to write a test to evaluate the coverage of this solution. Best regard. Jérôme.

2 6

[Maria-developers] MDEV-10142 - bb-10.2-compatibility
by jerome brauge 27 Apr '17

27 Apr '17

Hello Alexander, This is the patch for length/lengthb. Regard, Jérôme. PS : I share this code under terms of MCA. > -----Message d'origine----- > De : Alexander Barkov [mailto:bar@mariadb.org] > Envoyé : mercredi 26 avril 2017 06:12 > À : jerome brauge > Cc : MariaDB Developers > Objet : Re: MDEV-10142 - bb-10.2-compatibility / MDEV-10574 > > Hello Jerome, > > Thanks for working on this! > > On 04/25/2017 11:42 PM, jerome brauge wrote: > > Hello Alexander, > > Some times ago, we have had the below discussion. > > I made a patch following my first idea (it's enough from our point of view) : > replace empty string by null in literals and in make_empty_string. > > I think this patch can be complementary to your suggestion of adding > > an Item_func_eq_oracle (for data inserted in default sql_mode) > > > > This patch also contains following some functions changes: > > - replace ('ab','a',null) returns 'a' instead null > > - trim / ltrim / rtrim returns null when input only contains spaces > > - lpad /rpad can accept 2 args (default padding char is space) and if > > length equals to 0, returns null -substring : if start index is equal > > to 0, act as if it's equals to 1 -add a function chr() as a synonym of > > char() (but accept only one arg) -add a function lengthb() as a > > synonym of lentgh() -change the semantic of length for oracle (where > > length() is the character length and not the byte length) > > > > What do you think about this patch ? > > > > (Perhaps I have to make 2 patchs, one for empty string, and one for > > functions changes with separate test file for each modified function) > > Right, can you please split the patch into pieces? > I suggest even more pieces than two. One patch for one function. > I'll create MDEVs, one MDEV per function and put examples how Oracle > works. > > Can we do it in the following order: > > 1. LPAD / RPAD - making the third parameter optional. > This extension will be good for all sql_modes. > I checked IBM DB2, Oracle PostgreSQL, Firebird - they all have > the third parameter to LPAD/RPAD optional. > > 2. CHR(). > > 3. LENGTH() and LENGTHB(). > > > Then we can continue with REPLACE, TRIM. SUBSTRING in no particular > order, one patch per function. > > > Some general notes: > > > 1. Unify *.yy files > > Please put sql_mode dependent code into LEX rather than *.yy files directly. > We're going to unify the two files soon. See here for details: > MDEV-12518) Unify sql_yacc.yy and sql_yacc_ora.yy > > So instead of doing: > > | SUBSTRING '(' expr ',' expr ',' expr ')' > { > $$= new (thd->mem_root) Item_func_substr(thd, $3, $5, $7); > if ($$ == NULL) > MYSQL_YYABORT; > } > > > Please do something like this in both sql_yacc.yy and sql_yacc_ora.yy: > > > | SUBSTRING '(' expr ',' expr ',' expr ')' > { > if (!($$= Lex->make_item_func_substr(thd, $3, $5, $7))) > MYSQL_YYABORT; > } > > where LEX::make_item_func_substr() is a new method, creating either > Item_func_substr or Item_func_substr_oracle depending on sql_mode. > > The same applies to all other functions, and to text_literal. > > > 2. Don't use Item::is_null() when you call val_str() anyway. > > There are two reasons that: > - Performance > is_null() involves the second val_str() call, which can be resource consuming > in case of complex functions and subselect. > > - Side effect > Expressions can have some side effects. > For example, in case if the argument is a stored function which counts how > many times it was called and stores the counter in a user variable, or writes > something to a table on every execution. > If we do is_null() followed by val_str(), the side effect will done two times. > We need one time only. > > So instead of: > > String *Item_func_replace_oracle::val_str(String *str) { > String * res; > if (res = replace_value(str, > args[1]->is_null() ? &tmp_emtpystr : > rgs[1]->val_str(&tmp_value2), > args[2]->is_null() ? &tmp_emtpystr : > rgs[2]->val_str(&tmp_value3)) > > return (res->length() == 0) ? make_empty_result () : res; > else > return 0; > } > > > Please do something like this: > > String *Item_func_replace_oracle::val_str(String *str) { > String * res; > String *str1= args[1]->val_str(&tmp_value1); > String *str2= args[2]->val_str(&tmp_value2); > if (res= replace_value(str, > str1 ? str1 : &tmp_emptystr, > str2 ? str2 : &tmp_emptystr) > return (res->length() == 0) ? make_empty_result () : res; > else > return 0; > } > > > Thanks! > > > > > > > Best regards, > > Jérôme. > > > >> -----Message d'origine----- > >> De : Alexander Barkov [mailto:bar@mariadb.org] Envoyé : lundi 23 > >> janvier 2017 12:49 À : jerome brauge Cc : MariaDB Developers Objet : > >> Re: MDEV-10142 - bb-10.2-compatibility / MDEV-10574 > >> > >> Hi Jerome, > >> > >> On 01/18/2017 01:22 PM, jerome brauge wrote: > >>> Hello Alexander, > >>> Sometime ago, when I ask you about plan for MDEV-10574, you replied : > >>> > >>>> The current plan is to do these transformations: > >>>> > >>>> 1. Transform Insert > >>>> - insert values ("") -> insert values (null) > >>>> > >>>> 2. Transform Select > >>>> > >>>> - where v=x => (v <> "" and V=X) > >>>> - where v is null => (v="" or v is null) > >>>> > >>>> We didn't plan to change functions yet. Thanks for bringing this up. > >>>> We'll discuss this. > >>> > >>> I've done some tests just by changing : > >>> - insert an Item_null instead of an Item_string when $1.length==0 in > >>> rule text_literal of sql_yacc_ora.yy > >>> - return null instead of an empty string in > >>> Item_str_func::make_empty_result > >>> > >>> My first tests seem promising. > >>> > >>> Of course this solution does not allow to "see" the records created > >>> with > >> empty strings as null values. > >>> I don't see the importance of being able to do this in a transparent way. > >>> We can explicitly select these row by adding rtrim on these columns. > >>> > >>> If you are interesting, I can begin to write a test to evaluate the > >>> coverage of > >> this solution. > >> > >> I'm afraid this will fix the behavior only for literals. > >> > >> This script: > >> > >> DROP TABLE t1; > >> CREATE TABLE t1 (a VARCHAR(10), b VARCHAR(10)); INSERT INTO t1 > VALUES > >> ('',''); SELECT COUNT(*) FROM t1 WHERE a=''; SELECT COUNT(*) FROM t1 > >> WHERE a=b; > >> > >> returns 0 for both SELECT queries in Oracle, and returns 1 for both > >> SELECT queries in MariaDB, Your approach will fix the first SELECT only. > >> The second will still return 1. > >> > >> > >> For now, I'm not sure how to do this correctly. Some ways possible: > >> > >> 1. We could add new classes, e.g. Item_func_eq_oracle as a pair for > >> Item_func_eq, which will handle both empty strings and NULLs inside > >> their > >> val_int() implementations. > >> > >> 2. Or there could be some after-processing on the created Item tree > >> which will replace all things like Item_func_eq to Item_cond_and with > >> Item_func_eq and Item_func_null_predicate passed as arguments. > >> > >> Currently I'm inclined to #1, because we don't have a good mechanism > >> to clone items to implement #2 correctly. We'd need such cloning to > >> pass arguments both to Item_func_eq and Item_func_null_predicate > correctly. > >> Some places in the code don't expect that the same arguments can be > >> passed to two different Item_func instances. > >> > >> On the other hand, implementing #1 will need more changes in the > >> optimizer. > >> > >> We were going to think on this later. > >> > >>> > >>> Best regard. > >>> Jérôme. > >>>

1 0

[Maria-developers] Please review MDEV-12588 Add Type_handler::type_handler_for_tmp_table() and Type_handler::type_handler_for_union()
by Alexander Barkov 25 Apr '17

25 Apr '17

Hello Sanja, Please review a patch for MDEV-12588. Thanks!

1 0

[Maria-developers] Please review MDEV-12426 Add Field::type_handler()
by Alexander Barkov 25 Apr '17

25 Apr '17

Hello Vicențiu, Please review a patch for: MDEV-12426 Add Field::type_handler() also fixing: MDEV-12432 Range optimizer for ENUM and SET does not return "Impossible WHERE" in some case Thank you!

2 2

[Maria-developers] MDEV-9217 Split Item::tmp_table_field_from_field_type() into virtual methods in Type_handler
by Alexander Barkov 24 Apr '17

24 Apr '17

Hello Sanja, Please review a patch for MDEV-9217. Thanks!

2 3

Re: [Maria-developers] MDEV-12506 Split Item_func_min_max::fix_length_and_dec() into methods in Type_handler
by Alexander Barkov 22 Apr '17

22 Apr '17

Hi Alexey, Thanks for your review! On 04/22/2017 06:49 PM, Alexey Botchkov wrote: > Ok to push. > I'd only move the bool Item_func_min_max::fix_attributes(Item **items, > uint nitems) function > to the item_func.h as it's just the typehandler method call now. I'm afraid this is not possible. This method uses current_thd->is_error(), and the structure of THD is not visible in item_func.h. > > Best regards. > HF > > > On Sat, Apr 15, 2017 at 7:23 AM, Alexander Barkov <bar(a)mariadb.org > <mailto:bar@mariadb.org>> wrote: > > Hello Alexey, > > Can you please review a patch for MDEV-12506? > > Thanks. > >

1 0

[Maria-developers] Refactoring in LEX_STRING and LEX_CSTRING
by Alexander Barkov 21 Apr '17

21 Apr '17

Hello all, Monty is now doing refactoring: 1. Replacing a lot of LEX_STRING to LEX_CSTRING. 2. Fixing functions and methods that have arguments of types: a. pointer "const LEX_STRING *name" and b. reference "const LEX_STRING &name". to the same style. Monty chose passing by pointer (2a). I think that: - #1 is good. - #2 is good, in the sense of changing to the same style. But I think that the choice of "passing by pointer" in #2 is bad. Passing by reference for LEX_STRING/LEX_CSTRING related things would be better. With the "passing by pointer" approach whenever one needs to pass a LEX_STRING to a function accepting LEX_CSTRING, one will have to do: void func1(const LEX_STRING *str) { LEX_CSTRING tmp= {str->str, str->length}; func2(&tmp); } I'd propose two things instead: 1. Add a class: class Lex_cstring: public LEX_CSTRING { public: Lex_cstring() {} /*Uninitialized */ Lex_cstring(const char *str, size_t length) { LEX_STRING::str= str; LEX_STRING::length= length; } Lex_cstring(const LEX_STRING *str) { LEX_STRING::str= str->str; LEX_STRING::length= str->length; } }; 2. Instead of "const LEX_CSTRING *str" we use "const Lex_cstring &str" all around C++ code. So the above example with passing a LEX_STRING to something that expect a LEX_CSTRING would be simplified to void func1(const LEX_STRING *str) { func2(str); } 3. In some classes now we have duplicate methods and constructors like this: class XXX { public: XXX(const char *str, size_t length) { ... } XXX(const LEX_STRING *str) { ... } void method(const char *str, size_t length) {...} void method(const LEX_STRING *str) {...} }; to make it possible to call like this: new XXX(str, length); new XXX(lex_str_ptr) new XXX(&lex_str) method(str, length); method(lex_str_ptr); method(&lex_str); We change this to "passing by reference" and remove duplicates: class XXX { public: XXX(const Lex_cstring &str) { ... } void method(const Lex_cstring &str) {...} }; After this change we can call like this: new XXX(Lex_cstring(str, length)); new XXX(*lex_str_ptr) new XXX(lex_str); method(Lex_cstring(str, length)); method(*lex_str_ptr); method(lex_str); Notice, not needs for separate methods accepting two arguments "const char *str, size_t length".

4 5

[Maria-developers] How to report progress for InnoDB background tasks (esp. at startup/shutdown)?
by Marko Mäkelä 18 Apr '17

18 Apr '17

Hi all, MariaDB Server features a nice-looking interface for reporting progress on long-running SQL operations: https://mariadb.com/kb/en/mariadb/progress-reporting/ MySQL 5.7 implemented something similar for reporting progress for ALTER TABLE…ALGORITHM=INPLACE. I should reimplement that for MariaDB using the existing MariaDB infrastructure. I wonder if the same interface could be used for tracking progress on periodic background tasks that might not be directly related to the execution of any SQL statements. InnoDB has a number of background tasks, such as: * purging unneeded history (removing undo log records and delete-marked index records) * rolling back recovered transactions (mainly, if the server was killed) * index defragmentation, encryption key rotation * redo log checkpoints (flush all dirty pages up to the checkpoint LSN; most interesting at shutdown) The above would need to be monitored also while the server is shutting down. I understand that at startup, we would probably need a different progress reporting interfaces for the very first InnoDB task of redo log apply. Everything else can be done in the background while the server is accepting connections. Related to this, I wonder if it is possible or feasible to change the SQL layer logic so that at shutdown, it will not issue ROLLBACK (as part of KILL QUERY) to the currently active connections, but let the subsequent startup do the rollback in the background. This should allow faster shutdowns and restarts. Best regards, Marko -- DON’T MISS M|17April 11 - 12, 2017 The Conrad Hotel New York City https://m17.mariadb.com/ Marko Mäkelä, Lead Developer InnoDB MariaDB Corporation

2 2

[Maria-developers] Item_func_group_concat::fix_fields - indentation and check positioning?
by Daniel Black 18 Apr '17

18 Apr '17

I came across the following gcc warning (6.3.1) sql/item_sum.cc: In member function ‘virtual bool Item_func_group_concat::fix_fields(THD*, Item**)’: sql/item_sum.cc:3478:5: warning: this ‘if’ clause does not guard... [-Wmisleading-indentation] if ((!args[i]->fixed && ^~ sql/item_sum.cc:3482:7: note: ...this statement, but the latter is misleadingly indented as if it is guarded by the ‘if’ with_subselect|= args[i]->with_subselect; ^~~~~~~~~~~~~~ The with_subselect line was added in 950abd526 In context it is: Item_func_group_concat::fix_fields(THD *thd, Item **ref) { ... for (i=0 ; i < arg_count ; i++) { if ((!args[i]->fixed && args[i]->fix_fields(thd, args + i)) || args[i]->check_cols(1)) return TRUE; with_subselect|= args[i]->with_subselect; } While removing the indent would fix the warning, I'd like someone to check this line should be after the IF statement rather than before.

2 1

Re: [Maria-developers] [JIRA] (MDEV-12499) ARCHIVE tables still don't show up in I_S.TABLES when storage engine is not loaded
by Jean-Francois B. Gagne 17 Apr '17

17 Apr '17

Hi Sergei, (sorry for cross-posting to MariaDB.dev and MariaDB.discuss: I think this interest both audiences.) Sergei Golubchik commented on MDEV-12499: > ----------------------------------------- > > This only happens if you created an ARCHIVE table and never used it before > the upgrade. No INSERT, no SELECT, not even {{SELECT * FROM > INFORMATION_SCHEMA.TABLES}} — nothing. > > ARCHIVE engine supports table discovery, that is, the engine is always the > authoritative source of table metadata, while {{.frm}} files are merely a > metadata cache. When you do {{CREATE TABLE}} the server does not know > whether the engine wants the table metadata to be cached. This happens when > the table is opened for the first time. > > If there is no {{.frm}} file and the ARCHIVE engine is not loaded, the > server has no way of knowing that the ARCHIVE table exists. That's why it's > not shown anywhere. > > If you do {{SELECT * FROM test.a1}} before the upgrade, the engine will > tell the server to create the {{.frm}} file and on the upgrade the plugin > will be loaded. In a practical use case, it's enough to have at least one > ARCHIVE table that was accessed at least once after it was created, and the > ARCHIVE plugin will be loaded on upgrade. > Thanks for the detailed info above. In this, you write ".frm files are merely a metadata cache", is there any sort of documentation about when .frm files are useful, useless, and might get out of sync ? This probably deserves a page in the KB (and Google does not know much on "mariadb frm file site:mariadb.com"). However, your comment above raises the following question: if "ARCHIVE engine supports table discovery, that is, the engine is always the authoritative source of table metadata", where is this information stored ? I guess that unless the plugin is loaded, this information is not accessible. This causes a chicken and egg problem which MDEV-12499 was trying to solve: how can mysql_upgrade knows if it should add BLACKHOLE and ARCHIVE to the list of plugin to load (see MDEV-11942). Finally, you marked MDEV-12499 as "Won't Fix", do you have another solution for ARCHIVE and MDEV-11942 ? Many thanks, JFG

2 1

[Maria-developers] MDEV-12514 Split Item_temporal_func::fix_length_and_dec()
by Alexander Barkov 17 Apr '17

17 Apr '17

Hi Sanja, Please review a patch for MDEV-12514, also fixing this bug: MDEV-12515 Wrong value when storing DATE_ADD() and ADDTIME() to a numeric field Thanks!

1 1

[Maria-developers] MDEV-12506 Split Item_func_min_max::fix_length_and_dec() into methods in Type_handler
by Alexander Barkov 15 Apr '17

15 Apr '17

Hello Alexey, Can you please review a patch for MDEV-12506? Thanks.

1 0

[Maria-developers] Please review patch for 12017 and 12018.
by Sachin Setiya 09 Apr '17

09 Apr '17

Hi Monty!, Please review patch for 12017 and 12018 . this patch solves both the problem. -- Regards Sachin Setiya Software Engineer at MariaDB

2 1

[Maria-developers] compilation failure: my_atomic.h
by Alexander Barkov 08 Apr '17

08 Apr '17

Hi Sergey, kvm-bintar-centos5-x86 fails to compile: http://buildbot.askmonty.org/buildbot/grid?category=main&branch=10.3 with this error: /home/buildbot/buildbot/build/include/my_atomic.h:121:2: error: #error atomic ops for this platform are not implemented Can you please fix this?

3 4

[Maria-developers] Adding "proxy protocol" support to MariaDB
by Jose 07 Apr '17

07 Apr '17

Hi! As you may know, the proxy protocol [1,2] is a way to communicate a client's IP address to a backend server sitting behind a proxy front-end. It is the equivalent of the X-Forwarded-For HTTP header, but done at Layer 4. This protocol was originally proposed by one of the HAProxy developers and its support is becoming more widespread. Percona, Amazon AWS, HAProxy, NGINX, and others. Mariadb's Jira shows an unassigned feature request for it, but with no one apparently assigned to it [3]. Percona supports proxy protocol [4]. For the configuration part, Percona defines a global variable that gives the IP address of the proxy front-ends that are authorized to send a proxy protocol header. Percona's handler code for the proxy protocol (minus the global variable handling) is in the vio/viosockets.c file [4] I looked at Percona's implementation. This file looks very similar to MariaDb's own viosockets.c file [5] so it may a potential good insertion point. Percona's software license seems to be permisive enough for reusing / adapting their code [6]. Please correct me if I'm wrong but it seems no one is currently working on this feature right now. I'd like to know if you would be interested in help side-porting Percona's proxy protocol code to MariaDB. I'll probably need some help or pointers on how to add a global variable, though. Best regards, --jose [1] Proxy protocol documentation https://blog.haproxy.com/haproxy/proxy-protocol/ [2] Proxy protocol specification www.haproxy.org/download/1.8/doc/proxy-protocol.txt [3] Mariadb feature request for proxy protocol https://jira.mariadb.org/browse/MDEV-11159?jql=project%20%3D%20MDEV%20AND%2… [3] Percona proxy protocol manual page https://www.percona.com/doc/percona-server/5.6/flexibility/proxy_protocol_s… [4] Percona proxy protocol handler code https://github.com/percona/percona-server/blob/1e2f003a5bd48763c27e37542d97… https://github.com/percona/percona-server/blob/1e2f003a5bd48763c27e37542d97… [5] mariadb potential insertion pode for proxy protocol code https://github.com/MariaDB/server/blob/bb2c1a52c61706dde8c525a8887f2d364c0d… [6] Percona software license https://www.percona.com/doc/percona-server/LATEST/copyright.html

3 3