- developers - lists.mariadb.org

Re: [Maria-developers] 8790c095ab0: MDEV-21704 Add a new JSON field "version_id" into mysql.global_priv.priv
by Sergei Golubchik 20 Feb '20

20 Feb '20

Hi, Alexander! Looks ok, just one test related comment. On Feb 20, Alexander Barkov wrote: > revision-id: 8790c095ab0 (mariadb-10.5.0-225-g8790c095ab0) > parent(s): 2c34315df6e > author: Alexander Barkov <bar(a)mariadb.com> > committer: Alexander Barkov <bar(a)mariadb.com> > timestamp: 2020-02-14 21:56:12 +0400 > message: > > MDEV-21704 Add a new JSON field "version_id" into mysql.global_priv.priv > > +--log-error=$MYSQLTEST_VARDIR/tmp/system_mysql_db_error_log.err ... > +CREATE TABLE error_log (msg TEXT); > +--disable_query_log > +--eval LOAD DATA INFILE "$MYSQLTEST_VARDIR/tmp/system_mysql_db_error_log.err" INTO TABLE error_log; > +--enable_query_log > + > +# On some platforms hours less than 10 can be logged without the leading zero: > +# '2020-02-13 8:16:56' rather than > +# '2020-02-13 08:16:56' > +# Hence the space character in the hour part of the regular expression. > +# Also, replace '\r' to '', so the output is equal on Linux and Windows. > + > +SELECT > + REPLACE( > + REGEXP_REPLACE(msg, > + '^[0-9]{4}-[0-9]{2}-[0-9]{2} [ 0-9]{2}:[0-9]{2}:[0-9]{2}', > + 'YYYY-MM-DD hh:mm:ss'), > + '\r','' > + ) AS msg > +FROM error_log WHERE msg LIKE '%user%entry%'; instead of --log-error and LOAD DATA with REGEXP_REPLACE, please use something like --let SEARCH_FILE=$MYSQLTEST_VARDIR/log/mysqld.1.err --let SEARCH_PATTERN="'user' entry 'bad_.*has a wrong" --source include/search_pattern_in_file.inc this will print that there were 4 matches. Alternatively you can use four very specific SEARCH_PATTERN, one for each expected error. It's easier than what you've used, keeps the error log in one file, and doesn't run unnecessary sql on the server (otherwise debugging quickly becomes very annoying, grr, replication tests). Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] bd98ef106c8: MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT
by Sergei Golubchik 20 Feb '20

20 Feb '20

Hi, Aleksey! 1. It should be possible to enable/disable auto-creation. For example, CREATE TABLE ... PARTITION BY SYSTEM_TIME ... PARTITIONS AUTO; this solves few problems at once: * a user explicitly tells when auto-creation should work * you don't need to worry "if this name is already occupied" * you can define that these partitions can never overflow (for INTERVAL) * if you know that AUTO partitions never overflow, you can keep the old behavior for ALTER TABLE ADD PARTITION. 2. Separate thread is an interesting solution. Nicely avoids lots of problems. Still: * it's asynchronous, all tests have to take it into account * it's racy. "low probability" or not, with our number of users they'll hit it and rightfully will report it as a bug * if LOCK TABLE prevents it, partitions can overflow But I think these problems are easier to solve than those we'll face if auto-creation will happen in the main connection thread. So, let's keep your approach with a thread. But instead of going through the parser and mysql_execute_command, create a function that takes a TABLE or a TABLE_SHARE. And adds a partition there. It'll fix the "racy" part. This function can be called from a new thread. As for the LOCK TABLES - if you're under LOCK TABLES, you can simply call that function (to add a partition) directly at the end of the main INSERT/UPDATE/DELETE statement. It'll solve the last problem, LOCK TABLES won't prevent auto-creation. A couple of style comments: 1. create a one-liner make_partition_name (or something) with: sprintf(move_ptr, "p%u", i) and use it both in create_default_partition_names() and in your new code. Just to make sure partition names are always generated uniformly by the same function and if we'll want to change it to "pp%u" it should be done only in one place. 2. don't overload operators, use, say, bool greater_than(size_t seconds) But overall it's pretty good. A nice idea with a separate thread. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org On Nov 07, Aleksey Midenkov wrote: > revision-id: bd98ef106c8 (mariadb-10.4.7-55-gbd98ef106c8) > parent(s): 6d1fa01e8f1 > author: Aleksey Midenkov <midenok(a)gmail.com> > committer: Aleksey Midenkov <midenok(a)gmail.com> > timestamp: 2019-08-19 12:01:14 +0300 > message: > > MDEV-17554 Auto-create new partition for system versioned tables with history partitioned by INTERVAL/LIMIT > > When there are E empty partitions left, auto-create N new empty > partitions. > > This scheme must not allow partition overflow. I.e. E-fill time must > not exceed N-creation time. This means that low values for INTERVAL > and LIMIT must not be allowed for auto-creation. In case when overflow > is detected there is no need to do anything special: a warning will be > issued and the user will run manual rebuild to redistribute records > correctly. This is important because automatic ADD must be done fast, > without forced rebuild, by the obvious reason. > > Initial version implements hard-coded values of 1 for E and N. As well > as auto-creation threshold MinInterval = 1 hour, MinLimit = 1000. > > The name for newly added partition will be first chosen as "pX", where > X is partition number and "p" is hard-coded name prefix. If this name > is already occupied, the X will be incremented until the resulting > name will be free to use. > > Auto-creation mechanism is applied to every table having LIMIT or > INTERVAL clause. Note that there is no much sense in specifying > explicit partition list in this case and this is covered by > MDEV-19903. The syntax to explicitly turn it on/off as well as > user-defined values for E, N and name prefix is subject for further > discussion and feature requests. > > ALTER TABLE ADD PARTITION is now always fast. If there some history > partition overflow occurs manual ALTER TABLE REBUILD PARTITION is > needed.

2 5

[Maria-developers] Fixing 10.4+ build on OpenBSD
by Brad Smith 15 Feb '20

15 Feb '20

Hi, Looking for some assistance in getting a patch commited to resolve MDEV-19603. Since Galera was upgraded from 3 to 4 the CMake bits include a hardcoded libdl which broke the build of 10.4+ on OpenBSD.

1 0

Re: [Maria-developers] [Commits] a906aae: MDEV-21610 Different query results from 10.4.11 to 10.4.12
by Sergey Petrunia 14 Feb '20

14 Feb '20

Hello Igor, I've found a deficiency in the scheme used by the patch: it doesn't work if the join buffer is re-filled multiple times. On the first execution, everything works as intended. dsmrr_init() executes these lines: rowid_filter= h_arg->pushed_rowid_filter; h_arg->cancel_pushed_rowid_filter(); then I can see that Mrr_ordered_rndpos_reader::refill_from_index_reader uses the filter. The MRR scan continues until it finishes. Then, SQL layer fills the join buffer again, and calls multi_range_read_init() again, which calls dsmrr_init(). And here, h_arg->pushed_rowid_filter==NULL (as we've cleared it previously), and the second MRR scan is not used anymore. Example that I used for debugging (maybe it's larger than necessary): create table t10 ( pk int primary key, a int, b int, filler char(100), key(a), key(b) ); insert into t10 select A.a + 1000 *B.a, A.a + 1000 *B.a, A.a + 1000 *B.a, 'filler-data=FILLER=DATA' from one_k A, one_k B; create table t11 (a int); insert into t11 select a from one_k where a < 200; set optimizer_switch='mrr=on'; set join_cache_level=6; set join_buffer_size=128; MariaDB [test]> explain select * from t11, t10 where t10.a=t11.a and t10.b < 300; +------+-------------+-------+------------+---------------+------+---------+------------+--------+-----------------------------------------------------------------------------------------+ | id | select_type | table | type | possible_keys | key | key_len | ref | rows | Extra | +------+-------------+-------+------------+---------------+------+---------+------------+--------+-----------------------------------------------------------------------------------------+ | 1 | SIMPLE | t11 | ALL | NULL | NULL | NULL | NULL | 200 | Using where | | 1 | SIMPLE | t10 | ref|filter | a,b | a|b | 5|5 | test.t11.a | 1 (0%) | Using where; Using join buffer (flat, BKA join); Rowid-ordered scan; Using rowid filter | +------+-------------+-------+------------+---------------+------+---------+------------+--------+-----------------------------------------------------------------------------------------+ select * from t11, t10 where t10.a=t11.a and t10.b < 300; Ideas about the solution: 1. DS-MRR code should put the rowid filter back into the 'h_arg' handler when the scan finishes (i.e. returns HA_ERR_END_OF_FILE) 2. DS-MRR code should store the rowid filter internally (and clear it up when dsmrr_close() is called. I haven't investigated which is better. On Thu, Feb 13, 2020 at 10:55:56PM -0800, IgorBabaev wrote: > revision-id: a906aaee26a7be57fe2db62214179476ec124486 (mariadb-10.4.11-38-ga906aae) > parent(s): 7ea413ac2d80c7f03d1dbad90ac30ecddd8b2835 > author: Igor Babaev > committer: Igor Babaev > timestamp: 2020-02-13 22:55:56 -0800 > message: > > MDEV-21610 Different query results from 10.4.11 to 10.4.12 > > This patch fixes the following defects/bugs. > 1. If BKA[H] algorithm was used to join a table for which the optimizer > had decided to employ a rowid filter the filter actually was not built. > 2. The patch for the bug MDEV-21356 that added the code canceling pushing > rowid filter into an engine for the table joined with employment of > BKA[H] and MRR was not quite correct for Innodb engine because this > cancellation was done after InnoDB code had already bound the the pushed > filter to internal InnoDB structures. > > --- > mysql-test/main/rowid_filter_innodb.result | 333 +++++++++++++++++++++++++++++ > mysql-test/main/rowid_filter_innodb.test | 153 +++++++++++++ > sql/multi_range_read.cc | 39 ++-- > sql/multi_range_read.h | 5 +- > sql/opt_range.cc | 3 +- > sql/sql_join_cache.cc | 2 + > 6 files changed, 515 insertions(+), 20 deletions(-) > > diff --git a/mysql-test/main/rowid_filter_innodb.result b/mysql-test/main/rowid_filter_innodb.result > index c59b95b..9423fb1 100644 > --- a/mysql-test/main/rowid_filter_innodb.result > +++ b/mysql-test/main/rowid_filter_innodb.result > @@ -2522,3 +2522,336 @@ id select_type table type possible_keys key key_len ref rows r_rows filtered r_f > 1 SIMPLE t1 index a,b PRIMARY 4 NULL 3008 3008.00 1.36 0.00 Using where > DROP TABLE t1; > SET global innodb_stats_persistent= @stats.save; > +# > +# MDEV-21610: Using rowid filter with BKA+MRR > +# > +set @stats.save= @@innodb_stats_persistent; > +set global innodb_stats_persistent=on; > +CREATE TABLE acli ( > +id bigint(20) NOT NULL, > +rid varchar(255) NOT NULL, > +tp smallint(6) NOT NULL DEFAULT 0, > +PRIMARY KEY (id), > +KEY acli_rid (rid), > +KEY acli_tp (tp) > +) ENGINE=InnoDB DEFAULT CHARSET=utf8; > +insert into acli(id,rid,tp) values > +(184929059698905997,'ABABABABABABABABAB',103), > +(184929059698905998,'ABABABABABABABABAB',121), > +(283586039035985921,'00000000000000000000000000000000',103), > +(2216474704108064678,'020BED6D07B741CE9B10AB2200FEF1DF',103), > +(2216474704108064679,'020BED6D07B741CE9B10AB2200FEF1DF',121), > +(3080602882609775593,'B5FCC8C7111E4E3CBC21AAF5012F59C2',103), > +(3080602882609775594,'B5FCC8C7111E4E3CBC21AAF5012F59C2',121), > +(3080602882609776594,'B5FCC8C7111E4E3CBC21AAF5012F59C2',121), > +(3080602882609777595,'B5FCC8C7111E4E3CBC21AAF5012F59C2',121), > +(4269412446747236214,'SCSCSCSCSCSCSCSC',103), > +(4269412446747236215,'SCSCSCSCSCSCSCSC',121), > +(6341490487802728356,'6072D47E513F4A4794BBAB2200FDB67D',103), > +(6341490487802728357,'6072D47E513F4A4794BBAB2200FDB67D',121); > +CREATE TABLE acei ( > +id bigint(20) NOT NULL, > +aclid bigint(20) NOT NULL DEFAULT 0, > +atp smallint(6) NOT NULL DEFAULT 0, > +clus smallint(6) NOT NULL DEFAULT 0, > +PRIMARY KEY (id), > +KEY acei_aclid (aclid), > +KEY acei_clus (clus) > +) ENGINE=InnoDB DEFAULT CHARSET=utf8; > +insert into acei(id,aclid,atp,clus) values > +(184929059698905999,184929059698905997,0,1), > +(184929059698906000,184929059698905997,0,1), > +(184929059698906001,184929059698905997,1,1), > +(184929059698906002,184929059698905998,1,1), > +(283586039035985922,283586039035985921,1,1), > +(2216474704108064684,2216474704108064678,0,1), > +(2216474704108064685,2216474704108064678,0,1), > +(2216474704108064686,2216474704108064678,1,1), > +(2216474704108064687,2216474704108064679,1,1), > +(3080602882609775595,3080602882609775593,0,1), > +(3080602882609775596,3080602882609775593,0,1), > +(3080602882609775597,3080602882609775593,1,1), > +(3080602882609775598,3080602882609775594,1,1), > +(3080602882609776595,3080602882609776594,1,1), > +(3080602882609777596,3080602882609777595,1,1), > +(4269412446747236216,4269412446747236214,0,1), > +(4269412446747236217,4269412446747236214,0,1), > +(4269412446747236218,4269412446747236214,1,1), > +(4269412446747236219,4269412446747236215,1,1), > +(6341490487802728358,6341490487802728356,0,1), > +(6341490487802728359,6341490487802728356,0,1), > +(6341490487802728360,6341490487802728356,1,1), > +(6341490487802728361,6341490487802728357,1,1); > +CREATE TABLE filt ( > +id bigint(20) NOT NULL, > +aceid bigint(20) NOT NULL DEFAULT 0, > +clid smallint(6) NOT NULL DEFAULT 0, > +fh bigint(20) NOT NULL DEFAULT 0, > +PRIMARY KEY (id), > +KEY filt_aceid (aceid), > +KEY filt_clid (clid), > +KEY filt_fh (fh) > +) ENGINE=InnoDB DEFAULT CHARSET=utf8; > +insert into filt(id,aceid,clid,fh) values > +(184929059698905999,184929059698905999,1,8948400944397203540), > +(184929059698906000,184929059698906000,1,-3516039679025944536), > +(184929059698906001,184929059698906001,1,-3516039679025944536), > +(184929059698906002,184929059698906001,1,2965370193075218252), > +(184929059698906003,184929059698906001,1,8948400944397203540), > +(184929059698906004,184929059698906002,1,2478709353550777738), > +(283586039035985922,283586039035985922,1,5902600816362013271), > +(2216474704108064686,2216474704108064684,1,8948400944397203540), > +(2216474704108064687,2216474704108064685,1,-7244708939311117030), > +(2216474704108064688,2216474704108064686,1,-7244708939311117030), > +(2216474704108064689,2216474704108064686,1,7489060986210282479), > +(2216474704108064690,2216474704108064686,1,8948400944397203540), > +(2216474704108064691,2216474704108064687,1,-3575268945274980038), > +(3080602882609775595,3080602882609775595,1,8948400944397203540), > +(3080602882609775596,3080602882609775596,1,-5420422472375069774), > +(3080602882609775597,3080602882609775597,1,-5420422472375069774), > +(3080602882609775598,3080602882609775597,1,8518228073041491534), > +(3080602882609775599,3080602882609775597,1,8948400944397203540), > +(3080602882609775600,3080602882609775598,1,6311439873746261694), > +(3080602882609775601,3080602882609775598,1,6311439873746261694), > +(3080602882609776595,3080602882609776595,1,-661101805245999843), > +(3080602882609777596,3080602882609777596,1,-661101805245999843), > +(3080602882609777597,3080602882609777596,1,2216865386202464067), > +(4269412446747236216,4269412446747236216,1,8948400944397203540), > +(4269412446747236217,4269412446747236217,1,-1143096194892676000), > +(4269412446747236218,4269412446747236218,1,-1143096194892676000), > +(4269412446747236219,4269412446747236218,1,5313391811364818290), > +(4269412446747236220,4269412446747236218,1,8948400944397203540), > +(4269412446747236221,4269412446747236219,1,7624499822621753835), > +(6341490487802728358,6341490487802728358,1,8948400944397203540), > +(6341490487802728359,6341490487802728359,1,8141092449587136068), > +(6341490487802728360,6341490487802728360,1,8141092449587136068), > +(6341490487802728361,6341490487802728360,1,1291319099896431785), > +(6341490487802728362,6341490487802728360,1,8948400944397203540), > +(6341490487802728363,6341490487802728361,1,6701841652906431497); > +analyze table filt, acei, acli; > +Table Op Msg_type Msg_text > +test.filt analyze status Engine-independent statistics collected > +test.filt analyze status OK > +test.acei analyze status Engine-independent statistics collected > +test.acei analyze status OK > +test.acli analyze status Engine-independent statistics collected > +test.acli analyze status OK > +set @save_optimizer_switch=@@optimizer_switch; > +set @save_join_cache_level=@@join_cache_level; > +set optimizer_switch='mrr=off'; > +set join_cache_level=2; > +set statement optimizer_switch='rowid_filter=off' for explain extended select t.id, fi.* > +from (acli t inner join acei a on a.aclid = t.id) > +inner join filt fi on a.id = fi.aceid > +where > +t.rid = 'B5FCC8C7111E4E3CBC21AAF5012F59C2' and > +t.tp = 121 and > +a.atp = 1 and > +fi.fh in (6311439873746261694,-397087483897438286, > +8518228073041491534,-5420422472375069774); > +id select_type table type possible_keys key key_len ref rows filtered Extra > +1 SIMPLE t index_merge PRIMARY,acli_rid,acli_tp acli_tp,acli_rid 2,767 NULL 2 100.00 Using intersect(acli_tp,acli_rid); Using where; Using index > +1 SIMPLE a ref PRIMARY,acei_aclid acei_aclid 8 test.t.id 1 100.00 Using where > +1 SIMPLE fi ref filt_aceid,filt_fh filt_aceid 8 test.a.id 1 17.14 Using where > +Warnings: > +Note 1003 select `test`.`t`.`id` AS `id`,`test`.`fi`.`id` AS `id`,`test`.`fi`.`aceid` AS `aceid`,`test`.`fi`.`clid` AS `clid`,`test`.`fi`.`fh` AS `fh` from `test`.`acli` `t` join `test`.`acei` `a` join `test`.`filt` `fi` where `test`.`t`.`tp` = 121 and `test`.`a`.`atp` = 1 and `test`.`fi`.`aceid` = `test`.`a`.`id` and `test`.`a`.`aclid` = `test`.`t`.`id` and `test`.`t`.`rid` = 'B5FCC8C7111E4E3CBC21AAF5012F59C2' and `test`.`fi`.`fh` in (6311439873746261694,-397087483897438286,8518228073041491534,-5420422472375069774) > +set statement optimizer_switch='rowid_filter=off' for select t.id, fi.* > +from (acli t inner join acei a on a.aclid = t.id) > +inner join filt fi on a.id = fi.aceid > +where > +t.rid = 'B5FCC8C7111E4E3CBC21AAF5012F59C2' and > +t.tp = 121 and > +a.atp = 1 and > +fi.fh in (6311439873746261694,-397087483897438286, > +8518228073041491534,-5420422472375069774); > +id id aceid clid fh > +3080602882609775594 3080602882609775600 3080602882609775598 1 6311439873746261694 > +3080602882609775594 3080602882609775601 3080602882609775598 1 6311439873746261694 > +set statement optimizer_switch='rowid_filter=on' for explain extended select t.id, fi.* > +from (acli t inner join acei a on a.aclid = t.id) > +inner join filt fi on a.id = fi.aceid > +where > +t.rid = 'B5FCC8C7111E4E3CBC21AAF5012F59C2' and > +t.tp = 121 and > +a.atp = 1 and > +fi.fh in (6311439873746261694,-397087483897438286, > +8518228073041491534,-5420422472375069774); > +id select_type table type possible_keys key key_len ref rows filtered Extra > +1 SIMPLE t index_merge PRIMARY,acli_rid,acli_tp acli_tp,acli_rid 2,767 NULL 2 100.00 Using intersect(acli_tp,acli_rid); Using where; Using index > +1 SIMPLE a ref PRIMARY,acei_aclid acei_aclid 8 test.t.id 1 100.00 Using where > +1 SIMPLE fi ref|filter filt_aceid,filt_fh filt_aceid|filt_fh 8|8 test.a.id 1 (17%) 17.14 Using where; Using rowid filter > +Warnings: > +Note 1003 select `test`.`t`.`id` AS `id`,`test`.`fi`.`id` AS `id`,`test`.`fi`.`aceid` AS `aceid`,`test`.`fi`.`clid` AS `clid`,`test`.`fi`.`fh` AS `fh` from `test`.`acli` `t` join `test`.`acei` `a` join `test`.`filt` `fi` where `test`.`t`.`tp` = 121 and `test`.`a`.`atp` = 1 and `test`.`fi`.`aceid` = `test`.`a`.`id` and `test`.`a`.`aclid` = `test`.`t`.`id` and `test`.`t`.`rid` = 'B5FCC8C7111E4E3CBC21AAF5012F59C2' and `test`.`fi`.`fh` in (6311439873746261694,-397087483897438286,8518228073041491534,-5420422472375069774) > +set statement optimizer_switch='rowid_filter=on' for select t.id, fi.* > +from (acli t inner join acei a on a.aclid = t.id) > +inner join filt fi on a.id = fi.aceid > +where > +t.rid = 'B5FCC8C7111E4E3CBC21AAF5012F59C2' and > +t.tp = 121 and > +a.atp = 1 and > +fi.fh in (6311439873746261694,-397087483897438286, > +8518228073041491534,-5420422472375069774); > +id id aceid clid fh > +3080602882609775594 3080602882609775600 3080602882609775598 1 6311439873746261694 > +3080602882609775594 3080602882609775601 3080602882609775598 1 6311439873746261694 > +set optimizer_switch='mrr=on'; > +set join_cache_level=6; > +set statement optimizer_switch='rowid_filter=off' for explain extended select t.id, fi.* > +from (acli t inner join acei a on a.aclid = t.id) > +inner join filt fi on a.id = fi.aceid > +where > +t.rid = 'B5FCC8C7111E4E3CBC21AAF5012F59C2' and > +t.tp = 121 and > +a.atp = 1 and > +fi.fh in (6311439873746261694,-397087483897438286, > +8518228073041491534,-5420422472375069774); > +id select_type table type possible_keys key key_len ref rows filtered Extra > +1 SIMPLE t index_merge PRIMARY,acli_rid,acli_tp acli_tp,acli_rid 2,767 NULL 2 100.00 Using intersect(acli_tp,acli_rid); Using where; Using index > +1 SIMPLE a ref PRIMARY,acei_aclid acei_aclid 8 test.t.id 1 100.00 Using where; Using join buffer (flat, BKA join); Rowid-ordered scan > +1 SIMPLE fi ref filt_aceid,filt_fh filt_aceid 8 test.a.id 1 17.14 Using where; Using join buffer (incremental, BKA join); Rowid-ordered scan > +Warnings: > +Note 1003 select `test`.`t`.`id` AS `id`,`test`.`fi`.`id` AS `id`,`test`.`fi`.`aceid` AS `aceid`,`test`.`fi`.`clid` AS `clid`,`test`.`fi`.`fh` AS `fh` from `test`.`acli` `t` join `test`.`acei` `a` join `test`.`filt` `fi` where `test`.`t`.`tp` = 121 and `test`.`a`.`atp` = 1 and `test`.`fi`.`aceid` = `test`.`a`.`id` and `test`.`a`.`aclid` = `test`.`t`.`id` and `test`.`t`.`rid` = 'B5FCC8C7111E4E3CBC21AAF5012F59C2' and `test`.`fi`.`fh` in (6311439873746261694,-397087483897438286,8518228073041491534,-5420422472375069774) > +set statement optimizer_switch='rowid_filter=off' for select t.id, fi.* > +from (acli t inner join acei a on a.aclid = t.id) > +inner join filt fi on a.id = fi.aceid > +where > +t.rid = 'B5FCC8C7111E4E3CBC21AAF5012F59C2' and > +t.tp = 121 and > +a.atp = 1 and > +fi.fh in (6311439873746261694,-397087483897438286, > +8518228073041491534,-5420422472375069774); > +id id aceid clid fh > +3080602882609775594 3080602882609775600 3080602882609775598 1 6311439873746261694 > +3080602882609775594 3080602882609775601 3080602882609775598 1 6311439873746261694 > +set statement optimizer_switch='rowid_filter=on' for explain extended select t.id, fi.* > +from (acli t inner join acei a on a.aclid = t.id) > +inner join filt fi on a.id = fi.aceid > +where > +t.rid = 'B5FCC8C7111E4E3CBC21AAF5012F59C2' and > +t.tp = 121 and > +a.atp = 1 and > +fi.fh in (6311439873746261694,-397087483897438286, > +8518228073041491534,-5420422472375069774); > +id select_type table type possible_keys key key_len ref rows filtered Extra > +1 SIMPLE t index_merge PRIMARY,acli_rid,acli_tp acli_tp,acli_rid 2,767 NULL 2 100.00 Using intersect(acli_tp,acli_rid); Using where; Using index > +1 SIMPLE a ref PRIMARY,acei_aclid acei_aclid 8 test.t.id 1 100.00 Using where; Using join buffer (flat, BKA join); Rowid-ordered scan > +1 SIMPLE fi ref|filter filt_aceid,filt_fh filt_aceid|filt_fh 8|8 test.a.id 1 (17%) 17.14 Using where; Using join buffer (incremental, BKA join); Rowid-ordered scan; Using rowid filter > +Warnings: > +Note 1003 select `test`.`t`.`id` AS `id`,`test`.`fi`.`id` AS `id`,`test`.`fi`.`aceid` AS `aceid`,`test`.`fi`.`clid` AS `clid`,`test`.`fi`.`fh` AS `fh` from `test`.`acli` `t` join `test`.`acei` `a` join `test`.`filt` `fi` where `test`.`t`.`tp` = 121 and `test`.`a`.`atp` = 1 and `test`.`fi`.`aceid` = `test`.`a`.`id` and `test`.`a`.`aclid` = `test`.`t`.`id` and `test`.`t`.`rid` = 'B5FCC8C7111E4E3CBC21AAF5012F59C2' and `test`.`fi`.`fh` in (6311439873746261694,-397087483897438286,8518228073041491534,-5420422472375069774) > +set statement optimizer_switch='rowid_filter=on' for select t.id, fi.* > +from (acli t inner join acei a on a.aclid = t.id) > +inner join filt fi on a.id = fi.aceid > +where > +t.rid = 'B5FCC8C7111E4E3CBC21AAF5012F59C2' and > +t.tp = 121 and > +a.atp = 1 and > +fi.fh in (6311439873746261694,-397087483897438286, > +8518228073041491534,-5420422472375069774); > +id id aceid clid fh > +3080602882609775594 3080602882609775600 3080602882609775598 1 6311439873746261694 > +3080602882609775594 3080602882609775601 3080602882609775598 1 6311439873746261694 > +set statement optimizer_switch='rowid_filter=on' for analyze format=json select t.id, fi.* > +from (acli t inner join acei a on a.aclid = t.id) > +inner join filt fi on a.id = fi.aceid > +where > +t.rid = 'B5FCC8C7111E4E3CBC21AAF5012F59C2' and > +t.tp = 121 and > +a.atp = 1 and > +fi.fh in (6311439873746261694,-397087483897438286, > +8518228073041491534,-5420422472375069774); > +ANALYZE > +{ > + "query_block": { > + "select_id": 1, > + "r_loops": 1, > + "r_total_time_ms": "REPLACED", > + "table": { > + "table_name": "t", > + "access_type": "index_merge", > + "possible_keys": ["PRIMARY", "acli_rid", "acli_tp"], > + "key_length": "2,767", > + "index_merge": { > + "intersect": { > + "range": { > + "key": "acli_tp", > + "used_key_parts": ["tp"] > + }, > + "range": { > + "key": "acli_rid", > + "used_key_parts": ["rid"] > + } > + } > + }, > + "r_loops": 1, > + "rows": 2, > + "r_rows": 3, > + "r_total_time_ms": "REPLACED", > + "filtered": 100, > + "r_filtered": 100, > + "attached_condition": "t.tp = 121 and t.rid = 'B5FCC8C7111E4E3CBC21AAF5012F59C2'", > + "using_index": true > + }, > + "block-nl-join": { > + "table": { > + "table_name": "a", > + "access_type": "ref", > + "possible_keys": ["PRIMARY", "acei_aclid"], > + "key": "acei_aclid", > + "key_length": "8", > + "used_key_parts": ["aclid"], > + "ref": ["test.t.id"], > + "r_loops": 1, > + "rows": 1, > + "r_rows": 3, > + "r_total_time_ms": "REPLACED", > + "filtered": 100, > + "r_filtered": 100 > + }, > + "buffer_type": "flat", > + "buffer_size": "8Kb", > + "join_type": "BKA", > + "mrr_type": "Rowid-ordered scan", > + "attached_condition": "a.atp = 1", > + "r_filtered": 100 > + }, > + "block-nl-join": { > + "table": { > + "table_name": "fi", > + "access_type": "ref", > + "possible_keys": ["filt_aceid", "filt_fh"], > + "key": "filt_aceid", > + "key_length": "8", > + "used_key_parts": ["aceid"], > + "ref": ["test.a.id"], > + "rowid_filter": { > + "range": { > + "key": "filt_fh", > + "used_key_parts": ["fh"] > + }, > + "rows": 6, > + "selectivity_pct": 17.143, > + "r_rows": 5, > + "r_selectivity_pct": 40, > + "r_buffer_size": "REPLACED", > + "r_filling_time_ms": "REPLACED" > + }, > + "r_loops": 1, > + "rows": 1, > + "r_rows": 2, > + "r_total_time_ms": "REPLACED", > + "filtered": 17.143, > + "r_filtered": 100 > + }, > + "buffer_type": "incremental", > + "buffer_size": "603", > + "join_type": "BKA", > + "mrr_type": "Rowid-ordered scan", > + "attached_condition": "fi.fh in (6311439873746261694,-397087483897438286,8518228073041491534,-5420422472375069774)", > + "r_filtered": 100 > + } > + } > +} > +set optimizer_switch=@save_optimizer_switch; > +set join_cache_level=@save_join_cache_level; > +drop table filt, acei, acli; > +set global innodb_stats_persistent= @stats.save; > diff --git a/mysql-test/main/rowid_filter_innodb.test b/mysql-test/main/rowid_filter_innodb.test > index 30e0ede..74349b8 100644 > --- a/mysql-test/main/rowid_filter_innodb.test > +++ b/mysql-test/main/rowid_filter_innodb.test > @@ -381,3 +381,156 @@ ORDER BY pk LIMIT 1; > > DROP TABLE t1; > SET global innodb_stats_persistent= @stats.save; > + > +--echo # > +--echo # MDEV-21610: Using rowid filter with BKA+MRR > +--echo # > + > +set @stats.save= @@innodb_stats_persistent; > +set global innodb_stats_persistent=on; > + > +CREATE TABLE acli ( > + id bigint(20) NOT NULL, > + rid varchar(255) NOT NULL, > + tp smallint(6) NOT NULL DEFAULT 0, > + PRIMARY KEY (id), > + KEY acli_rid (rid), > + KEY acli_tp (tp) > +) ENGINE=InnoDB DEFAULT CHARSET=utf8; > + > +insert into acli(id,rid,tp) values > +(184929059698905997,'ABABABABABABABABAB',103), > +(184929059698905998,'ABABABABABABABABAB',121), > +(283586039035985921,'00000000000000000000000000000000',103), > +(2216474704108064678,'020BED6D07B741CE9B10AB2200FEF1DF',103), > +(2216474704108064679,'020BED6D07B741CE9B10AB2200FEF1DF',121), > +(3080602882609775593,'B5FCC8C7111E4E3CBC21AAF5012F59C2',103), > +(3080602882609775594,'B5FCC8C7111E4E3CBC21AAF5012F59C2',121), > +(3080602882609776594,'B5FCC8C7111E4E3CBC21AAF5012F59C2',121), > +(3080602882609777595,'B5FCC8C7111E4E3CBC21AAF5012F59C2',121), > +(4269412446747236214,'SCSCSCSCSCSCSCSC',103), > +(4269412446747236215,'SCSCSCSCSCSCSCSC',121), > +(6341490487802728356,'6072D47E513F4A4794BBAB2200FDB67D',103), > +(6341490487802728357,'6072D47E513F4A4794BBAB2200FDB67D',121); > + > +CREATE TABLE acei ( > + id bigint(20) NOT NULL, > + aclid bigint(20) NOT NULL DEFAULT 0, > + atp smallint(6) NOT NULL DEFAULT 0, > + clus smallint(6) NOT NULL DEFAULT 0, > + PRIMARY KEY (id), > + KEY acei_aclid (aclid), > + KEY acei_clus (clus) > +) ENGINE=InnoDB DEFAULT CHARSET=utf8; > + > +insert into acei(id,aclid,atp,clus) values > +(184929059698905999,184929059698905997,0,1), > +(184929059698906000,184929059698905997,0,1), > +(184929059698906001,184929059698905997,1,1), > +(184929059698906002,184929059698905998,1,1), > +(283586039035985922,283586039035985921,1,1), > +(2216474704108064684,2216474704108064678,0,1), > +(2216474704108064685,2216474704108064678,0,1), > +(2216474704108064686,2216474704108064678,1,1), > +(2216474704108064687,2216474704108064679,1,1), > +(3080602882609775595,3080602882609775593,0,1), > +(3080602882609775596,3080602882609775593,0,1), > +(3080602882609775597,3080602882609775593,1,1), > +(3080602882609775598,3080602882609775594,1,1), > +(3080602882609776595,3080602882609776594,1,1), > +(3080602882609777596,3080602882609777595,1,1), > +(4269412446747236216,4269412446747236214,0,1), > +(4269412446747236217,4269412446747236214,0,1), > +(4269412446747236218,4269412446747236214,1,1), > +(4269412446747236219,4269412446747236215,1,1), > +(6341490487802728358,6341490487802728356,0,1), > +(6341490487802728359,6341490487802728356,0,1), > +(6341490487802728360,6341490487802728356,1,1), > +(6341490487802728361,6341490487802728357,1,1); > + > +CREATE TABLE filt ( > + id bigint(20) NOT NULL, > + aceid bigint(20) NOT NULL DEFAULT 0, > + clid smallint(6) NOT NULL DEFAULT 0, > + fh bigint(20) NOT NULL DEFAULT 0, > + PRIMARY KEY (id), > + KEY filt_aceid (aceid), > + KEY filt_clid (clid), > + KEY filt_fh (fh) > +) ENGINE=InnoDB DEFAULT CHARSET=utf8; > + > +insert into filt(id,aceid,clid,fh) values > +(184929059698905999,184929059698905999,1,8948400944397203540), > +(184929059698906000,184929059698906000,1,-3516039679025944536), > +(184929059698906001,184929059698906001,1,-3516039679025944536), > +(184929059698906002,184929059698906001,1,2965370193075218252), > +(184929059698906003,184929059698906001,1,8948400944397203540), > +(184929059698906004,184929059698906002,1,2478709353550777738), > +(283586039035985922,283586039035985922,1,5902600816362013271), > +(2216474704108064686,2216474704108064684,1,8948400944397203540), > +(2216474704108064687,2216474704108064685,1,-7244708939311117030), > +(2216474704108064688,2216474704108064686,1,-7244708939311117030), > +(2216474704108064689,2216474704108064686,1,7489060986210282479), > +(2216474704108064690,2216474704108064686,1,8948400944397203540), > +(2216474704108064691,2216474704108064687,1,-3575268945274980038), > +(3080602882609775595,3080602882609775595,1,8948400944397203540), > +(3080602882609775596,3080602882609775596,1,-5420422472375069774), > +(3080602882609775597,3080602882609775597,1,-5420422472375069774), > +(3080602882609775598,3080602882609775597,1,8518228073041491534), > +(3080602882609775599,3080602882609775597,1,8948400944397203540), > +(3080602882609775600,3080602882609775598,1,6311439873746261694), > +(3080602882609775601,3080602882609775598,1,6311439873746261694), > +(3080602882609776595,3080602882609776595,1,-661101805245999843), > +(3080602882609777596,3080602882609777596,1,-661101805245999843), > +(3080602882609777597,3080602882609777596,1,2216865386202464067), > +(4269412446747236216,4269412446747236216,1,8948400944397203540), > +(4269412446747236217,4269412446747236217,1,-1143096194892676000), > +(4269412446747236218,4269412446747236218,1,-1143096194892676000), > +(4269412446747236219,4269412446747236218,1,5313391811364818290), > +(4269412446747236220,4269412446747236218,1,8948400944397203540), > +(4269412446747236221,4269412446747236219,1,7624499822621753835), > +(6341490487802728358,6341490487802728358,1,8948400944397203540), > +(6341490487802728359,6341490487802728359,1,8141092449587136068), > +(6341490487802728360,6341490487802728360,1,8141092449587136068), > +(6341490487802728361,6341490487802728360,1,1291319099896431785), > +(6341490487802728362,6341490487802728360,1,8948400944397203540), > +(6341490487802728363,6341490487802728361,1,6701841652906431497); > + > +analyze table filt, acei, acli; > + > +let $q= > +select t.id, fi.* > +from (acli t inner join acei a on a.aclid = t.id) > + inner join filt fi on a.id = fi.aceid > + where > + t.rid = 'B5FCC8C7111E4E3CBC21AAF5012F59C2' and > + t.tp = 121 and > + a.atp = 1 and > + fi.fh in (6311439873746261694,-397087483897438286, > + 8518228073041491534,-5420422472375069774); > + > +set @save_optimizer_switch=@@optimizer_switch; > +set @save_join_cache_level=@@join_cache_level; > + > +set optimizer_switch='mrr=off'; > +set join_cache_level=2; > +eval $without_filter explain extended $q; > +eval $without_filter $q; > +eval $with_filter explain extended $q; > +eval $with_filter $q; > + > +set optimizer_switch='mrr=on'; > +set join_cache_level=6; > +eval $without_filter explain extended $q; > +eval $without_filter $q; > +eval $with_filter explain extended $q; > +eval $with_filter $q; > +--source include/analyze-format.inc > +eval $with_filter analyze format=json $q; > + > +set optimizer_switch=@save_optimizer_switch; > +set join_cache_level=@save_join_cache_level; > + > +drop table filt, acei, acli; > + > +set global innodb_stats_persistent= @stats.save; > diff --git a/sql/multi_range_read.cc b/sql/multi_range_read.cc > index 7e4c2ed..daeb53d 100644 > --- a/sql/multi_range_read.cc > +++ b/sql/multi_range_read.cc > @@ -702,7 +702,8 @@ static int rowid_cmp_reverse(void *file, uchar *a, uchar *b) > int Mrr_ordered_rndpos_reader::init(handler *h_arg, > Mrr_index_reader *index_reader_arg, > uint mode, > - Lifo_buffer *buf) > + Lifo_buffer *buf, > + Rowid_filter *filter) > { > file= h_arg; > index_reader= index_reader_arg; > @@ -710,19 +711,7 @@ int Mrr_ordered_rndpos_reader::init(handler *h_arg, > is_mrr_assoc= !MY_TEST(mode & HA_MRR_NO_ASSOCIATION); > index_reader_exhausted= FALSE; > index_reader_needs_refill= TRUE; > - > - /* > - Currently usage of a rowid filter within InnoDB engine is not supported > - if the table is accessed by the primary key. > - With optimizer switches ''mrr' and 'mrr_sort_keys' are both enabled > - any access by a secondary index is converted to the rndpos access. In > - InnoDB the rndpos access is always uses the primary key. > - Do not use pushed rowid filter if the table is accessed actually by the > - primary key. Use the rowid filter outside the engine code (see > - Mrr_ordered_rndpos_reader::refill_from_index_reader). > - */ > - if (file->pushed_rowid_filter && file->primary_key_is_clustered()) > - file->cancel_pushed_rowid_filter(); > + rowid_filter= filter; > > return 0; > } > @@ -817,10 +806,8 @@ int Mrr_ordered_rndpos_reader::refill_from_index_reader() > index_reader->position(); > > /* > - If the built rowid filter cannot be used at the engine level use it here. > + If the built rowid filter cannot be used at the engine level, use it here. > */ > - Rowid_filter *rowid_filter= > - file->get_table()->reginfo.join_tab->rowid_filter; > if (rowid_filter && !file->pushed_rowid_filter && > !rowid_filter->check((char *)index_rowid)) > continue; > @@ -967,6 +954,7 @@ int DsMrr_impl::dsmrr_init(handler *h_arg, RANGE_SEQ_IF *seq_funcs, > handler *h_idx; > Mrr_ordered_rndpos_reader *disk_strategy= NULL; > bool do_sort_keys= FALSE; > + Rowid_filter *rowid_filter= NULL; > DBUG_ENTER("DsMrr_impl::dsmrr_init"); > /* > index_merge may invoke a scan on an object for which dsmrr_info[_const] > @@ -1015,6 +1003,21 @@ int DsMrr_impl::dsmrr_init(handler *h_arg, RANGE_SEQ_IF *seq_funcs, > if (!(keyno == table->s->primary_key && h_idx->primary_key_is_clustered())) > { > strategy= disk_strategy= &reader_factory.ordered_rndpos_reader; > + if (h_arg->pushed_rowid_filter) > + { > + /* > + Currently usage of a rowid filter within InnoDB engine is not supported > + if the table is accessed by the primary key. > + With optimizer switches ''mrr' and 'mrr_sort_keys' are both enabled > + any access by a secondary index is converted to the rndpos access. In > + InnoDB the rndpos access is always uses the primary key. > + Do not use pushed rowid filter if the table is accessed actually by the > + primary key. Use the rowid filter outside the engine code (see > + Mrr_ordered_rndpos_reader::refill_from_index_reader). > + */ > + rowid_filter= h_arg->pushed_rowid_filter; > + h_arg->cancel_pushed_rowid_filter(); > + } > } > > full_buf= buf->buffer; > @@ -1101,7 +1104,7 @@ int DsMrr_impl::dsmrr_init(handler *h_arg, RANGE_SEQ_IF *seq_funcs, > n_ranges, mode, &keypar, key_buffer, > &buf_manager)) || > (res= disk_strategy->init(primary_file, index_strategy, mode, > - &rowid_buffer))) > + &rowid_buffer, rowid_filter))) > { > goto error; > } > diff --git a/sql/multi_range_read.h b/sql/multi_range_read.h > index 0473fef..6be9537 100644 > --- a/sql/multi_range_read.h > +++ b/sql/multi_range_read.h > @@ -364,7 +364,7 @@ class Mrr_ordered_rndpos_reader : public Mrr_reader > { > public: > int init(handler *file, Mrr_index_reader *index_reader, uint mode, > - Lifo_buffer *buf); > + Lifo_buffer *buf, Rowid_filter *filter); > int get_next(range_id_t *range_info); > int refill_buffer(bool initial); > private: > @@ -399,6 +399,9 @@ class Mrr_ordered_rndpos_reader : public Mrr_reader > /* Buffer to store (rowid, range_id) pairs */ > Lifo_buffer *rowid_buffer; > > + /* Rowid filter to be checked against (if any) */ > + Rowid_filter *rowid_filter; > + > int refill_from_index_reader(); > }; > > diff --git a/sql/opt_range.cc b/sql/opt_range.cc > index c47da28..5f034c6 100644 > --- a/sql/opt_range.cc > +++ b/sql/opt_range.cc > @@ -2902,7 +2902,8 @@ int SQL_SELECT::test_quick_select(THD *thd, key_map keys_to_use, > remove_nonrange_trees(&param, tree); > > /* Get best 'range' plan and prepare data for making other plans */ > - if ((range_trp= get_key_scans_params(&param, tree, FALSE, TRUE, > + if ((range_trp= get_key_scans_params(&param, tree, > + only_single_index_range_scan, TRUE, > best_read_time))) > { > best_trp= range_trp; > diff --git a/sql/sql_join_cache.cc b/sql/sql_join_cache.cc > index 3a509b3..e9ad538 100644 > --- a/sql/sql_join_cache.cc > +++ b/sql/sql_join_cache.cc > @@ -2248,6 +2248,8 @@ enum_nested_loop_state JOIN_CACHE::join_matching_records(bool skip_last) > if ((rc= join_tab_execution_startup(join_tab)) < 0) > goto finish2; > > + join_tab->build_range_rowid_filter_if_needed(); > + > /* Prepare to retrieve all records of the joined table */ > if (unlikely((error= join_tab_scan->open()))) > { > _______________________________________________ > commits mailing list > commits(a)mariadb.org > https://lists.askmonty.org/cgi-bin/mailman/listinfo/commits -- BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] MDEV-7381
by Gabriel Ciciliani 11 Feb '20

11 Feb '20

Hi MariaDB developers, I want to start working on this ticket but I can't assing it to me with my current JIRA privileges. Someone higher in the chain should do it for me or I nee to get my JIRA account fixed? Thanks!

2 1

Re: [Maria-developers] 6887f16b61a: MDEV-17399 Add support for JSON_TABLE.
by Sergei Golubchik 06 Feb '20

06 Feb '20

Hi, Alexey! There isn't much to review yet, I don't see how different parts will work together yet. But here you are, some preliminary comments. On Feb 06, Alexey Botchkov wrote: > revision-id: 6887f16b61a (mariadb-10.5.0-153-g6887f16b61a) > parent(s): 74f76206369 > author: Alexey Botchkov <holyfoot(a)mariadb.com> > committer: Alexey Botchkov <holyfoot(a)mariadb.com> > timestamp: 2020-02-04 17:12:55 +0400 > message: > > MDEV-17399 Add support for JSON_TABLE. > > Syntax for the JSON_TABLE added. Standard JSON_TABLE syntax is quite complex. You are doing a subset of it, I presume. It would be good to specify here, in the comment, what syntax exactly are you implementing. > --- > sql/lex.h | 5 ++ > sql/sql_digest.cc | 1 + > sql/sql_lex.h | 3 + > sql/sql_yacc.yy | 164 ++++++++++++++++++++++++++++++++++++++++++++++++++- > sql/table.cc | 1 + > sql/table.h | 2 + > sql/table_function.h | 89 ++++++++++++++++++++++++++++ > 7 files changed, 264 insertions(+), 1 deletion(-) > > diff --git a/sql/lex.h b/sql/lex.h > index f3fc1513369..774f9c1517d 100644 > --- a/sql/lex.h > +++ b/sql/lex.h > @@ -208,6 +208,7 @@ static SYMBOL symbols[] = { > { "ELSE", SYM(ELSE)}, > { "ELSEIF", SYM(ELSEIF_MARIADB_SYM)}, > { "ELSIF", SYM(ELSIF_MARIADB_SYM)}, > + { "EMPTY", SYM(EMPTY_SYM)}, > { "ENABLE", SYM(ENABLE_SYM)}, > { "ENCLOSED", SYM(ENCLOSED)}, > { "END", SYM(END)}, > @@ -414,6 +415,7 @@ static SYMBOL symbols[] = { > { "NATIONAL", SYM(NATIONAL_SYM)}, > { "NATURAL", SYM(NATURAL)}, > { "NCHAR", SYM(NCHAR_SYM)}, > + { "NESTED", SYM(NESTED_SYM)}, > { "NEVER", SYM(NEVER_SYM)}, > { "NEW", SYM(NEW_SYM)}, > { "NEXT", SYM(NEXT_SYM)}, > @@ -448,6 +450,7 @@ static SYMBOL symbols[] = { > { "OPTIONALLY", SYM(OPTIONALLY)}, > { "OR", SYM(OR_SYM)}, > { "ORDER", SYM(ORDER_SYM)}, > + { "ORDINALITY", SYM(ORDINALITY_SYM)}, > { "OTHERS", SYM(OTHERS_MARIADB_SYM)}, > { "OUT", SYM(OUT_SYM)}, > { "OUTER", SYM(OUTER)}, > @@ -460,6 +463,7 @@ static SYMBOL symbols[] = { > { "PAGE_CHECKSUM", SYM(PAGE_CHECKSUM_SYM)}, > { "PARSER", SYM(PARSER_SYM)}, > { "PARSE_VCOL_EXPR", SYM(PARSE_VCOL_EXPR_SYM)}, > + { "PATH", SYM(PATH_SYM)}, > { "PERIOD", SYM(PERIOD_SYM)}, > { "PARTIAL", SYM(PARTIAL)}, > { "PARTITION", SYM(PARTITION_SYM)}, > @@ -742,6 +746,7 @@ static SYMBOL sql_functions[] = { > { "FIRST_VALUE", SYM(FIRST_VALUE_SYM)}, > { "GROUP_CONCAT", SYM(GROUP_CONCAT_SYM)}, > { "JSON_ARRAYAGG", SYM(JSON_ARRAYAGG_SYM)}, > + { "JSON_TABLE", SYM(JSON_TABLE_SYM)}, > { "JSON_OBJECTAGG", SYM(JSON_OBJECTAGG_SYM)}, > { "LAG", SYM(LAG_SYM)}, > { "LEAD", SYM(LEAD_SYM)}, you've added them all as reserved words, is it intentonal? > diff --git a/sql/sql_lex.h b/sql/sql_lex.h > index 308bebd9fa1..591de1e1583 100644 > --- a/sql/sql_lex.h > +++ b/sql/sql_lex.h > @@ -33,6 +33,7 @@ > #include "sql_tvc.h" > #include "item.h" > #include "sql_limit.h" // Select_limit_counters > +#include "table_function.h" // Json_table_column > > /* Used for flags of nesting constructs */ > #define SELECT_NESTING_MAP_SIZE 64 > @@ -3249,6 +3250,8 @@ struct LEX: public Query_tables_list > SQL_I_List<ORDER> proc_list; > SQL_I_List<TABLE_LIST> auxiliary_table_list, save_list; > Column_definition *last_field; > + Json_table_column *cur_json_table_column, *json_table_column_nest; > + Table_function_json_table *json_table; Did you have to put everything in LEX? It's pretty big already and it's never fully used, its members allocate memory all together at the same time, but used for different parts of the grammar, different statements. Could you try to pass that through the parser stack? > Item_sum *in_sum_func; > udf_func udf; > HA_CHECK_OPT check_opt; // check/repair options > diff --git a/sql/sql_yacc.yy b/sql/sql_yacc.yy > index c00ba42846a..5ef1bbd0721 100644 > --- a/sql/sql_yacc.yy > +++ b/sql/sql_yacc.yy > @@ -207,6 +208,8 @@ void _CONCAT_UNDERSCORED(turn_parser_debug_on,yyparse)() > Lex_for_loop_st for_loop; > Lex_for_loop_bounds_st for_loop_bounds; > Lex_trim_st trim; > + Json_table_column::On_response json_on_response; > + > vers_history_point_t vers_history_point; > struct > { > @@ -225,6 +228,7 @@ void _CONCAT_UNDERSCORED(turn_parser_debug_on,yyparse)() > > /* pointers */ > Create_field *create_field; > + Json_table_column *json_table_column; > Spvar_definition *spvar_definition; > Row_definition_list *spvar_definition_list; > const Type_handler *type_handler; > @@ -308,6 +312,7 @@ void _CONCAT_UNDERSCORED(turn_parser_debug_on,yyparse)() > enum vers_kind_t vers_range_unit; > enum Column_definition::enum_column_versioning vers_column_versioning; > enum plsql_cursor_attr_t plsql_cursor_attr; > + enum Json_table_column::enum_on_type json_on_type; > } > > %{ > @@ -489,6 +494,7 @@ End SQL_MODE_ORACLE_SPECIFIC */ > %token <kwd> ELSEIF_MARIADB_SYM > %token <kwd> ELSE /* SQL-2003-R */ > %token <kwd> ELSIF_ORACLE_SYM /* PLSQL-R */ > +%token <kwd> EMPTY_SYM /* SQL-2016-R */ > %token <kwd> ENCLOSED > %token <kwd> ESCAPED > %token <kwd> EXCEPT_SYM /* SQL-2003-R */ > @@ -506,6 +512,7 @@ End SQL_MODE_ORACLE_SPECIFIC */ > %token <kwd> GRANT /* SQL-2003-R */ > %token <kwd> GROUP_CONCAT_SYM > %token <rwd> JSON_ARRAYAGG_SYM > +%token <rwd> JSON_TABLE_SYM > %token <rwd> JSON_OBJECTAGG_SYM > %token <kwd> GROUP_SYM /* SQL-2003-R */ > %token <kwd> HAVING /* SQL-2003-R */ > @@ -564,6 +571,7 @@ End SQL_MODE_ORACLE_SPECIFIC */ > %token <kwd> MOD_SYM /* SQL-2003-N */ > %token <kwd> NATURAL /* SQL-2003-R */ > %token <kwd> NEG > +%token <kwd> NESTED_SYM /* SQL-2003-N */ No, NESTED is not in the SQL:2003 list of non-reserved words. Please specify a correct comment tag for every keyword you're adding here. > %token <kwd> NOT_SYM /* SQL-2003-R */ > %token <kwd> NO_WRITE_TO_BINLOG > %token <kwd> NOW_SYM > @@ -575,6 +583,7 @@ End SQL_MODE_ORACLE_SPECIFIC */ > %token <kwd> OPTIMIZE > %token <kwd> OPTIONALLY > %token <kwd> ORDER_SYM /* SQL-2003-R */ > +%token <kwd> ORDINALITY_SYM /* SQL-2003-N */ > %token <kwd> OR_SYM /* SQL-2003-R */ > %token <kwd> OTHERS_ORACLE_SYM /* SQL-2011-N, PLSQL-R */ > %token <kwd> OUTER > @@ -585,6 +594,7 @@ End SQL_MODE_ORACLE_SPECIFIC */ > %token <kwd> PAGE_CHECKSUM_SYM > %token <kwd> PARSE_VCOL_EXPR_SYM > %token <kwd> PARTITION_SYM /* SQL-2003-R */ > +%token <kwd> PATH_SYM /* SQL-2003-N */ > %token <kwd> PERCENTILE_CONT_SYM > %token <kwd> PERCENTILE_DISC_SYM > %token <kwd> PERCENT_RANK_SYM > @@ -1336,12 +1346,16 @@ End SQL_MODE_ORACLE_SPECIFIC */ > > %type <type_handler> int_type real_type > > +%type <json_on_type> json_empty_or_error > +%type <json_on_response> json_on_response > + > %type <Lex_field_type> type_with_opt_collate field_type > field_type_numeric > field_type_string > field_type_lob > field_type_temporal > field_type_misc > + json_table_field_type > > %type <Lex_dyncol_type> opt_dyncol_type dyncol_type > numeric_dyncol_type temporal_dyncol_type string_dyncol_type > @@ -1487,7 +1501,7 @@ End SQL_MODE_ORACLE_SPECIFIC */ > table_primary_derived table_primary_derived_opt_parens > derived_table_list table_reference_list_parens > nested_table_reference_list join_table_parens > - update_table_list > + update_table_list table_function > %type <date_time_type> date_time_type; > %type <interval> interval > > @@ -1642,6 +1656,9 @@ End SQL_MODE_ORACLE_SPECIFIC */ > opt_delete_gtid_domain > asrow_attribute > opt_constraint_no_id > + json_table_columns_clause json_table_columns_list json_table_column > + json_table_column_type json_opt_on_empty_or_error > + > > %type <NONE> call sp_proc_stmts sp_proc_stmts1 sp_proc_stmt > %type <NONE> sp_if_then_statements sp_case_then_statements > @@ -11320,6 +11337,150 @@ join_table_list: > derived_table_list { MYSQL_YYABORT_UNLESS($$=$1); } > ; > > +json_table_columns_clause: > + COLUMNS '(' json_table_columns_list ')' > + {} > + ; > + > +json_table_columns_list: > + json_table_column > + | json_table_columns_list ',' json_table_column > + {} > + ; > + > +json_table_column: > + ident > + { > + LEX *lex=Lex; > + Create_field *f= new (thd->mem_root) Create_field(); > + > + if (unlikely(check_string_char_length(&$1, 0, NAME_CHAR_LEN, > + system_charset_info, 1))) > + my_yyabort_error((ER_TOO_LONG_IDENT, MYF(0), $1.str)); > + > + lex->cur_json_table_column= > + new (thd->mem_root) Json_table_column(f, lex->json_table_column_nest); > + > + if (unlikely(!f || !lex->cur_json_table_column)) > + MYSQL_YYABORT; > + > + lex->init_last_field(f, &$1, NULL); > + } > + json_table_column_type > + { > + LEX *lex=Lex; > + lex->json_table->m_columns.push_back(lex->cur_json_table_column, > + thd->mem_root); > + } > + ; > + > +json_table_column_type: > + FOR_SYM ORDINALITY_SYM > + { > + Lex->cur_json_table_column->set(Json_table_column::FOR_ORDINALITY); > + } > + | json_table_field_type PATH_SYM TEXT_STRING_sys > + json_opt_on_empty_or_error > + { > + Lex->last_field->set_attributes(thd, $1, Lex->charset, > + COLUMN_DEFINITION_TABLE_FIELD); > + Lex->cur_json_table_column->set(Json_table_column::PATH, &$3); > + } > + | json_table_field_type EXISTS PATH_SYM TEXT_STRING_sys > + { > + Lex->last_field->set_attributes(thd, $1, Lex->charset, > + COLUMN_DEFINITION_TABLE_FIELD); > + Lex->cur_json_table_column->set(Json_table_column::EXISTS_PATH, &$4); > + } > + | NESTED_SYM PATH_SYM TEXT_STRING_sys > + { > + LEX *lex=Lex; > + lex->cur_json_table_column->set(Json_table_column::NESTED_PATH); > + lex->json_table_column_nest= lex->cur_json_table_column; > + } > + json_table_columns_clause > + { > + LEX *lex=Lex; > + lex->cur_json_table_column= lex->json_table_column_nest; > + lex->json_table_column_nest= lex->cur_json_table_column->m_nest; > + } > + ; > + > +json_table_field_type: > + field_type_numeric > + | field_type_temporal > + | field_type_string > + ; > + > +json_opt_on_empty_or_error: > + /* none */ > + {} > + | json_on_response ON json_empty_or_error json_opt_on_empty_or_error > + { > + if ($3 == Json_table_column::ON_EMPTY) > + { > + Lex->cur_json_table_column->m_on_empty= $1; > + } > + else /* ON_ERROR */ > + { > + Lex->cur_json_table_column->m_on_error= $1; > + } > + } > + ; > + > +json_on_response: > + ERROR_SYM > + { > + $$.m_response= Json_table_column::RESPONSE_ERROR; > + } > + | NULL_SYM > + { > + $$.m_response= Json_table_column::RESPONSE_NULL; > + } > + | DEFAULT TEXT_STRING_sys > + { > + $$.m_response= Json_table_column::RESPONSE_DEFAULT; > + $$.m_default= $1; > + } > + ; > + > +json_empty_or_error: > + ERROR_SYM > + { > + $$= Json_table_column::ON_EMPTY; > + } > + | EMPTY_SYM > + { > + $$= Json_table_column::ON_ERROR; > + } > + ; > + > +table_function: > + JSON_TABLE_SYM '(' expr ',' TEXT_STRING_sys difficult to see what you're parsing without a syntax specs in the commit comment. > + { > + Table_function_json_table *jt= > + new (thd->mem_root) Table_function_json_table($3, &$5); > + if (unlikely(!jt)) > + MYSQL_YYABORT; > + Lex->json_table= jt; > + Lex->json_table_column_nest= NULL; > + } > + json_table_columns_clause ')' AS ident_table_alias > + { > +LEX_CSTRING tn; > + SELECT_LEX *sel= Select; > + sel->table_join_options= 0; > +tn.str= "t1"; > +tn.length= 2; what's that? just to make the code compile for now? why not to use $10 - this is your ident_table_alias, currently it's ignored. > + if (!($$= Select->add_table_to_list(thd, > + new (thd->mem_root) Table_ident(&tn), &tn, > + Select->get_table_join_options(), > + YYPS->m_lock_type, > + YYPS->m_mdl_type))) > + MYSQL_YYABORT; > + } > + ; > + > /* > The ODBC escape syntax for Outer Join is: '{' OJ join_table '}' > The parser does not define OJ as a token, any ident is accepted > @@ -11527,6 +11688,7 @@ table_factor: > $$= $1; > } > | table_reference_list_parens { $$= $1; } > + | table_function { $$= $1; } > ; > > table_primary_ident_opt_parens: Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

[Maria-developers] [DISCUSS] Don't see ARM release information in MariaDB download page
by bo zhaobo 03 Feb '20

03 Feb '20

Hi Team, I want to make clear a thing about the Mariadb download page. Hope team could give some advices or help. As I had sent out an discussion to maria-docs, but didn't get some help. So I think I may get something in dev channel. When I want to download a aarch64 mariadb package, I'm failed to search any information in the content of page [1]. But I tried to search the RPM/DEB package, such as in '*Red Hat, Fedora, and CentOS Packages*' and '*Debian and Ubuntu Packages*', and I can see there are ARM(aarch64/arm64) specific packages in these linked pages [2]. And documentation in [1] just say OS/CPU with "*RedHat/CentOS/Fedora (x86, x86_64, ppc64, ppc64le)*". It seems that there is difference between the documentation with real released packages. That's why I'm here to want to make it clear and discuss with team. *How about add 'aarch64' in the OS/CPU description?* I think that would be great if community could fix that, because it is good for users. They could get what they want from MariaDB download page(MariaDB docs related). Thanks very much. BR ZhaoBo [1] https://downloads.mariadb.org/mariadb/10.5.0/ [2] http://ftp.igh.cnrs.fr/pub/mariadb//mariadb-10.5.0/yum/ http://ftp.igh.cnrs.fr/pub/mariadb//mariadb-10.5.0/repo/ [image: Mailtrack] <https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=sig…> Sender notified by Mailtrack <https://mailtrack.io?utm_source=gmail&utm_medium=signature&utm_campaign=sig…> 20/01/06 下午07:31:54

3 7

778e96749bc: MDEV-20923:UBSAN: member access within address … which does not point to an object of type 'xid_count_per_binlog'
by sujatha 23 Jan '20

23 Jan '20

revision-id: 778e96749bcadc5528bb7f214a711272f7413b96 (mariadb-10.1.43-66-g778e96749bc) parent(s): 982294ac1680938ac9223fb64a64e21f0cbc322a author: Sujatha committer: Sujatha timestamp: 2020-01-23 16:17:55 +0530 message: MDEV-20923:UBSAN: member access within address … which does not point to an object of type 'xid_count_per_binlog' Problem: ------- Accessing a member within 'xid_count_per_binlog' structure results in following error when 'UBSAN' is enabled. member access within address 0xXXX which does not point to an object of type 'xid_count_per_binlog' Analysis: --------- The problem appears to be that no constructor for 'xid_count_per_binlog' is being called, and thus the vtable will not be initialized. Fix: --- Defined a parameterized constructor for 'xid_count_per_binlog' class. --- sql/log.cc | 27 +++++++++++---------------- sql/log.h | 13 ++++++++++++- 2 files changed, 23 insertions(+), 17 deletions(-) diff --git a/sql/log.cc b/sql/log.cc index acf1f8f8a9c..0efef6d1e29 100644 --- a/sql/log.cc +++ b/sql/log.cc @@ -3216,7 +3216,7 @@ void MYSQL_BIN_LOG::cleanup() DBUG_ASSERT(!binlog_xid_count_list.head()); WSREP_XID_LIST_ENTRY("MYSQL_BIN_LOG::cleanup(): Removing xid_list_entry " "for %s (%lu)", b); - my_free(b); + delete b; } mysql_mutex_destroy(&LOCK_log); @@ -3580,17 +3580,9 @@ bool MYSQL_BIN_LOG::open(const char *log_name, */ uint off= dirname_length(log_file_name); uint len= strlen(log_file_name) - off; - char *entry_mem, *name_mem; - if (!(new_xid_list_entry = (xid_count_per_binlog *) - my_multi_malloc(MYF(MY_WME), - &entry_mem, sizeof(xid_count_per_binlog), - &name_mem, len, - NULL))) + new_xid_list_entry= new xid_count_per_binlog(log_file_name+off, (int)len); + if (!new_xid_list_entry) goto err; - memcpy(name_mem, log_file_name+off, len); - new_xid_list_entry->binlog_name= name_mem; - new_xid_list_entry->binlog_name_len= len; - new_xid_list_entry->xid_count= 0; /* Find the name for the Initial binlog checkpoint. @@ -3607,7 +3599,10 @@ bool MYSQL_BIN_LOG::open(const char *log_name, mysql_mutex_unlock(&LOCK_xid_list); if (!b) b= new_xid_list_entry; - strmake(buf, b->binlog_name, b->binlog_name_len); + if (b->binlog_name) + strmake(buf, b->binlog_name, b->binlog_name_len); + else + goto err; Binlog_checkpoint_log_event ev(buf, len); DBUG_EXECUTE_IF("crash_before_write_checkpoint_event", flush_io_cache(&log_file); @@ -3711,7 +3706,7 @@ bool MYSQL_BIN_LOG::open(const char *log_name, { WSREP_XID_LIST_ENTRY("MYSQL_BIN_LOG::open(): Removing xid_list_entry for " "%s (%lu)", b); - my_free(binlog_xid_count_list.get()); + delete binlog_xid_count_list.get(); } mysql_cond_broadcast(&COND_xid_list); WSREP_XID_LIST_ENTRY("MYSQL_BIN_LOG::open(): Adding new xid_list_entry for " @@ -3758,7 +3753,7 @@ Turning logging off for the whole duration of the MySQL server process. \ To turn it on again: fix the cause, \ shutdown the MySQL server and restart it.", name, errno); if (new_xid_list_entry) - my_free(new_xid_list_entry); + delete new_xid_list_entry; if (file >= 0) mysql_file_close(file, MYF(0)); close(LOG_CLOSE_INDEX); @@ -4252,7 +4247,7 @@ bool MYSQL_BIN_LOG::reset_logs(THD *thd, bool create_new_log, DBUG_ASSERT(b->xid_count == 0); WSREP_XID_LIST_ENTRY("MYSQL_BIN_LOG::reset_logs(): Removing " "xid_list_entry for %s (%lu)", b); - my_free(binlog_xid_count_list.get()); + delete binlog_xid_count_list.get(); } mysql_cond_broadcast(&COND_xid_list); reset_master_pending--; @@ -9736,7 +9731,7 @@ TC_LOG_BINLOG::mark_xid_done(ulong binlog_id, bool write_checkpoint) break; WSREP_XID_LIST_ENTRY("TC_LOG_BINLOG::mark_xid_done(): Removing " "xid_list_entry for %s (%lu)", b); - my_free(binlog_xid_count_list.get()); + delete binlog_xid_count_list.get(); } mysql_mutex_unlock(&LOCK_xid_list); diff --git a/sql/log.h b/sql/log.h index b4c9b24a3a9..277e5c6f69c 100644 --- a/sql/log.h +++ b/sql/log.h @@ -587,7 +587,18 @@ class MYSQL_BIN_LOG: public TC_LOG, private MYSQL_LOG long xid_count; /* For linking in requests to the binlog background thread. */ xid_count_per_binlog *next_in_queue; - xid_count_per_binlog(); /* Give link error if constructor used. */ + xid_count_per_binlog(char *log_file_name, uint log_file_name_len) + :binlog_id(0), xid_count(0) + { + binlog_name_len= log_file_name_len; + binlog_name= (char *) my_malloc(binlog_name_len, MYF(MY_ZEROFILL)); + if (binlog_name) + memcpy(binlog_name, log_file_name, binlog_name_len); + } + ~xid_count_per_binlog() + { + my_free(binlog_name); + } }; I_List<xid_count_per_binlog> binlog_xid_count_list; mysql_mutex_t LOCK_binlog_background_thread;

1 0

d9716cfadb8: MDEV-20923:UBSAN: member access within address … which does not point to an object of type 'xid_count_per_binlog'
by sujatha 23 Jan '20

23 Jan '20

revision-id: d9716cfadb8cd3448f25d8b411451f87064cecaf (mariadb-10.1.43-66-gd9716cfadb8) parent(s): 982294ac1680938ac9223fb64a64e21f0cbc322a author: Sujatha committer: Sujatha timestamp: 2020-01-23 16:00:41 +0530 message: MDEV-20923:UBSAN: member access within address … which does not point to an object of type 'xid_count_per_binlog' Problem: ------- Accessing a member within 'xid_count_per_binlog' structure results in following error when 'UBSAN' is enabled. member access within address 0xXXX which does not point to an object of type 'xid_count_per_binlog' Analysis: --------- The problem appears to be that no constructor for 'xid_count_per_binlog' is being called, and thus the vtable will not be initialized. Fix: --- Defined a parameterized constructor for 'xid_count_per_binlog' class. --- sql/log.cc | 27 +++++++++++---------------- sql/log.h | 14 +++++++++++++- 2 files changed, 24 insertions(+), 17 deletions(-) diff --git a/sql/log.cc b/sql/log.cc index acf1f8f8a9c..0efef6d1e29 100644 --- a/sql/log.cc +++ b/sql/log.cc @@ -3216,7 +3216,7 @@ void MYSQL_BIN_LOG::cleanup() DBUG_ASSERT(!binlog_xid_count_list.head()); WSREP_XID_LIST_ENTRY("MYSQL_BIN_LOG::cleanup(): Removing xid_list_entry " "for %s (%lu)", b); - my_free(b); + delete b; } mysql_mutex_destroy(&LOCK_log); @@ -3580,17 +3580,9 @@ bool MYSQL_BIN_LOG::open(const char *log_name, */ uint off= dirname_length(log_file_name); uint len= strlen(log_file_name) - off; - char *entry_mem, *name_mem; - if (!(new_xid_list_entry = (xid_count_per_binlog *) - my_multi_malloc(MYF(MY_WME), - &entry_mem, sizeof(xid_count_per_binlog), - &name_mem, len, - NULL))) + new_xid_list_entry= new xid_count_per_binlog(log_file_name+off, (int)len); + if (!new_xid_list_entry) goto err; - memcpy(name_mem, log_file_name+off, len); - new_xid_list_entry->binlog_name= name_mem; - new_xid_list_entry->binlog_name_len= len; - new_xid_list_entry->xid_count= 0; /* Find the name for the Initial binlog checkpoint. @@ -3607,7 +3599,10 @@ bool MYSQL_BIN_LOG::open(const char *log_name, mysql_mutex_unlock(&LOCK_xid_list); if (!b) b= new_xid_list_entry; - strmake(buf, b->binlog_name, b->binlog_name_len); + if (b->binlog_name) + strmake(buf, b->binlog_name, b->binlog_name_len); + else + goto err; Binlog_checkpoint_log_event ev(buf, len); DBUG_EXECUTE_IF("crash_before_write_checkpoint_event", flush_io_cache(&log_file); @@ -3711,7 +3706,7 @@ bool MYSQL_BIN_LOG::open(const char *log_name, { WSREP_XID_LIST_ENTRY("MYSQL_BIN_LOG::open(): Removing xid_list_entry for " "%s (%lu)", b); - my_free(binlog_xid_count_list.get()); + delete binlog_xid_count_list.get(); } mysql_cond_broadcast(&COND_xid_list); WSREP_XID_LIST_ENTRY("MYSQL_BIN_LOG::open(): Adding new xid_list_entry for " @@ -3758,7 +3753,7 @@ Turning logging off for the whole duration of the MySQL server process. \ To turn it on again: fix the cause, \ shutdown the MySQL server and restart it.", name, errno); if (new_xid_list_entry) - my_free(new_xid_list_entry); + delete new_xid_list_entry; if (file >= 0) mysql_file_close(file, MYF(0)); close(LOG_CLOSE_INDEX); @@ -4252,7 +4247,7 @@ bool MYSQL_BIN_LOG::reset_logs(THD *thd, bool create_new_log, DBUG_ASSERT(b->xid_count == 0); WSREP_XID_LIST_ENTRY("MYSQL_BIN_LOG::reset_logs(): Removing " "xid_list_entry for %s (%lu)", b); - my_free(binlog_xid_count_list.get()); + delete binlog_xid_count_list.get(); } mysql_cond_broadcast(&COND_xid_list); reset_master_pending--; @@ -9736,7 +9731,7 @@ TC_LOG_BINLOG::mark_xid_done(ulong binlog_id, bool write_checkpoint) break; WSREP_XID_LIST_ENTRY("TC_LOG_BINLOG::mark_xid_done(): Removing " "xid_list_entry for %s (%lu)", b); - my_free(binlog_xid_count_list.get()); + delete binlog_xid_count_list.get(); } mysql_mutex_unlock(&LOCK_xid_list); diff --git a/sql/log.h b/sql/log.h index b4c9b24a3a9..e69c9c39eaf 100644 --- a/sql/log.h +++ b/sql/log.h @@ -587,7 +587,19 @@ class MYSQL_BIN_LOG: public TC_LOG, private MYSQL_LOG long xid_count; /* For linking in requests to the binlog background thread. */ xid_count_per_binlog *next_in_queue; - xid_count_per_binlog(); /* Give link error if constructor used. */ + xid_count_per_binlog(char *log_file_name, uint log_file_name_len) + :binlog_name(log_file_name), binlog_name_len(log_file_name_len), + binlog_id(0), xid_count(0) + { + binlog_name_len= log_file_name_len; + binlog_name= (char *) my_malloc(binlog_name_len, MYF(MY_ZEROFILL)); + if (binlog_name) + memcpy(binlog_name, log_file_name, binlog_name_len); + } + ~xid_count_per_binlog() + { + my_free(binlog_name); + } }; I_List<xid_count_per_binlog> binlog_xid_count_list; mysql_mutex_t LOCK_binlog_background_thread;

1 0

[Maria-developers] 43f27ae4778: MDEV-21490: binlog tests fail with valgrind: Conditional jump or move depends on uninitialised value in sql_ex_info::init
by sujatha 23 Jan '20

23 Jan '20

revision-id: 43f27ae4778b3e7be86897e49c182dc05080fd88 (mariadb-10.1.43-66-g43f27ae4778) parent(s): 982294ac1680938ac9223fb64a64e21f0cbc322a author: Sujatha committer: Sujatha timestamp: 2020-01-23 11:57:46 +0530 message: MDEV-21490: binlog tests fail with valgrind: Conditional jump or move depends on uninitialised value in sql_ex_info::init Problem: ======= P1) Conditional jump or move depends on uninitialised value(s) sql_ex_info::init(char const*, char const*, bool) (log_event.cc:3083) code: All the following variables are not initialized. ---- return ((cached_new_format != -1) ? cached_new_format : (cached_new_format=(field_term_len > 1 || enclosed_len > 1 || line_term_len > 1 || line_start_len > 1 || escaped_len > 1))); P2) Conditional jump or move depends on uninitialised value(s) Rows_log_event::Rows_log_event(char const*, unsigned int, Format_description_log_event const*) (log_event.cc:9571) Code: Uninitialized values is reported for 'var_header_len' variable. ---- if (var_header_len < 2 || event_len < static_cast<unsigned int>(var_header_len + (post_start - buf))) P3) Conditional jump or move depends on uninitialised value(s) Table_map_log_event::pack_info(Protocol*) (log_event.cc:11553) code:'m_table_id' is uninitialized. ---- void Table_map_log_event::pack_info(Protocol *protocol) ... size_t bytes= my_snprintf(buf, sizeof(buf), "table_id: %lu (%s.%s)", m_table_id, m_dbnam, m_tblnam); Fix: === P1 - Fix) Initialize cached_new_format,field_term_len, enclosed_len, line_term_len, line_start_len, escaped_len members in default constructor. P2 - Fix) "var_header_len" is initialized by reading the event buffer. In case of an invalid event the buffer will contain invalid data. Hence added a check to validate the event data. If event_len is smaller than valid header length return immediately. P3 - Fix) 'm_table_id' within Table_map_log_event is initialized by reading data from the event buffer. Use 'VALIDATE_BYTES_READ' macro to validate the current state of the buffer. If it is invalid return immediately. --- sql/log_event.cc | 7 +++++++ sql/log_event.h | 9 ++++++++- 2 files changed, 15 insertions(+), 1 deletion(-) diff --git a/sql/log_event.cc b/sql/log_event.cc index e8881c77f2b..9924b9a0493 100644 --- a/sql/log_event.cc +++ b/sql/log_event.cc @@ -9535,6 +9535,12 @@ Rows_log_event::Rows_log_event(const char *buf, uint event_len, uint8 const post_header_len= description_event->post_header_len[event_type-1]; + if (event_len < (uint)(common_header_len + post_header_len)) + { + m_cols.bitmap= 0; + DBUG_VOID_RETURN; + } + DBUG_PRINT("enter",("event_len: %u common_header_len: %d " "post_header_len: %d", event_len, common_header_len, @@ -11043,6 +11049,7 @@ Table_map_log_event::Table_map_log_event(const char *buf, uint event_len, const char *post_start= buf + common_header_len; post_start+= TM_MAPID_OFFSET; + VALIDATE_BYTES_READ(post_start, buf, event_len); if (post_header_len == 6) { /* Master is of an intermediate source tree before 5.1.4. Id is 4 bytes */ diff --git a/sql/log_event.h b/sql/log_event.h index 2c8dc3d7353..04fce65faeb 100644 --- a/sql/log_event.h +++ b/sql/log_event.h @@ -2057,7 +2057,14 @@ class Query_log_event: public Log_event ****************************************************************************/ struct sql_ex_info { - sql_ex_info() {} /* Remove gcc warning */ + sql_ex_info(): + cached_new_format(-1), + field_term_len(0), + enclosed_len(0), + line_term_len(0), + line_start_len(0), + escaped_len(0) + {} /* Remove gcc warning */ const char* field_term; const char* enclosed; const char* line_term;

1 0