- developers - lists.mariadb.org

[Maria-developers] Progress Report GSoC 2020 (1-6 June)
by Anton Mikhaylenko 08 Jun '20

08 Jun '20

Good day, everybody. In this post, I want to describe what I did this week, June 1-6. I have added the initial implementation of the hill_climbing class. It includes an update function - the main function that changes the required number of threads. To implement this function, I added the Complex class - the base class for working with complex numbers. https://github.com/sapiest/server/commits/10.5-update_threadpool_concurrency The main work is not completed yet, as the algorithm requires adjustments and adaptation in MariaDB environment. Regards, Anton MIkhailenko

1 0

[Maria-developers] JSON aggregate functions working on JSON documents
by Sergey Petrunia 08 Jun '20

08 Jun '20

Hi Alexey, I'm looking at how JSON_ARRAYAGG and JSON_OBJECTAGG work at processing JSON documents. I'm trying these examples, first on MySQL-8: create table t30 (col1 json); insert into t30 values('{"color":"red", "size":1}' ); insert into t30 values('{"color":"blue", "size":2}' ); select JSON_ARRAYAGG(col1) from t30; +-------------------------------------------------------------+ | JSON_ARRAYAGG(col1) | +-------------------------------------------------------------+ | [{"size": 1, "color": "red"}, {"size": 2, "color": "blue"}] | +-------------------------------------------------------------+ Good. create table t31 (name varchar(100), contents json); insert into t31 values('obj1', '{"color":"blue", "size":2}' ); insert into t31 values('obj2', '{"color":"red", "size":1}' ); select JSON_OBJECTAGG(name, contents) from t31; +-----------------------------------------------------------------------------+ | JSON_OBJECTAGG(name, contents) | +-----------------------------------------------------------------------------+ | {"obj1": {"size": 2, "color": "blue"}, "obj2": {"size": 1, "color": "red"}} | +-----------------------------------------------------------------------------+ Good. Now, trying these on MariaDB, I get: MariaDB [test]> select JSON_ARRAYAGG(col1) from t30; +------------------------------------------------------------------------+ | JSON_ARRAYAGG(col1) | +------------------------------------------------------------------------+ | ["{\"color\":\"red\", \"size\":1}","{\"color\":\"blue\", \"size\":2}"] | +------------------------------------------------------------------------+ MariaDB [test]> select JSON_OBJECTAGG(name, contents) from t31; +---------------------------------------------------------------------------------------+ | JSON_OBJECTAGG(name, contents) | +---------------------------------------------------------------------------------------+ | {"obj1":"{\"color\":\"blue\", \"size\":2}", "obj2":"{\"color\":\"red\", \"size\":1}"} | +---------------------------------------------------------------------------------------+ The JSON documents are coverted into strings. I think, the users will prefer MySQL's behavior. But which behaivor is "correct"? (i.e. required by the standard, etc) Is MariaDBs's behaivor a bug? Is this expected? BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] MDEV-11563: GROUP_CONCAT(DISTINCT ...) may produce a non-distinct list
by Sergey Petrunia 08 Jun '20

08 Jun '20

Hi Varun, > revision-id: c408c41230c4a9de0114d13aaf861d3ced65f4c7 (mariadb-10.5.0-424-gc408c41230c) > parent(s): 95da2113a050ad739fdaf60ee871329468a01554 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2020-03-31 11:57:37 +0530 > message: > > MDEV-11563: GROUP_CONCAT(DISTINCT ...) may produce a non-distinct list > The commit message should describe the error and the fix made. (especially important here as this patch fixes only a part of the issue with GROUP_CONCAT) > --- > mysql-test/main/func_gconcat.result | 53 ++++++++++++++++++++++--------------- > mysql-test/main/func_gconcat.test | 7 +++++ > sql/item_sum.cc | 40 +++++++++++++++++----------- > sql/item_sum.h | 3 ++- > 4 files changed, 65 insertions(+), 38 deletions(-) > > diff --git a/sql/item_sum.h b/sql/item_sum.h > index e221d04397d..b10d9f5974d 100644 > --- a/sql/item_sum.h > +++ b/sql/item_sum.h > @@ -1879,7 +1879,8 @@ class Item_func_group_concat : public Item_sum > bool warning_for_row; > bool always_null; > bool force_copy_fields; > - bool no_appended; > + /** True if result has been written to output buffer. */ > + bool m_result_finalized; Why does this variable use the m_ convention when others do not? I think this is confusing (I stop for a moment to wonder how is it different from all others every time I see the name). Please remove this. What's "output buffer"? this->result ? And what does "result" mean here - the entire return value of GROUP_CONCAT function or some part of it? Please improve the comment. > /** Limits the rows in the result */ > Item *row_limit; > /** Skips a particular number of rows in from the result*/ Ok to push after this is addressed. BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

[Maria-developers] Is it okay that JSON_OBJECTAGG creates objects with duplicate elements?
by Sergey Petrunia 07 Jun '20

07 Jun '20

Hi Alexey, Consider this example: create table t20 (a varchar(10), b int); insert into t20 values ('red', 1), ('blue', 2), ('red',1001), ('blue', 1002); select json_objectagg(a,b) from t20; +----------------------------------------------+ | json_objectagg(a,b) | +----------------------------------------------+ | {"red":1, "blue":2, "red":1001, "blue":1002} | +----------------------------------------------+ the created object has multiple members with the same name. Is this expected or should be reported as a bug? In MySQL-8, the above example produces this: +-----------------------------+ | json_objectagg(a,b) | +-----------------------------+ | {"red": 1001, "blue": 1002} | +-----------------------------+ BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] 0f91c62d332: MDEV-22722 Assertion "inited==NONE" failed in handler::ha_index_init on the slave during UPDATE
by Sergei Golubchik 07 Jun '20

07 Jun '20

Hi, Sachin! Two questions: 1. This looks like there was no replication tests for periods of long uniques at all? Please add them (create an MDEV, like "replication tests of long unique") and add various relevant replication tests in 10.4. 2. It feels a bit risky to backport my changes to 10.4. At least you definitely should take the latest code from 10.5. Not cherry-pick one commit, but copy handler::prepare_for_insert() and other relevant code from 10.5. May be it'd be safer not to backport? What would your fix look like if you wouldn't have prepare_for_insert()? Also, a couple of comments to the code, see below. On Jun 03, Sachin wrote: > revision-id: 0f91c62d332 (mariadb-10.4.11-225-g0f91c62d332) > parent(s): c01b3d91690 > author: Sachin <sachin.setiya(a)mariadb.com> > committer: Sachin <sachin.setiya(a)mariadb.com> > timestamp: 2020-06-02 08:24:24 +0530 > message: > > MDEV-22722 Assertion "inited==NONE" failed in handler::ha_index_init on the slave during UPDATE > > Add missing call for handler->prepare_for_insert() in rows event. > > diff --git a/mysql-test/main/long_unique_bugs_replication.test b/mysql-test/main/long_unique_bugs_replication.test > new file mode 100644 > index 00000000000..5348be3e7d3 > --- /dev/null > +++ b/mysql-test/main/long_unique_bugs_replication.test > @@ -0,0 +1,22 @@ > +# > +# Long unique bugs related to master slave replication > +# > + > +# > +# MDEV-22722 Assertion "inited==NONE" failed in handler::ha_index_init on the slave during UPDATE > +# > + > +--source include/master-slave.inc > +--source include/have_binlog_format_row.inc master-slave.inc should always be included last, after all have_xxx.inc files that may cause the test to be skipped. > + > +create table t1 (i1 int, a1 text, unique key i1 (a1)) engine=myisam; > +insert into t1 values (1,1); > +update t1 set a1 = 'd' limit 1; > + > +sync_slave_with_master; > +connection slave; > + > +connection master; > +drop table t1; > + > +--source include/rpl_end.inc > diff --git a/sql/log_event.cc b/sql/log_event.cc > index 3707f73a716..f4c0c546a38 100644 > --- a/sql/log_event.cc > +++ b/sql/log_event.cc > @@ -11568,6 +11568,7 @@ int Rows_log_event::do_apply_event(rpl_group_info *rgi) > { > master_had_triggers= table->master_had_triggers; > bool transactional_table= table->file->has_transactions(); > + table->file->prepare_for_insert(); > /* > table == NULL means that this table should not be replicated > (this was set up by Table_map_log_event::do_apply_event() > @@ -14344,6 +14345,7 @@ int Rows_log_event::find_row(rpl_group_info *rgi) > issue_long_find_row_warning(get_general_type_code(), m_table->alias.c_ptr(), > is_index_scan, rgi); > table->default_column_bitmaps(); > + table->file->prepare_for_insert(); Why do you need it here, I'd think that do_apply_event() should be enough? > DBUG_RETURN(error); > } Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

2 1

Re: [Maria-developers] MDEV-16230: Server crashes when Analyze format=json is run with a window function with empty PARTITION BY and ORDER BY clauses
by Sergey Petrunia 04 Jun '20

04 Jun '20

Hi Varun, * Please limit the commit comment to ~80 chars wide. * The .result file has r_total_time_ms. Use analyze-format.inc to avoid this. Ok to push after this is addressed. > commit 9e755e21799695b9a837a240a86ba5ef863ae7b3 > Author: Varun Gupta <varun.gupta(a)mariadb.com> > Date: Tue Jun 2 16:31:53 2020 +0530 > > MDEV-16230: Server crashes when Analyze format=json is run with a window function with empty PARTITION BY and ORDER BY clauses > > Currently when both PARTITION BY and ORDER BY clauses are empty then we create a Item with the first field > in the select list and sort with that field. It should be created as an Item_temptable_field instead of > Item_field because the print() function continues to work even if the table has been dropped. > ... BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] e14dcfe0bc7: MDEV-22715: SIGSEGV in radixsort_for_str_ptr and in native_compare/my_qsort2 (optimized builds)
by Sergey Petrunia 04 Jun '20

04 Jun '20

Hi Varun, Please add a comment above the declaration of Field::sort_string() that some implementations actually may write up to 8 bytes regardless of what size was requested. Ok to push after that. > revision-id: e14dcfe0bc743aa6b4ed2b3c4f7e9314aa01d6eb (mariadb-10.1.43-158-ge14dcfe0bc7) > parent(s): db1f73b26edaace41345f19bca507a7848aa39a8 > author: Varun Gupta > committer: Varun Gupta > timestamp: 2020-05-29 21:13:38 +0530 > message: > > MDEV-22715: SIGSEGV in radixsort_for_str_ptr and in native_compare/my_qsort2 (optimized builds) > > For DECIMAL[(M[,D])] datatype max_sort_length was not being honoured which was leading to buffer > overflow while making the sort key. The fix to this problem would be to create sort keys for decimals > with atmost max_sort_key bytes > > Important: > The minimum value of max_sort_length has been raised to 8 (previously was 4), > so fixed size datatypes like DOUBLE and BIGINIT are not truncated for > lower values of max_sort_length. > ... BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] 7df57cc8206: MDEV-22719 Long unique keys are not created when individual key_part->length < max_key_length but SUM(key_parts->length) > max_key_length
by Sergei Golubchik 03 Jun '20

03 Jun '20

Hi, Sachin! ok to push On Jun 03, Sachin wrote: > revision-id: 7df57cc8206 (mariadb-10.4.11-226-g7df57cc8206) > parent(s): 0f91c62d332 > author: Sachin <sachin.setiya(a)mariadb.com> > committer: Sachin <sachin.setiya(a)mariadb.com> > timestamp: 2020-06-03 13:47:39 +0530 > message: > > MDEV-22719 Long unique keys are not created when individual key_part->length < max_key_length but SUM(key_parts->length) > max_key_length > > Make UNIQUE HASH key in case when key_info->key_length > max_key_length > Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] 1e9ea675a18: MDEV-21282 Assertion 'mariadb_table' failed in gcol.innodb_virtual_debug_purge
by Sergei Golubchik 03 Jun '20

03 Jun '20

Hi, Thirunarayanan! Sure, it's purely InnoDB functionality. If you say that purge thread already hold MDL for the table, then open_purge_table() doesn't need to. Ok to push as you see fit. On Jun 03, Thirunarayanan Balathandayuthapani wrote: > revision-id: 1e9ea675a18 (mariadb-10.5.2-304-g1e9ea675a18) > parent(s): f22093ad392 > author: Thirunarayanan Balathandayuthapani <thiru(a)mariadb.com> > committer: Thirunarayanan Balathandayuthapani <thiru(a)mariadb.com> > timestamp: 2020-06-01 18:02:17 +0530 > message: > > MDEV-21282 Assertion 'mariadb_table' failed in gcol.innodb_virtual_debug_purge > > - commit ea37b144094a0c2ebfc6774047fd473c1b2a8658 (MDEV-16678) caused > a regression. when purge thread tries to open the table for virtual > column computation, there is no need to acquire MDL for the table. > Because purge thread already hold MDL for the table Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0

Re: [Maria-developers] c01b3d91690: MDEV-22551 Server crashes in skip_trailing_space upon ANALYZE on table with long unique, or Assertion `marked_for_read()' fails
by Sergei Golubchik 03 Jun '20

03 Jun '20

Hi, Sachin! On Jun 03, Sachin wrote: > revision-id: c01b3d91690 (mariadb-10.4.11-224-gc01b3d91690) > parent(s): 0552ed86782 > author: Sachin <sachin.setiya(a)mariadb.com> > committer: Sachin <sachin.setiya(a)mariadb.com> > timestamp: 2020-06-02 08:24:13 +0530 > message: > > MDEV-22551 Server crashes in skip_trailing_space upon ANALYZE on table with long unique, or Assertion `marked_for_read()' fails > > If Blob(or parent) field read_set is not set then Do not calc hash in > that case. No, I think this is wrong. The problem, again, is that read_set bits are set incorrectly, and the fix is, again, to set them correctly, not to perform more read_set checks at run time. Think about what's happening here. The server tries to calculate a generated column value (because gcol is present in read_set) when not all argument columns were read (because they are not in read_set). This is an impossible situation, a gcol can _only_ be in read_set if all its dependencies are also there, you've seen in TABLE::mark_virtual_columns_for_write() that for every marked gcol all dependencies are also recursively marked. So in this case ANALYZE breaks this rule and should be fixed not to do it. There are two possible fixes - either ANALYZE should mark dependencies for all gcols (marking blobs in this bug) or ANALYZE should skip and not mark gcols. Better ask somebody who's more famiiar with EITS about what fix is preferred. Regards, Sergei VP of MariaDB Server Engineering and security(a)mariadb.org

1 0