January 2018 - developers - lists.mariadb.org

[Maria-developers] Windows line endings in MariaDB sources again (
by Otto Kekäläinen 23 Jul '19

23 Jul '19

Hello! While trying to release 10.0.26 for Debian and Ubuntu I noticed git chokes again on Windows line endings. As our .gitattributes states that by default git should clean up the line endings, why do we keep getting new Windows line endings in new code? mariadb-10.0$ cat .gitattributes # Set the default behavior, in case people don't have core.autocrlf set. * text=auto ... storage/connect/mysql-test/connect/std_data/*.txt -text mariadb-10.0.26$ find . | xargs file | grep CRLF ./storage/connect/mysql-test/connect/std_data/boyswin.txt: ASCII text, with CRLF line terminators ./storage/connect/mysql-test/connect/std_data/expenses.txt: ASCII text, with CRLF line terminators ./storage/connect/mysql-test/connect/std_data/emp.txt: ASCII text, with CRLF line terminators ./mysql-test/std_data/loaddata7.dat: ASCII text, with CRLF line terminators ./mysql-test/r/loadxml.result: ASCII text, with CRLF, LF line terminators ./mysql-test/r/perror-win.result: ASCII text, with CRLF, LF line terminators ./mysql-test/r/mysql_binary_mode.result: ASCII English text, with CRLF, LF line terminators ./mysql-test/r/func_regexp_pcre.result: UTF-8 Unicode C++ program text, with CRLF, CR, LF line terminators ./pcre/testdata/grepoutputN: ASCII text, with CRLF, CR, LF line terminators ./pcre/testdata/greppatN4: ASCII text, with CRLF line terminators This leads for me to: dpkg-source: info: building mariadb-10.0 using existing ./mariadb-10.0_10.0.26.orig.tar.gz dpkg-source: info: local changes detected, the modified files are: mariadb-10.0/storage/connect/mysql-test/connect/std_data/boyswin.txt mariadb-10.0/storage/connect/mysql-test/connect/std_data/emp.txt mariadb-10.0/storage/connect/mysql-test/connect/std_data/expenses.txt dpkg-source: error: aborting due to unexpected upstream changes, see /tmp/mariadb-10.0_10.0.26-1.diff.Yzwwhw Which means I have to do extra work with importing the upstream sources.. - Otto

2 4

[Maria-developers] 10.2 and merges (e.g. TokuDB)
by Phil Sweeney 28 Aug '18

28 Aug '18

Hi there, It seems 10.2.6 is still shipping with TokuDB from PS 5.6.35-80 (Feb 8th 2017). I can't see a jira issue for a 10.2 merge since 10.2.4. Perhaps intentional to help stabilise 10.2 for release? 10.0.30 also shipped with the same Feb version (in March). 10.1.23 however was released in May and still has this version from Feb, which seems unfortunate, but I know it's because it has to flow through 10.0 first. I see 10.0.31 is due to get TokuDB from PS 5.6.36-82 And that should flow into 10.1.24 soon too. However, 10.2.7 isn't due until July, so by then TokuDB version will be 5 months old in 10.2 I'm waiting on a fairly important row count fix that should appear in the next Percona Server release. When it does, if the current merging strategy continues, do I have to first wait for 10.0.32 to get it in late July, then 10.1.26 in August, then 10.2 after that? Or with 10.0, 10.1 and 10.2 all being supported and needing merges, is there a new strategy on the cards to shorten this lag? Thanks! Phil

3 7

[Maria-developers] MDEV-13118 Wrong results with LOWER and UPPER and subquery
by Alexander Barkov 12 Jul '18

12 Jul '18

Hello Sergei, Please review a patch for MDEV-13118. Thanks!

2 4

Re: [Maria-developers] 980a5c31880: MDEV-14572: Assertion `! is_set()' failed in Diagnostics_area::set_eof_status upon EXPLAIN UPDATE in PS
by Sergei Golubchik 19 Jun '18

19 Jun '18

Hi, Oleksandr! On Jan 28, Oleksandr Byelkin wrote: > Hi, Sergei! > > Flag in SELECT_LEX options dropped at the end of execution (in our case > prepare). All commands where EXPLAIN is possible restore the flag in main > SELECT_LEX (in mysql_explain_union()) except UPDATE (because it do > not use mysql_explain_union() > for explain). Without the flag it first try to send OK (as part of update) > then EOF as part of EXPLAIN. Ok. 1. Why does it not use mysql_explain_union() ? 2. Why EXPLAIN flag is need to be removed and restored anyway? 3. Please append the following to the commit comment: " Without it, mysql_update() didn't know that there will be EXPLAIN result set and was sending OK at the end of the update, which conflicted with the EOF sent later by EXPLAIN. " > On Sun, Jan 28, 2018 at 2:45 PM, Sergei Golubchik <serg(a)mariadb.org> wrote: > > > Hi, Oleksandr! > > > > On Dec 22, Oleksandr Byelkin wrote: > > > revision-id: 980a5c318803d6aa7355e019869597976dcf19bf (mariadb-10.0.33-37-g980a5c31880) > > > parent(s): 042f763268c0f209e7c12e0a6a72bb5d204dfe29 > > > author: Oleksandr Byelkin > > > committer: Oleksandr Byelkin > > > timestamp: 2017-12-22 14:00:10 +0100 > > > message: > > > > > > MDEV-14572: Assertion `! is_set()' failed in Diagnostics_area::set_eof_status upon EXPLAIN UPDATE in PS > > > > > > Restore EXPAIN flag in SELECT_LEX before execution multi-update by flag in LEX > > > (the same but in other way made before INSERT/DELETE/SELECT) > > > > Please explain what was the reason for the bug, why the status was set > > twice, and how your change fixes it. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

2 3

Re: [Maria-developers] Mdev-10664 Add statuses about optimistic parallel replication stalls.
by Sachin Setiya 19 Apr '18

19 Apr '18

Hi Kristian! Sorry for late reply , I was on vacations. On Tue, Jan 10, 2017 at 7:31 PM, Kristian Nielsen <knielsen(a)knielsen-hq.org> wrote: > Sachin Setiya <sachin.setiya(a)mariadb.com> writes: > > >> Right, but you still need _some_ way to synchronise multiple threads > > > I used atomic builtins my_atomic_add64_explicit. > > That seems fine (but see comment below). > > > Documentation:- > > > > Slave_DDL_Events:- > > > > Description: Number of DDL statements slave have executed in > > replication channel X. > > Scope: Per Replication Channel > > Data Type: numeric > > Introduced: MariaDB 10.1 > > > > Slave_Non_Transactional_Events:- > > > > Description: Number of Non Transactional "no idea what should be the > > next word " slave have executed in replication channel X. > > Scope: Per Replication Channel > > Data Type: numeric > > Introduced: MariaDB 10.1 > > I checked, the term "event group" is used already multiple places in the > documentation, so I think it can be used here also. An event group can be a > transaction (possibly with multiple transactional DML statements), or a DDL > or non-transactional statement. > > What you are doing is introducing status variables to help understand > optimistic (and aggressive) parallel replication modes. > > In optimistic parallel replication, a non-transactional event group is not > safe to run in parallel with other non-transactional events. So that event > is made to wait for all prior event groups to reach commit state, before > being allowed to execute. However, other transactional events are not > delayed by the non-transactional event group. > > Slave_Non_Transactional_Events counts the occurences of this. > > Then in case of a DDL statement, this can potentially be unsafe to run in > parallel even with other transactional events. So for a DDL, the DDL has to > wait for all prior event groups, _and_ subsequent event groups have to wait > for the DDL. > > Slave_DDL_Events counts these occurences. > > This is what the documentation, and commit messages, should explain, I > think. > > I think the status variables describe the values for > @@default_master_connection?, this should also be explained in the > documentation. > > Why did you decide to put this information into a status variable? > Normally, > slave status is seen in SHOW SLAVE STATUS, which supports showing status > for > a particular connection without using @@default_master_connection. > > Sorry for this , I guess It was a confusion I was looking at Slave_Running this is avaliable in show status as well as show slave status. So I thought for if I want to show some information in SHOW slave status this has to be in show status. And this is wrong. Now we can see non trans events / ddl events in show slave status. Show I remove this vars from Show Status, or it is okay ? > > diff --git a/sql/log_event.h b/sql/log_event.h > > index 90900f6..579607f 100644 > > --- a/sql/log_event.h > > +++ b/sql/log_event.h > > @@ -50,6 +50,7 @@ > > #endif > > > > #include "rpl_gtid.h" > > +#include "my_atomic.h" > > Why include my_atomic.h in the header file, when you only use it in the .cc > file? > Changed. > > > + mysql_mutex_lock(&LOCK_active_mi); > > + if (master_info_index) > > + { > > + mi= master_info_index-> > > + get_master_info(&thd->variables.default_master_connection, > > + Sql_condition::WARN_LEVEL_NOTE); > > + if (mi) > > + tmp= mi->total_ddl_events; > > > + if (mi) > > + tmp= mi->total_non_trans_events; > > Better use an atomic primitive here to read the value. Taking a mutex does > not help synchronise against my_atomic_add64_explicit(). > > Changed. > > + } > > + mysql_mutex_unlock(&LOCK_active_mi); > > + if (mi) > > + *((longlong *)buff)= tmp; > > + else > > + var->type= SHOW_UNDEF; > > + > > + return 0; > > +} > > #endif /* HAVE_REPLICATION */ > > > > static int show_open_tables(THD *thd, SHOW_VAR *var, char *buff, > > > index 9365c06..a474aa5 100644 > > --- a/sql/rpl_mi.h > > +++ b/sql/rpl_mi.h > > @@ -303,6 +303,12 @@ class Master_info : public > Slave_reporting_capability > > > > /* The parallel replication mode. */ > > enum_slave_parallel_mode parallel_mode; > > + > > + /* No of DDL statements */ > > + long total_ddl_events; > > + > > + /* No of non-transactional statements*/ > > + long total_non_trans_events; > > "long" is not the appropriate type here. Certainly not since you are using > my_atomic_add64_explicit(). Use a type that is the same size on all > platforms (and why use a signed type?). > > Changed to uint64. > - Kristian. > Attached a newer version of patch , please review it. -- Regards Sachin

3 7

[Maria-developers] CREATE SEQUENCE is coming
by Michael Widenius 18 Apr '18

18 Apr '18

Hi! For those that are trying to port applications to MariaDB from other databases or need more features than the current AUTO_INCREMENT provides, the following may be of interest: First version of sequence is pushed to bb-10.2-sequence. It should be merged to bb-10.2-compatilbity and then to 10.3 shortly (1-2 weeks). What is working for the moment: CREATE OR REPLACE [TEMPORARY] SEQUENCE [IF NOT EXISTS] name [ INCREMENT [ BY | = ] increment ] [ MINVALUE [=] minvalue | NO MINVALUE ] [ MAXVALUE [=] maxvalue | NO MAXVALUE ] [ START [ WITH | = ] start ] [ CACHE [=] cache ] [ [ NO ] CYCLE ] ENGINE=xxx COMMENT=".." SELECT NEXT VALUE FOR sequence_name; SELECT NEXTVAL(sequence_name); SELECT PREVIOUS VALUE FOR sequence_name; SELECT LASTVAL(sequence_name); SHOW CREATE SEQUENCE sequence_name; SHOW CREATE TABLE sequence_name; CREATE TABLE sequence-structure ... SEQUENCE=1 ALTER TABLE sequence RENAME TO sequence2; RENAME TABLE sequence RENAME TO sequence2; DROP [TEMPORARY] SEQUENCE sequence_name [IF EXISTS] See https://jira.mariadb.org/browse/MDEV-10139 for progress. See commit messages in bb-10.2-sequence for what is still to be done. Some documentation can be found at: https://mariadb.com/kb/en/mariadb/create-sequence/ This will be improved as the work progress. One of the goals with the SEQUENCE implementation is that all old tools, like mysqldump, should work unchanged, while still keeping normal usage of sequence standard compatibly. The make this possible, the sequence is currently implemented as a table with a few exclusive properties. The main disadvantage of having sequence as a table is that it uses the same name space as tables. In other words, you can't have a table and a sequence of the same name. The benefit is that sequences shows up in 'show tables', one can create a sequence also with 'create table' and drop it with 'drop table'. One can select from it as from any other table. This ensures that all old tools that works with tables should work with sequences. The special properties for sequence tables are: - A sequence table has always one row. - When one creates a sequence, either with CREATE TABLE or CREATE SEQUENCE, one row will be inserted. - Normal table options works for CREATE SEQUENCE. One can use ENGINE=xxx, COMMENT=xxx etc. - If one tries to insert into a sequence table, the single row will be updated. This allows mysqldump to work but also gives the additional benefit that one can change all properties of a sequence with a single insert. New applications can of course also use ALTER SEQUENCE. - Updates to the sequence table will change the single row. - Doing a select on the sequence shows the current state of the sequence, except the values that are reserved in the cache. The column 'next_value' shows the next value not reserved by the cache. - Truncate on a sequence table will give an error. - Alter table and rename works on sequences. - If one creates a sequence with INCREMENT 0, then the sequence will use auto_increment_increment and auto_increment_offset for the sequence, just like AUTO_INCREMENT. This allows sequences to work reliable in a master-master environment and with Galera. Internally sequence tables are created as a normal table without rollback (InnoDB, Aria and MySAM supports this) with is wrapped by a sequence engine. This allowed me to create sequences with almost no performance impact for normal tables. (The cost is one 'if' per insert if binary log is enabled). MariaDB 10.3 will support both the ANSI SQL and Oracle syntax for creating and accessing sequences. As ANSI SQL doesn't have an easy access to the last generated value, 103 also supports 'PREVIOUS VALUE FOR sequence_name', like IBM DB2, and LASTVAL(sequence_name) as PostgreSQL. I want to thank Jianwe Zhao from Aliyun for his work on SEQUENCE in AliSQL which gave me ideas and inspiration for this work. I also want to thank Marko Mäkelä for his help in the InnoDB part of the code. There is still a lot of work to fix edge cases, but in the current implementation most major things seams to work... Comments, suggestions or questions? Regards, Monty

2 1

[Maria-developers] Partial indexes?
by Jared Beck 03 Feb '18

03 Feb '18

Dear Maria, :p Thanks for all your hard work! Any plans to implement partial indexes? (https://www.postgresql.org/docs/10/static/indexes-partial.html) I hope this is the right place to suggest new features. Please redirect me if not. -- Jared Beck • (607) 216-5373 • jared(a)jaredbeck.com

3 7

Re: [Maria-developers] [Commits] a0ea3a58224: MDEV-14857: problem with 10.2.11 server crashing when executing stored procedure
by Sergey Petrunia 31 Jan '18

31 Jan '18

Hi Sanja, Please find my review below. Lots of typos, but the two important comments are at the bottom, for the changes in sql_select.cc and sql_trigger.cc On Sat, Jan 27, 2018 at 11:39:10AM +0100, Oleksandr Byelkin wrote: > revision-id: a0ea3a5822486fa1702d6615f1be30684c3d412f (mariadb-10.1.30-26-ga0ea3a58224) > parent(s): 524221e7a34d42214e82dd868348b453f2ef1591 > author: Oleksandr Byelkin > committer: Oleksandr Byelkin > timestamp: 2018-01-27 11:38:38 +0100 > message: > > MDEV-14857: problem with 10.2.11 server crashing when executing stored procedure > > Counter for select numbering made stored with the statement (before was global) > So now it does have always accurate value which does not depend on > interruption of statement prepare by errors like lack of table in > a view definition. > > --- > diff --git a/sql/sql_class.h b/sql/sql_class.h > index 4556487bdfe..2dc2fb61e9d 100644 > --- a/sql/sql_class.h > +++ b/sql/sql_class.h > @@ -1027,6 +1027,21 @@ class Statement: public ilink, public Query_arena > > LEX_STRING name; /* name for named prepared statements */ > LEX *lex; // parse tree descriptor > + /* > + LEX whith represent curent statement (conventional, SP or PS) s/whith/which/ s/curent/current/ s/represent/represents/ > + > + For example during view parsing THD::lex will point to the views LEX and > + THD::stmt_lex will point on LEX of the statement where the view will be s/point on/point to/ > + included > + > + Now used to have alwais correct select numbering inside statemet s/Now used/Currently it is used/ s/alwais/always/ s/statemet/statement/ > + (LEX::current_select_number) without storing and restoring a global > + counter which was THD::select_number. > + > + TODO: make some unified statement representation (now SP has different) > + to store such data like LEX::current_select_number. > + */ > + LEX *stmt_lex; > /* > Points to the query associated with this statement. It's const, but > we need to declare it char * because all table handlers are written > diff --git a/sql/sql_lex.h b/sql/sql_lex.h > index b57fba08b47..a2da6f672ca 100644 > --- a/sql/sql_lex.h > +++ b/sql/sql_lex.h > @@ -727,7 +727,13 @@ class st_select_lex: public st_select_lex_node > Item *prep_having;/* saved HAVING clause for prepared statement processing */ > /* Saved values of the WHERE and HAVING clauses*/ > Item::cond_result cond_value, having_value; > - /* point on lex in which it was created, used in view subquery detection */ > + /* > + Point on lex in which it was created, used in view subquery detection. "Points to the LEX...". Also please shift the whole comment 1 position to the left. > + > + TODO: make also st_select_lex::parent_stmt_lex (see THD::stmt_lex) > + and use st_select_lex::parent_lex & st_select_lex::parent_stmt_lex > + instead of global (from THD) references where it is possible. > + */ > LEX *parent_lex; > enum olap_type olap; > /* FROM clause - points to the beginning of the TABLE_LIST::next_local list. */ > @@ -2452,7 +2458,7 @@ struct LEX: public Query_tables_list > /** SELECT of CREATE VIEW statement */ > LEX_STRING create_view_select; > > - uint number_of_selects; // valid only for view > + uint current_select_number; // valid for statmet LEX (not view) s/statmet/statement/ > > /** Start of 'ON table', in trigger statements. */ > const char* raw_trg_on_table_name_begin; > diff --git a/sql/sql_parse.cc b/sql/sql_parse.cc > index 6084c59a257..8d3116cd8b3 100644 > --- a/sql/sql_parse.cc > +++ b/sql/sql_parse.cc > @@ -6903,7 +6903,12 @@ void THD::reset_for_next_command(bool do_clear_error) > clear_error(1); > > thd->free_list= 0; > - thd->select_number= 1; > + /* > + The same thd->stmt_lex made in lex_start, but during bootstrup this "We also assign thd->stmt_lex in lex_start()" s/bootstrup/bootstrap > + code executed first code is executed > + */ > + thd->stmt_lex= &main_lex; thd->stmt_lex->current_select_number= 1; > + DBUG_PRINT("info", ("Lex %p stmt_lex: %p", thd->lex, thd->stmt_lex)); > /* > Those two lines below are theoretically unneeded as > THD::cleanup_after_query() should take care of this already. > diff --git a/sql/sql_select.cc b/sql/sql_select.cc > index b9fe8f3162a..7a6a028ee9c 100644 > --- a/sql/sql_select.cc > +++ b/sql/sql_select.cc > @@ -2462,6 +2462,17 @@ void JOIN::save_explain_data(Explain_query *output, bool can_overwrite, > bool need_tmp_table, bool need_order, > bool distinct) > { > + /* > + If there is SELECT in this statemet with the same number it must be the > + same SELECT > + */ > + DBUG_ASSERT(select_lex->select_number == UINT_MAX || > + select_lex->select_number == INT_MAX || > + !output || Is it really possible that output==NULL? If yes, that's probably a bug. > + !output->get_select(select_lex->select_number) || > + output->get_select(select_lex->select_number)->select_lex == > + select_lex); > + > if (select_lex->select_number != UINT_MAX && > select_lex->select_number != INT_MAX /* this is not a UNION's "fake select */ && > have_query_plan != JOIN::QEP_NOT_PRESENT_YET && > @@ -24601,6 +24612,11 @@ int JOIN::save_explain_data_intern(Explain_query *output, bool need_tmp_table, > { > explain= new (output->mem_root) Explain_select(output->mem_root, > thd->lex->analyze_stmt); > + if (!explain) > + DBUG_RETURN(1); // EoM > +#ifndef DBUG_OFF > + explain->select_lex= select_lex; > +#endif > join->select_lex->set_explain_type(true); > > explain->select_id= join->select_lex->select_number; > diff --git a/sql/sql_trigger.cc b/sql/sql_trigger.cc > index 0b4978b2862..878cd5a3b57 100644 > --- a/sql/sql_trigger.cc > +++ b/sql/sql_trigger.cc > @@ -1378,7 +1378,7 @@ bool Table_triggers_list::check_n_load(THD *thd, const char *db, > ulonglong save_sql_mode= thd->variables.sql_mode; > LEX_STRING *on_table_name; > > - thd->lex= &lex; > + thd->lex= thd->stmt_lex= &lex; > The same comment as before: we change thd->stmt_lex here. The function has code below that restores the value of thd->lex: // QQ: anything else ? lex_end(&lex); thd->lex= old_lex; But the value of thd->stmt_lex is not restored? > save_db.str= thd->db; > save_db.length= thd->db_length; BR Sergei -- Sergei Petrunia, Software Developer MariaDB Corporation | Skype: sergefp | Blog: http://s.petrunia.net/blog

1 0

Re: [Maria-developers] Wording suggestion?
by Sergei Golubchik 30 Jan '18

30 Jan '18

Hi, Russell! On Jan 30, Russell J.T. Dyer wrote: > How about concurrent_assign ? I think "concurrent" already has a clear meaning, at least in MariaDB - operations are done "concurrently" if they're executed truly in parallel, usually in different threads. For example, both MyISAM and InnoDB allow concurrent SELECTs, but only InnoDB allows concurrent UPDATEs. I considered simultaneous_assign, as a slightly shorter alternative to simultaneous_assignment. But it looked somewhat unnatural to me. Still, this is the case when I happily yield to the opinion of a documentation writer :) If you think that simultaneous_assign looks just fine, I'll use it. > > It's not only for update. There is a patch to fix SELECT ... INTO, > > and I suppose we'll have "simultaneous assignment" behavior > > everywhere eventually. > > > > So, I'm leaning towards either simultaneous_assignment or > > simultaneous_set as mode names. The first is more exact, but it's > > rather long (sql_mode names range from 3 to 26 characters, and this > > one is 23). The second (with 16) fits right in the middle of the > > range, length-wise. But there's no SET keyword in SELECT ... INTO. > > On the other hand, there's no ASSIGNMENT keyword anywhere either :) > > Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0

Re: [Maria-developers] Mdev-15085 Invisible Column Non-constant Default value results...
by Sergei Golubchik 30 Jan '18

30 Jan '18

Hi, Sachin! ok to push! On Jan 30, Sachin Setiya wrote: > commit a2e6d3a2fe6842bd3c35dffb1cd6f6d8a3117107 > Author: Sachin Setiya <sachin.setiya(a)mariadb.com> > Date: Tue Jan 30 14:04:05 2018 +0530 > > Mdev-15085 Invisible Column Non-constant Default value results... > > Problem:- If we create table field with dynamic default value then that > field always gets NULL value. > > Analyze:- This is because in fill_record we simple continue at Invisible > column because we though that share->default_values(default value is > always copied into table->record[0] before insert) will have a default > value for them(which is true for constant defaults , but not for dynamic > defaults). > > Solution:- We simple set all_fields_have_value to null , and this will > make call to update_default_fields (in the case of dynamic default), And > default expr will be evaluted and value will be set in field. Regards, Sergei Chief Architect MariaDB and security(a)mariadb.org

1 0